libatomic trouble after tce-update

[GNUser]

Are you using an unsanctioned script for updates? 

Hi Rich. update-everything is part of base system and the current improved version can do everything that the four separate "sanctioned" commands can do.

The big advantage of update-everything is that it updates all of the user's .dep files with a single download from the TCL repo (dep.db.gz), rather than potentially hundreds of downloads (redownloading each and every individual .dep file in tce/optional/).

P.S. Other than the special logic to use dep.db.gz, update-everything is 100% powered by tce-audit and tce-update.

[jazzbiker]

Hi GNUser!

Nice that update-everything is finally improved, i.e. works correctly ;-) I think that use of tce-update separately from tce-audit magic must be deprecated in some way, as it can cause problems.

Thanks, I've loaded the improved version.

[GNUser]

Hi jazzbiker! Thanks for grabbing the improved version. If you can think of any way to make it better, please let me know. I want it to be as fast and efficient as possible. You provided valuable tips when this script was originally created

P.S.
    While on the topic of system updates, one situation that comes up rarely is that Juanito will deprecate an extension (i.e., completely remove it from the repository). This recently happened with icu67-bin.tcz, for example. If that extension is present on user's system, tce-update shows an error saying that the extension cannot be updated. This error also shows up when using update-everything, of course.
    I think this situation is rare enough that it is not worth complicating tce-update and/or update-everything to ask the user what to do. In other words, I think it's fine for the tce-update step of update-everything to show an error when an extension has been deprecated, and let the user figure out the best course of action (which is usually to remove the deprecated extension from their system and run update-everything again).

[jazzbiker]

By the way individual .dep files are not necessary and even may produce conflicts since dep.db.gz is present in the repo.

[GNUser]

I agree. I'm a huge fan of all the benefits made possible by dep.db.gz, obviously

[Rich]

Hi GNUser

Are you using an unsanctioned script for updates? 

Hi Rich. update-everything is part of base system ...

I didn't know it had been incorporated into the base.

... can do everything that the four separate "sanctioned" commands can do. ...

The term  unsanctioned  was the result of my twisted sense of humor. 

[GNUser]

I didn't know it had been incorporated into the base.
...
The term  unsanctioned  was the result of my twisted sense of humor. 

Haha, no worries. My wife often doesn't understand my own sense of humor, so I know how you feel!

The script is simply a variation on the four commands that you recommended to me a few years ago (thank you), tweaked to take advantage of dep.db.gz.

I hope you find the script to be useful and convenient. Please let me know if you can think of a way to make it better.

[Paul_123]

The new updated update-everything script has a little danger in it. 

1) If there is a problem downloading or unpacking dep.db.gz, then it will remove all of your depfiles from your system.
2) It will also remove dep files from any custom extensions a user might have.  FOr example and extension downloaded from a 3rd party location, as long as it has an md5.txt file.

Suggestions:
1) the function get_depdb needs some sort of error trap to make sure the downloaded deb.db is somewhat sensible before continuing.
2) the list if extensions to be updated should be checked to make sure they exist on the current mirror.
    - Get a list of md5 files from the optional dir.  Check that list against md5.db
    - use the checked list to find dep file changes

Edit: Updated with suggestions
 

[GNUser]

Hi Paul_123. The two dangers are real, thank you for pointing them out.

I like your suggestion for how to mitigate danger #2. I will implement a check so that .dep file is only considered superfluous and deleted if all these conditions are true:
* the extension has an md5 file on user's system
* there is no dep file for the extension in dep.db
* extension has been verified to exist in the repository (because there is an entry for it in md5.db)

What do you think would be a sensible mitigation for danger #1? How about checking that the download of dep.db.gz completed successfully and the downloaded file is a valid .gz archive with nonzero size?

[Paul_123]

I would think as long as gunzip has an exit code of zero, it’s probably fine.  A quick check to make sure it contains a few common entries that should always exist.

For example check for zstd.tcz.dep.  That would also check to make sure something didn’t happen during server file generation creating a short read.

[Rich]

Hi GNUser
This is how I once handled a similar situation:

Code: [Select]

# If we have a working URL, fetch the list containing each extensions md5.
[ "$URLreachable" == "Yes" ] && wget -q -O  $Dest.md5.db.gz $RepoURL/md5.db.gz > /dev/null 2>&1
# Assume the file was received and intact. Attempt to unzip it and then test for errors.
gunzip $Dest.md5.db.gz > /dev/null 2>&1

if [ "$?" == 0 ]
then    # Do a case sensitive sort on md5.db. The current sort is case insensitive.
        sort -k2 $Dest.md5.db > $Dest.md5.sorted
        mv $Dest.md5.sorted $Dest.md5.db
else    # Set a flag if gunzip failed.
        md5RepoSkipped="1"
fi

[GNUser]

Thank you both.
https://github.com/tinycorelinux/Core-scripts/pull/64