Proposal for upgrading .tcz on .tcz.dep changed

[jazzbiker]

Still if the upgraded .dep file refers at the missing extensions the

Code: [Select]

tce-audit builddb
tce-audit fetchmissing
may be necessary. But builddb and fetchmissing seems to take less time than updatedeps:

Code: [Select]

tc@box:~$ time tce-audit builddb
Building Extension Database from /etc/sysconfig/tcedir/optional
.........................................................................................
real    0m 3.73s
user    0m 1.52s
sys     0m 1.92s
tc@box:~$ time tce-audit updatedeps
Updating .dep files from /etc/sysconfig/tcedir/optional
...
Building Extension Database from /etc/sysconfig/tcedir/optional
.........................................................................................
real    0m 53.06s
user    0m 3.20s
sys     0m 6.83s
tc@box:~$ time tce-audit fetchmissing
real    0m 30.21s
user    0m 13.68s
sys     0m 16.80s


[jazzbiker]

tce-update making the tce-audit builddb and fetchmissing job:

Code: [Select]

--- tce-update.orig
+++ tce-update
@@ -17,6 +17,7 @@
        [ -f "$1" ] && rm -f "$1"
        wget -cq "$MIRROR"/"$1"
 }
+
 fetchzsync(){
        [ -f "$1" ] && rm -f "$1"
        if zsync -i "$UPGRADE_DIR"/"$1" -q "$MIRROR"/"$1".zsync; then
@@ -78,39 +79,27 @@
        fi
 }

-
 check_dependencies(){
-#      echo "${BLUE}Checking dependencies:${NORMAL}"
-       DEPS="$(/bin/busybox find "$UPGRADE_DIR" -regex '.*\.tcz\.dep$' | sort)"
-       [ -f  $TCEDIR/tcz-black.lst ] && DEPS=$(echo $DEPS | grep -v -f $TCEDIR/tcz-black.lst)
+       echo "${BLUE}Checking dependencies:${NORMAL}"
+       DEPS="$(/bin/busybox find "$TARGET_DIR" -regex '.*\.tcz\.dep$' | sort)"
+
+       DEPS_TO_LOAD=""
+
        for F in $DEPS; do
                for TARGET in `cat "$F"`; do
-                       if [ ! -f "$UPGRADE_DIR"/"$TARGET" ] && [ ! -f "$UPGRADE_DIR"/upgrade/"$TARGET" ]; then
-                               REPO=`grep -w "$TARGET" md5.db`
-                               if [ -n "$REPO" ]; then
-                                       echo -n  "Fetching required dependency: $TARGET "
-                                       if [ -n "$USEZSYNC" ]; then
-                                               fetchzsync "$TARGET" 2>/dev/null
-                                       else
-                                               fetch "$TARGET" 2>/dev/null
-                                       fi
-                                       echo  "$REPO"  > "$TARGET".md5.txt
-                                       md5sum -cs "$TARGET".md5.txt
-                                       if [ "$?" == 0 ]; then
-                                               mv /tmp/"$TARGET".md5.txt "$TARGET_DIR"/. 2>/dev/null
-                                               mv /tmp/"$TARGET" "$TARGET_DIR"/. 2>/dev/null
-                                               UPDATED_APPS=1
-                                               echo "OK"
-                                       else
-                                               echo "Failed"
-                                               echo "Error occurred while fetching dependency: $TARGET" >> "$ERRLIST"
-                                               rm /tmp/"$TARGET".md5.txt
-                                               rm /tmp/"$TARGET"
-                                       fi
-                               fi
+                       if [ ! -f "$UPGRADE_DIR"/"$TARGET" ]; then
+                               DEPS_TO_LOAD="$DEPS_TO_LOAD $TARGET"
                        fi
                done
        done
+
+       if [ "$DEPS_TO_LOAD" ] && ! tce-load -w $DEPS_TO_LOAD > "$CHECKDEP_LIST"
+       then
+               echo "Failed"
+               cat "$CHECKDEP_LIST" >> "$ERRLIST"
+       else
+               echo "OK"
+       fi
 }

 check_bootoptions(){
@@ -204,6 +193,8 @@
 cd /tmp
 ERRLIST="upgrade_errors.lst"
 > "$ERRLIST"
+CHECKDEP_LIST="checkdep.lst"
+> "$CHECKDEP_LIST"
 UPDATED_APPS=0
 [ -e md5.db ] && rm -f md5.db
 tce-fetch.sh md5.db.gz



[GNUser]

everything is nice but execution time. Something like 1m50s against 15s for tce-update.

The long execution time is because the naive script I shared in Reply #3 redownloads every .dep file that exists on your system and always runs fetchmissing.

I improved update-everything so that:
1. The only dep file that's downloaded is dep.db.gz*
2. Only changed .dep files are copied to your tcedir/optional directory
3. fetchmissing only runs if at least one of your .dep files has changed

jazzbiker, the new (smart and fast) version of update-everything is attached. I'd be happy if you'd test it and let me know your thoughts.

* This is thanks to the incredible power of awk, which can recreate every .dep file that exists on the official mirror from a single file (deb.db.gz), in a fraction of a second.

[jazzbiker]

Hi GNUser,

I think Your hack is nice. Though dep.db.gz is accessible only for tc14.

Here are the benchmarks on my tce14.

Plain update.sh by Rich:

Code: [Select]

real    0m 45.78s
user    0m 8.47s
sys     0m 14.76s
Your update-everything:

Code: [Select]

real    0m 13.72s
user    0m 4.69s
sys     0m 5.46s
And my tce-update from post #16

Code: [Select]

real    0m 9.67s
user    0m 3.62s
sys     0m 2.62s

Aren't You using tce-fetch.sh?

[jazzbiker]

By the way, Your update-everything and update.sh (as I named Rich's version) are not equivalent!
First I run update.sh and it found nothing to upgrade.
Then I run update-everything, and it reported:

Code: [Select]

local shared-mime-info.tcz.dep is different from one in mirror, fixing it now...
local dbus-glib.tcz.dep is different from one in mirror, fixing it now...
local atk.tcz.dep is different from one in mirror, fixing it now...
local dbus.tcz.dep is different from one in mirror, fixing it now...
If I run update-everything once again, it tells nothing.

But! If I run update.sh, and then again update-everything, it reports about the same differing .dep files once again! I think dep.db.dz in TC14 x86 repo is not updated.

What are Your thoughts?

[GNUser]

Wow, jazzbiker, nice catch. My assumption is that dep.db.tz on the official repo is updated anytime a .dep file is added or modified. If this is not the case, then the efficient way my new update-everything script handles .dep files is, unfortunately, unreliable.

Question for the developers (curaga, Juanito, Rich):
Does dep.db.tz always reflect the latest version of repo's .dep files, or does dep.db.tz only get updated sporadically? If keeping it updated is not the current practice but possible to implement, pretty please implement it.

P.S. jazzbiker, regarding tce-fetch.sh, I've never had to call it directly. I think tce-update and tce call tce-fetch.sh where needed.

[jazzbiker]

Hi GNUser,

I've made a little investigation and found that for all 4 files which were told by update-everything as rotten the difference consists of empty lines. It means that dep.db.gz is up-to-date and You need to make the processing of the .dep files a little bit smarter. You are triggering on the empty lines, that's not good.

[GNUser]

Hi jazzbiker. Thank you for your investigation. I'm on it!

[GNUser]

jazzbiker, where were the blank lines?
did awk generate them in /tmp/depfiles/foo.tcz or were they present in your original /etc/sysconfig/tcedir/optional/foo.tcz?

[GNUser]

Hi jazzbiker. Never mind where the blank lines are. It's more efficient to ignore them in the comparison than to remove them before comparing.

I changed the comparison method. Fixed script is attached

[jazzbiker]

Hi GNUser,

Life is tough, forget relaxing:

Code: [Select]

tc@box:~$ hexdump -C /mnt/sdb3/tce14/optional/dbus.tcz.dep
00000000  65 78 70 61 74 32 2e 74  63 7a 0a 20 0a           |expat2.tcz. .|
0000000d
tc@box:~$ hexdump -C /tmp/depfiles/dbus.tcz.dep
00000000  65 78 70 61 74 32 2e 74  63 7a 0a                 |expat2.tcz.|
0000000b

Extra line consisting of the single space ...

[GNUser]

Hi jazzbiker. Fortunately, the creators of  diff  thought of everything.
-Bbw flags should ignore absolutely all whitespace.

Unfortunately, in Busybox diff, -Bbw is broken (try diff -Bbwsq on two files that differ only by whitespace and you'll see that files are reported as being different). In GNU diff, -Bbw works exactly as expected.

diffutils.tcz is now a dependency of the smart and fast update-everything script, unfortunately.

Here is the (hopefully) final version and time to relax

[curaga]

dep.db.gz is updated whenever the repo files are updated. This does mean if someone edits a .dep file manually, it may be out of sync until a new extension gets added etc.

[GNUser]

Thanks for the information, curaga.

I think the efficiency gained by downloading dep.db.gz (rather than redownloading every .dep present on one's system) is worth the minimal risk of dep.db.gz being temporarily out of sync due to a manual .dep edit on the repo.

For full system update, using dep.db.gz translates into a win for the user (update process takes seconds rather than, potentially, minutes) and a win for your server (one wget request per user rather than, potentially, hundreds).

[Rich]

Hi curaga
If I place an updated .dep file in staging and then run update repo, does
dep.db.gz get updated then?