Why does sce-update take so much time (even more than sce-import)?

[sm8ps]

Greetings and happy new year! I have not been very successful with my dCore adventures last year and so I am hoping that 2019 will allow more break-throughs, for myself and also for everybody else!

I have been tinkering with dCore upgrades for a long time and while at it I had quite enough opportunities for wondering why sce-update takes so long. It even seems that it takes more time to update a package than to simply re-import it. Today I decided to take a closer look on a netbook with the a base extension called X-LIST that contains the following packages:

Code: [Select]

pm-utils, graphics-4.14.10-tinycore, xorg, xorg-intel, xserver-xorg-input-synaptics,
arandr, openbox, flwm, lxpanel, xinput, dbus, dbus-x11, suru-icon-theme
Including dependencies, there are 380 packages to be imported. I compared updating with 'sce-update -rn' to (re-)importing with 'sce-import -rln'. I repeated the test twice in order to eliminate the effect of downloading (because packages are taken from 'tce/import/debs/' if available).

Furthermore, I compared two different sets of repositories under '/opt/debextra/': A) only three PPAs, B) the same three PPAs plus all the following Ubuntu repositories:

Code: [Select]

bionic-backports-main, bionic-backports-multiverse,bionic-backports-restricted, bionic-backports-universe,
bionic-main, bionic-multiverse, bionic-restricted, bionic-universe,
bionic-updates-main, bionic-updates-multiverse, bionic-updates-restricted, bionic-updates-universe
The PPAs just happened to be there and do not play any particular role.

Here the results:

Code: [Select]

sce-update -rn X-LIST    A)  4'55"    B) 13'29"
sce-import -rln X-LIST   A)  1'59"    B)  6'54"
Interpretation:

• vertically: sce-update takes two and a half times longer than sce-import
• horizontally: extra repositories increase the execution time for both commands considerably

This is on dCore-bionic:2018.08.10.22.24. Grep is imported and thus not Busybox-grep.

Is this all expected behavior?

[Jason W]

Hi sm8ps

The short answer is yes, sce-update by it's nature takes longer than sce-import.  And extra repos do add to that time for both sce-update and sce-import. 

Much effort has been spent in streamlining sce-update and sce-import for performance.  100% accuracy is essential, but performance is also important.  I have tried different routines using variations of awk, grep, sed, and other commands in the scripts to speed things up but the current is the best performing way found so far.

[sm8ps]

Thanks for your answer, JasonW! I am aware that there is much effort behind all the sce-tools. Nevertheless, I am wondering why to bother with using sce-update at all? Probably because it does tell when there are no updates necessary as opposed to sce-import. Even so, could a combination of sce-update check and "blindly" sce-importing anew perform better? -- This is all very naive as I do not know the code behind the tools. So I would be grateful for another short answer.

Cheers!

[Jason W]

Here is my result from using sce-update on my SCE directory, all SCEs being checked, with below the size of my SCE directory, which is about 9.2 GB not counting the update subdirectory:

Code: [Select]

jason@box:/mnt/sda1/tceimport-bionic64/sce$ du -h
1.8G ./update
11G .


Below is the sce-update of the SCE directory, with checking all SCEs and no re-importing:

Code: [Select]

jason@box:~$ time sce-update -an
* Using nice level 19.
* Using the -a option.
* Using the -n option.
* DEBINX sync based on ubuntu bionic.
* Debian index sync: ubuntu_bionic_main_amd64_Packages
* Debian security index sync: ubuntu_bionic_security_amd64_Packages
* Using repo: http://security.ubuntu.com/ubuntu bionic main
Checking all system SCEs for updates:
 
         200-bionic.sce update check.
         alsa-utils.sce update check.
         atril.sce update check.
         audacity.sce update check.
         avidemux2.6-qt.sce update check.
         bionic-desk.sce update check.
         brasero.sce update check.
         cdrdao.sce update check.
         cheese.sce update check.
         chromium-browser.sce update check.
         dCore-chrome-stable-installer.sce update check.
         dCore-firefox-installer.sce update check.
         dCore-google-chrome-stable-installer.sce update check.
         dCorePlus-bionic64.sce update check.
         dCore-usbinstall.sce update check.
         devede.sce update check.
         dosfstools.sce update check.
         e17.sce update check.
         e3.sce update check.
         emelfm2.sce update check.
         enlightenment.sce update check.
         evince.sce update check.
         ffmpeg.sce update check.
         file.sce update check.
         firefox.sce update check.
         firefox-latest.sce update check.
         fonts-freefont-ttf.sce update check.
         gdb-dbg.sce update check.
         git.sce update check.
         gnome-screensaver.sce update check.
         google-chrome.sce update check.
         gparted.sce update check.
         graphics-4.14.10-tinycore64.sce update check.
         grep.sce update check.
         icewm.sce update check.
         imagemagick.sce update check.
         k3b.sce update check.
         kernel-all-4.14.10-tinycore64.sce update check.
         leafpad.sce update check.
         libbullet2.87.sce update check.
         libgail-3-0.sce update check.
         libgconf2-4.sce update check.
         libgles2.sce update check.
         libgtk-3-bin.sce update check.
         libk3b7-extracodecs.sce update check.
         libmadlib.sce update check.
         libmad-ocaml.sce update check.
         libmagickwand-6.q16-3.sce update check.
         libopenjp2-7.sce update check.
         libpaper1.sce update check.
         libpaper-utils.sce update check.
         libpoppler73.sce update check.
         libpoppler-glib8.sce update check.
         libpostproc54.sce update check.
         libraw16.sce update check.
         libreoffice-gtk3.sce update check.
         librsvg2-bin.sce update check.
         libspectre1.sce update check.
         linux-headers-generic.sce update check.
         mesa-utils.sce update check.
         minitube-2.9.sce update check.
         mkvtoolnix.sce update check.
         module-assistant.sce update check.
         ndiswrapper-modules-4.14.10-tinycore64.sce update check.
         nouveau-4.14.10-tinycore64.sce update check.
         nvidia-340.106-4.14.10-tinycore64.sce update check.
         nvidia-340-dev.sce update check.
         picard.sce update check.
         poppler-data.sce update check.
         python3-sip.sce update check.
         python-sip.sce update check.
         sce-ppa-add.sce update check.
         sce-update.sce update check.
         smplayer.sce update check.
         ssh.sce update check.
         suru-icon-theme.sce update check.
         terminology.sce update check.
         tumbler.sce update check.
         unetbootin.sce update check.
         usb-creator-gtk.sce update check.
         veracrypt-cli.sce update check.
         veracrypt-gtk.sce update check.
         virtualbox-5.2.10-host-modules-4.14.10-tinycore64.sce update check.
         virtualbox-dkms.sce update check.
         virtualbox-qt.sce update check.
         vlc.sce update check.
         webp.sce update check.
         wifi.sce update check.
         wine1.6.sce update check.
         wireless.sce update check.
         wireless-4.14.10-tinycore64.sce update check.
         xdg-user-dirs.sce update check.
         xdg-utils.sce update check.
         xfburn.sce update check.
         xfig.sce update check.
         xorg-all.sce update check.
         xorg-dev.sce update check.
 
No updates available for main or any dependency SCEs.
Command exited with non-zero status 1
real 8m 28.30s
user 5m 40.03s
sys 1m 53.19s
jason@box:~$

8 and a half minutes to check the entire SCE directory, 9.2 GB of files.

And below is simply re-importing all the above mentioned SCEs with the below script run in the SCE directory:

Code: [Select]

#!/bin/sh

for I in `ls *.sce`; do E=`basename "$I" .sce`; sce-import -n "$E"; done



Results:

Code: [Select]

* Imported xorg-dev.sce.
real 1h 17m 34s
user 45m 34.98s
sys 11m 48.18s



Only the last snippet was posted as it was too long.  1 hour and 17 min versus 8 and a half minutes.  So simply re-importing is good for an install with few or small SCEs, but the savings in time with sce-update shows with larger installs. 

Hope this explains the usefulness of sce-update. 

[sm8ps]

Many thanks for taking the time to do the test on your own, JasonW! Obviously, there is a great advantage in not re-importing extensions for which there is no update available. This is what sce-update is really useful for, no doubt about it and your results make that very clear.

Though If updates are available, I expected that sce-update and sce-import would perform about the same with a slight penalty for sce-update due to its searching for updates. What struck me is how big the penalty is, increasing the total time by a factor of 2.5. That lead me to the preliminary conclusion that sce-update was fast at checking but slow at importing.

After re-considering your statements, I realized that I was confused by the fact that sce-update worked almost instantly in my cases. It took less than 20" even with all additional repositories. This must be due to the checking for changes in the debinx-files which had not happened when I tested it after performing the update/import.

In reversed order, however, it shows that the checking part of sce-update indeed does take considerable time which explains it all. So I step back from my suspicion that sce-update might not perform well enough. I had been hoping for shorter system upgrade times but I understand now why this won't easily happen. All in all, sce-update is an indispensable tool and does work well. Thanks again for all your time and effort!

Cheers!

[Jason W]

Hi sm8ps,

I will look into the performance of sce-update when there are updates available, which I did not test in my last post.  I am always looking for better performance.  Thanks for your observations. 

[sm8ps]

Thanks for the follow-up, Jason! -- I was going to say: "why bother?" Though after re-considering the results, I do think it would be interesting to compare.

In my case, it seems that sce-update adds 150% of the time of sce-import for checking if it should upgrade or not. This does sound inefficient but only pertains to one single extension with updates indeed available. If no updates are available, it does save a lot of time, of course.

I do not really see how to compare your results to mine because the difference in performance is so striking in your case. For a one-to-one comparison, you could add or remove some extra repositories and compare sce-update to sce-import for one single extension.

I have not really understood the code, although I took a closer look at it. (My avatar says it all when it comes to my understanding of Awk.) So maybe there is no much more efficient way of finding out if an extension is due for an update or not. Still I do not really understand why the searching for an update should take more time (+150%) than the actual upgrade.

Cheers!

[Jason W]

Hi sm8ps,

I don't have extra repos, but sce-import and sce-update use the same routines in parsing them.  But I did test 'sce-import -n firefox', 'sce-update -n firefox' with no updates, and tweaked the resulting firefox.sce to trigger an update and used 'sce-update -n firefox' that caused an update.  The results are below, and none of these required downloads:

Code: [Select]

dCore-stretch:

firefox initial import:
real    1m 12.98s
user    0m 38.45s
sys     0m 12.56s

'sce-update -n firefox' full checking but with no re-import:
real    1m 32.82s
user    0m 14.15s
sys     0m 5.76s

'sce-update -n firefox' full checking with re-import:
real    2m 36.13s
user    0m 51.99s
sys     0m 17.67s

I am seeing an expected time result in sce-update that triggers an import that is roughly equal to the times of the sce-update checking plus sce-import.  I am running a machine with 6GB of RAM (~4GB recoginzed on dCore-stretch) and a 3 Ghz dual core.

[sm8ps]

Thanks again for taking the time, Jason! Your results do look like the expected behavior and there is no real point in striking a debate. Doing so nevertheless is not for the sake of argument but because you said you were always looking for ways to increase performance. Unfortunately, my Bash- and Awk-foo is not so strong that I could simply understand the source code (see my avatar ...) so I can only formulate conjectures, unfortunately.

I would expect sce-update to be quicker in discovering if an extension can be updated. Because it is enough to discover any single package in the chain of dependencies that can be updated and there is no need for checking all of the packages. sce-import on the other hand does have to touch every single package and therefore should take considerably more time. The difference in performance may vary by the position of the first package that can be updated but on average it should be about 50%.

Judging from the results, it looks to me like sce-update does not only check if some single package can be updated but seems to to a full checking for all the upgradeable packages. I do not know how exactly your tweaking of firefox.sce worked but in my case of a base extension consisting of 300+ individual packages it almost certainly was not the last one to which triggered the upgrade. I would have expected sce-update to signal much quicker that an update has been found. In my case, however, the full process of checking for updates takes more time than the full of importing the extension anew.

It is acceptable for sce-update to take (at most) as much time as sce-import to determine that there is no update available if this query is so complex. However, it should be no more and, at best, much less. Does that make sense? As I said, I have not understood the programming logic involved in sce-update and my reasoning pure speculation. If I am going astray then I shall be just happy to learn so.

Cheers!

[Jason W]

Hi sm8ps,

You make a point in that sce-update can simply trigger an update on the first found instance that requires an update on any possible package, startup scripts, or data.tar.gz.  Rather than having to go through the full checking for updates in the SCE.  I think most folks just want to know if an SCE needs updating, as any startup script, data.tar.gz, or .deb package can be a very important change but don't need to know why.

I will look into how it can be done, perhaps with a "-f" "full check" option for sce-update for those who want a full checking and output. 

[sm8ps]

That sounds promissing! I am looking forward to testing any new versions.

Cheers!

[Jason W]

It seems it is not so much the checking of all packages in an SCE that was causing the performance difference between sce-import and sce-update, but the sce-update routines needed to be adjusted with the changes made recently to sce-import.  Below is a comparison of the performance of sce-update before and after the changes made now.  All seems well, so sce-update has been updated, please test.

Code: [Select]

Size of 200-bionic.sce:
$ ls -lh 200-bionic.sce  ... 1.3G

Number of packages in 200-bionic.sce (lines in below file):
$ wc -l /usr/local/sce/200-bionic/200-bionic.md5sum
1498 /usr/local/sce/200-bionic/200-bionic.md5sum

--------

New sce-update, simple mode:
$ time sce-update -c 200-bionic.sce
....
real    1m 30.34s
user    1m 7.85s
sys     0m 25.47s

$ cat /tmp/updateavailable
====  200-bionic updates  ====
chromium-browser    ubuntu bionic security-updates

--------

New sce-update, full check mode:
$ time sce-update -cf 200-bionic.sce
....
real    2m 1.88s
user    1m 27.87s
sys     0m 36.10s

$ cat /tmp/updateavailable
====  200-bionic updates  ====
chromium-browser    ubuntu bionic security-updates
chromium-codecs-ffmpeg    ubuntu bionic security-updates
chromium-codecs-ffmpeg-extra    ubuntu bionic security-updates
dirmngr    ubuntu bionic security-updates
gnupg    ubuntu bionic security-updates
gnupg-l10n    ubuntu bionic security-updates
gnupg-utils    ubuntu bionic security-updates
gpg    ubuntu bionic security-updates
gpg-agent    ubuntu bionic security-updates
gpgconf    ubuntu bionic security-updates
gpgsm    ubuntu bionic security-updates
gpgv    ubuntu bionic security-updates
gpg-wks-client    ubuntu bionic security-updates
gpg-wks-server    ubuntu bionic security-updates
libarchive13    ubuntu bionic security-updates
libcaca0    ubuntu bionic security-updates
libexiv2-14    ubuntu bionic security-updates
libgssapi-krb5-2    ubuntu bionic security-updates
libjavascriptcoregtk-4.0-18    ubuntu bionic security-updates
libk5crypto3    ubuntu bionic security-updates
libkrb5-3    ubuntu bionic security-updates
libkrb5support0    ubuntu bionic security-updates
libnss3    ubuntu bionic security-updates
libpam-systemd    ubuntu bionic security-updates
libpolkit-agent-1-0    ubuntu bionic security-updates
libpolkit-backend-1-0    ubuntu bionic security-updates
libpolkit-gobject-1-0    ubuntu bionic security-updates
libsystemd0    ubuntu bionic security-updates
libudev1    ubuntu bionic security-updates
libwebkit2gtk-4.0-37    ubuntu bionic security-updates
libzmq5    ubuntu bionic security-updates
policykit-1    ubuntu bionic security-updates
systemd-sysv    ubuntu bionic security-updates

--------

Old sce-update:
$ time sce-update -c 200-bionic.sce
....
real    7m 27.23s
user    1m 56.17s
sys     0m 44.86s

$ cat /tmp/updateavailable
====  200-bionic updates  ====
chromium-browser    ubuntu bionic security-updates
chromium-codecs-ffmpeg    ubuntu bionic security-updates
chromium-codecs-ffmpeg-extra    ubuntu bionic security-updates
dirmngr    ubuntu bionic security-updates
gnupg    ubuntu bionic security-updates
gnupg-l10n    ubuntu bionic security-updates
gnupg-utils    ubuntu bionic security-updates
gpg    ubuntu bionic security-updates
gpg-agent    ubuntu bionic security-updates
gpgconf    ubuntu bionic security-updates
gpgsm    ubuntu bionic security-updates
gpgv    ubuntu bionic security-updates
gpg-wks-client    ubuntu bionic security-updates
gpg-wks-server    ubuntu bionic security-updates
libarchive13    ubuntu bionic security-updates
libcaca0    ubuntu bionic security-updates
libexiv2-14    ubuntu bionic security-updates
libgssapi-krb5-2    ubuntu bionic security-updates
libjavascriptcoregtk-4.0-18    ubuntu bionic security-updates
libk5crypto3    ubuntu bionic security-updates
libkrb5-3    ubuntu bionic security-updates
libkrb5support0    ubuntu bionic security-updates
libnss3    ubuntu bionic security-updates
libpam-systemd    ubuntu bionic security-updates
libpolkit-agent-1-0    ubuntu bionic security-updates
libpolkit-backend-1-0    ubuntu bionic security-updates
libpolkit-gobject-1-0    ubuntu bionic security-updates
libsystemd0    ubuntu bionic security-updates
libudev1    ubuntu bionic security-updates
libwebkit2gtk-4.0-37    ubuntu bionic security-updates
libzmq5    ubuntu bionic security-updates
policykit-1    ubuntu bionic security-updates
systemd-sysv    ubuntu bionic security-updates
 



[sm8ps]

Now that does sound exciting! The time savings really do look spectacular.

[jls]

Hi
Thanks Jason for the update and sm8ps for pushing Jason
-f needs to be adjusted in sce-update -h

[Jason W]

Thanks jls, -f option has been corrected in the -h help menu.