Why does sce-update take so much time (even more than sce-import)?

dCore Import Debian Packages to Mountable SCE extensions > dCore X86

(1/6) > >>

sm8ps:
Greetings and happy new year! I have not been very successful with my dCore adventures last year and so I am hoping that 2019 will allow more break-throughs, for myself and also for everybody else!

I have been tinkering with dCore upgrades for a long time and while at it I had quite enough opportunities for wondering why sce-update takes so long. It even seems that it takes more time to update a package than to simply re-import it. Today I decided to take a closer look on a netbook with the a base extension called X-LIST that contains the following packages:

--- Code: ---pm-utils, graphics-4.14.10-tinycore, xorg, xorg-intel, xserver-xorg-input-synaptics,
arandr, openbox, flwm, lxpanel, xinput, dbus, dbus-x11, suru-icon-theme
--- End code ---

Including dependencies, there are 380 packages to be imported. I compared updating with 'sce-update -rn' to (re-)importing with 'sce-import -rln'. I repeated the test twice in order to eliminate the effect of downloading (because packages are taken from 'tce/import/debs/' if available).

Furthermore, I compared two different sets of repositories under '/opt/debextra/': A) only three PPAs, B) the same three PPAs plus all the following Ubuntu repositories:

--- Code: ---bionic-backports-main, bionic-backports-multiverse,bionic-backports-restricted, bionic-backports-universe,
bionic-main, bionic-multiverse, bionic-restricted, bionic-universe,
bionic-updates-main, bionic-updates-multiverse, bionic-updates-restricted, bionic-updates-universe

--- End code ---
The PPAs just happened to be there and do not play any particular role.

Here the results:

--- Code: ---sce-update -rn X-LIST A) 4'55" B) 13'29"
sce-import -rln X-LIST A) 1'59" B) 6'54"

--- End code ---
Interpretation:

* vertically: sce-update takes two and a half times longer than sce-import
* horizontally: extra repositories increase the execution time for both commands considerablyThis is on dCore-bionic:2018.08.10.22.24. Grep is imported and thus not Busybox-grep.

Is this all expected behavior?

Jason W:
Hi sm8ps

The short answer is yes, sce-update by it's nature takes longer than sce-import. And extra repos do add to that time for both sce-update and sce-import.

Much effort has been spent in streamlining sce-update and sce-import for performance. 100% accuracy is essential, but performance is also important. I have tried different routines using variations of awk, grep, sed, and other commands in the scripts to speed things up but the current is the best performing way found so far.

sm8ps:
Thanks for your answer, JasonW! I am aware that there is much effort behind all the sce-tools. Nevertheless, I am wondering why to bother with using sce-update at all? Probably because it does tell when there are no updates necessary as opposed to sce-import. Even so, could a combination of sce-update check and "blindly" sce-importing anew perform better? -- This is all very naive as I do not know the code behind the tools. So I would be grateful for another short answer.

Cheers!

Jason W:
Here is my result from using sce-update on my SCE directory, all SCEs being checked, with below the size of my SCE directory, which is about 9.2 GB not counting the update subdirectory:

--- Code: ---jason@box:/mnt/sda1/tceimport-bionic64/sce$ du -h
1.8G ./update
11G .

--- End code ---

Below is the sce-update of the SCE directory, with checking all SCEs and no re-importing:

--- Code: ---jason@box:~$ time sce-update -an
* Using nice level 19.
* Using the -a option.
* Using the -n option.
* DEBINX sync based on ubuntu bionic.
* Debian index sync: ubuntu_bionic_main_amd64_Packages
* Debian security index sync: ubuntu_bionic_security_amd64_Packages
* Using repo: http://security.ubuntu.com/ubuntu bionic main
Checking all system SCEs for updates:

200-bionic.sce update check.
alsa-utils.sce update check.
atril.sce update check.
audacity.sce update check.
avidemux2.6-qt.sce update check.
bionic-desk.sce update check.
brasero.sce update check.
cdrdao.sce update check.
cheese.sce update check.
chromium-browser.sce update check.
dCore-chrome-stable-installer.sce update check.
dCore-firefox-installer.sce update check.
dCore-google-chrome-stable-installer.sce update check.
dCorePlus-bionic64.sce update check.
dCore-usbinstall.sce update check.
devede.sce update check.
dosfstools.sce update check.
e17.sce update check.
e3.sce update check.
emelfm2.sce update check.
enlightenment.sce update check.
evince.sce update check.
ffmpeg.sce update check.
file.sce update check.
firefox.sce update check.
firefox-latest.sce update check.
fonts-freefont-ttf.sce update check.
gdb-dbg.sce update check.
git.sce update check.
gnome-screensaver.sce update check.
google-chrome.sce update check.
gparted.sce update check.
graphics-4.14.10-tinycore64.sce update check.
grep.sce update check.
icewm.sce update check.
imagemagick.sce update check.
k3b.sce update check.
kernel-all-4.14.10-tinycore64.sce update check.
leafpad.sce update check.
libbullet2.87.sce update check.
libgail-3-0.sce update check.
libgconf2-4.sce update check.
libgles2.sce update check.
libgtk-3-bin.sce update check.
libk3b7-extracodecs.sce update check.
libmadlib.sce update check.
libmad-ocaml.sce update check.
libmagickwand-6.q16-3.sce update check.
libopenjp2-7.sce update check.
libpaper1.sce update check.
libpaper-utils.sce update check.
libpoppler73.sce update check.
libpoppler-glib8.sce update check.
libpostproc54.sce update check.
libraw16.sce update check.
libreoffice-gtk3.sce update check.
librsvg2-bin.sce update check.
libspectre1.sce update check.
linux-headers-generic.sce update check.
mesa-utils.sce update check.
minitube-2.9.sce update check.
mkvtoolnix.sce update check.
module-assistant.sce update check.
ndiswrapper-modules-4.14.10-tinycore64.sce update check.
nouveau-4.14.10-tinycore64.sce update check.
nvidia-340.106-4.14.10-tinycore64.sce update check.
nvidia-340-dev.sce update check.
picard.sce update check.
poppler-data.sce update check.
python3-sip.sce update check.
python-sip.sce update check.
sce-ppa-add.sce update check.
sce-update.sce update check.
smplayer.sce update check.
ssh.sce update check.
suru-icon-theme.sce update check.
terminology.sce update check.
tumbler.sce update check.
unetbootin.sce update check.
usb-creator-gtk.sce update check.
veracrypt-cli.sce update check.
veracrypt-gtk.sce update check.
virtualbox-5.2.10-host-modules-4.14.10-tinycore64.sce update check.
virtualbox-dkms.sce update check.
virtualbox-qt.sce update check.
vlc.sce update check.
webp.sce update check.
wifi.sce update check.
wine1.6.sce update check.
wireless.sce update check.
wireless-4.14.10-tinycore64.sce update check.
xdg-user-dirs.sce update check.
xdg-utils.sce update check.
xfburn.sce update check.
xfig.sce update check.
xorg-all.sce update check.
xorg-dev.sce update check.

No updates available for main or any dependency SCEs.
Command exited with non-zero status 1
real 8m 28.30s
user 5m 40.03s
sys 1m 53.19s
jason@box:~$

--- End code ---

8 and a half minutes to check the entire SCE directory, 9.2 GB of files.

And below is simply re-importing all the above mentioned SCEs with the below script run in the SCE directory:

--- Code: ---#!/bin/sh

for I in `ls *.sce`; do E=`basename "$I" .sce`; sce-import -n "$E"; done

--- End code ---

Results:

--- Code: ---* Imported xorg-dev.sce.
real 1h 17m 34s
user 45m 34.98s
sys 11m 48.18s

--- End code ---

Only the last snippet was posted as it was too long. 1 hour and 17 min versus 8 and a half minutes. So simply re-importing is good for an install with few or small SCEs, but the savings in time with sce-update shows with larger installs.

Hope this explains the usefulness of sce-update.

sm8ps:
Many thanks for taking the time to do the test on your own, JasonW! Obviously, there is a great advantage in not re-importing extensions for which there is no update available. This is what sce-update is really useful for, no doubt about it and your results make that very clear.

Though If updates are available, I expected that sce-update and sce-import would perform about the same with a slight penalty for sce-update due to its searching for updates. What struck me is how big the penalty is, increasing the total time by a factor of 2.5. That lead me to the preliminary conclusion that sce-update was fast at checking but slow at importing.

After re-considering your statements, I realized that I was confused by the fact that sce-update worked almost instantly in my cases. It took less than 20" even with all additional repositories. This must be due to the checking for changes in the debinx-files which had not happened when I tested it after performing the update/import.

In reversed order, however, it shows that the checking part of sce-update indeed does take considerable time which explains it all. So I step back from my suspicion that sce-update might not perform well enough. I had been hoping for shorter system upgrade times but I understand now why this won't easily happen. All in all, sce-update is an indispensable tool and does work well. Thanks again for all your time and effort!

Cheers!

Navigation

[0] Message Index

[#] Next page

Go to full version