sce-update, possible quick check feature

[nitram]

sce-update has been under revision, so just food for thought or maybe future feature. Running sce-update is CPU intensive and time consuming, especially on old hardware, anticipate longish even on newer system. Takes ~10 minutes with 100% processor use to systematically check 14 SCEs for updates, 6 of which are dCore pre-built that rarely change...even if the DEBINX hasn't changed.

To help with testing got into habit of backing up old DEBINX files, running diff against new versions to anticipate update flags. If this feature was built-in, a quick preliminary update check could take seconds not minutes.

When a user runs sce-update, why not archive the old DEBINX (eg. debian_jessie_main_i386_Packages.old), new DEBINX downloaded, run diff, depending on outcome:

1. Neither main or security package indexes have changed since last sce-update, SCE updates unlikely, quit or perform thorough check anyway?
2. Both the main and security package indexes have changed since last sce-update, perform thorough check?
2. The main package index has changed since last sce-update, SCE updates undetermined, perform thorough check?
3. The security package index has changed since last sce-update, security updates may be pending, perform thorough check?

Of course the non-interactive option would not pause for this. Running sce-import should also not automatically backup DEBINX, just when sce-update is run. If the user inadvertently deleted the archived DEBINX.old file(s), then sce-update would just backup the new and run through. Total backed up DEBINX files only ~10mb, potentially lots of time and CPU cycles saved checking updates.

My logic may be flawed or require revision. Maybe the DEBINX files change so frequently that adding such a feature is just a waste of time and effort. Just a thought, thanks.

[Jason W]

We have to consider the debinx.* files in /etc/sysconfig/tcedir/import/debinx, as well as the PREBUILTMD5SUMLIST and PKGDATAFILEMD5SUMLIST.  But here is a simple way to tell if anything has changed.  It takes .05 seconds for me to run " cat debinx.* *Packages /tmp/PREBUILTMD5SUMLIST /tmp/PKGDATAFILEMD5SUMLIST  > NEWFILE" to get a file of the cobined needed lists to check for change.  When running debGetEnv again, if no change in the debinx or the prebuilt stuff, then no md5sum change of the resulting file.  Any changes mean a change in the md5sum, of course, as well as any added or deleted /opt/debextra entries.

Costing .05 seconds and less than 15mb is small compared to sce-update running all the way through to determine if even one package anywhere has changed.  Should be simple for me to implement.  Thanks for the idea.

[nitram]

Costing .05 seconds and less than 15mb is small compared to sce-update running all the way through to determine if even one package anywhere has changed.  Should be simple for me to implement.  Thanks for the idea.

My CPU says thanks for knowing how to implement this stuff. Wondered how prebuilt packages could be captured. Will gladly test when available.

[Jason W]

Availble now, please test.

[Jason W]

Oh, and to test the concept the first time without having to run a full sce-update, just do CTRL-C after the updating begins, and then run sce-update again.  Should turn up no updates without having to run a full sce-update.

EDIT: Found a bug, will test and upload tonight.

EDIT2 per bugfix: After commencing update and CTRL-C, then cd /etc/sysconfig/tcedir/import
then
sudo cp NEWDEBINX OLDDEBINX

and re-run sce-update for a quick test and should return no updates.

[Jason W]

Bugfix uploaded, please test.

[Jason W]

Ok, had to fix a routine on the server that is not reflected in the "version" command as for testing the up to date RC.  Should be good now, please test.

[netnomad]

hi friends,

tc@box:~$ version -r
You are running dCore-jessie:2016.01.11.23.47, the latest release candidate.

some errors, perhaps only cosmetic...
at the first try a missing md5sum-file was reported in /tmp.
i attached two screenshots.

thank you for your interest.

[nitram]

Followed NEWDEBINX instructions and encountered difficulties. Haven't been able to test the new feature or it's not working as expected. Deleted all old DEBINX and /tmp files, retried, Found Intel update so okay, Intel update failed see below. No idea if relevant, just wanted to paste, reboot and try again. My connection is fine, don't done why it timed out. Also, remember just two ####### progress bars before (main and security), not sure exactly now what's happening. Thanks.

Code: [Select]

tc@box:/mnt/sdb4/tce/import/debinx$ sce-update -a
debian
#################### 100.0% 0.0 kBps DONE   

verifying download...checksum matches OK
used 11409408 local, fetched 0
#################### 100.0% 0.0 kBps DONE   

verifying download...checksum matches OK
used 362496 local, fetched 0
Using the repo http://security.debian.org jessie main
debian
#################### 100.0% 0.0 kBps DONE   

verifying download...checksum matches OK
used 11409408 local, fetched 0
#################### 100.0% 0.0 kBps DONE   
xorg-intel
 
The above SCEs are about to be updated.  Press Enter to proceed, y to view package updates, or Ctrl-C to abort..

Using the -u option.
Using the -n option.
 
 
Importing xorg-intel.
 
 
Using Package Index: /etc/sysconfig/tcedir/import/debinx/debian_jessie_main_i386_Packages
 
Using Package Index: /etc/sysconfig/tcedir/import/debinx/debian_jessie_security_i386_Packages
 
Using debian Mirror: http://ftp.us.debian.org/debian
 
Using debian Security Mirror: http://ftp.us.debian.org/debian

xorg-intel is a meta package of related debian ones.
Gathering dependency info..
Connecting to ftp.us.debian.org (64.50.233.100:80)
libxvmc1_1.0.8-2+b1_ 100% |*******************************| 24230   0:00:00 ETA
Merging libxvmc1
Connecting to ftp.us.debian.org (64.50.236.52:80)
libxcb-util0_0.3.8-3 100% |*******************************| 23030   0:00:00 ETA
Merging libxcb-util0
Merging libdrm-intel1
Connecting to ftp.us.debian.org (128.30.2.36:80)
wget: can't connect to remote host (128.30.2.36): Connection timed out
failed on download of pool/main/x/xserver-xorg-video-intel/xserver-xorg-video-intel_2.21.15-2+b2_i386.deb
wget: can't connect to remote host (128.30.2.36): Connection timed out
ar: can't open '/etc/sysconfig/tcedir/import/debs/xserver-xorg-video-intel_2.21.15-2+b2_i386.deb': No such file or directory
ar: can't open '/etc/sysconfig/tcedir/import/debs/xserver-xorg-video-intel_2.21.15-2+b2_i386.deb': No such file or directory
/usr/bin/debExtract: line 42: can't create /tmp/: Is a directory
Failure to extract xserver-xorg-video-intel_2.21.15-2+b2_i386.deb, exiting..



[nitram]

Reboot, retest, now says no updates. Don't trust yet so will go back to my old diff checks for a while. Also some output errors below. This output only took a few seconds.

Code: [Select]

tc@box:/mnt/sdb4/tce/import/debinx$ sce-update -a
debian
#################### 100.0% 0.0 kBps DONE   

verifying download...checksum matches OK
used 11409408 local, fetched 0
#################### 100.0% 0.0 kBps DONE   

verifying download...checksum matches OK
used 362496 local, fetched 0
Using the repo http://security.debian.org jessie main
cat: debinx.*: No such file or directory
cat: /tmp/PKGEXTRAFILEMD5SUMLIST: No such file or directory
Your SCEs are up to date.

Figured i would run  sce-update -a  again (no reboot). Hangs for a while, then mirror not available. Never had this issue with dCore. Paste http://ftp.us.debian.org/debian into Dillo and it loads instantly.

Code: [Select]

tc@box:/mnt/sdb4/tce/import/debinx$ sce-update -a
debian
#################### 100.0% 0.0 kBps DONE   

verifying download...checksum matches OK
used 11409408 local, fetched 0
#################### 100.0% 0.0 kBps DONE   

verifying download...checksum matches OK
used 362496 local, fetched 0
The debian jessie main repo of mirror http://ftp.us.debian.org/debian is not available, exiting..
Error in updating needed DEBINX files.  Exiting...

/debinx directory presently looks like this:

Code: [Select]

tc@box:/mnt/sdb4/tce/import/debinx$ ll
total 23M
-rw-rw-r-- 1 tc   staff  12M Jan 11 23:52 OLDDEBINX
-rw------- 1 root root   11M Jan 12  2016 debian_jessie_main_i386_Packages
-rw------- 1 root root  354K Jan 12  2016 debian_jessie_security_i386_Packages

Sorry for live commentary. Appears to hang ~30% of the time, maybe server protection. Able to run through most of the time, reports no updates but still these errors:

Code: [Select]

cat: debinx.*: No such file or directory
cat: /tmp/PKGEXTRAFILEMD5SUMLIST: No such file or directory
Your SCEs are up to date.


[nitram]

The connection issue was coincidental, changing Debian mirror fixed.

sce-update -a now runs through quickly, indicates SCEs up to date, just the cat issues already noted.

Code: [Select]

tc@box:/etc/sysconfig$ sce-update
debian
#################### 100.0% 0.0 kBps DONE   

verifying download...checksum matches OK
used 11409408 local, fetched 0
#################### 100.0% 0.0 kBps DONE   

verifying download...checksum matches OK
used 362496 local, fetched 0
Using the repo http://security.debian.org jessie main
cat: debinx.*: No such file or directory
cat: /tmp/PKGEXTRAFILEMD5SUMLIST: No such file or directory
Your SCEs are up to date.


[Jason W]

I had made some changes after my last post here, but didn't want to keep posting.  And the missing debinx.* is cosmetic, will fix tonight. 

Please do:

sudo rm /etc/sysconfig/tcedir/import/debinx/OLDDEBINX

sudo rm /etc/sysconfig/tcedir/import/debinx/NEWDEBINX

And try again.

[Jason W]

I see a couple issues, but mostly cosmetic.  Will make a new RC tonight.

[Jason W]

Uploaded fix to RC area, hopefully all is well now.

[nitram]

dCore-jessie:2016.01.12.21.09

There was an xorg-intel update there, updated. On first run,  sce-update -a  launched directly into checking individual extensions, presumably as updates were potentially available. Unable to attach output due to screen clear.

Just thinking outloud. When potential updates available, NEWDEBINX doesn't immediately become OLDDEBINX. In my perfect world, if changes were detected then sce-update would ask user to view diff, user can make infomed decision whether it's worth proceeding with all SCE checks. Or maybe even exit and then just check a particular SCE to save time/CPU (eg sce-update xorg-intel). Some diffs would be quite obvious that installed SCEs are not affected. Think that's why i was thinking diff in the OP, not md5sum check.

The OLDDEBINX and NEWDEBINX diff with the xorg-intel update, presumably xtrlock or maybe more triggered the update:

Code: [Select]

tc@box:/mnt/sdb4/tce/import/debinx$ diff OLDDEBINX NEWDEBINX
--- OLDDEBINX
+++ NEWDEBINX
@@ -323546,3 +323546,22 @@
 xtrlock-deb2sce: 76ddbcf61e954f57de4cbc9b9f4fc4b5
 xulrunner-10.0-deb2sce: cb87e753c5f157e03c5d6c2d76a43e2f
 xz-utils-deb2sce: 6d0a099c18e8d49667d774b1e9d0549a
+clamav: 7f0b89a396b0ae92184d21f113b748f6
+e17: 4d8793fac534235e9ef0102c0a98679b
+e17-stable: 4d8793fac534235e9ef0102c0a98679b
+e18-plain: 4d8793fac534235e9ef0102c0a98679b
+e19-plain: 4d8793fac534235e9ef0102c0a98679b
+e20: 5864e145a570ff8ed66ea795e22371d5
+firefox: c9a6ade22ef6ca73127fe2e55349dd7c
+fluxbox: 0bcbb6918bb8adbdfe9d9addaf7b470a
+gparted: 593b3259914148c3a7d6d79c783a154b
+icewm: efd0a4ff93970f79842eafc8370f43ed
+jwm: 04f4fa5d9613bdfee73dd9d7f58d26a2
+libpam-modules: 848f814a7a61c8a5ed08cda56268029f
+libreoffice: bbbc7ed99d1683598d3cc5e79d71ea5f
+lxde: 5249893e10ea4698e48eb7de349963b0
+midori: b02713f85b579033d836432e93dac16b
+openbox: fc81a6233002989d9a539fda88e057da
+python-minimal: 28cd80c3fe790316583a1d3c1170ce3f
+xfce4: 8407272ea0fb442718436657292c1126
+xtrlock: 4f77bf4d10b6d350128eeef18f8541d8

Upon reboot, xorg-intel worked well. Re-ran  sce-import -a  but still getting this:

Code: [Select]

tc@box:/mnt/sdb4/tce/import/debinx$ sce-update -a
/usr/bin/sce-update: line 116: can't open /tmp/debinx: no such file
cat: /tmp/PKGEXTRAFILEMD5SUMLIST: No such file or directory
Your SCEs are up to date.