Tiny Core Linux
dCore Import Debian Packages to Mountable SCE extensions => dCore X86 => Topic started by: nitram on January 11, 2016, 08:06:14 PM
-
sce-update has been under revision, so just food for thought or maybe future feature. Running sce-update is CPU intensive and time consuming, especially on old hardware, anticipate longish even on newer system. Takes ~10 minutes with 100% processor use to systematically check 14 SCEs for updates, 6 of which are dCore pre-built that rarely change...even if the DEBINX hasn't changed.
To help with testing got into habit of backing up old DEBINX files, running diff against new versions to anticipate update flags. If this feature was built-in, a quick preliminary update check could take seconds not minutes.
When a user runs sce-update, why not archive the old DEBINX (eg. debian_jessie_main_i386_Packages.old), new DEBINX downloaded, run diff, depending on outcome:
1. Neither main or security package indexes have changed since last sce-update, SCE updates unlikely, quit or perform thorough check anyway?
2. Both the main and security package indexes have changed since last sce-update, perform thorough check?
2. The main package index has changed since last sce-update, SCE updates undetermined, perform thorough check?
3. The security package index has changed since last sce-update, security updates may be pending, perform thorough check?
Of course the non-interactive option would not pause for this. Running sce-import should also not automatically backup DEBINX, just when sce-update is run. If the user inadvertently deleted the archived DEBINX.old file(s), then sce-update would just backup the new and run through. Total backed up DEBINX files only ~10mb, potentially lots of time and CPU cycles saved checking updates.
My logic may be flawed or require revision. Maybe the DEBINX files change so frequently that adding such a feature is just a waste of time and effort. Just a thought, thanks.
-
We have to consider the debinx.* files in /etc/sysconfig/tcedir/import/debinx, as well as the PREBUILTMD5SUMLIST and PKGDATAFILEMD5SUMLIST. But here is a simple way to tell if anything has changed. It takes .05 seconds for me to run " cat debinx.* *Packages /tmp/PREBUILTMD5SUMLIST /tmp/PKGDATAFILEMD5SUMLIST > NEWFILE" to get a file of the cobined needed lists to check for change. When running debGetEnv again, if no change in the debinx or the prebuilt stuff, then no md5sum change of the resulting file. Any changes mean a change in the md5sum, of course, as well as any added or deleted /opt/debextra entries.
Costing .05 seconds and less than 15mb is small compared to sce-update running all the way through to determine if even one package anywhere has changed. Should be simple for me to implement. Thanks for the idea.
-
Costing .05 seconds and less than 15mb is small compared to sce-update running all the way through to determine if even one package anywhere has changed. Should be simple for me to implement. Thanks for the idea.
My CPU says thanks for knowing how to implement this stuff. Wondered how prebuilt packages could be captured. Will gladly test when available.
-
Availble now, please test.
-
Oh, and to test the concept the first time without having to run a full sce-update, just do CTRL-C after the updating begins, and then run sce-update again. Should turn up no updates without having to run a full sce-update.
EDIT: Found a bug, will test and upload tonight.
EDIT2 per bugfix: After commencing update and CTRL-C, then cd /etc/sysconfig/tcedir/import
then
sudo cp NEWDEBINX OLDDEBINX
and re-run sce-update for a quick test and should return no updates.
-
Bugfix uploaded, please test.
-
Ok, had to fix a routine on the server that is not reflected in the "version" command as for testing the up to date RC. Should be good now, please test.
-
hi friends,
tc@box:~$ version -r
You are running dCore-jessie:2016.01.11.23.47, the latest release candidate.
some errors, perhaps only cosmetic...
at the first try a missing md5sum-file was reported in /tmp.
i attached two screenshots.
thank you for your interest.
-
Followed NEWDEBINX instructions and encountered difficulties. Haven't been able to test the new feature or it's not working as expected. Deleted all old DEBINX and /tmp files, retried, Found Intel update so okay, Intel update failed see below. No idea if relevant, just wanted to paste, reboot and try again. My connection is fine, don't done why it timed out. Also, remember just two ####### progress bars before (main and security), not sure exactly now what's happening. Thanks.
tc@box:/mnt/sdb4/tce/import/debinx$ sce-update -a
debian
#################### 100.0% 0.0 kBps DONE
verifying download...checksum matches OK
used 11409408 local, fetched 0
#################### 100.0% 0.0 kBps DONE
verifying download...checksum matches OK
used 362496 local, fetched 0
Using the repo http://security.debian.org jessie main
debian
#################### 100.0% 0.0 kBps DONE
verifying download...checksum matches OK
used 11409408 local, fetched 0
#################### 100.0% 0.0 kBps DONE
xorg-intel
The above SCEs are about to be updated. Press Enter to proceed, y to view package updates, or Ctrl-C to abort..
Using the -u option.
Using the -n option.
Importing xorg-intel.
Using Package Index: /etc/sysconfig/tcedir/import/debinx/debian_jessie_main_i386_Packages
Using Package Index: /etc/sysconfig/tcedir/import/debinx/debian_jessie_security_i386_Packages
Using debian Mirror: http://ftp.us.debian.org/debian
Using debian Security Mirror: http://ftp.us.debian.org/debian
xorg-intel is a meta package of related debian ones.
Gathering dependency info..
Connecting to ftp.us.debian.org (64.50.233.100:80)
libxvmc1_1.0.8-2+b1_ 100% |*******************************| 24230 0:00:00 ETA
Merging libxvmc1
Connecting to ftp.us.debian.org (64.50.236.52:80)
libxcb-util0_0.3.8-3 100% |*******************************| 23030 0:00:00 ETA
Merging libxcb-util0
Merging libdrm-intel1
Connecting to ftp.us.debian.org (128.30.2.36:80)
wget: can't connect to remote host (128.30.2.36): Connection timed out
failed on download of pool/main/x/xserver-xorg-video-intel/xserver-xorg-video-intel_2.21.15-2+b2_i386.deb
wget: can't connect to remote host (128.30.2.36): Connection timed out
ar: can't open '/etc/sysconfig/tcedir/import/debs/xserver-xorg-video-intel_2.21.15-2+b2_i386.deb': No such file or directory
ar: can't open '/etc/sysconfig/tcedir/import/debs/xserver-xorg-video-intel_2.21.15-2+b2_i386.deb': No such file or directory
/usr/bin/debExtract: line 42: can't create /tmp/: Is a directory
Failure to extract xserver-xorg-video-intel_2.21.15-2+b2_i386.deb, exiting..
-
Reboot, retest, now says no updates. Don't trust yet so will go back to my old diff checks for a while. Also some output errors below. This output only took a few seconds.
tc@box:/mnt/sdb4/tce/import/debinx$ sce-update -a
debian
#################### 100.0% 0.0 kBps DONE
verifying download...checksum matches OK
used 11409408 local, fetched 0
#################### 100.0% 0.0 kBps DONE
verifying download...checksum matches OK
used 362496 local, fetched 0
Using the repo http://security.debian.org jessie main
cat: debinx.*: No such file or directory
cat: /tmp/PKGEXTRAFILEMD5SUMLIST: No such file or directory
Your SCEs are up to date.
Figured i would run sce-update -a again (no reboot). Hangs for a while, then mirror not available. Never had this issue with dCore. Paste http://ftp.us.debian.org/debian into Dillo and it loads instantly.
tc@box:/mnt/sdb4/tce/import/debinx$ sce-update -a
debian
#################### 100.0% 0.0 kBps DONE
verifying download...checksum matches OK
used 11409408 local, fetched 0
#################### 100.0% 0.0 kBps DONE
verifying download...checksum matches OK
used 362496 local, fetched 0
The debian jessie main repo of mirror http://ftp.us.debian.org/debian is not available, exiting..
Error in updating needed DEBINX files. Exiting...
/debinx directory presently looks like this:
tc@box:/mnt/sdb4/tce/import/debinx$ ll
total 23M
-rw-rw-r-- 1 tc staff 12M Jan 11 23:52 OLDDEBINX
-rw------- 1 root root 11M Jan 12 2016 debian_jessie_main_i386_Packages
-rw------- 1 root root 354K Jan 12 2016 debian_jessie_security_i386_Packages
Sorry for live commentary. Appears to hang ~30% of the time, maybe server protection. Able to run through most of the time, reports no updates but still these errors:
cat: debinx.*: No such file or directory
cat: /tmp/PKGEXTRAFILEMD5SUMLIST: No such file or directory
Your SCEs are up to date.
-
The connection issue was coincidental, changing Debian mirror fixed.
sce-update -a now runs through quickly, indicates SCEs up to date, just the cat issues already noted.
tc@box:/etc/sysconfig$ sce-update
debian
#################### 100.0% 0.0 kBps DONE
verifying download...checksum matches OK
used 11409408 local, fetched 0
#################### 100.0% 0.0 kBps DONE
verifying download...checksum matches OK
used 362496 local, fetched 0
Using the repo http://security.debian.org jessie main
cat: debinx.*: No such file or directory
cat: /tmp/PKGEXTRAFILEMD5SUMLIST: No such file or directory
Your SCEs are up to date.
-
I had made some changes after my last post here, but didn't want to keep posting. And the missing debinx.* is cosmetic, will fix tonight.
Please do:
sudo rm /etc/sysconfig/tcedir/import/debinx/OLDDEBINX
sudo rm /etc/sysconfig/tcedir/import/debinx/NEWDEBINX
And try again.
-
I see a couple issues, but mostly cosmetic. Will make a new RC tonight.
-
Uploaded fix to RC area, hopefully all is well now.
-
dCore-jessie:2016.01.12.21.09
There was an xorg-intel update there, updated. On first run, sce-update -a launched directly into checking individual extensions, presumably as updates were potentially available. Unable to attach output due to screen clear.
Just thinking outloud. When potential updates available, NEWDEBINX doesn't immediately become OLDDEBINX. In my perfect world, if changes were detected then sce-update would ask user to view diff, user can make infomed decision whether it's worth proceeding with all SCE checks. Or maybe even exit and then just check a particular SCE to save time/CPU (eg sce-update xorg-intel). Some diffs would be quite obvious that installed SCEs are not affected. Think that's why i was thinking diff in the OP, not md5sum check.
The OLDDEBINX and NEWDEBINX diff with the xorg-intel update, presumably xtrlock or maybe more triggered the update:
tc@box:/mnt/sdb4/tce/import/debinx$ diff OLDDEBINX NEWDEBINX
--- OLDDEBINX
+++ NEWDEBINX
@@ -323546,3 +323546,22 @@
xtrlock-deb2sce: 76ddbcf61e954f57de4cbc9b9f4fc4b5
xulrunner-10.0-deb2sce: cb87e753c5f157e03c5d6c2d76a43e2f
xz-utils-deb2sce: 6d0a099c18e8d49667d774b1e9d0549a
+clamav: 7f0b89a396b0ae92184d21f113b748f6
+e17: 4d8793fac534235e9ef0102c0a98679b
+e17-stable: 4d8793fac534235e9ef0102c0a98679b
+e18-plain: 4d8793fac534235e9ef0102c0a98679b
+e19-plain: 4d8793fac534235e9ef0102c0a98679b
+e20: 5864e145a570ff8ed66ea795e22371d5
+firefox: c9a6ade22ef6ca73127fe2e55349dd7c
+fluxbox: 0bcbb6918bb8adbdfe9d9addaf7b470a
+gparted: 593b3259914148c3a7d6d79c783a154b
+icewm: efd0a4ff93970f79842eafc8370f43ed
+jwm: 04f4fa5d9613bdfee73dd9d7f58d26a2
+libpam-modules: 848f814a7a61c8a5ed08cda56268029f
+libreoffice: bbbc7ed99d1683598d3cc5e79d71ea5f
+lxde: 5249893e10ea4698e48eb7de349963b0
+midori: b02713f85b579033d836432e93dac16b
+openbox: fc81a6233002989d9a539fda88e057da
+python-minimal: 28cd80c3fe790316583a1d3c1170ce3f
+xfce4: 8407272ea0fb442718436657292c1126
+xtrlock: 4f77bf4d10b6d350128eeef18f8541d8
Upon reboot, xorg-intel worked well. Re-ran sce-import -a but still getting this:
tc@box:/mnt/sdb4/tce/import/debinx$ sce-update -a
/usr/bin/sce-update: line 116: can't open /tmp/debinx: no such file
cat: /tmp/PKGEXTRAFILEMD5SUMLIST: No such file or directory
Your SCEs are up to date.
-
Ok, thanks for testing. Here is what I see. Debinx needs to be run anyway to produce the /tmp/debinx but without producing a new OLDDEBINX, I have not rebooted in testing this so I overlooked it. Will be added.
Md5sum is checked to see if the NEWDEBINX and OLDDEBINX are different in any way, and if so triggering a normal sce-update. But a diff could be offered for those who would like to view as an expert option as to whether to proceed with sce-update. Today dCore-wily had libvirt-bin and related packages added to the debinx that were not there an hour ago but a newly introduced package and not relevant to my install.
Viewing your diff, the + entries appear to be what is changed in the NEWDEBINX. Correct me if I am wrong.
If the OLDDEBINX and NEWDEBINX are different at all per md5sum, then checking an individual SCE or all or a menu selected list is still just like before. I am for the diff viewing and an informed decision for expert users, but let's first make this a stable cut after some more testing, I need to upload to a stable cut to release and then we can further add features.
-
Fixed needed detGetEnv on first run, please check.
-
Yes as per the diff output above:
--- OLDDEBINX
+++ NEWDEBINX
I don't understand how this DEBINX diff works, however. The diff from above listed Fluxbox which i use, why not updated? Already deleted, next time will open in vi, go to line #323546 or whatever and take a closer look. Maybe because Fluxbox is dCore pre-built, overrides Debian repository. But then incorrectly, i would feel obligated to run sce-update fluxbox.
Agree with getting stable first, hence code words 'sce-update has been under revision, so just food for thought or maybe future feature' and 'Just thinking outloud'. Certainly don't intend to disrupt your life with a new RC within hours, just something to think about while the topic is fresh. The work you do is incredible. This stuff is just nice to have...and save unnecessary CPU cycles. Don't consider myself expert, still learning, but i think most dCore users would appreciate this feature.
Just read your note, will re-test later tonight. Thanks.
-
Ok, I see the issue with prebuilt versus Debian as far as needing updates. This diff or md5sum check of catted debinx, prebuilt, files can only determine if something was changed, not what and where as in what repo. Is only good to determine if nothing at all was changed, then no need to run regular sce-update. Otherwise, sce-update as usual.
We can determine if this is really useful as if something is updated in Debian/Ubuntu but the prebuilt supercedes it, then an update is really not needed but is indicated by the change in Debian/Ubuntu.
It's all good, because certainly nothing lost by this addition and can save a full "sce-update -a" when no changes in debinx or perbuilt have been made, but when changes in the debinx files, it is not clear whether really needed.
With existing routines using the OLD and NEW debinx diff info, we can tell what repo packages available for update come from, and even get which SCEs are affected. And much quicker than the current sce-update routine does as it would be grepping one word per package change rather than doing a full md5sum checking of each or every SCE. Lets stay with this thought.
-
Oh, and when I see new ideas and concepts I tend to go with them when they seem they may make things more efficient or useful. Sometimes they don't work, other times they do, time and testing will tell in the end. Everyone keep the ideas coming.
-
hi friends,
i attached a screenshot about two missing files.
tc@box:~$ version -r
You are running dCore-jessie:2016.01.13.01.02, the latest release candidate.
thank you for your interest.
-
Sorry still something not right, excuse if netnomad indicated same - Dillo can't download attached screenshots properly.
After a fresh boot, didn't run any other sce-* commands or anything previously:
tc@box:~$ version -r
You are running dCore-jessie:2016.01.13.01.02, the latest release candidate.
tc@box:~$ sce-update -a
/usr/bin/sce-update: line 116: can't open /tmp/debinx: no such file
debian
#################### 100.0% 0.0 kBps DONE
verifying download...checksum matches OK
used 11409408 local, fetched 0
Debian Index synced: debian_jessie_main_i386_Packages
#################### 100.0% 0.0 kBps DONE
verifying download...checksum matches OK
used 362496 local, fetched 0
Debian Security Index synced: debian_jessie_security_i386_Packages
Using the repo http://security.debian.org jessie main
cat: /tmp/PKGEXTRAFILEMD5SUMLIST: No such file or directory
Your SCEs are up to date.
-
Ok, I will check into it tonight.
-
Bugs hopefully fixed, and added option to view diff of new and old debinx data.
-
Here is an issue I see with this. Unless "sce-update -a" is run and completed, and instead updating only one or several SCEs, the NEWDEBINX when copied to OLDDEBINX at the completion of sce-update, it gives a false impression all SCEs are up to date and therefore until a new change in DEBINX data occurs, it kind of assumes all SCEs have been updated to the latest OLDDEBINX state. Don't know how to fix or work around. Ideas welcome.
-
Just logged in to report this issue (stomach sinks). In OP told you my logic might be flawed. Never thought of it earlier as i always choose to sce-update -a , but earlier today ran regular sce-update just for testing.
Looks like either the entire OLDDEBINX idea should be scrapped or sce-update -a should be the only option. Having only update all may simplify the update script a lot. IMO dCore's strength is the ability to fairly easily run a fully up to date system. So for me personally i would never update Dillo or Iceweasel without worrying about iptables or whatever. Would want them all checked and updated.
Hopefully other regular members will provide feedback. Sorry if this idea caused a lot of work or disruption. May make things better in the end or may just be discarded. Sounded like a good idea at the time...
Other feedback that may no longer be relevant. Ran sce-update but decided to check all (pressed 1 multiple times). Almost half SCEs updated due to libpng update. Upon completion OLDDEBINX didn't move to NEWDEBINX, so running sce-update again afterwards continued to flag potential updates (perpetual updates as working off of now outdated DEBINX). Reboot still flagged the same updates, so my /debinx still looks like this (think should just be OLDDEBINX):
tc@box:/mnt/sdb4/tce/import/debinx$ ll
total 35M
-rw-rw-r-- 1 tc staff 12M Jan 13 21:31 NEWDEBINX
-rw-rw-r-- 1 tc staff 12M Jan 12 19:20 OLDDEBINX
-rw--w---- 1 tc staff 11M Jan 14 2016 debian_jessie_main_i386_Packages
-rw--w---- 1 tc staff 357K Jan 14 2016 debian_jessie_security_i386_Packages
Not sure, maybe an argument for running sce-update -a only. Maybe libpng bad example, but couldn't there potentially be version conflicts if multiple versions of some dependencies loaded at the same time.
In my recent libpng update all these were affected, never would have thought, but now i know they are all up to date:
SCE: Xprogs
Available package updates:
None
SCE: conky
Available package updates:
libpng12-0 debian jessie main
SCE: dillo
Available package updates:
libpng12-0 debian jessie main
SCE: emelfm
Available package updates:
None
SCE: fluxbox
Available package updates:
libpng12-0 debian jessie main
SCE: graphics-3.16.6-tinycore
Available package updates:
None
SCE: gtkfind
Available package updates:
None
SCE: iceweasel
Available package updates:
libpng12-0 debian jessie main
SCE: iptables
Available package updates:
None
SCE: netfilter-3.16.6-tinycore
None
SCE: xorg-intel
Available package updates:
libpng12-0 debian jessie main
-
hi friends,
i attached my screenshot with a probably small cosmetic issue.
the overall functions are working now fine for me.
thank you for your work.
-
If we use this for sce-update -a, all should be good. But only then, of course. Nothing is lost, nor is there any misspent time. In fact, much gained for the sce-update -a routine. Because with sce-update -a is where the most time and machine requirements is called upon, and is probably the most used option of sce-update, at least for me.
Let's work this into sce-update -a and this should save much time and resourse requirements. Tomorrow I will add this to only sce-update -a and we should be good. Sorry for the false "It may not work" alarm. Any bugs as to moving NEWDEBINX to OLDDEBINX can be fixed.
But I just tonight noticed that it may not work for all sce-update options.
-
Think you're correct, sorry for jumping the gun. Additional feedback. Previous cat type errors appear fixed. Like ability to review DEBINX diff, worked well.
Re-ran with sce-update -a , duplicate 'Checking for updates of existing SCEs..' line:
Using the repo http://security.debian.org jessie main
Sce-update is about to proceed. Press Enter to proceed, y to view the
difference in new and old debinx data, or Ctrl-C to abort..
Checking for updates of existing SCEs..
Checking for updates of existing SCEs..
Searching for available updates for all existing SCEs.
Xprogs: Checkup for updates..
After running sce-update -a same DEBINX issue noted above. NEWDEBINX didn't move to and overwrite OLDDEBINX upon completion, which would trigger perpetual update checks. [Edit] ...but when i run sce-update -a yet again, then it finally overwrites NEWDEBINX to OLDDEBINX but get this error:
tc@box:/mnt/sdb4/tce/import/debinx$ sce-update -a
debian
#################### 100.0% 24.6 kBps DONE
verifying download...checksum matches OK
used 11409408 local, fetched 0
Debian Index synced: debian_jessie_main_i386_Packages
#################### 100.0% 0.0 kBps DONE
verifying download...checksum matches OK
used 366592 local, fetched 0
Debian Security Index synced: debian_jessie_security_i386_Packages
Using the repo http://security.debian.org jessie main
Your SCEs are up to date.
/usr/bin/sce-update: exit: line 175: Illegal number: 0-
-
Uploaded a fix, hopefully all is good now, the OLDDEBINX/NEWDEBINX thing is only done with "sce-update -a".
-
Solid. Tested a few times with reboots. No errors reported. Running sce-update then selecting items leaves DEBINX untouched. Running sce-update -a updates extensions as required and updates OLDDEBINX. Re-testing sce-update -a after system fully up to date exits quickly with 'no updates available'.
The new DEBINX system for sce-update -a is sweet. When no potential updates available, even on slow hardware takes only ~5 seconds to report 'No updates'. Only tweak might be to reduce the sleep command upon exit to get cursor back faster.
Thanks for your hard work Jason :)
-
Good news all appears to be working. I think the sleep command can be removed, will do so next cut.
-
I can confirm it as flawless for me.
-
I think we are good for a release, this one is surely as or more stable than the current relase.
-
Hi
is the old / new debix thingh eliminated if one after an sce-update -a does a sce-update only on some or one sce?, so a subequent sce-update -a doesn't use old/new debinx and instead check all the sces?
thanks
-
'sce-update -a' updates any SCEs needing updating, whether one or all. Either way, all are up to date with the NEWDEBINX, which is then copied to OLDDEBINX after the updating is complete.
All 'sce-update -a' sessions use the old/new debinx checking. And the NEWDEBINX is copied to old only after the updating of any needed SCEs is complete.
I actually have a plan to make this quick check available to all SCEs with any use of sce-update by the use of a 'packagename'.sce.debinx file containing the md5sum of the debinx it was created with, which would be checked against the NEWDEBINX.
-
New RC uploaded that now creates on sce-import packagename.sce.debinx files that contain the md5sum of the NEWDEBINX data they were created with. This way, when the .debinx file exists, any SCE gets a quick update check with any use of sce-update.
-
Why is development slowing, 24 hours from stable release to new RC! Appears to work well. New sce-import creates an *.sce.debinx file. Running sce-update then selecting that SCE for update check gets almost immediate 'No updates available...' message. Just cosmetic, output indicates no updates available twice.
nano
You are about to check for updates to the above SCEs. y to continue, n to exit. (y/N): y
Checking for updates for nano.sce..
nano is up to date.
No updates available for main or dependency SCEs.
No updates available for chosen SCEs at this time.
-
lol. Cosmetic fixes to sce-update now in the latest RC. Hopefully all is good, we will know when there are updates available to the dCore port we are testing with. But being a simple md5sum check, the quick just be working. And the updating aside from that when there are package updates hopefully is now solid.
-
New RC uploaded that now creates on sce-import packagename.sce.debinx files that contain the md5sum of the NEWDEBINX data they were created with. This way, when the .debinx file exists, any SCE gets a quick update check with any use of sce-update.
This new packagename.sce.debinx system seems pretty good, but as described only gets created with a fresh import. Haven't had any recent updates, if an SCE gets updated then it presumably gets re-imported along with a fresh packagename.sce.debinx file?
If so that's great. Would there be any value or even possible to also create this packagename.sce.debinx file when an individual SCE is simply update checked, then rechecking again should be instant. Maybe irrelevant as new dCore users will build new SCEs with the helpful *.sce.debinx files already in place. More an issue for existing users that haven't needed to update or import the extension recently. Just wondering...
-
A new *.sce.debinx file gets created with any import whether from 'sce-import filename' or an update that includes re-importing when running 'sce-update'.
I will look into creating an *.sce.debinx when a successful update check is run and there are no updates available, makes sense.
-
This would involve a bit of code change inside of loops and involving SCE dependencies. Would probably as with any code change introduce bugs and mean time for us spent in testing, fixes, etc. Since this is a temporary situation, the lack of .sce.debinx files, probably better to leave the code alone and soon the missing .sce.debinx files will work itself out. And then we can spend our time on other issues that won't just work themselves out. Thanks for the input, and as always, keep the ideas coming.
Oh, and what I did was a similar below command to create the needed .sce.debinx files to in the future update quickly with though it takes some machine power:
for I in `ls /etc/sysconfig/tcedir/sce/*.sce`; do D=`basename "$I" .sce`; sce-import -n "$D"; done
-
No problem, thanks for checking it out, keeping it simple probably better.
-
Interesting experimented, wanted to share. Slowly been re-importing SCEs, couple every reboot, to produce the new /sce/*.sce.debinx files. Last SCE without a debinx was Iceweasel:
Iceweasel update check old way/without debinx file:
time sce-update iceweasel
.
No updates available for iceweasel.sce or dependency SCEs.
real 4m 11.78s
user 2m 40.74s
sys 0m 57.27s
Re-importing Iceweasel (local DEBs re-merged/no updates available):
time sce-import -n iceweasel
.
Finished importing iceweasel.sce.
real 4m 53.10s
user 2m 15.04s
sys 1m 26.07s
Icweasel update check after iceweasel.sce.debinx created:
time sce-update iceweasel
.
iceweasel.sce is up to date.
real 0m 5.39s
user 0m 0.92s
sys 0m 1.27s
The results speak for themselves, thanks again Jason for your hard work. Though re-importing was marginally slower than the old update check, once the *.sce.debinx file is created the re-import process has already paid for itself before a 2nd update check can be completed (if no updates available). Maybe Jason knows of a quicker way to produce this debinx, i don't and this works well. Possible reasons to hold off on re-imports:
- it doesn't matter, you never check for updates
- DEBs are not stored locally, re-downloading would slow process and eat bandwidth
- you're expecting a pending update anyway, then just wait
-
*,sce.debinx files created upon successful sce-update with or without having to re-import. On further looking it was just adding 2 or 3 lines to importupdatecheck. Also found a couple bugs while testing. Available in new RC.
-
Just been playing around a bit with scripts, you wanted ideas, didn't expend much effort as i'm likely off base, logic flaw and obviously don't understand the entire update process. The new DEBINX system is good but everytime there's the slightest update/change the long version SCE update check is initiated. Not sure if this would be any better than the current method.
Just a cursory emelfm pre-built example. Adding this to ~line 500 of deb2sce creates a /tmp/emelfm.sce.md5.list file:
sudo md5sum "${DEBINFO}".tar.gz >> /tmp/"$TARGET".sce.md5.list
if [ -f "${DEBINFO}".tar.gz ] && [ `/bb/md5sum "${DEBINFO}".tar.gz | cut -f1 -d" "` == `grep "^$DEBINFO": /tmp/PREBUILTMD5SUMLIST | cut -f2 -d" "` ]; then
echo "${DEBINFO}: `/bb/md5sum "${DEBINFO}".tar.gz | cut -f1 -d" "`" >> "$IMPORT"/"$TARGET"/usr/local/sce/"$TARGET"/"$TARGET".md5sum
echo "Merging "${DEBINFO}""
sudo md5sum "${DEBINFO}".tar.gz >> /tmp/"$TARGET".sce.md5.list
tar xf "${DEBINFO}".tar.gz -C "$IMPORT"/"$TARGET"
if [ "$?" != 0 ]
then
echo "Failed merging of "${DEBINFO}".tar.gz."
echo "${DEBINFO}".tar.gz >> /tmp/import.log
fi
emelfm.sce.md5.list file contents after import:
61b2f7d81ad473e3ffe9a47c5a750344 glib1.tar.gz
5ed16326225ff7c35742e5318fd298a8 gtk1.tar.gz
a17f00793e04af874a8a2ce40953a99d emelfm.tar.gz
/sce:
tc@box:/mnt/sdb4/tce/sce$ ls | grep emelfm
emelfm.sce
emelfm.sce.debinx
emelfm.sce.md5.list
emelfm.sce.md5.txt
Each SCE would have this emelfm.sce.md5.list file in /sce, which would contain md5sums of every dependency contained in the extension. Some fancy grep, sed loop could relatively quickly compare individual dependency md5sums against NEWDEBINX, first difference triggers re-import. The current NEWDEBINX format is, however, inconsistent so it wouldn't work well at present. Just a thought, thanks.
-
Actually, what you have just spoke of is the long way, the correct order of debinx files is checked against.
Load emelfm.sce and check the contents of /usr/local/sce/emelfm/emelfm.md5sum, that is the md5sum data that is used.
-
I see, thanks! Storing this md5sum file outside the *.sce probably not beneficial for speed as the extension must only be mounted to retrieve the md5sum file, which takes only fraction of a second, as opposed to a slow unsquash. So i guess barking up wrong tree. Fun delving deeper into scripts, now at importupdatecheck. Thanks.