sce.db ?

[jls]

Hi
what do you think about a file where the first columm is the package name the 2nd the sce comtaining it the 3rd the version.
If a pkg is in more then an sce is listed more times. This could maybe speed up the sce creation and update.
Thanks

[Jason W]

We have the list of packages an sce contains along with the md5sum of that package in /usr/local/sce/scename/scename.md5sum, and a file for the same info for startup scripts and -data.tar.gz files.  In any case, the time spent checking for updates is parsing the DEBINX files for the info needed, whether it be size, md5sum, version, etc.  Parsing the md5sum file is almost instantaneous.

Are you menaing a separate file like sce.list that has this data for the relevent sce? 

Thanks for the idea,

[jls]

Hi
I mean all the pkg which are included in all the extensions present in the sce dir could be listed in a single sce.db file, so there is no need to mount the sces to check them, also I think the way now is that the sce are mounted one by one.
Actually my question is also how does debian check the updates?.
since Jason prefer using md5sum rather then version numbers, this could be stored in the sce.db instead of the version numbers, even this is more user friendly, since it can happen that a user could easily check the version of a program looking in this file
thanks

[Jason W]

With Debian using APT, that is a totally different thing.  Md5sum is much more specific than version number to use as a check for file differences and that is why it was chosen.  And mounting takes a small fraction of a second.  Below is a script that takes .03 seconds to complete on my machine:

Code: [Select]

#!/bin/sh

[ -d 1 ] || mkdir 1
sudo mount wodim.sce 1
cat 1/usr/local/sce/wodim/wodim.md5sum
sudo umount 1


If you are wanting a file like sce.db to be able to find things like version number of packages, and be able to see what is in what sce, tree of deps, etc, I can make a utility to create it one day.

[jls]

I don'know how the update  routine works but i think it is sufficient to check if the version of each  package is changed. The problem is when the dep list of a pkg changes and if there is a way to recognize it.
Using this way a change of a list file without a reimport  will not work and can also cause the need of reimport of the sces that depends on it. But since a list file is made by user is human responsibility .
Thank you

[Jason W]

The existing sce.dep file is copied into the sce upon import so any changes found during update are noted and an update is suggested.  Same with the sce.lst file.  And if any new dependencies to packages in an sce during update have been added upstream they are considered and an update is triggered if they don't already exist in the deps of the packages in the sce.

[nitram]

...the time spent checking for updates is parsing the DEBINX files for the info needed, whether it be size, md5sum, version, etc.

I also don't fully understand update process, despite reviewing scripts, so just offering a query.

At ~11MB each, processing OLDDEBINX and NEWDEBINX takes time, even to run  diff  if there have been lots of changes. Something like the egrep below reduces OLDDEBINX size considerably and takes <1 second to complete on slow hardware. From previous playing around, running  diff  on these smaller grepped DEBINX files is almost instantaneous compared to the full 11mb versions. Would something like this be useful to reduce update check time?

However in it's current format, the DEBINX doesn't contain only DEB packages but also pre-built so the format is inconsistent and not all data is properly captured.

Quick to run, considerable size reduction:

Code: [Select]

tc@box:/mnt/sdb4/tce/import/debinx$ time egrep '(Package|MD5sum)' OLDDEBINX >> OLDDEBINX_grepped
real    0m 0.92s
user    0m 0.23s
sys     0m 0.29s
tc@box:/mnt/sdb4/tce/import/debinx$ ll | grep OLD
-rw-rw-r-- 1 tc staff  12M Feb 10 01:47 OLDDEBINX
-rw-r--r-- 1 tc staff 2.8M Feb 11 23:16 OLDDEBINX_grepped

OLDEBINX format:

Code: [Select]

Package: 0ad
Installed-Size: 9700
Depends: 0ad-data , 0ad-data , 0ad-data-common , 0ad-data-common , libboost-filesystem1.55.0, libc6 , libcurl3-gnutls , libenet7, libgcc1 , libgl1-mesa-glx | libgl1, libgloox12, libicu52 , libjpeg62-turbo , libminiupnpc10 , libmozjs-24-0, libnvtt2, libopenal1 , libpng12-0 , libsdl1.2debian , libstdc++6 , libvorbisfile3 , libwxbase3.0-0 , libwxgtk3.0-0 , libx11-6, libxcursor1 , libxml2 , zlib1g
Filename: pool/main/0/0ad/0ad_0.0.17-1_i386.deb
Size: 2838756
MD5sum: 47f9f448490623194647262d2a0f6243

Package: 0ad-dbg
Installed-Size: 55187
Depends: 0ad
Filename: pool/main/0/0ad/0ad-dbg_0.0.17-1_i386.deb
Size: 54226022
MD5sum: a521787684d0c22824e8a11e4341ef36

OLDDEBINX_grepped format:

Code: [Select]

Package: 0ad
MD5sum: 47f9f448490623194647262d2a0f6243
Package: 0ad-dbg
MD5sum: a521787684d0c22824e8a11e4341ef36


[Jason W]

GNU grep checks the 11mb DEBINX very fast, busybox grep is very slow.  The below script will give a comparison.  And at the speed of GNU grep, I am not sure it is worth further trimming down DEBINX any further as it is already trimmed to only contain needed info for dCore.  Try this script, one time with grep being grep in the script, then use /bb/grep in place of grep and notice the difference. 

Code: [Select]

#!/bin/sh

read DEBINX < /etc/sysconfig/tcedir/import/debinx/debinx
DEBINX="/etc/sysconfig/tcedir/import/debinx/$DEBINX"

while :; do sudo grep "^Package: file$" "$DEBINX" > /dev/null 2>&1; echo "$RANDOM"; done


[nitram]

Thanks for detailed feedback, see what you mean regarding /bb/grep, quite a difference.

From this it appears dCore already uses the fastest grep:

Code: [Select]

tc@box:~$ which grep
/bin/grep

Didn't realize DEBINX was already stripped to only what was required, thanks.