Trying to provide better to provides.db

[CentralWare]

Geed evening everyone!

I have a set of scripts which rebuild our local repository and I'm looking to see if anyone can help us tweak the one which creates provides.db

1. We get a list of files (filename.tcz) and the filename is sent to provides.db
2. After the filename, comes the content of filename.tcz.list (listing of all files found inside filename.tcz)
3. We add an empty line into provides.db and rinse, lather and repeat for the remaining files in the repo.

Our command which does all of this is as follows:

Code: [Select]

find *.tcz -type f -print -exec cat "${SRC}/{}.list" >> "$OUT/provides.db" \; -exec echo >> "$OUT/provides.db" \;SRC is our path to /mnt/array/tinycorelinux/(VERSION).x/(PLATFORM)/tcz
OUT is where we're going to save provides.db for processing before it's moved to SRC

Creating a FOR loop seems to take just as long as -exec above.
Any thoughts on how to make it better?

Thanks!

[Rich]

Hi CentralWare
I couldn't get your version to run, so I didn't have a benchmark
to compare to. I attached a copy of what I came up with.
I ran it against the TC14 x86 repo 3 times with a cold cache
for each run. These were the times:

Code: [Select]

rich@tcbox:~$ ./ClearCache.sh
rich@tcbox:~$ time ./BuildProvides.sh

real    0m4.728s
user    0m0.088s
sys     0m0.592s
rich@tcbox:~$ ./ClearCache.sh
rich@tcbox:~$ time ./BuildProvides.sh

real    0m4.754s
user    0m0.068s
sys     0m0.596s
rich@tcbox:~$ ./ClearCache.sh
rich@tcbox:~$ time ./BuildProvides.sh

real    0m4.715s
user    0m0.096s
sys     0m0.620s
rich@tcbox:~$

[CentralWare]

Benchmarks are about the same

Code: [Select]

# time ./BuildProvides.sh

real    0m13.649s
user    0m4.320s
sys     0m8.220s


# time ./build_prov
Creating Provides Database...
cat: can't open '/mnt/array1/tinycorelinux/14.x/x86/tcz/Xorg-7.7-lib.tcz.list': No such file or directory
cat: can't open '/mnt/array1/tinycorelinux/14.x/x86/tcz/alsa-plugins-dev.tcz.list': No such file or directory
cat: can't open '/mnt/array1/tinycorelinux/14.x/x86/tcz/geany-plugins-dev.tcz.list': No such file or directory
cat: can't open '/mnt/array1/tinycorelinux/14.x/x86/tcz/gnome-keyring-dev.tcz.list': No such file or directory
cat: can't open '/mnt/array1/tinycorelinux/14.x/x86/tcz/hplip-dev.tcz.list': No such file or directory
cat: can't open '/mnt/array1/tinycorelinux/14.x/x86/tcz/original-modules-6.1.2-tinycore.tcz.list': No such file or directory
cat: can't open '/mnt/array1/tinycorelinux/14.x/x86/tcz/original-modules-6.1.2-tinycore64.tcz.list': No such file or directory
cat: can't open '/mnt/array1/tinycorelinux/14.x/x86/tcz/strongswan-dev.tcz.list': No such file or directory

real    0m13.952s
user    0m4.970s
sys     0m8.440s
Mine throws MIA errors because of meta extensions and actual "missing" files.

Example: 14.x x86 strongswan-dev.tcz
This one's broken (missing .list file where there should be one)
NOTE: This could be just a missing file on THIS END -- which is why this system is being updated in the first place; to ensure things are properly (and efficiently) synchronized!
Nope, it's missing on that end as well: https://distro.ibiblio.org/tinycorelinux/14.x/x86/tcz/strongswan-dev.tcz.list

...whereas original-modules-6.1.2-tinycore64.tcz.list "should" be missing, but has a file size and md5 >0bytes so there's no "visible" marker to work with yet without extracting the tcz and seeing whether files exist inside it.

Thank you for your efforts!

[Rich]

Hi CentralWare

... so there's no "visible" marker to work with yet without extracting the tcz and seeing whether files exist inside it. ...

No need to extract, get a listing:

Code: [Select]

rich@tcbox:~$ unsquashfs -ll tinycorelinux/15.x/x86_64/tcz/original-modules-6.6.8-tinycore64.tcz
Parallel unsquashfs: Using 6 processors
0 inodes (0 blocks) to write

drwx------ root/root                26 2023-12-23 17:54 squashfs-root
drwxr-xr-x root/root                 3 2023-12-23 17:54 squashfs-root/lib
rich@tcbox:~$

[Rich]

Hi CentralWare
Test if extension contains files.

This contains files:

Code: [Select]

rich@tcbox:~$ unsquashfs -ll tinycorelinux/14.x/x86_64/tcz/alsamixergui.tcz | grep -q '^-'
rich@tcbox:~$ echo $?
0
rich@tcbox:~$
This contains only empty directories:

Code: [Select]

rich@tcbox:~$ unsquashfs -ll tinycorelinux/14.x/x86_64/tcz/alsa-plugins-dev.tcz | grep -q '^-'
rich@tcbox:~$ echo $?
1
rich@tcbox:~$

[CentralWare]

I'll have to see about building a squash-tools for the units, first

[Rich]

Hi CentralWare
Build notes and patch if you are interested:
http://tinycorelinux.net/14.x/x86_64/tcz/src/squashfs-tools/

[CentralWare]

Thank you, sir!  It would be an ARM build instead of an x64, but appreciation none the less!
(...or I may be able to just steal a copy of piCore's arm7 squash build IF it doesn't require kernel headers... )
Then again, even 7-zip is supposed to support squash...  never know, may be an easy-peasy fix!

[Rich]

Hi CentralWare
Oh, so maybe you want this:
http://tinycorelinux.net/14.x/armv7/tcz/src/squashfs-tools/

[Rich]

Hi CentralWare

... Then again, even 7-zip is supposed to support squash... 

I didn't know that, so I tried it.

It worked for grep.tcz.

It failed for Xorg-7.7-lib.tcz (meta-extension):

Code: [Select]

tc@E310:~$ 7z l /mnt/sda1/tce/optional/Xorg-7.7-lib.tcz

7-Zip 9.20  Copyright (c) 1999-2010 Igor Pavlov  2010-11-18
p7zip Version 9.20 (locale=C,Utf16=off,HugeFiles=on,2 CPUs)

Error: /mnt/sda1/tce/optional/Xorg-7.7-lib.tcz: Can not open file as archive

Errors: 1tc@E310:~$ echo $?
2
tc@E310:~$
Unfortunately, it also failed for numlockx.tcz which contains one file:

Code: [Select]

tc@E310:~$ 7z l /mnt/sda1/tce/optional/numlockx.tcz

7-Zip 9.20  Copyright (c) 1999-2010 Igor Pavlov  2010-11-18
p7zip Version 9.20 (locale=C,Utf16=off,HugeFiles=on,2 CPUs)

Error: /mnt/sda1/tce/optional/numlockx.tcz: Can not open file as archive

tc@E310:~$ echo $?
2
tc@E310:~$
The extension itself appears to be OK:

Code: [Select]

tc@E310:~$ unsquashfs -ll /mnt/sda1/tce/optional/numlockx.tcz
Parallel unsquashfs: Using 2 processors
1 inodes (2 blocks) to write

drwxr-xr-x tc/staff                 26 2010-12-04 21:48 squashfs-root
drwxr-xr-x tc/staff                 28 2010-12-04 21:48 squashfs-root/usr
drwxr-xr-x tc/staff                 26 2010-12-04 21:48 squashfs-root/usr/local
drwxr-xr-x tc/staff                 31 2010-12-04 21:48 squashfs-root/usr/local/bin
-rwxr-xr-x tc/staff               5700 2010-12-04 21:48 squashfs-root/usr/local/bin/numlockx
tc@E310:~$

[patrikg]

Something says to me that when you compiling apps/programs you can provide what is being supported and you can provide many options set that when compiling.

Many times you can see how many dep and includes being used for let say how to extract tar zip files, from the result of configure.

So it maybe depends on zlib, don't know if there some lib for tar.

If not the apps/programs is freestanding, supporting all things internal.
And are maybe compiled static.

That's all for me, for now. So don't be fooled by the fact that the programs support e.g. unpacking algorithms, they may depend on other libraries that they link in when the program is run.

But now i think it's internel busybox commands that are compatible with each other.

[CentralWare]

@Rich: LOL...  go figure. '7z with an attitude.

I just finished the updates to the Local Repo update utility (testing on 9.x due to the "unique" index that causes even wget not to be able to get a directory listing for /tcz) and thus far all looks good on that front (download any TCZ whose MD5.txt signature doesn't match the signature from the live repo) and then download the "listing" files (md5, sizelist, info.list, etc.) to ensure the local copy is "most recent."  The self-check/self-heal will be a work in progress as it needs to be entirely TCL portable (x86/x64/arm) and the walk through squashfs dependencies may be a bit of a stretch for the moment since the intended usage device on this end for the moment is an arm based NAS with an antique-ish operating system.

[patrikg]

Yes 7z is the best!!! , and support's many formats.
I use it a lot, even with windows files, to extract files from installer files with files within.

[Rich]

Hi CentralWare

... (testing on 9.x due to the "unique" index that causes even wget not to be able to get a directory listing for /tcz) ...

There's a simple way around that:

Code: [Select]

http://tinycorelinux.net/9.x/x86/tcz/info.lstThat works on all of the repos. I use that anytime I want to look
at a 9.x repo with my web browser.

[CentralWare]

@Rich: Yes, info.lst works as a directory listing -- I have to use md5.txt for the project, though as I need something to diff our files with the repos.
I'm guessing 9.x was an experiment on www to reduce repo downloading (as it was not repeated in 10.x) or maybe a web based file app?
I'm tempted to ask @curaga to delete those index files, but honestly...  9.x is so old it really doesn't matter on our end. (2017-ish?)
Originally I used 4.x as my intended testing ground...  I had NO idea just how many changes existed there (40% of the entire repo at least)
as we haven't done a 4.x update since ~2018 or so.  Either that, or something happened on our end where the files (file time stamps, etc.) somehow
were seen as stagnant by wget.

Nostalgic fun fact (which I now feel incredibly antiquated...)  We've been "hands dirty" with Tiny Core since somewhere shortly after 2.x was released;
I just came across a couple initrd and a kernel build from way-back (though 3.x is where we really started taking it seriously with our old PXE managed
workstations.)  September 11, 2010 ~ May ye rest in peace, my little projects...   (My login here = 2009...  how time flies!)

@patrikg: Yes, when I'm in Windows land 7-Zip is my go-to archive application - it just seems as though we've asked it to do something it doesn't like.
Then again, it's very possible WE are doing something with SquashFS that doesn't conform to exactly how SFS was first intended "by the book"
   -- who knows at this point.  (...though I doubt it!)

Okay my friends, it's been a pleasure chatting it up -- now it's time for me to get to work!  Take care!