Tiny Core Linux
General TC => General TC Talk => Topic started by: CentralWare on January 26, 2024, 01:53:01 PM
-
Geed evening everyone!
I have a set of scripts which rebuild our local repository and I'm looking to see if anyone can help us tweak the one which creates provides.db
1. We get a list of files (filename.tcz) and the filename is sent to provides.db
2. After the filename, comes the content of filename.tcz.list (listing of all files found inside filename.tcz)
3. We add an empty line into provides.db and rinse, lather and repeat for the remaining files in the repo.
Our command which does all of this is as follows:
find *.tcz -type f -print -exec cat "${SRC}/{}.list" >> "$OUT/provides.db" \; -exec echo >> "$OUT/provides.db" \;SRC is our path to /mnt/array/tinycorelinux/(VERSION).x/(PLATFORM)/tcz
OUT is where we're going to save provides.db for processing before it's moved to SRC
Creating a FOR loop seems to take just as long as -exec above.
Any thoughts on how to make it better?
Thanks!
-
Hi CentralWare
I couldn't get your version to run, so I didn't have a benchmark
to compare to. I attached a copy of what I came up with.
I ran it against the TC14 x86 repo 3 times with a cold cache
for each run. These were the times:
rich@tcbox:~$ ./ClearCache.sh
rich@tcbox:~$ time ./BuildProvides.sh
real 0m4.728s
user 0m0.088s
sys 0m0.592s
rich@tcbox:~$ ./ClearCache.sh
rich@tcbox:~$ time ./BuildProvides.sh
real 0m4.754s
user 0m0.068s
sys 0m0.596s
rich@tcbox:~$ ./ClearCache.sh
rich@tcbox:~$ time ./BuildProvides.sh
real 0m4.715s
user 0m0.096s
sys 0m0.620s
rich@tcbox:~$
-
Benchmarks are about the same
# time ./BuildProvides.sh
real 0m13.649s
user 0m4.320s
sys 0m8.220s
# time ./build_prov
Creating Provides Database...
cat: can't open '/mnt/array1/tinycorelinux/14.x/x86/tcz/Xorg-7.7-lib.tcz.list': No such file or directory
cat: can't open '/mnt/array1/tinycorelinux/14.x/x86/tcz/alsa-plugins-dev.tcz.list': No such file or directory
cat: can't open '/mnt/array1/tinycorelinux/14.x/x86/tcz/geany-plugins-dev.tcz.list': No such file or directory
cat: can't open '/mnt/array1/tinycorelinux/14.x/x86/tcz/gnome-keyring-dev.tcz.list': No such file or directory
cat: can't open '/mnt/array1/tinycorelinux/14.x/x86/tcz/hplip-dev.tcz.list': No such file or directory
cat: can't open '/mnt/array1/tinycorelinux/14.x/x86/tcz/original-modules-6.1.2-tinycore.tcz.list': No such file or directory
cat: can't open '/mnt/array1/tinycorelinux/14.x/x86/tcz/original-modules-6.1.2-tinycore64.tcz.list': No such file or directory
cat: can't open '/mnt/array1/tinycorelinux/14.x/x86/tcz/strongswan-dev.tcz.list': No such file or directory
real 0m13.952s
user 0m4.970s
sys 0m8.440s
Mine throws MIA errors because of meta extensions and actual "missing" files.
Example: 14.x x86 strongswan-dev.tcz
This one's broken (missing .list file where there should be one)
NOTE: This could be just a missing file on THIS END -- which is why this system is being updated in the first place; to ensure things are properly (and efficiently) synchronized! :)
Nope, it's missing on that end as well: https://distro.ibiblio.org/tinycorelinux/14.x/x86/tcz/strongswan-dev.tcz.list (https://distro.ibiblio.org/tinycorelinux/14.x/x86/tcz/strongswan-dev.tcz.list)
...whereas original-modules-6.1.2-tinycore64.tcz.list "should" be missing, but has a file size and md5 >0bytes so there's no "visible" marker to work with yet without extracting the tcz and seeing whether files exist inside it.
Thank you for your efforts!
-
Hi CentralWare
... so there's no "visible" marker to work with yet without extracting the tcz and seeing whether files exist inside it. ...
No need to extract, get a listing:
rich@tcbox:~$ unsquashfs -ll tinycorelinux/15.x/x86_64/tcz/original-modules-6.6.8-tinycore64.tcz
Parallel unsquashfs: Using 6 processors
0 inodes (0 blocks) to write
drwx------ root/root 26 2023-12-23 17:54 squashfs-root
drwxr-xr-x root/root 3 2023-12-23 17:54 squashfs-root/lib
rich@tcbox:~$
-
Hi CentralWare
Test if extension contains files.
This contains files:
rich@tcbox:~$ unsquashfs -ll tinycorelinux/14.x/x86_64/tcz/alsamixergui.tcz | grep -q '^-'
rich@tcbox:~$ echo $?
0
rich@tcbox:~$
This contains only empty directories:
rich@tcbox:~$ unsquashfs -ll tinycorelinux/14.x/x86_64/tcz/alsa-plugins-dev.tcz | grep -q '^-'
rich@tcbox:~$ echo $?
1
rich@tcbox:~$
-
I'll have to see about building a squash-tools for the units, first :)
-
Hi CentralWare
Build notes and patch if you are interested:
http://tinycorelinux.net/14.x/x86_64/tcz/src/squashfs-tools/
-
Thank you, sir! It would be an ARM build instead of an x64, but appreciation none the less!
(...or I may be able to just steal a copy of piCore's arm7 squash build IF it doesn't require kernel headers... :) )
Then again, even 7-zip is supposed to support squash... never know, may be an easy-peasy fix!
-
Hi CentralWare
Oh, so maybe you want this:
http://tinycorelinux.net/14.x/armv7/tcz/src/squashfs-tools/
-
Hi CentralWare
... Then again, even 7-zip is supposed to support squash...
I didn't know that, so I tried it.
It worked for grep.tcz.
It failed for Xorg-7.7-lib.tcz (meta-extension):
tc@E310:~$ 7z l /mnt/sda1/tce/optional/Xorg-7.7-lib.tcz
7-Zip 9.20 Copyright (c) 1999-2010 Igor Pavlov 2010-11-18
p7zip Version 9.20 (locale=C,Utf16=off,HugeFiles=on,2 CPUs)
Error: /mnt/sda1/tce/optional/Xorg-7.7-lib.tcz: Can not open file as archive
Errors: 1tc@E310:~$ echo $?
2
tc@E310:~$
Unfortunately, it also failed for numlockx.tcz which contains one file:
tc@E310:~$ 7z l /mnt/sda1/tce/optional/numlockx.tcz
7-Zip 9.20 Copyright (c) 1999-2010 Igor Pavlov 2010-11-18
p7zip Version 9.20 (locale=C,Utf16=off,HugeFiles=on,2 CPUs)
Error: /mnt/sda1/tce/optional/numlockx.tcz: Can not open file as archive
tc@E310:~$ echo $?
2
tc@E310:~$
The extension itself appears to be OK:
tc@E310:~$ unsquashfs -ll /mnt/sda1/tce/optional/numlockx.tcz
Parallel unsquashfs: Using 2 processors
1 inodes (2 blocks) to write
drwxr-xr-x tc/staff 26 2010-12-04 21:48 squashfs-root
drwxr-xr-x tc/staff 28 2010-12-04 21:48 squashfs-root/usr
drwxr-xr-x tc/staff 26 2010-12-04 21:48 squashfs-root/usr/local
drwxr-xr-x tc/staff 31 2010-12-04 21:48 squashfs-root/usr/local/bin
-rwxr-xr-x tc/staff 5700 2010-12-04 21:48 squashfs-root/usr/local/bin/numlockx
tc@E310:~$
-
Something says to me that when you compiling apps/programs you can provide what is being supported and you can provide many options set that when compiling.
Many times you can see how many dep and includes being used for let say how to extract tar zip files, from the result of configure.
So it maybe depends on zlib, don't know if there some lib for tar.
If not the apps/programs is freestanding, supporting all things internal.
And are maybe compiled static.
That's all for me, for now. So don't be fooled by the fact that the programs support e.g. unpacking algorithms, they may depend on other libraries that they link in when the program is run.
But now i think it's internel busybox commands that are compatible with each other. :)
-
@Rich: LOL... go figure. '7z with an attitude.
I just finished the updates to the Local Repo update utility (testing on 9.x due to the "unique" index that causes even wget not to be able to get a directory listing for /tcz) and thus far all looks good on that front (download any TCZ whose MD5.txt signature doesn't match the signature from the live repo) and then download the "listing" files (md5, sizelist, info.list, etc.) to ensure the local copy is "most recent." The self-check/self-heal will be a work in progress as it needs to be entirely TCL portable (x86/x64/arm) and the walk through squashfs dependencies may be a bit of a stretch for the moment since the intended usage device on this end for the moment is an arm based NAS with an antique-ish operating system.
-
Yes 7z is the best!!! :) , and support's many formats.
I use it a lot, even with windows files, to extract files from installer files with files within.
-
Hi CentralWare
... (testing on 9.x due to the "unique" index that causes even wget not to be able to get a directory listing for /tcz) ...
There's a simple way around that:
http://tinycorelinux.net/9.x/x86/tcz/info.lstThat works on all of the repos. I use that anytime I want to look
at a 9.x repo with my web browser.
-
@Rich: Yes, info.lst works as a directory listing -- I have to use md5.txt for the project, though as I need something to diff our files with the repos.
I'm guessing 9.x was an experiment on www to reduce repo downloading (as it was not repeated in 10.x) or maybe a web based file app?
I'm tempted to ask @curaga to delete those index files, but honestly... 9.x is so old it really doesn't matter on our end. (2017-ish?)
Originally I used 4.x as my intended testing ground... I had NO idea just how many changes existed there (40% of the entire repo at least)
as we haven't done a 4.x update since ~2018 or so. Either that, or something happened on our end where the files (file time stamps, etc.) somehow
were seen as stagnant by wget.
Nostalgic fun fact (which I now feel incredibly antiquated...) We've been "hands dirty" with Tiny Core since somewhere shortly after 2.x was released;
I just came across a couple initrd and a kernel build from way-back (though 3.x is where we really started taking it seriously with our old PXE managed
workstations.) September 11, 2010 ~ May ye rest in peace, my little projects... :) (My login here = 2009... how time flies!)
@patrikg: Yes, when I'm in Windows land 7-Zip is my go-to archive application - it just seems as though we've asked it to do something it doesn't like.
Then again, it's very possible WE are doing something with SquashFS that doesn't conform to exactly how SFS was first intended "by the book"
-- who knows at this point. (...though I doubt it!)
Okay my friends, it's been a pleasure chatting it up -- now it's time for me to get to work! Take care!
-
Hi CentralWare
... I'm guessing 9.x was an experiment on www to reduce repo downloading (as it was not repeated in 10.x) or maybe a web based file app? ...
You can read about it here:
https://forum.tinycorelinux.net/index.php/topic,22016.0.html
-
edit: We used to have an online search at packages.tinycorelinux.net, but bmarkus didn't have the time to maintain it.
The domain (DNS) entry for packages is still lit, it's just seemingly empty.
The concept of creating a STATIC web pages for repo file listings is not the way to go. This would explain why it was so short lived.
- IF the repo machine supports PHP, this would be a very simple enhancement
- IF nginx supports index file scripting, which I believe it does, again -- simple enhancement
(ie: sending HTML files through the PHP parser, not just .php files) - LOL We can even have it "themed" similar to the TinyCore website colors
I have a few ideas I'll run past everyone when time exists to act on 'em.
I have to give @Nathan_SR kudos, though... not too shabby of a job trimming the fat off his original! :)
nginx: To update the engine to parse HTML files through PHP I believe it's just by adding the following line to www.conf
security.limit_extensions = .php .html...but that also assumes cgi/fastcgi has already been added to the config, too.
-
PHP is intentionally not allowed on the main server for security reasons. Same for all other kinds of server-side scripting. Our mirrors are static hosts as well, and they should have the same functionality.
-
@curaga: Non-PHP... understood. (That would just have been the "easy way" :) ) As for the mirrors, they would have hosted the cache files (90% of the bandwidth spent) but "functionality" would still have called home for server-side support.
How about we resurrect the packages idea, then? I have to do some internal surgery on wiki/forum in the next couple weeks once testing on this end is complete and can lay the foundation for it while we're in there. If you're okay with the idea, please update DNS and add/point tce.tinycorelinux.net to the same IP as wiki and forum. (Once complete, we can bring "packages.tinycorelinux.net" into the fold.)
We'll also need to create a cron job (later) to build a couple files (per version/platform) which TCE can efficiently download to help run the show.
Take care!
-
Someone already made one on github?
-
Someone already made one on github?
Save for the one that was used back on TinyCore 9.x I had no idea. Okay, if there's already a working model, no sense reinventing the wheel! :)
-
Someone already made one on github?
Save for the one that was used back on TinyCore 9.x I had no idea. Okay, if there's already a working model, no sense reinventing the wheel! :)
presumably there is a url / link to this ...
-
Hi CentralWare
... (testing on 9.x due to the "unique" index that causes even wget not to be able to get a directory listing for /tcz) ...
There's a simple way around that:
http://tinycorelinux.net/9.x/x86/tcz/info.lstThat works on all of the repos. I use that anytime I want to look
at a 9.x repo with my web browser.
To bad these info.lst files are not simply created automatically with a computer like `ls -1 *.tcz > info.lst.new; mv info.lst.new info.lst;` (or similar, accounting for whatever is the exact specified sorting order).
Because if they were, then some computer must have had a darn lousy day when it added wxgtk-dev.tcz.dep to http://repo.tinycorelinux.net/5.x/armv6/tcz/info.lst
-
Hi Dies Irae
It seems a misnamed info file (wxgtk-dev.tcz.dep.info) confused
one of the scripts used for updating the repo. The issue has been
corrected and the repo updated.
Thank you for reporting this.