Tiny Core Linux
Tiny Core Base => Raspberry Pi => Topic started by: CentralWare on December 17, 2020, 04:31:05 AM
-
piCore 11.x (assuming 10.x and 12.x, too), v6, v7 and v7l
missing: nfs-utils, samba, dropbear, vblade
Samba and dropbear were seen in aarch64, so I'm guessing file hosting on a Pi wasn't seen as being worthwhile prior to Pi4 after v10 was launched?
Thanks guys!
~TJ~
-
piCore/piCore64-12.x is the current version.
It is possible that versions of dropbear, nfs-utils, samba or vblade from earlier versions of piCore might work in piCore-12.x - maybe you could give them a try and report back.
Otherwise, any extension submissions would be gratefully received :)
-
@Juanito I thought NFS-Utils was kernel specific (dependencies such as Device Manager and the likes?)
Regardless, for those items not kernel dependent, I'll see what I can do about replacing tce-ab :)Take care & stay safe!
~TJ~
-
Architecture inquiry:
Arm's aarch64 is in a category of its own to date... but what about armv7l/v7/v6? (ie: Are these assumed to be backward compatible? For example, it's assumed v6 extensions would work in v7 and that v7 work in 7L?) Mind you, kernel based extensions (firmware and the likes) would be release specific, so those are being filtered out. Additionally, is it safe to assume x86 extensions would not execute under x86_64?
Thanks!
-
Hi centralware
... Additionally, is it safe to assume x86 extensions would not execute under x86_64?
Yes.
-
but what about armv7l/v7/v6?
We use compiler flags such that almost all piCore exensions will run on armv6 - there are a few extensions where this is not the case, but it is mentioned in the info file.
-
Good evening, everyone!
The "tce-ab replacement" is listing extensions such as:1. filename.tcz v11.x
2. another.tcz v9.x...so whomever is using it will have the details up front, plus, like tce-ab, the .info file is displayed before giving options to install, onDemand, etc.
If it's not too much trouble, maybe you guys could help me with a scripting challenge??
I've taken the info.lst files from v11 down to v5 and merged them into a single file where each line contains FILENAME.TCZ:VERSION (duplicates obviously exist if older versions have the same extension; which is intentional) but I'm running into a brick wall trying to make the parsing process reasonably quick and efficient.
The goal here was to use `cat mergefile.lst | xargs (or awk)` to conditionally execute an echo statement (shown below), but my programming imagination seems to be failing me.
The logic is as follows:
# mergefile.txt is a combined copy of the repo's info.lst from version 5.x to current, with the numeric version number appended ( filename.tcz:5 )
# for each tcz listed. The version number is the highest version where this extension was found (earlier releases are disregarded.)
# /app/path/info.lst will be our output file containing FILENAME.TCZ:VERSION one per line
while read -r line
do
filename=$(echo ${line} | awk -F: '{print $1}')
version=$(echo ${line} | awk -F: '{print $2}')
if [ ! -f /tmp/files/${filename} ]; then
echo "${filename}:${version} >> /app/path/info.lst ## Add this entry to our output file
touch /tmp/files/${filename} ## Use these /tmp/files as flags to indicate which TCZs have already been processed.
fi
done < mergefile.txt
# rm /tmp/files and other clean-up thereafter
Any thoughts? Thanks guys!
-
Hi centralware
I had to read your post about a dozen times but I think I understand it now.
1. You created a file called mergefile.txt containing a list of available extensions for TC11 formatted as Extension.tcz:TC11.
2. To that, you appended the same information for TC10, then TC9, and so on until you got to TC5.
3. Now you want parse that list to find the most recent instance of Extension.tcz:TCversion and append that to /app/path/info.lst.
Hopefully I got that right.
See if this runs any faster:
# mergefile.txt is a combined copy of the repo's info.lst from version 5.x to current, with the numeric version
# number appended ( filename.tcz:5 ) for each tcz listed. The version number is the highest version where this
# extension was found (earlier releases are disregarded.) /app/path/info.lst will be our output file containing
# FILENAME.TCZ:VERSION one per line
while read -r line
do
# filename=$(echo ${line} | awk -F: '{print $1}')
## Too complicated. The key to speed is to do as little work as possible.
filename=${line%:*}
# version=$(echo ${line} | awk -F: '{print $2}')
## Not needed.
if [ ! -f /tmp/files/${filename} ]; then
# echo "${filename}:${version} >> /app/path/info.lst ## Add this entry to our output file
## You separated $line to create $filename and $version. Now you want to combine them again to create $line?
echo "${line} >> /app/path/info.lst ## Add this entry to our output file
touch /tmp/files/${filename} ## Use these /tmp/files as flags to indicate which TCZs have already been processed.
fi
done < mergefile.txt
# rm /tmp/files and other clean-up thereafter
-
Hello Rich. Yes, that sounds right. (Sorry if it caused you that many times to be read!!!)
Currently, the conditional loop takes too long to process all those lines in the merged info.lst file.
As it stands, when the new tce-ab searches through the output file info.lst for an extension name (or portion of one), such as "dropbear", it finds records such as "dropbear.tcz:7" where "7" indicates the most recent release of this extension can be found in version 7.x.
The parsing is done by:
cat mergefile.txt | xargs -I % sh -c "repo_mkfile % $version"where the script "/usr/local/bin/repo_mkfile [extension] [version]" does the conditional check
if [ ! -f /tmp/repo/$1 ]; then
echo "${1}:${2}" >> /tmp/repo/info.lst
touch /tmp/repo/$1
fi
Thanks for your efforts!
T.J.
-
Here's the most efficient/fastest I've come up with thus far; I know it could be so much quicker though.
The first half just downloads info.lst, tags.db, provides.db, etc. The second half parses our new info.lst which is where it needs attention.
### CONFIG FILE VARIABLES ###
HOME=/opt/repo
REPO="http://tinycorelinux.net"
MINVER=5
PROC=armv7
### CONFIG FILE VARIABLES ###
####################################
### MANAGES VER.x database files ###
####################################
steps=0
FILES="sizelist provides.db tags.db md5.db info.lst"
for v in $(seq 5 $VER)
do
cPATH=${HOME}/tinycorelinux/${v}.x/${PROC}/tcz; [ ! -d $cPATH ] && mkdir -p $cPATH
for file in $FILES
do
[ $update_repo -eq 1 ] && [ -f $cPATH/$file ] && rm $cPATH/$file -f
if [ ! -f $cPATH/$file ]; then
[ $steps -lt 1 ] && echo -n "${WHITE}Downloading ${YELLOW}${v}.x ${plat} ${WHITE}information file${NORMAL} " && steps=1
wget -O $cPATH/${file}.gz -q --no-check-certificate $REPO/${v}.x/${PROC}/tcz/${file}.gz >/dev/null 2>&1
[ -f $cPATH/${file}.gz ] && gunzip $cPATH/${file}.gz
[ ! -f $cPATH/$file ] && touch $cPATH/$file
fi
done
[ $steps -gt 0 ] && echo
steps=0
done
#################################
# Creates our new database file #
#################################
if [ ! -f ${HOME}/tinycorelinux/repo/info.lst ]; then
echo -n "Updating database "
dst=/tmp/info.lst #<-- This file should end up with filename.tcz:version
chk=/tmp/info.chk #<-- This file should contain ONLY filename.tcz
cp "${HOME}/tinycorelinux/${VER}.x/${PROC}/tcz/info.lst" $dst
cp $dst $chk
sed -i "s/.tcz/.tcz:${VER}/g" $dst
START=$(expr $VER - 1)
for counter in $(seq $START -1 $MINVER)
do
src=${HOME}/tinycorelinux/${counter}.x/${PROC}/tcz/info.lst
echo -n "${counter}.x "
while read -r line
do
test=$(grep -w $line $chk) #<-- Set grep to match only EXACT records
if [ "${test}" == "" ]; then
echo $line >> $chk
echo "${line}:${counter}" >> $dst
fi
done < $src
done
sync; sort /tmp/info.lst > ${HOME}/tinycorelinux/repo/info.lst # Sort the new file
echo >> ${HOME}/tinycorelinux/repo/info.lst # Pad the file with an extra LF
rm $dst $chk -f # Clean up
fi
-
Hi centralware
The attached script downloads the info.lst files for TC5 through TC11 from the x86 repo. It reduces the data to unique
entries, appends the correct TC version to the end of each line, and sorts the final result and leaves it in info.lst. The
script completes in less than 3 seconds on my low end hardware.
[EDIT]: Attachment updated. Added filter for piCore modules, bumped version to 0.2. Rich
-
@Rich: Thanks! Looks good! After taking a quick peek, the compare/trim concept didn't work for me on my first go-'round, but I also had a lot of filters added in to remove kernel mods and the likes (from older versions, since they're inherently incompatible) which could have thrown in a wrench with the results.
I still have to beat up on provides.db and tweak tags.db a little but so far, things are looking promising.
Versions supported: v5 to current (earlier than that, the directory structure skews), min defaults to 7.x, current=$(version)
Arch's supported: x86 x86_64 armv6 armv7 armv7l aarch64, defaults to the running os (my detection scheme seems to work well, but I haven't yet looked in tc-config, etc. to see if there's one already made.)
dCore and mips may be added at a later date.
Thanks again,
T.J.
-
Come to find out my comparative version worked, just with inaccurate results, but I was trying to kill two birds.
info.lst ends with 2,141 results/lines. tags.db ends with 2,148 results/lines.
alsa-utils* being 3 of the duplicates, versions 10 and 11 being listed, not sure what the other four are. Why 10x was seen as unique to 11x... the jury is still out.
-
Hi centralware
... Why 10x was seen as unique to 11x... the jury is still out.
I don't understand. Who said 10.x was unique?
-
info.lst and tags.db should contain the same number of records.
alsa-utils*.tcz (for some reason) from 10.x info.lst and 11.x info.lst are both in the final (merged) info.lst
Should be a simple fix once I trace things out.
...
alsa-oss-dev.tcz:9
alsa-oss-doc.tcz:9
alsa-oss.tcz:9
alsa-plugins-dev.tcz:11
alsa-plugins.tcz:11
alsa-utils-doc.tcz:10
alsa-utils-doc.tcz:11
alsa-utils-locale.tcz:10
alsa-utils-locale.tcz:11
alsa-utils.tcz:10
alsa-utils.tcz:11
alsa.tcz:11
alsaequal.tcz:7
amtk-dev.tcz:11
...
-
Hi centralware
For which architecture are you seeing that. What did you set MinVersion and MaxVersion to?
-
Hi centralware
I could not replicate your results. I ran it for armv6 from TC5 to TC11:
aircrack-ng.tcz:TC9
alpine-doc.tcz:TC7
alpine.tcz:TC7
alsa-config.tcz:TC7
alsa-dev.tcz:TC7
alsa-doc.tcz:TC7
alsa-locale.tcz:TC7
alsa-modules-4.19.81-piCore.tcz:TC11
alsa-oss-dev.tcz:TC9
alsa-oss-doc.tcz:TC9
alsa-oss.tcz:TC9
alsa-plugins-dev.tcz:TC11
alsa-plugins.tcz:TC11
alsa-utils-doc.tcz:TC11
alsa-utils-locale.tcz:TC11
alsa-utils.tcz:TC11
alsa.tcz:TC11
alsaequal.tcz:TC7
amtk-dev.tcz:TC11
amtk-gir.tcz:TC11
amtk.tcz:TC11
The only thing I can think of is you ran the comm command on lists that were not sorted identically.
I added a filter to deal with piCore modules. The script attached to reply #10 has been updated.
-
Min=5, Max=(current=11), OS=TCL ArmV7
I'll double-check the sorting, thanks. (Note: no attachment found :) )
Thanks!
T.J.
-
Sorry, misunderstood the attachment part.
-
Hi centralware
Min=5, Max=(current=11), OS=TCL ArmV7 ...
Still works correctly, even though ArmV7 did not exist until TC6:
aircrack-ng.tcz:TC9
alpine-doc.tcz:TC7
alpine.tcz:TC7
alsa-config.tcz:TC7
alsa-dev.tcz:TC7
alsa-doc.tcz:TC7
alsa-locale.tcz:TC7
alsa-modules-4.19.81-piCore-v7.tcz:TC11
alsa-modules-4.19.81-piCore-v7l.tcz:TC11
alsa-oss-dev.tcz:TC9
alsa-oss-doc.tcz:TC9
alsa-oss.tcz:TC9
alsa-plugins-dev.tcz:TC11
alsa-plugins.tcz:TC11
alsa-utils-doc.tcz:TC11
alsa-utils-locale.tcz:TC11
alsa-utils.tcz:TC11
alsa.tcz:TC11
alsaequal.tcz:TC7
amtk-dev.tcz:TC11
amtk-gir.tcz:TC11
amtk.tcz:TC11
-
ArmV7 / Version 6.x - correct; I just had to make it easy for anyone to set a static MinVer= (config file) and work around the errors when files don't exist, etc. and based on my personal experiences in x86, for example, some extensions dated back to v5.x when running v8, so this isn't the first time I had to build something to compensate between versions, it's just the first time I intended to submit the end results as a TCL update and/or extension itself.
Yes, I ran your script - seems to work flawlessly on info.lst; I'm incorporating it into tags.db right this moment which is just about done. Provides.db... that one's going to be a joy! :)
Currently, with info.lst & tags.db we're at ~3.25 seconds on a Pi3 running the show. A keyword search takes a couple seconds to respond with the new tags.db, but I imagine it's not much quicker with a single #.x file, let alone the combined one. Due to the need to script a loop to rebuild provides.db (/usr/bin/provides.sh) it'll likely run behind the scenes, that part of the new tce-ab being disabled until it completes. (Unless you can see a quicker method??)
Your assistance here is truly appreciated!
-
Good morning!
I took a quick peek at the provides.db concept/layout - is there any reason we're keeping the directory structure usr/local/somewhere/else/filename.so or do you think this can be truncated? (I'm assuming provides is used to find files within an extension - not find the path they'd be installed into had the extension been loaded.)
-
Hi centralware
The paths can provide context. If I enter this term for provides:
findI get 88 results, 87 of which I'm not interested in. If I enter this:
bin\/findI reduce that to 4 results.
-
Makes sense; thanks for the enlightenment.
Mind looking at something for me? I am trying to use our info.lst files to create a merged provides.db (example using TC=8, thus 8Unique.lst and 8.db is a copy of provides.db from 8.x/tcz
awk '{print $0}' 8Unique.lst | awk -v TARGET=$1 'BEGIN {FS="\n";RS=""} /$TARGET/{print $0}' 8.db
Obviously not working. Suggestions?
1. Read 8Unique.lst one line at a time
2. Pipe the results (something.tcz) into the second awk as TARGET by reading $1 <-- likely where the issue is
3. Read 8.db, searching for $TARGET and print the contents of that entry $0
** a match() may be necessary - files like mc.tcz reside inside filenames such as libXvmc.tcz
Thanks!
-
Hi centralware
I modified the script as follows:
1. It creates a providesOld.db file that includes only the extensions not found in $MaxVersion.
2. I added more robust error checking to deal with files/URLs not being found. $MinVersion can be left as 4 regardless
of which architecture it is run for.
I ran the script for armv7 and it took 3 mins and 37 secs to complete. 3 mins and 35 secs of that was from creating providesOld.db.
I included a modified version of the provides.sh script called ProvidesOld.sh. It looks for providesOld.db in the current directory.
In addition to printing out the extension name and TC version, it also prints the path/filename it matched on:
tc@E310:~/count$ ./ProvidesOld.sh libpng
libpng-doc.tcz:TC9
usr/local/share/man/man3/libpng.3
usr/local/share/man/man3/libpngpf.3
libpng12.tcz:TC7
usr/local/lib/libpng12.so.0
usr/local/lib/libpng12.so.0.51.0
vlc.tcz:TC7
usr/local/lib/vlc/plugins/codec/libpng_plugin.la
usr/local/lib/vlc/plugins/codec/libpng_plugin.so
tc@E310:~/count$
-
Happy Holidays!
There has to be a way to rip through provides.db without wget/looping.
Here's my mental concept:
- We start a loop while TC in $(seq $MaxVer -1 $MinVer)
- We have a copy of provides$TC.db for each version downloaded
- $TC"Unique.lst" has a complete list of tcz files which we're going to pass to awk as TARGET
awk 'BEGIN {FS="\n";RS=""} /'${TARGET}'/{print $0} END{print "\n"}' "provides$TC.db" > our_output$TC.db
I know this has to be do-able, I'm just having a time of it experimenting with different methods to either have a second instance of grep, awk, etc. feeding this instance -v TARGET=$1... It's a work in progress since I don't have all that much personal experience with awk. (The the past decade, I doubt I've used awk in a more complicated method beyond:ifconfig eth0 | grep Bcast | awk -F: '{print $2}' | awk '{print $1}'...and even that's probably not the best method to do what's being done ;) )
Many thanks and a Happy New Year!
-
Hi centralware
So, I decided to start from scratch and rethink my approach, and it payed off. These are the results:
tc@E310:~/count/armv7TC04TC11$ sudo cache-clear
tc@E310:~/count/armv7TC04TC11$ sync
tc@E310:~/count/armv7TC04TC11$ ./BuildProvidesAllDatabase.sh
Stopwatch reset. Fetching provides.db files.
Elapsed Time=00:00:15 Fetching provides.db files. Done.
Stopwatch reset. Convert records from multi line to single line.
Elapsed Time=00:00:01 Convert records from multi line to single line. Done.
Elapsed Time=00:00:00 Combine provides files and strip out kermel module records.
Elapsed Time=00:00:02 Combine provides files and strip out kermel module records. Done.
Elapsed Time=00:00:00 Do a case insensitive sort (-f).
Elapsed Time=00:00:05 Case insensitive sort (-f). Done.
Elapsed Time=00:00:00 Create combined info.lst from combined provides.db.
Elapsed Time=00:00:00 Create combined info.lst from combined provides.db. Done.
Elapsed Time=00:00:00 Remove duplicate names keeping the most recent TCversion, saving in infoAll.lst.
Elapsed Time=00:00:01 Remove duplicate names keeping the most recent TCversion, saving in infoAll.lst. Done.
Elapsed Time=00:00:00 Find providesSort.db entries matching infoAll.lst, saving results.
Elapsed Time=00:00:01 Find providesSort.db entries matching infoAll.lst, saving results. Done.
Elapsed Time=00:00:00 Convert back to multi line, save in providesAll.db file.
Elapsed Time=00:00:00 Convert back to multi line, save in providesAll.db file. Done.
Elapsed Time=00:00:10 Total time spent manipulating data.
Results were saved in the armv7TC04TC11 directory.
tc@E310:~/count/armv7TC04TC11$Time to download the provides.db files is listed separately. It varied from 15 to 28 seconds. Nothing I can do about that.
After that, the steps are"
1. Append :TCversion to each Name.tcz entry and change the field separators in the provides.db files from a carriage
return to a semicolon. This way each record in the file occupies only one line. This will simplify sorting later on.
2. Combine the provides.db files while stripping out the kernel module entries from the older TC versions.
3. Do a case insensitive sort on the first field of the combined provides.db file.
4. Create an info.lst that matches the combined provides.db file by copying the first field from each record.
5. Reduce the info.lst by keeping the most recent version of each extension name.
6. Reduce the combined provides.db file using the reduced info.lst file as a template.
7. Change the field separators in the reduced provides.db file from a semicolon back to a carriage return.
The process started with 6901 entries and reduced that to 2141 entries.
Total time for steps 1 through 7 is 10 seconds. All commands in the script are aliased to busybox. At 5 seconds, step 5
takes the most time. Using GNU sort, step 5 execution time drops from 5 seconds to 0.5 seconds.
Results are saved in a work directory formed by $ARCH $MinVersion and $MaxVersion like this armv7TC04TC11.
I decided to run this on a bigger data set, so I went with x86 TC4 through TC11.
Time to download the provides.db files was 47 seconds.
The process started with 20113 entries and reduced that to 5594 entries.
Total time for steps 1 through 7 was 41 seconds. Step 5 took 23 seconds. Using GNU sort would reduce step 5 to 2 seconds.
The BuildProvidesAllDatabase.sh script saves its results in 2 files, infoAll.lst and providesAll.db.
The ProvidesAll.sh script can be use to search providesAll.db which it looks for in the current directory. Example:
tc@E310:~/count/armv7TC04TC11$ ../ProvidesAll.sh "bin\/find"
findutils.tcz:TC11
usr/local/bin/find
libftdi.tcz:TC09
usr/local/bin/find_all
util-linux.tcz:TC11
usr/local/sbin/findfs
usr/local/bin/findmnt
tc@E310:~/count/armv7TC04TC11$
-
Good evening, Rich!
I must say, you're both persistent and dedicated! Kudos and thank you!
Time is against me this week; I'll take a look at the scripts this weekend and see how well/easy things incorporate into the tce-ab project.
(Note: info.lst, provides.db, etc. have .gz counterparts within the repository (ie: provides.db.gz) which obviously affect download times a good deal.)
I don't know if you have the answer to this, but I was setting up v12.x within our local repo copy and noticed 12.x/armv7l is missing releases/kernel sources? (It has armv7l/tcz populated, thus seemed out of the ordinary to have something that large and not use-able?)
-
Hi centralware
For some reason the sources are located here:
http://tinycorelinux.net/12.x/armv7/releases/RPi/src/kernel/
-
The 12.x/armv7l is a dead repo. You should not be using that.
-
@Paul_123 : Dead, as in, no longer supported - today and future releases? (I'm updating our local repo; versions which go to the graveyard (ie: mips) I usually put in notes (when/why/what to do instead) to conserve on the level of unnecessary chaos in our day :) Thanks to COVID, there's plenty already!)
@Rich : Thank you! I'm preparing a few RasPi4 devices this weekend, initially planning to test v12 armv7L and aarch64 (thus my noticing 7L "MIA") to see which would be best suited for the lab they're going to be part of and wanted to have the kernel source handy in case some tweaking was needed.(2) PiZero Controllers (DHCP/DNS Primary/Secondary)
Controlling (3) 8ch Relay modules to power the Pis and (1) 4ch Relay for the PSUs
Powered by (5) 14A PSU (one 24/7, the others relay controlled)
(24) RasPi2
(4) RasPi3 First Gen
(4) RasPi4
-- All running TCL --I'll ship in photos when she's finished; looks kind'a cute!
-
Hi centralware
... Dead, as in, no longer supported - today and future releases? ...
I believe this repo covers armv7, armv7l, and armv7l+:
http://tinycorelinux.net/12.x/armv7/tcz/
The difference being in the kernel module extensions, for example:
alsa-modules-5.4.51-piCore-v7l.tcz
alsa-modules-5.4.51-piCore-v7l+.tcz
alsa-modules-5.4.51-piCore-v7.tcz
The zip file found here contains multiple kernels:
http://tinycorelinux.net/12.x/armv7/releases/RPi/
-
@Rich : Thank you for the detailed information; our repo has been updated (and set to no longer look for v7*)