piCore Extension Request

[CentralWare]

piCore 11.x (assuming 10.x and 12.x, too), v6, v7 and v7l
missing: nfs-utils, samba, dropbear, vblade
Samba and dropbear were seen in aarch64, so I'm guessing file hosting on a Pi wasn't seen as being worthwhile prior to Pi4 after v10 was launched?

Thanks guys!
~TJ~

[Juanito]

piCore/piCore64-12.x is the current version.

It is possible that versions of dropbear, nfs-utils, samba or vblade from earlier versions of piCore might work in piCore-12.x - maybe you could give them a try and report back.

Otherwise, any extension submissions would be gratefully received 

[CentralWare]

@Juanito  I thought NFS-Utils was kernel specific (dependencies such as Device Manager and the likes?)
Regardless, for those items not kernel dependent, I'll see what I can do about replacing tce-ab :)Take care & stay safe!
~TJ~

[CentralWare]

Architecture inquiry:
   Arm's aarch64 is in a category of its own to date...  but what about armv7l/v7/v6?   (ie: Are these assumed to be backward compatible?  For example, it's assumed v6 extensions would work in v7 and that v7 work in 7L?)   Mind you, kernel based extensions (firmware and the likes) would be release specific, so those are being filtered out.   Additionally, is it safe to assume x86 extensions would not execute under x86_64?
Thanks!

[Rich]

Hi centralware

... Additionally, is it safe to assume x86 extensions would not execute under x86_64?

Yes.

[Juanito]

but what about armv7l/v7/v6?

We use compiler flags such that almost all piCore exensions will run on armv6 - there are a few extensions where this is not the case, but it is mentioned in the info file.

[CentralWare]

Good evening, everyone! 
The "tce-ab replacement" is listing extensions such as:

Code: [Select]

1. filename.tcz v11.x
2. another.tcz v9.x...so whomever is using it will have the details up front, plus, like tce-ab, the .info file is displayed before giving options to install, onDemand, etc.
If it's not too much trouble, maybe you guys could help me with a scripting challenge??
I've taken the info.lst files from v11 down to v5 and merged them into a single file where each line contains FILENAME.TCZ:VERSION (duplicates obviously exist if older versions have the same extension; which is intentional) but I'm running into a brick wall trying to make the parsing process reasonably quick and efficient. 

The goal here was to use `cat mergefile.lst | xargs (or awk)` to conditionally execute an echo statement (shown below), but my programming imagination seems to be failing me. 
The logic is as follows:

Code: [Select]

# mergefile.txt is a combined copy of the repo's info.lst from version 5.x to current, with the numeric version number appended ( filename.tcz:5 )
#     for each tcz listed. The version number is the highest version where this extension was found (earlier releases are disregarded.)
# /app/path/info.lst will be our output file containing FILENAME.TCZ:VERSION one per line
while read -r line
do
  filename=$(echo ${line} | awk -F: '{print $1}')
  version=$(echo ${line} | awk -F: '{print $2}')
  if [ ! -f /tmp/files/${filename} ]; then
    echo "${filename}:${version} >> /app/path/info.lst  ## Add this entry to our output file
    touch /tmp/files/${filename}  ## Use these /tmp/files as flags to indicate which TCZs have already been processed.
  fi
done < mergefile.txt
# rm /tmp/files and other clean-up thereafter

Any thoughts?  Thanks guys!

[Rich]

Hi centralware
I had to read your post about a dozen times but I think I understand it now.
1. You created a file called  mergefile.txt  containing a list of available extensions for TC11 formatted as Extension.tcz:TC11.
2. To that, you appended the same information for TC10, then TC9, and so on until you got to TC5.
3. Now you want parse that list to find the most recent instance of  Extension.tcz:TCversion  and append that to  /app/path/info.lst.

Hopefully I got that right.

See if this runs any faster:

Code: [Select]

# mergefile.txt is a combined copy of the repo's info.lst from version 5.x to current, with the numeric version
# number appended ( filename.tcz:5 )  for each tcz listed. The version number is the highest version where this
# extension was found (earlier releases are disregarded.) /app/path/info.lst will be our output file containing
# FILENAME.TCZ:VERSION one per line
while read -r line
do

#  filename=$(echo ${line} | awk -F: '{print $1}')
## Too complicated. The key to speed is to do as little work as possible.
  filename=${line%:*}

#  version=$(echo ${line} | awk -F: '{print $2}')
## Not needed.

  if [ ! -f /tmp/files/${filename} ]; then

#    echo "${filename}:${version} >> /app/path/info.lst  ## Add this entry to our output file
##  You separated  $line  to create  $filename  and  $version. Now you want to combine them again to create  $line?
    echo "${line} >> /app/path/info.lst  ## Add this entry to our output file

    touch /tmp/files/${filename}  ## Use these /tmp/files as flags to indicate which TCZs have already been processed.
  fi
done < mergefile.txt
# rm /tmp/files and other clean-up thereafter

[CentralWare]

Hello Rich. Yes, that sounds right.  (Sorry if it caused you that many times to be read!!!)

Currently, the conditional loop takes too long to process all those lines in the merged info.lst file.

As it stands, when the new tce-ab searches through the output file info.lst for an extension name (or portion of one), such as "dropbear", it finds records such as "dropbear.tcz:7" where "7" indicates the most recent release of this extension can be found in version 7.x.
The parsing is done by:

Code: [Select]

cat mergefile.txt | xargs -I % sh -c "repo_mkfile % $version"where the script "/usr/local/bin/repo_mkfile [extension] [version]" does the conditional check

Code: [Select]

if [ ! -f /tmp/repo/$1 ]; then
  echo "${1}:${2}" >> /tmp/repo/info.lst
  touch /tmp/repo/$1
fi

Thanks for your efforts!

T.J.

[CentralWare]

Here's the most efficient/fastest I've come up with thus far; I know it could be so much quicker though.
The first half just downloads info.lst, tags.db, provides.db, etc. The second half parses our new info.lst which is where it needs attention.

Code: [Select]


### CONFIG FILE VARIABLES ###
HOME=/opt/repo
REPO="http://tinycorelinux.net"
MINVER=5
PROC=armv7
### CONFIG FILE VARIABLES ###

####################################
### MANAGES VER.x database files ###
####################################
steps=0
FILES="sizelist provides.db tags.db md5.db info.lst"
for v in $(seq 5 $VER)
do
    cPATH=${HOME}/tinycorelinux/${v}.x/${PROC}/tcz; [ ! -d $cPATH ] && mkdir -p $cPATH
    for file in $FILES
    do
        [ $update_repo -eq 1 ] && [ -f $cPATH/$file ] && rm $cPATH/$file -f
        if [ ! -f $cPATH/$file ]; then
            [ $steps -lt 1 ] && echo -n "${WHITE}Downloading ${YELLOW}${v}.x ${plat} ${WHITE}information file${NORMAL} " && steps=1
            wget -O $cPATH/${file}.gz -q --no-check-certificate $REPO/${v}.x/${PROC}/tcz/${file}.gz >/dev/null 2>&1
            [ -f $cPATH/${file}.gz ] && gunzip $cPATH/${file}.gz
            [ ! -f $cPATH/$file ] && touch $cPATH/$file
        fi
    done
    [ $steps -gt 0 ] && echo
    steps=0
done

#################################
# Creates our new database file #
#################################
if [ ! -f ${HOME}/tinycorelinux/repo/info.lst ]; then
    echo -n "Updating database "
    dst=/tmp/info.lst             #<-- This file should end up with filename.tcz:version
    chk=/tmp/info.chk             #<-- This file should contain ONLY filename.tcz

    cp "${HOME}/tinycorelinux/${VER}.x/${PROC}/tcz/info.lst" $dst
    cp $dst $chk
    sed -i "s/.tcz/.tcz:${VER}/g" $dst

    START=$(expr $VER - 1)
    for counter in $(seq $START -1 $MINVER)
    do
        src=${HOME}/tinycorelinux/${counter}.x/${PROC}/tcz/info.lst
        echo -n "${counter}.x "
        while read -r line
        do
            test=$(grep -w $line $chk)       #<-- Set grep to match only EXACT records
            if [ "${test}" == "" ]; then
                echo $line >> $chk
                echo "${line}:${counter}" >> $dst
            fi
        done < $src
    done

    sync; sort /tmp/info.lst > ${HOME}/tinycorelinux/repo/info.lst # Sort the new file
    echo >> ${HOME}/tinycorelinux/repo/info.lst # Pad the file with an extra LF
    rm $dst $chk -f # Clean up

fi


[Rich]

Hi centralware
The attached script downloads the info.lst files for TC5 through TC11 from the x86 repo. It reduces the data to unique
entries, appends the correct TC version to the end of each line, and sorts the final result and leaves it in  info.lst.  The
script completes in less than 3 seconds on my low end hardware.

    [EDIT]: Attachment updated. Added filter for piCore modules, bumped version to 0.2.  Rich

[CentralWare]

@Rich: Thanks!  Looks good!  After taking a quick peek, the compare/trim concept didn't work for me on my first go-'round, but I also had a lot of filters added in to remove kernel mods and the likes (from older versions, since they're inherently incompatible) which could have thrown in a wrench with the results. 
I still have to beat up on provides.db and tweak tags.db a little but so far, things are looking promising.

Versions supported: v5 to current (earlier than that, the directory structure skews), min defaults to 7.x, current=$(version)
Arch's supported: x86 x86_64 armv6 armv7 armv7l aarch64, defaults to the running os (my detection scheme seems to work well, but I haven't yet looked in tc-config, etc. to see if there's one already made.) 
   dCore and mips may be added at a later date.

Thanks again,
T.J.

[CentralWare]

Come to find out my comparative version worked, just with inaccurate results, but I was trying to kill two birds.
info.lst ends with 2,141 results/lines. tags.db ends with 2,148 results/lines.
alsa-utils* being 3 of the duplicates, versions 10 and 11 being listed, not sure what the other four are. Why 10x was seen as unique to 11x...  the jury is still out.

[Rich]

Hi centralware

... Why 10x was seen as unique to 11x...  the jury is still out.

I don't understand. Who said 10.x was unique?

[CentralWare]

info.lst and tags.db should contain the same number of records.
alsa-utils*.tcz (for some reason) from 10.x info.lst and 11.x info.lst are both in the final (merged) info.lst
Should be a simple fix once I trace things out.

Code: [Select]

...

alsa-oss-dev.tcz:9
alsa-oss-doc.tcz:9
alsa-oss.tcz:9
alsa-plugins-dev.tcz:11
alsa-plugins.tcz:11
alsa-utils-doc.tcz:10
alsa-utils-doc.tcz:11
alsa-utils-locale.tcz:10
alsa-utils-locale.tcz:11
alsa-utils.tcz:10
alsa-utils.tcz:11
alsa.tcz:11
alsaequal.tcz:7
amtk-dev.tcz:11
...