Dependent TCZs Scanner for TCZ Maintainers

[Sashank999]

Hello.

I was trying to create and submit a TCZ to the repo when I just found out that we have to find out the list of TCZs that will be used by an extension's binaries (i. e., dependent TCZs) manually.
I just wrote a small script in Python that checks the files in a directory, scans the files to check if they are ELF files (both executables and shared objects have the same magic bytes IIRC) and then uses ldd to find the libraries they depend on.

It checks the /etc/sysconfig/tcedir/provides.db for the TCZs that provide those libraries. These are listed in an "initial scan" log.

Then, this script checks if there are any unnecessary dependencies, like listing both child and parent dependent TCZs. It does this by downloading the .tree files for one dependent TCZ at a time (it also has a local cache for .tree files so that the same .tree file does not get downloaded twice).

All these will be stored in a .dep file, named as "first argument.dep", in the directory of the call of the script. Thus your first parameter to the script must be "newextension.tcz".

It also has a verbose mode so that the .tree files that are downloaded and the redundant dependencies removed during resolution are printed to stdout.

The script is written and tested with Python 3.9 in the TCL v15 x86_64 repo.

Please let me know if there are any improvements that I can do so that this script suits other's needs.
Note: The script is currently unlicensed. I plan on MIT licensing it if necessary.

Thank you.

[polikuo]

Hi, Sashank999.

Dependency is not always that easy.
For small library, yes, such script will do.
If you browse the forum more, you can see tons of scripts written by different individuals.
In case you're interested, you can check out mine here on the forum, or on my github.
They are old, my skill was immature back then, sometimes I do want to rewrite it, but I always got occupied.
Anyway, if you find anything useful in my script, feel free to implement them into yours.

[Rich]

Hi Sashank999
I did something like that a couple of years ago:
https://forum.tinycorelinux.net/index.php/topic,26040.0.html
In my version you just told it which binary to check and and for which TC version.
It would create a .dep file for that TC version and report any missing libraries.

... uses ldd to find the libraries they depend on.

It checks the /etc/sysconfig/tcedir/provides.db for the TCZs that provide those libraries. These are listed in an "initial scan" log. ...

That's what I did.

... Then, this script checks if there are any unnecessary dependencies, like listing both child and parent dependent TCZs. It does this by downloading the .tree files for one dependent TCZ at a time (it also has a local cache for .tree files so that the same .tree file does not get downloaded twice). ...

I took a different approach:

Code: [Select]

search provides.db for library
if found
    echo FILE.TCZ >> $DependencyList
else
    # Check /lib and /usr/lib to see if it's part of the base system.
    [ -f "/lib/$LIB" ] && continue
    [ -f "/usr/lib/$LIB" ] && continue
    # LIB was not found.
    echo " ******************* Missing $LIB" >> "$MissingDeps"
fi

# This sorts the file in place and removes duplicates.
echo "$(sort -uf "$DependencyList")" > "$DependencyList"


mkdir "$NEEDEDDEPSDIR"
mkdir "$UNNEEDEDDEPSDIR"

# Create a directory of .dep files for further processing.
RUN=1

while [ $RUN -eq 1 ]
do
    RUN=0
    for F in `cat "$DependencyList"`
    do
        # Skip this entry if it already exists.
        [ -f "$NEEDEDDEPSDIR$F.dep" ] && continue
        wget -q -O "$NEEDEDDEPSDIR$F.dep" "$URL$F.dep" 2>&1 | grep -q "404 Not Found"
        if [ $? -eq 0 ]
        then    # We drop into here if grep succeeds in finding "404 Not Found".
            # Create a temporary dummy .dep file if one doesn't exist.
            touch "$NEEDEDDEPSDIR$F.dep"
        else
            cat "$NEEDEDDEPSDIR$F.dep" >> "$DepContents"
            # We continue running as long as more dependencies are found.
            RUN=1
        fi
    done

    echo "$(sort -uf "$DepContents")" > "$DepContents"
    cp "$DepContents" "$DependencyList"

done

# We now have a directory of .dep files and a list of their contents.
# If a file is on the list, its .dep file can be moved to $UNNEEDEDDEPSDIR.
for F in $(cat "$DepContents")
do
    # Skip this entry if it doesn't exist.
    [ ! -f "$NEEDEDDEPSDIR$F.dep" ] && continue
    mv "$NEEDEDDEPSDIR$F.dep" "$UNNEEDEDDEPSDIR"
done

# Create dependency file.
for F in $(ls -1 "$NEEDEDDEPSDIR" | sort -f)
do
    echo ${F%.dep} >> "$WORKDIR/Result.dep"
done
Kernel modules are listed as  ExtensionName-KERNEL.tcz  and will
show up as  ExtensionName-KERNEL.tcz.dep  with a file size of zero.
Since there is the possibility of more than one kernel version for
modules in a release, and the check you are running may be for a
different TC version, automating this is likely not possible.

[Sashank999]

Hello.

If you browse the forum more, you can see tons of scripts written by different individuals.
In case you're interested, you can check out mine here on the forum, or on my github.
They are old, my skill was immature back then, sometimes I do want to rewrite it, but I always got occupied.
Anyway, if you find anything useful in my script, feel free to implement them into yours.

I expected someone to be doing something like this, but there are really a lot of such scripts. I am also intrigued by your script's ideas, but I still don't know what I can add from your script. And we really thought of (almost) the same name for our scripts.
I would like to work on my script a bit more. I think we can think a bit more about the ways we can solve this problem.

In my version you just told it which binary to check and and for which TC version.
It would create a .dep file for that TC version and report any missing libraries.

Kernel modules are listed as  ExtensionName-KERNEL.tcz  and will
show up as  ExtensionName-KERNEL.tcz.dep  with a file size of zero.
Since there is the possibility of more than one kernel version for
modules in a release, and the check you are running may be for a
different TC version, automating this is likely not possible.

I did not understand this comment. What is the problem with ExtensionName-KERNEL.tcz ?
The problem that you described, which I think is what you are saying for your script, only occurs when you are using the .dep files. The .tree files contain the KERNEL part of the TCZ name replaced with the KERNEL version so I don't think there will be a problem with my script. As my script does not include the option of running it for a different TCL version than the hardcoded values, I do not think it has such a problem.

One thing that I would like to admire is that both of you, and many others as well, created shell scripts to do this, and not Python. I really cannot do this in shell. You guys are good at this !

[polikuo]

I did not understand this comment. What is the problem with ExtensionName-KERNEL.tcz ?
The problem that you described, which I think is what you are saying for your script, only occurs when you are using the .dep files. The .tree files contain the KERNEL part of the TCZ name replaced with the KERNEL version so I don't think there will be a problem with my script. As my script does not include the option of running it for a different TCL version than the hardcoded values, I do not think it has such a problem.

One thing that I would like to admire is that both of you, and many others as well, created shell scripts to do this, and not Python. I really cannot do this in shell. You guys are good at this !

The problem is, all kernel module should follow that convention for dep files.
Kernel release may change value.
Thus in all dep files, we use string -KERNEL to denote that the extension is a kernel module.
Your script doesn't process those files, neither reading nor writing.

For example

Code: [Select]

$ uname -r
6.6.8-tinycoreThis is the kernel release, sometimes there maybe some major bugs in kernel.
If kernel 6.6.8 has one of those and 6.6.9 fixes that problem.
Then, after some evaluation, TC team will release TC15.1 with kernel 6.6.9

On the other hand, whenever a newer major release occurs, we move on to newer kernel.
Most packages can be copied directly without any trouble.
It would be tedious if we put actual kernel string into dep files.

[Rich]

Hi Sashank999

... I did not understand this comment. What is the problem with ExtensionName-KERNEL.tcz ? ...

My script was designed to let you ask "will my TC10 version of ProgramXYZ run on TC13?"
In that case, you can't use  uname  to pick a  KERNEL  value.

Multiple kernels.
It hasn't happened often for Intel architectures, but it has happened.
TC3:

Code: [Select]

alsa-modules-2.6.33.3-l1-rt19.tcz
alsa-modules-2.6.33.3-tinycore.tczTC4:

Code: [Select]

alsa-modules-3.0.32-x1-rt52.tcz
alsa-modules-3.0.21-tinycore.tcz
alsa-modules-3.0.3-tinycore.tczAnd all of the x86 repos also contain 64 bit versions of KERNEL for
running Core64 (64 bit kernel and modules, 32 bit apps).

Now try it on piCore.
TC15 armv6 (armhf):

Code: [Select]

alsa-modules-6.6.28-piCore.tcz
alsa-modules-6.6.34-piCore.tcz
alsa-modules-6.6.47-piCore.tczOr for TC15 aarch64:

Code: [Select]

alsa-modules-6.6.28-piCore-v8.tcz
alsa-modules-6.6.34-piCore-v8.tcz
alsa-modules-6.6.47-piCore-v8.tcz
alsa-modules-6.6.28-piCore-v8-16k.tcz
alsa-modules-6.6.34-piCore-v8-16k.tcz
alsa-modules-6.6.47-piCore-v8-16k.tczI don't know if those first three are used or just leftovers.

By the way, a tree file can/will contain a version string, but which one?

[Paul_123]

just left overs from the beta releases.  Raspi typically has major hardware releases while in the middle of a kernel release cycle, so I often upgrade the piCore kernels between beta and release and sometimes on dot releases.

[Sashank999]

The problem is, all kernel module should follow that convention for dep files.
Kernel release may change value.
Thus in all dep files, we use string -KERNEL to denote that the extension is a kernel module.
Your script doesn't process those files, neither reading nor writing.

For example

Code: [Select]

$ uname -r
6.6.8-tinycoreThis is the kernel release, sometimes there maybe some major bugs in kernel.
If kernel 6.6.8 has one of those and 6.6.9 fixes that problem.
Then, after some evaluation, TC team will release TC15.1 with kernel 6.6.9

On the other hand, whenever a newer major release occurs, we move on to newer kernel.
Most packages can be copied directly without any trouble.
It would be tedious if we put actual kernel string into dep files.

Hello polikuo.
I remember that -KERNEL.tcz are parsed by substituting the placeholder with "uname -r" output in the tce-load script. Thanks for reminding me.

So, if my script recognizes whether a dependency is a kernel dependency, then it should specify the dependency as "KernelExtenion-KERNEL.tcz".
IMHO, this can be automated by first checking if the original dependency name has "uname -r" in it. Then, we can look in the .list of the original dependency TCZ if there are files that should be in folders starting with "usr/local/lib/modules/$(uname -r)/kernel". This would be validating the dependency as a kernel tcz. Then the script can mention the dependency as a -KERNEL.tcz in the .dep file.
Would this solve the problem ? What would be the issues with this approach ?

My script was designed to let you ask "will my TC10 version of ProgramXYZ run on TC13?"
In that case, you can't use  uname  to pick a  KERNEL  value.

Hello Rich.

My script is meant to just check the dependent libraries. It does not factor in placing a different kernel's TCZs into a new TCL version. I also read a changelog entry in submitqc which mentioned library inclusion checking so I thought my script can be used along with it.

By the way, a tree file can/will contain a version string, but which one?

That depends on the text placed in the .tree file. I don't know the conventions that are followed for the .tree files.

From your TC15 aarch64 listing, I searched for a .dep file that contains a -KERNEL.tcz as a dependency and has a .tree file. I found http://repo.tinycorelinux.net/15.x/aarch64/tcz/wireless_tools.tcz.tree. Its .dep file has "wireless-KERNEL.tcz" while the .tree file has "wireless-6.6.26-piCore-v8.tcz", which does not even exist in the repo.

Are .tree files unreliable ? I think we can do the initial check that I mentioned in the previous paragraphs with "uname -r" in package name to find if a TCZ is a -KERNEL.tcz.

BTW, how do we find the kernel modules required for a TCZ ?

[Rich]

Hi Sashank999

... From your TC15 aarch64 listing, I searched for a .dep file that contains a -KERNEL.tcz as a dependency and has a .tree file. I found http://repo.tinycorelinux.net/15.x/aarch64/tcz/wireless_tools.tcz.tree. Its .dep file has "wireless-KERNEL.tcz" while the .tree file has "wireless-6.6.26-piCore-v8.tcz", which does not even exist in the repo.

Are .tree files unreliable ? I think we can do the initial check that I mentioned in the previous paragraphs with "uname -r" in package name to find if a TCZ is a -KERNEL.tcz. ...

I would say yes. You have 3 kernel versions. You have only one tree file.
All the tree file can tell you is you need kernel modules, not which version.

This sets up a search expression for detecting module versions:

Code: [Select]

case $ARCH in
        # REGEX notes:
        # First - is escaped so it's not treated as an option by grep.
        # Wildcard needs to be preceded with a period like this:  .*
               
        x86*) # REGEX for finding Intel kernel strings.
                KString="\-[0-9]+.[0-9]+.[0-9]+-tinycore.*.tcz"
        ;;
        armv*|aarch*) # REGEX for finding ARM kernel strings.
                KString="\-[0-9]+.[0-9]+.[0-9]+-piCore.*.tcz"
        ;;                                                       
        *) echo "Invalid architecture specified."
           Cleanup 1
        ;;
esac
If you detect a kernel module version, you should probably
change it to  KERNEL.

[Sashank999]

Are .tree files unreliable ?

I would say yes.
This sets up a search expression for detecting module versions:

Hello Rich.
Thank you for the regular expression. I will look into implementing this into my script.
With this inspiration, I have added "redundant dependency elimination" into submitqc.
Checkout this PR: https://github.com/tinycorelinux/submitqc/pull/10.