script that generates .tree files

[GNUser]

Would one of the developers be able to please share the script that generates the .tree files on the official repo? While I could reimplement it myself, I'm trying to some save time and energy.

P.S. When submitting new extensions for the official repo, I sometimes also have to create and submit build dependencies. Being able to generate the .tree files of all of these new extensions on my local mirror--before submitting anything--would be useful to me. For example, tce-size, which depends on the .tree files, gives valuable information that can help me decide if an extension is too bloated to be worth using/submitting.

[GNUser]

I found the script in one of my own posts
https://forum.tinycorelinux.net/index.php/topic,26040.msg168124.html#msg168124

When I use the  deptree  script with the  -d  flag, the output exactly matches the .tree format.

Sorry for the noise.

[Paul_123]

We use a C program now, most of the recent changes has been in circular dependency monitoring, so it doesn’t fill up the disk space.  But otherwise it doesn’t do much more that the shell script.

[GNUser]

Hi Paul_123. May I please have a copy of the C program? I'd prefer it over the shell script.

[Paul_123]

sure,...  I'll get it for you.  next time I can login to the repo server.

[GNUser]

Thank you

[Paul_123]

Sorry,  its cpp.  here you go.  I also included the script that calls treegen.

[Rich]

Hi GNUser
I modified the attached  gen_tcz_trees_17.x  file so the path
on the server would not be exposed.

You'll need to modify this to match your system if you want to run it:

Code: [Select]

cd $REPO || exit 1
There are also lines that call out the kernel version for TC17.

[GNUser]

Thank you very much, Paul_123. I'm in your debt.

Got it, Rich. Thanks.

[GNUser]

I did a little test. Not surprisingly, the cpp version is over 500x faster than the shell script!

Code: [Select]

$ time sh -c "for i in labwc-config labwc-dev labwc-menu-generator labwc sfwbar foot appindicator-broker dino; do treegen $i $(uname -r) >$i.tcz.tree; done"
real 0m 0.03s
user 0m 0.01s
sys 0m 0.00s

$ time sh -c "for i in labwc-config labwc-dev labwc-menu-generator labwc sfwbar foot appindicator-broker dino; do deptree -d $i >$i.tcz.tree; done"
real 0m 16.11s
user 0m 11.03s
sys 0m 4.37s

[GNUser]

I wrote my own tiny versions, just for my edification. First I wrote one in shell, then rewrote it in awk.

The awk version is ~40x faster than the shell version, so I won't bother to post the shell version.

Here's my  treegen.awk  in case anybody finds it helpful. Caveat: It is extremely minimalistic, so there are no sanity checks for circular dependencies or the like.

Code: [Select]

#!/usr/bin/awk -f

# usage example: $ treegen.awk labwc.tcz

BEGIN {
LEVEL = 0
LINUX_VERSION = "6.18.2-tinycore64"
MIRROR_PATH = "/path/to/your/local/tinycorelinux/17.x/x86_64/tcz/"
get_dependencies(ARGV[1])
}

function get_dependencies(app,    depapp) {
gsub(/KERNEL/,LINUX_VERSION,app)
for (i = 0; i < LEVEL; i++)
printf("   ")
print app
while (getline depapp < MIRROR_PATH app ".dep" > 0) {
# if (depapp ~ /\.tcz/) is there because some .dep files have blank lines.
if (depapp ~ /\.tcz/) {
LEVEL++
get_dependencies(depapp)
}
}
close(MIRROR_PATH app ".dep")
LEVEL--
}
    [Edit]: Added comment from next post to code.  Rich

[GNUser]

I should have put in a comment:
if (depapp ~ /\.tcz/) is there because some .dep files have blank lines.

[Rich]

Hi GNUser
Comment added to code.

[GNUser]

Thanks, Rich.

Here's final tally of busybox ash vs. busybox awk vs. C++ implementations of treegen. All three implementations parse .dep files present on the local mirror (without using network/wget):

                                                                        busybox ash    busybox awk      C++
time it takes to generate labwc-dev.tcz.tree:   11 sec              0.43 sec             0.06 sec
time it takes to generate vlc-dev.tcz.tree:        49 sec              1.86 sec             0.31 sec

The fact that that a compiled executable (the C++ implementation) crushes the two scripts doesn't surprise me. What does surprise me is the vast difference in performance between the shell script and the awk script--especially considering that the logic and number of operations in the two scripts is exactly the same.

[Rich]

Hi GNUser

... What does surprise me is the vast difference in performance between the shell script and the awk script--especially considering that the logic and number of operations in the two scripts is exactly the same.

I'd like to address that by quoting myself from another thread we collaborated on:

... Given how quickly GNU awk is able to sort provides.db, I'd say this problem is more than solved. The problem is crushed.

There's a reason roberts liked to inject awk snippets into his scripts. When it
comes to data manipulation, it can be wicked fast.

I've had a few instances were I found the execution time of a script unacceptable
and was forced to add an awk function. None of my techniques could even touch
the speed of awk.

I would not be surprised if GNU awk was even faster.