WelcomeWelcome | FAQFAQ | DownloadsDownloads | WikiWiki

Author Topic: piCore Extension Request  (Read 8473 times)

Offline Rich

  • Administrator
  • Hero Member
  • *****
  • Posts: 11670
Re: piCore Extension Request
« Reply #15 on: December 22, 2020, 03:16:25 PM »
Hi centralware
For which architecture are you seeing that. What did you set  MinVersion  and  MaxVersion  to?

Offline Rich

  • Administrator
  • Hero Member
  • *****
  • Posts: 11670
Re: piCore Extension Request
« Reply #16 on: December 22, 2020, 04:50:52 PM »
Hi centralware
I could not replicate your results. I ran it for armv6 from TC5 to TC11:
Code: [Select]
aircrack-ng.tcz:TC9
alpine-doc.tcz:TC7
alpine.tcz:TC7
alsa-config.tcz:TC7
alsa-dev.tcz:TC7
alsa-doc.tcz:TC7
alsa-locale.tcz:TC7
alsa-modules-4.19.81-piCore.tcz:TC11
alsa-oss-dev.tcz:TC9
alsa-oss-doc.tcz:TC9
alsa-oss.tcz:TC9
alsa-plugins-dev.tcz:TC11
alsa-plugins.tcz:TC11
alsa-utils-doc.tcz:TC11
alsa-utils-locale.tcz:TC11
alsa-utils.tcz:TC11
alsa.tcz:TC11
alsaequal.tcz:TC7
amtk-dev.tcz:TC11
amtk-gir.tcz:TC11
amtk.tcz:TC11

The only thing I can think of is you ran the  comm  command on lists that were not sorted identically.

I added a filter to deal with piCore modules. The script attached to reply #10 has been updated.

Offline CentralWare

  • Retired Admins
  • Hero Member
  • *****
  • Posts: 770
Re: piCore Extension Request
« Reply #17 on: December 22, 2020, 09:06:27 PM »
Min=5, Max=(current=11), OS=TCL ArmV7
I'll double-check the sorting, thanks.  (Note: no attachment found :) )
Thanks!
T.J.

Offline CentralWare

  • Retired Admins
  • Hero Member
  • *****
  • Posts: 770
Re: piCore Extension Request
« Reply #18 on: December 22, 2020, 09:07:31 PM »
Sorry, misunderstood the attachment part.

Offline Rich

  • Administrator
  • Hero Member
  • *****
  • Posts: 11670
Re: piCore Extension Request
« Reply #19 on: December 22, 2020, 09:14:37 PM »
Hi centralware
Min=5, Max=(current=11), OS=TCL ArmV7 ...
Still works correctly, even though ArmV7 did not exist until TC6:
Code: [Select]
aircrack-ng.tcz:TC9
alpine-doc.tcz:TC7
alpine.tcz:TC7
alsa-config.tcz:TC7
alsa-dev.tcz:TC7
alsa-doc.tcz:TC7
alsa-locale.tcz:TC7
alsa-modules-4.19.81-piCore-v7.tcz:TC11
alsa-modules-4.19.81-piCore-v7l.tcz:TC11
alsa-oss-dev.tcz:TC9
alsa-oss-doc.tcz:TC9
alsa-oss.tcz:TC9
alsa-plugins-dev.tcz:TC11
alsa-plugins.tcz:TC11
alsa-utils-doc.tcz:TC11
alsa-utils-locale.tcz:TC11
alsa-utils.tcz:TC11
alsa.tcz:TC11
alsaequal.tcz:TC7
amtk-dev.tcz:TC11
amtk-gir.tcz:TC11
amtk.tcz:TC11

Offline CentralWare

  • Retired Admins
  • Hero Member
  • *****
  • Posts: 770
Re: piCore Extension Request
« Reply #20 on: December 22, 2020, 10:24:31 PM »
ArmV7 / Version 6.x - correct; I just had to make it easy for anyone to set a static MinVer= (config file) and work around the errors when files don't exist, etc. and based on my personal experiences in x86, for example, some extensions dated back to v5.x when running v8, so this isn't the first time I had to build something to compensate between versions, it's just the first time I intended to submit the end results as a TCL update and/or extension itself.
 
Yes, I ran your script - seems to work flawlessly on info.lst; I'm incorporating it into tags.db right this moment which is just about done. Provides.db...  that one's going to be a joy! :)
Currently, with info.lst & tags.db we're at ~3.25 seconds on a Pi3 running the show. A keyword search takes a couple seconds to respond with the new tags.db, but I imagine it's not much quicker with a single #.x file, let alone the combined one. Due to the need to script a loop to rebuild provides.db (/usr/bin/provides.sh) it'll likely run behind the scenes, that part of the new tce-ab being disabled until it completes.  (Unless you can see a quicker method??)


Your assistance here is truly appreciated!

Offline CentralWare

  • Retired Admins
  • Hero Member
  • *****
  • Posts: 770
Re: piCore Extension Request
« Reply #21 on: December 23, 2020, 06:16:21 AM »
Good morning!
I took a quick peek at the provides.db concept/layout - is there any reason we're keeping the directory structure usr/local/somewhere/else/filename.so or do you think this can be truncated?  (I'm assuming provides is used to find files within an extension - not find the path they'd be installed into had the extension been loaded.)

Offline Rich

  • Administrator
  • Hero Member
  • *****
  • Posts: 11670
Re: piCore Extension Request
« Reply #22 on: December 23, 2020, 08:48:09 AM »
Hi centralware
The paths can provide context. If I enter this term for provides:
Code: [Select]
findI get 88 results, 87 of which I'm not interested in. If I enter this:
Code: [Select]
bin\/findI reduce that to 4 results.

Offline CentralWare

  • Retired Admins
  • Hero Member
  • *****
  • Posts: 770
Re: piCore Extension Request
« Reply #23 on: December 23, 2020, 12:26:08 PM »
Makes sense; thanks for the enlightenment.
Mind looking at something for me?  I am trying to use our info.lst files to create a merged provides.db (example using TC=8, thus 8Unique.lst and 8.db is a copy of provides.db from 8.x/tcz
Code: [Select]
awk '{print $0}' 8Unique.lst | awk -v TARGET=$1 'BEGIN {FS="\n";RS=""} /$TARGET/{print $0}' 8.db
Obviously not working. Suggestions?
1. Read 8Unique.lst one line at a time
2. Pipe the results (something.tcz) into the second awk as TARGET by reading $1  <-- likely where the issue is
3. Read 8.db, searching for $TARGET and print the contents of that entry $0
** a match() may be necessary - files like mc.tcz reside inside filenames such as libXvmc.tcz


Thanks!


Offline Rich

  • Administrator
  • Hero Member
  • *****
  • Posts: 11670
Re: piCore Extension Request
« Reply #24 on: December 25, 2020, 04:08:36 PM »
Hi centralware
I modified the script as follows:
1. It creates a  providesOld.db  file that includes only the extensions not found in  $MaxVersion.
2. I added more robust error checking to deal with files/URLs not being found.  $MinVersion  can be left as 4 regardless
   of which architecture it is run for.

I ran the script for armv7 and it took 3 mins and 37 secs to complete. 3 mins and 35 secs of that was from creating  providesOld.db.

I included a modified version of the provides.sh script called  ProvidesOld.sh.  It looks for  providesOld.db  in the current directory.
In addition to printing out the extension name and TC version, it also prints the path/filename it matched on:
Code: [Select]
tc@E310:~/count$ ./ProvidesOld.sh libpng
libpng-doc.tcz:TC9
usr/local/share/man/man3/libpng.3
usr/local/share/man/man3/libpngpf.3
libpng12.tcz:TC7
usr/local/lib/libpng12.so.0
usr/local/lib/libpng12.so.0.51.0
vlc.tcz:TC7
usr/local/lib/vlc/plugins/codec/libpng_plugin.la
usr/local/lib/vlc/plugins/codec/libpng_plugin.so
tc@E310:~/count$

Offline CentralWare

  • Retired Admins
  • Hero Member
  • *****
  • Posts: 770
Re: piCore Extension Request
« Reply #25 on: December 26, 2020, 05:02:00 AM »
Happy Holidays!
There has to be a way to rip through provides.db without wget/looping.
Here's my mental concept:
  • We start a loop while TC in $(seq $MaxVer -1 $MinVer)
  • We have a copy of provides$TC.db for each version downloaded
  • $TC"Unique.lst" has a complete list of tcz files which we're going to pass to awk as TARGET
Code: [Select]
awk 'BEGIN {FS="\n";RS=""} /'${TARGET}'/{print $0} END{print "\n"}' "provides$TC.db" > our_output$TC.db
I know this has to be do-able, I'm just having a time of it experimenting with different methods to either have a second instance of grep, awk, etc. feeding this instance -v TARGET=$1... It's a work in progress since I don't have all that much personal experience with awk. (The the past decade, I doubt I've used awk in a more complicated method beyond:
Code: [Select]
ifconfig eth0 | grep Bcast | awk -F: '{print $2}' | awk '{print $1}'...and even that's probably not the best method to do what's being done ;) )

Many thanks and a Happy New Year!

Offline Rich

  • Administrator
  • Hero Member
  • *****
  • Posts: 11670
Re: piCore Extension Request
« Reply #26 on: January 03, 2021, 12:05:26 AM »
Hi centralware
So, I decided to start from scratch and rethink my approach, and it payed off. These are the results:
Code: [Select]
tc@E310:~/count/armv7TC04TC11$ sudo cache-clear
tc@E310:~/count/armv7TC04TC11$ sync
tc@E310:~/count/armv7TC04TC11$ ./BuildProvidesAllDatabase.sh
Stopwatch reset. Fetching provides.db files.
Elapsed Time=00:00:15 Fetching provides.db files. Done.

Stopwatch reset. Convert records from multi line to single line.
Elapsed Time=00:00:01 Convert records from multi line to single line. Done.

Elapsed Time=00:00:00 Combine provides files and strip out kermel module records.
Elapsed Time=00:00:02 Combine provides files and strip out kermel module records. Done.

Elapsed Time=00:00:00 Do a case insensitive sort (-f).
Elapsed Time=00:00:05 Case insensitive sort (-f). Done.

Elapsed Time=00:00:00 Create combined info.lst from combined provides.db.
Elapsed Time=00:00:00 Create combined info.lst from combined provides.db. Done.

Elapsed Time=00:00:00 Remove duplicate names keeping the most recent TCversion, saving in infoAll.lst.
Elapsed Time=00:00:01 Remove duplicate names keeping the most recent TCversion, saving in infoAll.lst. Done.

Elapsed Time=00:00:00 Find providesSort.db entries matching infoAll.lst, saving results.
Elapsed Time=00:00:01 Find providesSort.db entries matching infoAll.lst, saving results. Done.

Elapsed Time=00:00:00 Convert back to multi line, save in providesAll.db file.
Elapsed Time=00:00:00 Convert back to multi line, save in providesAll.db file. Done.

Elapsed Time=00:00:10 Total time spent manipulating data.


Results were saved in the armv7TC04TC11 directory.


tc@E310:~/count/armv7TC04TC11$
Time to download the provides.db files is listed separately. It varied from 15 to 28 seconds. Nothing I can do about that.
After that, the steps are"
1. Append  :TCversion  to each  Name.tcz  entry and change the field separators in the provides.db files from a carriage
    return to a semicolon. This way each record in the file occupies only one line. This will simplify sorting later on.
2. Combine the provides.db files while stripping out the kernel module entries from the older TC versions.
3. Do a case insensitive sort on the first field of the combined provides.db file.
4. Create an info.lst that matches the combined provides.db file by copying the first field from each record.
5. Reduce the info.lst by keeping the most recent version of each extension name.
6. Reduce the combined provides.db file using the reduced info.lst file as a template.
7. Change the field separators in the reduced provides.db file from a semicolon back to a carriage return.

The process started with 6901 entries and reduced that to 2141 entries.
Total time for steps 1 through 7 is 10 seconds. All commands in the script are aliased to busybox. At 5 seconds, step 5
takes the most time. Using GNU sort, step 5 execution time drops from 5 seconds to 0.5 seconds.

Results are saved in a work directory formed by $ARCH $MinVersion and $MaxVersion like this  armv7TC04TC11.

I decided to run this on a bigger data set, so I went with x86 TC4 through TC11.
Time to download the provides.db files was 47 seconds.
The process started with 20113 entries and reduced that to 5594 entries.
Total time for steps 1 through 7 was 41 seconds. Step 5 took 23 seconds. Using GNU sort would reduce step 5 to 2 seconds.

The BuildProvidesAllDatabase.sh script saves its results in 2 files,  infoAll.lst  and  providesAll.db.
The ProvidesAll.sh script can be use to search providesAll.db which it looks for in the current directory. Example:
Code: [Select]
tc@E310:~/count/armv7TC04TC11$ ../ProvidesAll.sh "bin\/find"
findutils.tcz:TC11
usr/local/bin/find
libftdi.tcz:TC09
usr/local/bin/find_all
util-linux.tcz:TC11
usr/local/sbin/findfs
usr/local/bin/findmnt
tc@E310:~/count/armv7TC04TC11$

Offline CentralWare

  • Retired Admins
  • Hero Member
  • *****
  • Posts: 770
Re: piCore Extension Request
« Reply #27 on: January 06, 2021, 12:15:23 AM »
Good evening, Rich!
I must say, you're both persistent and dedicated! Kudos and thank you!
Time is against me this week; I'll take a look at the scripts this weekend and see how well/easy things incorporate into the tce-ab project.
(Note: info.lst, provides.db, etc. have .gz counterparts within the repository (ie: provides.db.gz) which obviously affect download times a good deal.)

I don't know if you have the answer to this, but I was setting up v12.x within our local repo copy and noticed 12.x/armv7l is missing releases/kernel sources?  (It has armv7l/tcz populated, thus seemed out of the ordinary to have something that large and not use-able?)

Offline Rich

  • Administrator
  • Hero Member
  • *****
  • Posts: 11670
Re: piCore Extension Request
« Reply #28 on: January 06, 2021, 12:23:53 AM »
Hi centralware
For some reason the sources are located here:
http://tinycorelinux.net/12.x/armv7/releases/RPi/src/kernel/

Offline Paul_123

  • Administrator
  • Hero Member
  • *****
  • Posts: 1252
Re: piCore Extension Request
« Reply #29 on: January 06, 2021, 12:42:30 PM »
The 12.x/armv7l is a dead repo.  You should not be using that.