Tiny Core Linux

General TC => Programming & Scripting - Unofficial => Topic started by: Nathan_SR on June 26, 2018, 07:48:57 PM

Title: Enhancing the "Browse TCZs" Webpage with an Automated Script
Post by: Nathan_SR on June 26, 2018, 07:48:57 PM
Here is a small contribution from my side, to Enhance the "Browse TCZs" Webpage, with an Automated Script.

Enclosed is a tar file, containing a bash script called tabulate.sh,
which must be extracted to the folder containing all the *.tcz.info files and run from there. This will generate an index.html in the same folder ( also enclosed ) which can be viewed from any browser.

People viewing the "Browse TCZs" Webpage from various operating systems, will find it now easier, to locate information with this enhanced webpage.

If the *.tcz.info files are not present locally, then the second script,
called download_all_info_files.sh can be run to download them to a new folder. After downloading the info files, the tabulate.sh must be copied to and run from the new folder.

P.S. A few info files need to be fixed by a maintainer, to confirm to the tiny core standards. These files can be identified by first navigating to the extreme right of this webpage, using the right arrow key, and press the page down key repeatedly, to locate entries, which are shown in white color. These info files could be missing either 1. Data pertaining to anyone of these tags : Title,Description,Version,Author,Original-site,Copying-policy,Size,Extension_by,Tags OR 2. Missing the tag itself altogether OR 3. Could be missing a : symbol after the tag name. It is easy to identify what is missing in these files.
Title: Re: Enhancing the "Browse TCZs" Webpage with an Automated Script
Post by: Nathan_SR on June 26, 2018, 08:14:10 PM
Viewers of this new webpage can also install and use firefox addons like : https://addons.mozilla.org/en-US/firefox/addon/table-to-excel/?src=search to convert this webpage to excel file, which results in added benefits like Filtering on any column, to find related information. For example, filtering on the Tags column will help to search and identify related programs. For eg. how many TERMINAL or FILE MANAGER programs are available as a choice etc.
Title: Re: Enhancing the "Browse TCZs" Webpage with an Automated Script
Post by: Nathan_SR on June 29, 2018, 04:01:32 PM
The above proposed re-design of the "Browse TCZs" Webpage, with added fields like Description, Tags etc. and hyperlinks to Original-site, will make it ideal for Support and Documentation purposes and can serve as a Wiki Page as well.

I would like to get some feedback from TCE Maintainers / Wiki Admins on this re-design. Thanks.
Title: Re: Enhancing the "Browse TCZs" Webpage with an Automated Script
Post by: curaga on June 30, 2018, 01:30:51 AM
While certainly useful, it has some downsides that make it unsuitable for our use:
- the generated file is 10x the size. The index is loaded often, size matters
- it's too wide for my display, reducing the intended usability

edit: We used to have an online search at packages.tinycorelinux.net, but bmarkus didn't have the time to maintain it.
Title: Re: Enhancing the "Browse TCZs" Webpage with an Automated Script
Post by: Nathan_SR on June 30, 2018, 06:10:59 AM
Thank you curaga for the feedback. I highly appreciate it.

The tabulate.sh is easily customizable. Feel free to knock of any column that you think is un-necessary OR increasing the size / width of the webpage, by doing just three things, in that script :

1. The Variable : Search_Pattern='^Title|^Description|^Version|^Author|^Original-site|^Copying-policy|^Size|^Extension_by|^Tags'

    Suppose Author column is not required. Just knock off |^Author. Likewise for others.

2. For those columns knocked out, remove the corresponding html header lines, starting at line number 35, which has the th tag

3. Lastly, if you decide to keep the Original-site column ( now hyperlinked ), it might get displaced a little, after the above knockoffs, so need to adjust its index number at line number 52, which is currently reading :32:1. Use the echo statement below to find out its new index number, after modifying it, suitably

Re-generate the index.html webpage, by running the script on the .tcz.info files folder. If you do not have the info files to test locally, you can use the download script provided.   

Hope with these instructions, you get a highly fine tuned webpage, satisfying your requirements.
Title: Re: Enhancing the "Browse TCZs" Webpage with an Automated Script
Post by: Nathan_SR on June 30, 2018, 11:30:47 AM
Hi Curaga,

in addition to my new message above, to you, here are some more helpful pointers:

1. If you search for href in tabulate.sh script, then in that line, the second $i can be replaced with HomePage, to squeeze the Original-site column further

2. If you look at the index.html file, the Version column should not be so large, which indicates an issue with certain info files, missing some important data. For eg. the ccid.tcz.info file is missing a colon symbol, after the Version keyword, causing the Authors column data to overflow into the Version column. Likewise, there are some more info files, which are preventing small columns from getting squeezed. These can be identified by a white colored cell at the end of their rows.

3. Apart from the above points, there is also a Webserver Compression option, which can be enabled for html's only, to reduce the size of a html being transferred from the Server to the Client Browsers.

4. The above proposed re-design of the "Browse TCZs" Webpage is only for tinycorelinux.net website and not for websites like ibiblio/tinycorelinux, because the ibiblio site hosts other useful files like .info, .list, .dep etc., which must be browsable for a indepth analysis of any extension. If required, a seperate link can be created nearby to the "Browse TCZs" option for the ibiblio like sites.

5. The tabulate.sh will require about 3 minutes for processing nearly 2437 info files and generate the index.html file.

Regards,

Nathan SR
Title: Re: Enhancing the "Browse TCZs" Webpage with an Automated Script
Post by: Nathan_SR on June 30, 2018, 11:58:59 AM
Oops! A typing mistake in point 1. above. The HomePage string must be surrounded by double or single quotes.
Title: Re: Enhancing the "Browse TCZs" Webpage with an Automated Script
Post by: Nathan_SR on July 02, 2018, 11:10:55 AM
Hi Curaga,

I have further enhanced the tabulate.sh script, to generate an index.html{,.gz}, with the following features:

1. Dynamic Searching / Filtering of all Rows, based on an input
2. Fixed width table / columns, to fit the page width, at all times
3. Colored backgrounds for a pleasing appearance
4. Hover color for easy identification of any row

The index.html.gz generated has a size of ~82.7 KB only and will be served by the webserver ( packages.tinycorelinux.net ) directly to the client browsers, after doing some one time configurations at the server level, as outlined in the links below :

https://www.techrepublic.com/article/how-to-configure-apache-moddeflate-to-enhance-web-server-performance/

https://www.christianroessler.net/tech/2015/apache-and-mod-deflate-serve-pre-compressed-content-instead-of-deflate-on-every-request.html

http://httpd.apache.org/docs/current/mod/mod_deflate.html#precompressed  ( This link does not specify html file compression )

I suggest making a seperate link called "Search TCZs", on the right of "Browse TCZs" link, pointing to this index.html at packages.tinycorelinux.net, for all tiny core websites. The webserver will check with the client browser for support of .gz files ( which most modern browsers support ) and will serve it. The uncompressed index.html must also be kept on the server for supporting very old browsers.

The tabulate.sh script can be appended with cp commands, to copy the index.html, index.html.gz and search.png to the webserver's DocumentRoot folder. This script can be run on demand or scheduled by a cron job, as per the frequency required.

Please check the enclosed new files and give me your feedback. Thanks.

P.S. Extract the index.html.gz to a new folder and copy the search.png icon into it, before viewing the index.html file.

P.S. I have also enclosed a Corrected_info_files_folder.tar.gz which contains a folder with 32 corrected info files, to the tiny core standards. Please do a diff with their original files, to know the difference and then merge them into your webserver's info folder.
Title: Re: Enhancing the "Browse TCZs" Webpage with an Automated Script
Post by: curaga on July 03, 2018, 01:02:13 AM
I'm afraid it is still too large. The existing info.lst.gz is under 10kb, a 4x increase would be acceptable, but 8x is a bit too much. Can you try to shrink it more?

edit: Regarding the appearance, it looks very good now. The search however does not work in Fifth:
file:///tmp/index.html 20:0 SyntaxError: Unexpected identifier 'tdata'
file:///tmp/index.html 84:11 ReferenceError: Can't find variable: myFunction

I assume it's because the "let" keyword is only supported in new JS.
Title: Re: Enhancing the "Browse TCZs" Webpage with an Automated Script
Post by: Nathan_SR on July 03, 2018, 06:51:57 AM
Hi Curaga,

I am sorry, that I offended you, by sending you a PM. I was a little eager to get your feedback on the new version, asap.
Also, I really appreciate your selfless services to the forum queries, in general.

Tiny Core is Great and I would like to complete my contribution to tiny core, the maximum extent required, by myself now.

Kindly clarify the following in your reply :

1. The currently online "Browse TCZs" webpage has a size of 37 KB and does not use any compression, as reported by http://www.whatsmyip.org/http-compression-test/?url=aHR0cDovL3Rpbnljb3JlbGludXgubmV0LzkueC94ODYvdGN6Lw==

So, the newly proposed "Search TCZs" page is just 2x the size of the current one. Also, once you turn on compression exclusively on packages.tinycorelinux.net, the .gz file will be served to the client browsers directly.

2. I tested the newly proposed webpage on Firefox, Chrome, Opera and Safari, but I could not catch any error. They seem to work fine. BTW, I noticed that  your error message was pointing to a /tmp file. Does that mean you opened the webpage directly from the .gz file, without decompressing it first ? In case, if the error still persists, kindly let me know the browser and its version that you are testing it with, so that I can replicate it at my end. Thanks.

Regards,

Nathan SR
Title: Re: Enhancing the "Browse TCZs" Webpage with an Automated Script
Post by: curaga on July 03, 2018, 11:05:23 AM
It's possible our nginx config does not serve the .gz file properly - the uncompressed info.lst is 36kb. That indeed should be fixed if so. Size comparisons should naturally be compressed vs compressed.

I'm using Fifth git, and I unpacked the html file first. However, the error should happen in all older browsers: https://caniuse.com/#feat=let - we strive for the pages to work in as old browsers as possible, they work fine in text browsers such as lynx or links even. The search of course requires JS, but it shouldn't require the latest browsers.
Title: Re: Enhancing the "Browse TCZs" Webpage with an Automated Script
Post by: Nathan_SR on July 03, 2018, 12:44:49 PM
Thanks Curaga for the reply.

1. As you said before, the packages.tinycorelinux.net apache webserver can be used to offer a compressed / uncompressed version of this new webpage, as per the client browser's demand. Without affecting the existing "Browse TCZs" webpage, we can have a new "Search TCZs With Modern Browsers" weblink for giving Various OS Users, a choice to locate more information on TCE's quickly. Tiny core users already have the appbrowser inbuilt, so there is less chance of them getting affected.
2. Meanwhile, someone more knowledgable in JS can work this out for older browsers, as well and then we can remove the " With Modern Browsers" suffix . The tabulate.sh bash script has the JS at the top, so should be easy to locate and modify. Hope I have carried out most of the bulk work of converting all the info files to the html table format, with this bash script and even provided the corrected info files as well.

Regards,

Nathan SR
Title: Re: Enhancing the "Browse TCZs" Webpage with an Automated Script
Post by: Nathan_SR on July 08, 2018, 03:02:01 AM
Hi Curaga,

After a short refreshing break, I started working on this script, again and got some very good results, as follows:

1. I got the Search Facility working on the Fifth Browser, as well, thanks to your earlier inputs.

2. I have prepared 3 versions of the index.html.gz file, with each version, lesser in size, than the previous one. The last version is just 48.5 kB in size, with the most essential information, included, from the user's perspective. I have enclosed them all here. The first file alone is 7-zipped to meet the forum size restrictions of 192 KB total attachment size.

3. As Tiny Core defaults to vga=791 or 1024x768 resolution, I have re-worked the pixel sizes of the columns now, to fit within the page width, of this resolution.

Regards,

Nathan SR
Title: Re: Enhancing the "Browse TCZs" Webpage with an Automated Script
Post by: curaga on July 08, 2018, 11:41:41 AM
The last one looks nice, and search works well. Will deploy that later.
Title: Re: Enhancing the "Browse TCZs" Webpage with an Automated Script
Post by: Nathan_SR on July 08, 2018, 02:06:35 PM
Thank you Curaga. Hope our users feel the same way too.
Title: Re: Enhancing the "Browse TCZs" Webpage with an Automated Script
Post by: curaga on July 18, 2018, 09:30:53 AM
Now enabled for 9.x. Some info file errors may remain.
Title: Re: Enhancing the "Browse TCZs" Webpage with an Automated Script
Post by: Nathan_SR on July 18, 2018, 11:40:54 AM
Yes, I checked it now, on both my laptop and a smart phone. Wow !
It took just 1 second to load the entire page, in spite of carrying so much data. Credits to your webserver.

A couple of little things, that remain:

1. The search.png file needs to be copied to the new index.html folder. Once copied, the "Browse TCZs" page should display this icon on the left edge of "Search in all Fields" text box.

2. The version "a" of this script/index.html can extend as a Wiki Page named "List of Extensions" and will contribute to Users/Maintainers in different ways.

BTW, Congrats on getting this implemented, at such a rapid pace.
Title: Re: Enhancing the "Browse TCZs" Webpage with an Automated Script
Post by: Nathan_SR on July 18, 2018, 07:19:32 PM
I missed out an important point in my previous message.

3. If you are planning to deploy this script, as a Cron Job as well, then a couple of additional settings might be required, inside this script:

a. cd full_path_to_info_files  #### script will pick up the info files from here

b. export PATH=current_path_setting_to_all_binaries

Hope the first test run of the cron job will reveal, if anything else is required.
Title: Re: Enhancing the "Browse TCZs" Webpage with an Automated Script
Post by: curaga on July 19, 2018, 01:28:51 AM
The PNG file will not display at all in old browsers, so copying it to every folder didn't seem necessary. Browsers new enough to display it also display the placeholder text.
Title: Re: Enhancing the "Browse TCZs" Webpage with an Automated Script
Post by: Nathan_SR on July 19, 2018, 03:00:24 AM
I got what you are saying. Atleast for the newer browsers, it will display a nice search icon, like the one at : http://wiki.tinycorelinux.net/ search box. It will require a one time effort though.
Title: Re: Enhancing the "Browse TCZs" Webpage with an Automated Script
Post by: aus9 on July 20, 2018, 08:17:06 PM
Hi Nathan_SR

1) I have 64 and often run firefox on 64 bit

so firstly I installed the FF addon as per
https://addons.mozilla.org/en-US/firefox/addon/table-to-excel/?src=search

and pointed my FF to
https://distro.ibiblio.org/tinycorelinux/9.x/x86_64/tcz/

and that appears to work altho, its not useful enough for me at this stage......so

2) I downloaded tabulate scripts version 4 and download_all_info_files.sh

I have not downloaded any index html file as I assume they are 32 bit, so am attempting to generate a 64 bit html

I installed it to my .local/bin and made it executable
I amended it to read
Code: [Select]
#!/bin/bash
tce-load -i wget
mkdir tcz.info.files.folder
cd tcz.info.files.folder
pwd
# Check URL at the End of the Below Command For Any Modification Required
wget -e robots=off --no-check-certificate -r -l1 -np -nd -A ".info" https://distro.ibiblio.org/tinycorelinux/9.x/x86_64/tcz/

I had to load wget as my busybox wget was not able to handle no-check-certificate and as you can see just the 64 bit repo changed too

I get this error?

Code: [Select]
sh download_all_info_files.sh
wget is already installed!
/home/tc/.local/bin/tcz.info.files.folder
--2018-07-21 10:57:36--  https://distro.ibiblio.org/tinycorelinux/9.x/x86_64/tcz/
Resolving distro.ibiblio.org... 152.19.134.43
Connecting to distro.ibiblio.org|152.19.134.43|:443... connected.
WARNING: The certificate of ‘distro.ibiblio.org’ is not trusted.
WARNING: The certificate of ‘distro.ibiblio.org’ hasn't got a known issuer.
HTTP request sent, awaiting response... 200 OK
Length: 241884 (236K) [text/html]
Saving to: ‘index.html’

100%[=====================================================>] 241,884      176KB/s   in 1.3s   

2018-07-21 10:57:38 (176 KB/s) - ‘index.html’ saved [241884/241884]

Removing index.html since it should be rejected.

FINISHED --2018-07-21 10:57:38--
Total wall clock time: 2.4s
Downloaded: 1 files, 236K in 1.3s (176 KB/s)
tc@box:~/.local/bin$ ls
download_all_info_files.sh  get-dep-list.sh  sound    wifi
elog     kernel      tabulate_script_version_c.sh
fox     search.png      tcz.info.files.folder/

I attempted a root search for index.html get no hits

how can I stop it from "Removing index.html since it should be rejected." ?

thanks in advance
Title: Re: Enhancing the "Browse TCZs" Webpage with an Automated Script
Post by: Rich on July 20, 2018, 08:29:50 PM
Hi aus9
... I have not downloaded any index html file as I assume they are 32 bit, so am attempting to generate a 64 bit html ...
To the best of my knowledge, HTML files are text based and do not come in 32 or 64 bit flavors.
Title: Re: Enhancing the "Browse TCZs" Webpage with an Automated Script
Post by: aus9 on July 20, 2018, 10:04:20 PM
fair enough

any clues why its created and then immediately deleted?
Title: Re: Enhancing the "Browse TCZs" Webpage with an Automated Script
Post by: Rich on July 20, 2018, 11:30:18 PM
Hi aus9
... any clues why its created and then immediately deleted?
Based on what you posted, it looks like it's being done by  download_all_info_files.sh.  It probably downloads the index file
to create the list of info files to download. Once it's done that, it probably no longer needs the index and thus deletes it.
Title: Re: Enhancing the "Browse TCZs" Webpage with an Automated Script
Post by: Nathan_SR on July 21, 2018, 02:33:47 AM
Thanks Rich for being very helpful, as always.

Hi Aus9,

The -A option of wget filters files on the extensions given and removes others that don’t match. You can give .html here if you require. BTW, are you preparing a 64 bit version of the new Browse TCZ webpage ? Then, downloading all the 64 bit info files first and then running the tabulate.sh in that folder, should generate a new index.html in that folder. Hope this helps.
Title: Re: Enhancing the "Browse TCZs" Webpage with an Automated Script
Post by: aus9 on July 21, 2018, 04:12:29 AM
I got the index.html to download via below command but I am attempting to get all info files from repo  and not my TCEDIR
Code: [Select]
wget -e robots=off --no-check-certificate -r -l1 -np -nd -A ".html" https://distro.ibiblio.org/tinycorelinux/9.x/x86_64/tcz/
sorry but I am running it from .local/bin and not my tcedir...sorry if not clear early

anyhow new error is
Code: [Select]
sh tcz.info.files.folder/tabulate_script_version_c.sh
cat: can't open '*.tcz.info': No such file or directory
tcz.info.files.folder/tabulate_script_version_c.sh: line 141: syntax error: unexpected "(" (expecting "then")

I am not a real coder so umm I have other things to do. sorry to have troubled you
Title: Re: Enhancing the "Browse TCZs" Webpage with an Automated Script
Post by: Nathan_SR on July 21, 2018, 05:01:09 AM
Hi Aus9,

Yes, the error message is pretty clear.

First, type this on your terminal :

cd tcz.info.files.folder

Then:

sh tabulate_script_version_c.sh

Hope, you have got some info files already in that folder. Just check:

ls -lt *.tcz.info | wc -l

which should return a count greater than 1.
Title: Re: Enhancing the "Browse TCZs" Webpage with an Automated Script
Post by: Greg Erskine on July 21, 2018, 07:13:59 PM
I am not sure if I am following this 100%, but tce-fetch.sh is handy for getting .info files as well as others.

Here is a list of standard Tiny Core commands. When I developed a web interface for manipulating extensions for pCP I found the Tiny Core guys had already written *all* the required scripts.

http://wiki.tinycorelinux.net/wiki:tc_commands

Note: An important script,  search.sh is missing from the list.
Title: Re: Enhancing the "Browse TCZs" Webpage with an Automated Script
Post by: Nathan_SR on July 21, 2018, 07:39:45 PM
Thanks Greg for the info....