Enhancing the "Browse TCZs" Webpage with an Automated Script

[Nathan_SR]

Here is a small contribution from my side, to Enhance the "Browse TCZs" Webpage, with an Automated Script.

Enclosed is a tar file, containing a bash script called tabulate.sh,
which must be extracted to the folder containing all the *.tcz.info files and run from there. This will generate an index.html in the same folder ( also enclosed ) which can be viewed from any browser.

People viewing the "Browse TCZs" Webpage from various operating systems, will find it now easier, to locate information with this enhanced webpage.

If the *.tcz.info files are not present locally, then the second script,
called download_all_info_files.sh can be run to download them to a new folder. After downloading the info files, the tabulate.sh must be copied to and run from the new folder.

P.S. A few info files need to be fixed by a maintainer, to confirm to the tiny core standards. These files can be identified by first navigating to the extreme right of this webpage, using the right arrow key, and press the page down key repeatedly, to locate entries, which are shown in white color. These info files could be missing either 1. Data pertaining to anyone of these tags : Title,Description,Version,Author,Original-site,Copying-policy,Size,Extension_by,Tags OR 2. Missing the tag itself altogether OR 3. Could be missing a : symbol after the tag name. It is easy to identify what is missing in these files.

[Nathan_SR]

Viewers of this new webpage can also install and use firefox addons like : https://addons.mozilla.org/en-US/firefox/addon/table-to-excel/?src=search to convert this webpage to excel file, which results in added benefits like Filtering on any column, to find related information. For example, filtering on the Tags column will help to search and identify related programs. For eg. how many TERMINAL or FILE MANAGER programs are available as a choice etc.

[Nathan_SR]

The above proposed re-design of the "Browse TCZs" Webpage, with added fields like Description, Tags etc. and hyperlinks to Original-site, will make it ideal for Support and Documentation purposes and can serve as a Wiki Page as well.

I would like to get some feedback from TCE Maintainers / Wiki Admins on this re-design. Thanks.

[curaga]

While certainly useful, it has some downsides that make it unsuitable for our use:
- the generated file is 10x the size. The index is loaded often, size matters
- it's too wide for my display, reducing the intended usability

edit: We used to have an online search at packages.tinycorelinux.net, but bmarkus didn't have the time to maintain it.

[Nathan_SR]

Thank you curaga for the feedback. I highly appreciate it.

The tabulate.sh is easily customizable. Feel free to knock of any column that you think is un-necessary OR increasing the size / width of the webpage, by doing just three things, in that script :

1. The Variable : Search_Pattern='^Title|^Description|^Version|^Author|^Original-site|^Copying-policy|^Size|^Extension_by|^Tags'

    Suppose Author column is not required. Just knock off |^Author. Likewise for others.

2. For those columns knocked out, remove the corresponding html header lines, starting at line number 35, which has the th tag

3. Lastly, if you decide to keep the Original-site column ( now hyperlinked ), it might get displaced a little, after the above knockoffs, so need to adjust its index number at line number 52, which is currently reading :32:1. Use the echo statement below to find out its new index number, after modifying it, suitably

Re-generate the index.html webpage, by running the script on the .tcz.info files folder. If you do not have the info files to test locally, you can use the download script provided.   

Hope with these instructions, you get a highly fine tuned webpage, satisfying your requirements.

[Nathan_SR]

Hi Curaga,

in addition to my new message above, to you, here are some more helpful pointers:

1. If you search for href in tabulate.sh script, then in that line, the second $i can be replaced with HomePage, to squeeze the Original-site column further

2. If you look at the index.html file, the Version column should not be so large, which indicates an issue with certain info files, missing some important data. For eg. the ccid.tcz.info file is missing a colon symbol, after the Version keyword, causing the Authors column data to overflow into the Version column. Likewise, there are some more info files, which are preventing small columns from getting squeezed. These can be identified by a white colored cell at the end of their rows.

3. Apart from the above points, there is also a Webserver Compression option, which can be enabled for html's only, to reduce the size of a html being transferred from the Server to the Client Browsers.

4. The above proposed re-design of the "Browse TCZs" Webpage is only for tinycorelinux.net website and not for websites like ibiblio/tinycorelinux, because the ibiblio site hosts other useful files like .info, .list, .dep etc., which must be browsable for a indepth analysis of any extension. If required, a seperate link can be created nearby to the "Browse TCZs" option for the ibiblio like sites.

5. The tabulate.sh will require about 3 minutes for processing nearly 2437 info files and generate the index.html file.

Regards,

Nathan SR

[Nathan_SR]

Oops! A typing mistake in point 1. above. The HomePage string must be surrounded by double or single quotes.

[Nathan_SR]

Hi Curaga,

I have further enhanced the tabulate.sh script, to generate an index.html{,.gz}, with the following features:

1. Dynamic Searching / Filtering of all Rows, based on an input
2. Fixed width table / columns, to fit the page width, at all times
3. Colored backgrounds for a pleasing appearance
4. Hover color for easy identification of any row

The index.html.gz generated has a size of ~82.7 KB only and will be served by the webserver ( packages.tinycorelinux.net ) directly to the client browsers, after doing some one time configurations at the server level, as outlined in the links below :

https://www.techrepublic.com/article/how-to-configure-apache-moddeflate-to-enhance-web-server-performance/

https://www.christianroessler.net/tech/2015/apache-and-mod-deflate-serve-pre-compressed-content-instead-of-deflate-on-every-request.html

http://httpd.apache.org/docs/current/mod/mod_deflate.html#precompressed  ( This link does not specify html file compression )

I suggest making a seperate link called "Search TCZs", on the right of "Browse TCZs" link, pointing to this index.html at packages.tinycorelinux.net, for all tiny core websites. The webserver will check with the client browser for support of .gz files ( which most modern browsers support ) and will serve it. The uncompressed index.html must also be kept on the server for supporting very old browsers.

The tabulate.sh script can be appended with cp commands, to copy the index.html, index.html.gz and search.png to the webserver's DocumentRoot folder. This script can be run on demand or scheduled by a cron job, as per the frequency required.

Please check the enclosed new files and give me your feedback. Thanks.

P.S. Extract the index.html.gz to a new folder and copy the search.png icon into it, before viewing the index.html file.

P.S. I have also enclosed a Corrected_info_files_folder.tar.gz which contains a folder with 32 corrected info files, to the tiny core standards. Please do a diff with their original files, to know the difference and then merge them into your webserver's info folder.

[curaga]

I'm afraid it is still too large. The existing info.lst.gz is under 10kb, a 4x increase would be acceptable, but 8x is a bit too much. Can you try to shrink it more?

edit: Regarding the appearance, it looks very good now. The search however does not work in Fifth:
file:///tmp/index.html 20:0 SyntaxError: Unexpected identifier 'tdata'
file:///tmp/index.html 84:11 ReferenceError: Can't find variable: myFunction

I assume it's because the "let" keyword is only supported in new JS.

[Nathan_SR]

Hi Curaga,

I am sorry, that I offended you, by sending you a PM. I was a little eager to get your feedback on the new version, asap.
Also, I really appreciate your selfless services to the forum queries, in general.

Tiny Core is Great and I would like to complete my contribution to tiny core, the maximum extent required, by myself now.

Kindly clarify the following in your reply :

1. The currently online "Browse TCZs" webpage has a size of 37 KB and does not use any compression, as reported by http://www.whatsmyip.org/http-compression-test/?url=aHR0cDovL3Rpbnljb3JlbGludXgubmV0LzkueC94ODYvdGN6Lw==

So, the newly proposed "Search TCZs" page is just 2x the size of the current one. Also, once you turn on compression exclusively on packages.tinycorelinux.net, the .gz file will be served to the client browsers directly.

2. I tested the newly proposed webpage on Firefox, Chrome, Opera and Safari, but I could not catch any error. They seem to work fine. BTW, I noticed that  your error message was pointing to a /tmp file. Does that mean you opened the webpage directly from the .gz file, without decompressing it first ? In case, if the error still persists, kindly let me know the browser and its version that you are testing it with, so that I can replicate it at my end. Thanks.

Regards,

Nathan SR

[curaga]

It's possible our nginx config does not serve the .gz file properly - the uncompressed info.lst is 36kb. That indeed should be fixed if so. Size comparisons should naturally be compressed vs compressed.

I'm using Fifth git, and I unpacked the html file first. However, the error should happen in all older browsers: https://caniuse.com/#feat=let - we strive for the pages to work in as old browsers as possible, they work fine in text browsers such as lynx or links even. The search of course requires JS, but it shouldn't require the latest browsers.

[Nathan_SR]

Thanks Curaga for the reply.

1. As you said before, the packages.tinycorelinux.net apache webserver can be used to offer a compressed / uncompressed version of this new webpage, as per the client browser's demand. Without affecting the existing "Browse TCZs" webpage, we can have a new "Search TCZs With Modern Browsers" weblink for giving Various OS Users, a choice to locate more information on TCE's quickly. Tiny core users already have the appbrowser inbuilt, so there is less chance of them getting affected.
2. Meanwhile, someone more knowledgable in JS can work this out for older browsers, as well and then we can remove the " With Modern Browsers" suffix . The tabulate.sh bash script has the JS at the top, so should be easy to locate and modify. Hope I have carried out most of the bulk work of converting all the info files to the html table format, with this bash script and even provided the corrected info files as well.

Regards,

Nathan SR

[Nathan_SR]

Hi Curaga,

After a short refreshing break, I started working on this script, again and got some very good results, as follows:

1. I got the Search Facility working on the Fifth Browser, as well, thanks to your earlier inputs.

2. I have prepared 3 versions of the index.html.gz file, with each version, lesser in size, than the previous one. The last version is just 48.5 kB in size, with the most essential information, included, from the user's perspective. I have enclosed them all here. The first file alone is 7-zipped to meet the forum size restrictions of 192 KB total attachment size.

3. As Tiny Core defaults to vga=791 or 1024x768 resolution, I have re-worked the pixel sizes of the columns now, to fit within the page width, of this resolution.

Regards,

Nathan SR

[curaga]

The last one looks nice, and search works well. Will deploy that later.

[Nathan_SR]

Thank you Curaga. Hope our users feel the same way too.