Is TCL's philosophy of minimalism going the wrong way!?

[gadget42]

@CentralWare, i just used TCL 1.x as an example. keep us posted as you progress and hopefully we can come to some viable solution(s)!

[CentralWare]

@gadget42: TCL 1.x is kind'a in the ether as it was blended throughout the main repository website (unless someone has a backup pre-2009?? we cleaned house quite a long time ago and TCL 1.x went with it.) HOWEVER, 2.x is currently uploading as we speak!  (If memory serves, 1.x pointed to http://tinycorelinux.net/tcz instead of our current method of tinycorelinux.net/[VERSION]/tcz but it's been SO long ago I cannot be 100% certain.)

ATTEMPT #1 : Caveats
1. I'm using GIT as a storage container for the first attempt; thus far there's no problems...  "except..."
     * There IS a file size limitation of 100MB for any one file -- and there's only ONE file that falls into that category for 2.x ...  OpenOffice (so for the moment it's being left out, possibly to be broken into smaller pieces later which should be reasonably simple to do - just add openoffice-part2.tcz to the .dep and create part2 to contain half of the original )
     * TCL 2.x is showing 11,723 files being pushed at the moment and we've been kicked (connection) three times now, so pushing commits has to be done in small doses.
2. GIT is public-open-door which we want... but probably not the "ideal" platform for noob/rookie users...  then again, rookies probably should be pressed to update if they're still running single-digit TCL as it's deemed unsupported.
3. I could easily be wrong, but I thought GIT had a 2GB per repo ceiling once upon a time.  IF SO I'll be looking at one of the other options OR possibly a Split-GIT as size dictates...  we'll see!

If the above is successful and if our only real limitation is the 100mb file size @MAINTAINERS please consider this for all future submissions as it's a small price to pay for archival services!

[CentralWare]

Tiny Core Linux 2.x x86 (the only platform of the time) is almost finished uploading to GH.
1. I've been pushing ~1,000 files at a time without issue - beyond taking a long time to do so manually (hurry up 'n wait...)
2. We're already at ~6GB on GIT without complaint about storage space, but too soon to tell if ALL of the archives will hit an unforeseen cap
The last batch of ~1,500 files is going up now just to press my luck

If successful, each release will be found in its own repository (ie: 5.x x64 will have its own apartment next to 5.x x86)

The archives will NOT include older ISO releases (Version tinycore_2.11.iso, for example, should be plenty...  2.07 doesn't make sense TO ME to hang onto.)  If anyone disagrees please post your thoughts here.



SUCCESS!
https://github.com/centralware/TinyCoreLinux_2.x_x86
I'm going to scour through 2.x and see if there are any remaining files that would be necessary if I were running a device on 2.x, otherwise it looks like we're good to go!

@Rich @Curaga @Paul_123 @Etc please review your thoughts as to how you want (C)opy and README to read

@xor: This work for you??

[Stefann]

@Stefann: Please note which version(s) of TCL you want/need from the archive (v4.x and older)
@CNK: Please do the same
@xor: Please do the same
Gadget42: Ditto

Thanks!
But I’m super happy with 15.
That was the whole point of my post.
I really liked that I was able to upgrade my board from 2008 to a Linux kernel from 2024 as that discloses basically all new software to it.

I very much appreciate the drive for tiny core to stay small footprint because that allows to run frugal in memory within the memory size of my board.

[Rich]

Hi CentralWare

... SUCCESS!
https://github.com/centralware/TinyCoreLinux_2.x_x86 ...

Just a couple of observations:
1. When I looked in the tcz directory, I saw this message:

Sorry, we had to truncate this directory to 1,000 files. 9,593 entries were omitted from the list. Latest commit info may be omitted.

   The tce directory displayed a similar message.

2. There's a separate dep directory that looks like it might contain duplicate
   .dep files that are already in the tcz directory.

[CentralWare]

@Rich:
Yes, their WEBSITE truncates at 1,000 listed files - I'm thinking it's a browser related limitation so as not to overwhelm a single session.
I just tested a few minutes ago using git command-line and the entire repo is there and downloaded successfully.
(Just saw the 1,000 limit ~20 minutes ago myself )

For 2.x, I am not sure why there's a "deps" directory but instead of trying to trace back that far into history I just left it in.
We can run a file comparison and if all of the needed .dep files are inside ./tcz we can rid ourselves of it once it's confirmed

[CNK]

The problem with GitHub is that although you can access it like a repo mirror via URLs like this:

Code: [Select]

https://raw.githubusercontent.com/centralware/TinyCoreLinux_2.x_x86/master/tcz/bootchart-pl.tcz

If you try with http:// instead of https:// it redirects to https://. This means it won't work to fetch extensions on systems running old TC versions because they won't support the new HTTPS encryption methods so the downloads will fail. You could manually copy the files over, but that's never easy and now we see there's no way to list the full directories in order to manually select files to download in a browser.

Since I thought it was getting off-topic, I cut out my suggestion for archiving the old extension repos from my last post:

-----

On the old repos going offline - if I ever get enough time in the presence of a fast, unlimited, internet connection (this opportunity hasn't presented itself to me in a few years, I usually just have a few GB/month of mobile broadband data), I intend to do the following:

• Download all of the repos set to go in the next new years, say up to TC 10
• Unless I can find a way to avoid duplicate extension downloads with Rsync, run a file de-duplicator program (eg. rdfind) that turns copies of extensions or source code archives that were reused from earlier versions into symlinks
• Compress it all in an archive format supported by the Internet Archive's "view contents" function - zip or maybe tar.xz. This does still allow individual files to be accessed via direct HTTP URLs, such as for the /opt/tcemirror setting. Need to confirm whether this works with symlinks though.
• Upload it to the Internet Archive, maybe using their command-line uploader tool if the size of the file causes issues with Firefox

I suspect that many people here are in a better position connectivity-wise to do this than me, but although some have stated intentions to do so, I haven't seen any sign of people doing it. For now I've just managed to keep saving each version before it goes offline in preparation to do this one day.

-----

Currently Archive.org downloads work over HTTP and HTTPS (even though browsing the main website redirects to HTTPS). Or did before they recently got hacked and DDoSed, leaving the archive offline at the moment.

I've also been thinking about using a free Oracle Cloud VPS to run a HTTP server from its 200GB free "block" storage, but I think I measured that the later TC repos have a lot more data than that (although I don't know how much of that is duplicate extensions).

P.S. I guess the main TC extension repo server does itself de-duplicate extensions that are the same between TC versions to save space? I sure hope that would be attempted before hitting the delete button on the old versions.

[CNK]

I've also been thinking about using a free Oracle Cloud VPS to run a HTTP server from its 200GB free "block" storage, but I think I measured that the later TC repos have a lot more data than that (although I don't know how much of that is duplicate extensions).

Damn, I just went to check the sizes and the HDD I put them on has died. I remember they didn't fit on my backup drive, and I don't think I backed them up elsewhere, so I think my hours downloading them while 'on holiday' before they went offline are all down the drain.

Excuse me while I walk around swearing loudly for the rest of the day...

[CentralWare]

P.S. I guess the main TC extension repo server does itself de-duplicate extensions that are the same between TC versions to save space?

Sorry, I don't have access to the live repository server to know what's an actual file and what's a link, but I do know that a lot of copying goes into play when a new version is released (where extensions are copied from their predecessor into the new TCZ directory - whether it's real or linked maybe @curaga could shed a little light on the subject?
De-duping...  to be honest I doubt it is going to be handled in such a fashion as that takes a good deal of time...  but like I said, I don't have server access to know one way or the other for certain.

@CNK: If you would like a hard copy, please send me a private message with your full name and mailing address along with the version(s) of TCL you want to have shipped and I can send you a quote based on the required media size and shipping from the States. Please include a note as to whether or not you require source code files as this will definitely affect media size!

PS: Sorry about your HDD

[gadget42]

...
Excuse me while I walk around swearing loudly for the rest of the day...

glad to see i am not the only one who does that...still...argh!

[curaga]

There is no dedup between versions. Not that it would affect anything, as our mirrors do their own thing, and the space usage affects them as well.

[Stefann]

My 2 cents,
I guess you do not need "all" older versions.
I just got v15 working on a board that launched 2005 so I kind of feel V15 covers most post 2005 hardware.
With CentralWare succeeding to get the oldest practically functional version to live (v2) that should either be enough to cover hardware until 2005 OR maximally 1 other "smartly selected release" would make sense.

[mocore]

@Curaga: @mocore brings up a good point regarding sources and build scripts
For the time being, at least until we have a solid source repository/archive, please LEAVE source directories when disposing of antique versions.
x86/x64: ./release/src and ./tcz/src
arm/etc: ./releases/RPi/src and ./tcz/src

some of the reasoning wrt to build scripts  quoted in the below link 
https://forum.tinycorelinux.net/index.php/topic,27148.msg174366.html#msg174366

just to make the logic behind this reasoning it explicit ( idk if this (useful info) is in the fqa or wiki but imho it should be! )

If you don't  see the source it probably has not changed since the previous release, so look in the download area of previous releases.

---> so N.x/ version directory's do not necessarily contain all relevant data   <----
       extensions might be copied from previous version
       leaving build scripts
     
EG you can not " look in the download area of previous releases" if it has been deleted  ( see also : https://forum.tinycorelinux.net/index.php/topic,18712.msg175870.html )

.... perhaps im missing the point and this is desirable outcome
...  but i struggle to see the benefit of repo purge
...  archiving (eg copy to some other dir like /unsuported/*.x ) i understand deletion seams nu-advantageous
fwiw 

 (imho)  it seams a shame to loose the work/data of the tiny core team and community
however useful(or not) old versions might be to run on old hw ect
im not shore there is any disadvantage to  keep the data available (even if not in `working` mirror/dir structure ) for historical reference
 
 
... i dont set out to "go against the grain" , it just seams that often that is the way things are 

[CNK]

@CNK: If you would like a hard copy, please send me a private message with your full name and mailing address along with the version(s) of TCL you want to have shipped and I can send you a quote based on the required media size and shipping from the States. Please include a note as to whether or not you require source code files as this will definitely affect media size!

Thanks, but that's probably more trouble for you than uploading them to the Internet Archive which is what I've had in mind to do all along - just limited by internet access. Then I don't need everything myself, I can just set the repo to archive.org URLs in old TC versions.

There is no dedup between versions. Not that it would affect anything, as our mirrors do their own thing, and the space usage affects them as well.

Well Rsync can copy symlinks, so I'm not sure why it has to be that way. If that's just how distro mirrors work traditionally then that's unfortunate.

My 2 cents,
I guess you do not need "all" older versions.
I just got v15 working on a board that launched 2005 so I kind of feel V15 covers most post 2005 hardware.
With CentralWare succeeding to get the oldest practically functional version to live (v2) that should either be enough to cover hardware until 2005 OR maximally 1 other "smartly selected release" would make sense.

I want to be able to get extensions for any version that was installed before, and try extensions that weren't carried forward to later versions. A lot of TC4 extensions (next to go) aren't available for later TC versions. If others don't care about that, OK. I was hoping to sort it all out for myself, but I failed. So in the end maybe it's just my bad luck and no point discussing it further at this point. I'm pretty grumpy about it now anyway.

[Rich]

Hi CNK
I just finished creating backups of TC4 - TC8  x86 and x86_64
from distro.ibiblio.org/tinycorelinux. I currently have TC1 - TC8.

In case someone is interested in the storage required for this:

Code: [Select]

tc@E310:/mnt/sdb1$ for C in `seq 1 8`; do du -hs "$C".x/; done
2.3G    1.x/
8.4G    2.x/
14G     3.x/
26G     4.x/
9.1G    5.x/
13G     6.x/
15G     7.x/
16G     8.x/
tc@E310:/mnt/sdb1$ calc 2.3+8.4+14+26+9.1+13+15+16
103.8
tc@E310:/mnt/sdb1$
This is a breakdown of storage used just for tcz/ (files only), tcz/src/, and tcz/backup/.
Add in release_candidates/, release/, and archive/ and you have another 500M to 3.5G.

x86:

Code: [Select]

tcz/=8.6G       tcz/src/=4.9G   tcz/backup/=1.4G        x86 Ver. 4
tcz/=2.5G       tcz/src/=1.5G   tcz/backup/=954M        x86 Ver. 5
tcz/=2.6G       tcz/src/=1.2G   tcz/backup/=1.6G        x86 Ver. 6
tcz/=3.3G       tcz/src/=942M   tcz/backup/=1.2G        x86 Ver. 7
tcz/=3.5G       tcz/src/=182M   tcz/backup/=754M        x86 Ver. 8
tcz/=3.9G       tcz/src/=514M   tcz/backup/=887M        x86 Ver. 9
tcz/=4.6G       tcz/src/=396M   tcz/backup/=1.9G        x86 Ver. 10
tcz/=4.1G       tcz/src/=565M   tcz/backup/=1.5G        x86 Ver. 11
tcz/=4.1G       tcz/src/=447M   tcz/backup/=332M        x86 Ver. 12
tcz/=4.3G       tcz/src/=260M   tcz/backup/=385M        x86 Ver. 13
tcz/=5.2G       tcz/src/=290M   tcz/backup/=1.2G        x86 Ver. 14
tcz/=5.3G       tcz/src/=379M   tcz/backup/=1.5G        x86 Ver. 15
x86_64:

Code: [Select]

tcz/=374M       tcz/src/=343M   tcz/backup/=25M         x86_64 Ver. 4
tcz/=1.1G       tcz/src/=1.4G   tcz/backup/=481M        x86_64 Ver. 5
tcz/=1.6G       tcz/src/=1.5G   tcz/backup/=1.5G        x86_64 Ver. 6
tcz/=2.7G       tcz/src/=2.3G   tcz/backup/=2.2G        x86_64 Ver. 7
tcz/=3.4G       tcz/src/=2.4G   tcz/backup/=1.8G        x86_64 Ver. 8
tcz/=4.5G       tcz/src/=2.4G   tcz/backup/=2.7G        x86_64 Ver. 9
tcz/=5.2G       tcz/src/=2.3G   tcz/backup/=2.7G        x86_64 Ver. 10
tcz/=5.6G       tcz/src/=3.6G   tcz/backup/=2.2G        x86_64 Ver. 11
tcz/=6.6G       tcz/src/=2.5G   tcz/backup/=1.4G        x86_64 Ver. 12
tcz/=7.3G       tcz/src/=1.4G   tcz/backup/=2.0G        x86_64 Ver. 13
tcz/=8.4G       tcz/src/=3.3G   tcz/backup/=3.1G        x86_64 Ver. 14
tcz/=8.4G       tcz/src/=3.9G   tcz/backup/=3.2G        x86_64 Ver. 15
If anyone wants to download a repo from a mirror:

Code: [Select]

# Copy repos using wget.
# ipv4 only, quiet, recurse, no clobber, no parent, reject index.html files, no host directories, ignore NUMBER remote directory components
# The --cut-dirs prevents wget from recursing up the tree.
tc@E310:/mnt/sdb1/8.x$ wget -4 -q -r -nc -np -R "index.html*" -nH --cut-dirs 2 http://distro.ibiblio.org/tinycorelinux/8.x/x86/
As soon as it finishes, run this and make sure it returns zero:

Code: [Select]

tc@E310:/mnt/sdb1/8.x$ echo $?
0
If it does not return zero, this will provide a list of which files were problematic:

Code: [Select]

# Find download errors:
# ipv4 only, no verbose, recurse, no clobber, no parent, reject index.html files, no host directories, ignore NUMBER remote directory components
# The --cut-dirs prevents wget from recursing up the tree.
tc@E310:/mnt/sdb1/8.x$ wget -4 -nv -r -nc -np -R "index.html*" -nH --cut-dirs 2 http://distro.ibiblio.org/tinycorelinux/8.x/x86/ 2>&1 | grep -v "index.html.tmp"