Tiny Core Linux
Tiny Core Extensions => TCE Talk => Topic started by: jazzbiker on July 30, 2023, 02:37:50 AM
-
Hi Core!
Some extensions already use this convention. If all extensions including the system libraries will follow it then in future TinyCore will withstand the coming dependency hell easier. Packages and their dependency trees are growing, thriving and evolving.
This proposal has the subproposal part concerning the *-dev.tcz directories naming. I have a dream, that dev files will be placed in the manner, which will not overlap with the next version of the same package dev files.
As an example let's take readline-dev.tcz. Now I'm in TC10 x86. readline's version is 7.0.
readline-dev.tcz.list:
/usr/local/include/readline/keymaps.h
/usr/local/include/readline/readline.h
/usr/local/include/readline/rlconf.h
/usr/local/include/readline/rlstdc.h
/usr/local/include/readline/rltypedefs.h
/usr/local/include/readline/tilde.h
So in TC13 for example the dev files will conflict with the TC13's 8 version of readline-dev.tcz.
The library extension may be renamed and used in newer versions. But not the dev files because they will overlap with the native version dev files. It means that in order to use some package linked against one of the previous versions I must build it inside the previous TC version, which may lack some other packages present in the newer TC version.
Returning to the readline-dev.tcz example, if readline-dev.tcz will be named as readline7-dev.tcz and its files will be located in a slightly different way:
/usr/local/include/readline7/keymaps.h
/usr/local/include/readline7/readline.h
/usr/local/include/readline7/rlconf.h
/usr/local/include/readline7/rlstdc.h
/usr/local/include/readline7/rltypedefs.h
/usr/local/include/readline7/tilde.h
then it gives much more freedom for building dependent extensions.
This approach is implemented in the certain extensions (lua-dev, guile-dev as an examples) and I think it is not a kind of revolution if the system libs will follow it too. It will simply make TinyCore more convenient and developer friendly.
Have a nice Core!
-
Returning to the readline-dev.tcz example, if readline-dev.tcz will be named as readline7-dev.tcz and its files will be located in a slightly different way:
/usr/local/include/readline7/keymaps.h
/usr/local/include/readline7/readline.h
/usr/local/include/readline7/rlconf.h
/usr/local/include/readline7/rlstdc.h
/usr/local/include/readline7/rltypedefs.h
/usr/local/include/readline7/tilde.h
then it gives much more freedom for building dependent extensions.
Hi jazzbiker. The nix package manager (https://en.m.wikipedia.org/wiki/Nix_(package_manager)) does this to an extreme--every library and application is installed into its own directory with a unique name. It allows you to install multiple versions of any application and any library, if you wish. You may find the project interesting.
The package manager is distro-agnostic and can be used alongside the native package manager on any distro. I don't use it on TCL because nix assumes a traditional installation with persistent filesystem, but I used it with good results alongside apt back in my Devuan days.
-
Hi GNUser. Yes, I've heard of Nix and Guix. There are several threads on the forum touching the topic, but none of them seems to include reports about successful integration none of those package managers with TC. I never used none of these package managers and can say nothing why it is not possible to use them with TinyCore, though I suspect that if these package managers not yet are built for TinyCore there exist some significant complications to do it.
-
I don't like version numbers in extensions at all. If I write a script that loads/downloads/updates readline.tcz, I don't want to have to change the script from loading readline7.tcz to readline8.tcz to readline9.tcz to readline9.7.tcz to readline10.3.20280734.tcz, as TC versions and readline versions advance over the years. I also don't want to have to pick from ten different versioned readline packages in the extension manager.
I definately don't want to check all the time whether updates for programs are now applied to a new extension with the next version's name. Right now I've got ffmpeg4.tcz installed and have no reason not to upgrade to the latest version with an extension. Is there a ffmpeg5.tcz extension which I should switch to now? I don't know, but I've had to check before, and that's a waste of time for me. It also means that the automatically-generated "ondemand" script doesn't work because the ffmpeg executable isn't named ffmpeg4 (and renaming that executable would be an even worse idea!).
Some extensions are likely version-named for good reasons, where a compatibility issue means that both might be desired on the current version of TC. Renaming all of the extensions just for people cross-compiling for older TC versions is silly. If you want to do that, figure out your own solution. I suggest keeping TC10 extensions in a separate directory and manually loading them or symlinking them into tce/optional. Maybe keep extensions all in separate "TC(version)" directories and symlink all desired extensions to tce/optional as required. Or write your own version of tce-load. There are lots of ways to fix this problem at your end (more with Tiny Core than with most distros) without making other people's Tiny Core experience worse.
Yes, I've heard of Nix and Guix. There are several threads on the forum touching the topic, but none of them seems to include reports about successful integration none of those package managers with TC. I never used none of these package managers and can say nothing why it is not possible to use them with TinyCore, though I suspect that if these package managers not yet are built for TinyCore there exist some significant complications to do it.
Nix hasn't interested me particularly, but last night I tried installing Pkgsrc (http://www.pkgsrc.org) on TC14 (x86_64). With /usr/pkg and /usr/pkgsrc mounted to persistant storage using bind. It didn't work, with the bootstrap script getting stuck looking for /usr/pkg/bin/nawk (which it seems to have been supposed to create itself, but didn't, and symlinking to /usr/local/bin/gawk didn't work either). I tried various options to the bootstrap script, but always got the same result.
This morning though, I remembered a pre-built bootstrap tar.gz that's available for use in RHEL 7, so I'll try that when I get the chance.
-
Hi CNK!
Thanks for Your answer.
You wrote:
> I don't like version numbers in extensions at all.
I damn agree with You! Especially the versions of packages. I don't understand why they can not write the software and don't touch it again and again pushing twisted updates? Why not to think well before starting to write the software? Eight versions of the readline. Are You kidding me? What for the software is written with errors? I am serious by the way. But this is not the way of thinking of those who make this not-yet-finished-cool-buggy package1.x.y as the dependence of another cool-but-buggy-and-not-yet-ready package2.u.w. Riding the bike with the square wheels may not be named comfortable. Better to say it may be named comfortable, but it is not definitely comfortable as is.
You wrote:
> I definately don't want to check all the time whether updates for programs are now applied to a new extension with the next version's name.
I damn agree with You twice!
You wrote:
> Some extensions are likely version-named for good reasons, where a compatibility issue means that both might be desired on the current version of TC.
Oh no ... Why versions? What are these versions? I don't want to know anything about it.
I was a little bit ironic, this was intentional to show that I totally agree with You when You are describing what You want, like, prefer etc. And TinyCore developers as I can understand try to make the best to simplify the usage of TC. But even they are forced to keep the different versions of the same package in every release directory.
You wrote:
> Or write your own version of tce-load. There are lots of ways to fix this problem at your end (more with Tiny Core than with most distros) without making other people's Tiny Core experience worse.
Ok, got it. This is exactly the way I do. Sorry, if hypothetical appliance of the proposal in the topic name will really destroy the people's experience please ignore this thread. Unfortunately their experience is already partially destroyed by several versioned extensions.
Have a nice Core!
-
Hi Rich!
Can You please remove the topic? I am already not able to do it :-(
-
Are You kidding me? What for the software is written with errors? I am serious by the way. But this is not the way of thinking of those who make this not-yet-finished-cool-buggy package1.x.y as the dependence of another cool-but-buggy-and-not-yet-ready package2.u.w.
Hi jazzbiker. I feel your pain. In my opinion, it is a waste of user's time to chase "upgrades" when user's needs have not changed, underlying protocol has not changed, and current version works well enough.
Sorry if my previous recommendation (nix) was too abstract and ill-suited for TCL. I'll try to redeem myself by making a practical recommendation that promotes sanity and integrates seamlessly with TCL: AppImage.
I've been using an AppImage of Thunderbird 45.4.0 since 2016 with success, for example. I also use AppImages for gimp, libreoffice, and imagemagick, among others. Basically, if an application is complex (i.e., has GUI and/or vast dependency tree) and deals with very mature protocols or filetypes, then I'll use an AppImage for it.
An AppImage is a squashfs of an application plus all its dependencies, coupled with a little C utility that mounts the squashfs (using fuse) and launches the application. The main downside of AppImages is their large size (e.g., 161 MB for gimp, 268 MB for libreoffice), but that is the price to pay for bundling the entire dependency tree.
Call me boring and middle-aged, but I have zero interest in upgrading large applications unless I absolutely have to (no pros coupled with high probability of new bugs or dependency hell). AppImages allow me to use a frozen-in-time version of an application for as long as I want to, on any distro including TCL.
Have a nice Core (sprinkled with AppImages if it helps maintain sanity) :)
-
I don't understand why they can not write the software and don't touch it again and again pushing twisted updates?
TL;DR version of my previous post: I use AppImages to protect myself from twisted updates ;D
-
Hi GNUser. Yes, I've heard of Nix and Guix. There are several threads on the forum touching the topic, but none of them seems to include reports about successful integration none of those package managers with TC. I never used none of these package managers and can say nothing why it is not possible to use them with TinyCore, though I suspect that if these package managers not yet are built for TinyCore there exist some significant complications to do it.
i have some success , useing nix (not so recently ) and guix a bit more recently from pre-buit binary's
so it is possible , but ... afair :P the way they are designed/intended to work under their own isolated paths ect
makes packaging these package managers as squashed tcz less desirable (however! they can/could be made to produce tcz of the package's they manage!)
they are more suited to be installed under /opt
and linked bind mounted on to respective / root paths
this method worked well for me ,
and i had made some modifications (bind mounting and other tweaks) to the install scripts
to facilitate (hybrid)reinstall under various *live* environment's inc tiny core
[I suspect that if these package managers not yet are built for TinyCore there exist some significant complications to do it.]
they are designed to work on top of existing linux install , but as mentioned above
need a little convincing to operate on top of fixed ro root fs
basically that just consists of re-doing install initiation on boot (add users ect) and binding the paths from /opt/*whatever* to /*whatever*
i also looked at pkgsrc after playing with bsd 8) but as i never managed to push my guix install modification's as patch didnt want to get to much more distracted :P
-
Hi jazzbiker
Hi Rich!
Can You please remove the topic? I am already not able to do it :-(
Software versioning issues and how to deal with them sounds like
a valid topic to me. I see no reason to remove this thread.
-
As a maintainer of PHP and PostgreSQL, I use version numbers for the packages because the major new versions introduce breaking changes. But I don't number down to the patch levels because those are meant to be safe from such things. It makes sense that libraries have major numbers when they have a new ABI or significant changes, and the downstream packages look for particular versions in their config files. Yes, it's a pain, but again yes, it matters.
-
I've been using an AppImage of Thunderbird 45.4.0 since 2016 with success, for example. I also use AppImages for gimp, libreoffice, and imagemagick, among others. Basically, if an application is complex (i.e., has GUI and/or vast dependency tree) and deals with very mature protocols or filetypes, then I'll use an AppImage for it.
An AppImage is a squashfs of an application plus all its dependencies, coupled with a little C utility that mounts the squashfs (using fuse) and launches the application. The main downside of AppImages is their large size (e.g., 161 MB for gimp, 268 MB for libreoffice), but that is the price to pay for bundling the entire dependency tree.
If an AppImage exists that you like, then that's great. So far that's yet to happen for me.
A while ago I wanted to try out a program that required Qt 6, which no distro I'm running has available. I installed a recent Ubuntu on a VPS (so I didn't have to download many things over my limited home internet). First I tried to build a static binary, but it turns out you need a commercial version of Qt for static builds to work (yet another reason why I dislike Qt). So I tried to build an AppImage. Very long (228 lines of notes) story short, it took a hours of manual fiddling with the parameters to linuxdeploy (I also tried appimagetool, but didn't work for me at all) to get something that half worked. I still couldn't get the program to use the AppImage-included OpenSSL 3, so any functions requiring encryption failed because TC14 only has OpenSSL 1.1.1. Which is exactly one of those awkward version problems (OpenSSL seems to be very good at creating them lately) which AppImages are meant to solve!
Many AppImage developer's guides suggest building AppImages on older distros, which wouldn't use new versions of libraries like OpenSSL 3 and therefore only people on even older distros would encounter such problems. But old distros don't have Qt 6 packaged, so it's a catch-22.
Lesson learnt: If nobody's made an AppImage already, it's probably because it's just too damn hard to do so.
Static binaries are still better than AppImages in my optinion. I use them for pandoc and fsarchiver. But they're not always possible to build for complex software.
Pkgsrc isn't going much better. I wanted to use the binary packages available for either RHEL 7 or Slackware on x86_64, but binaries from either of those sources won't run on TC 14 Pure64. The dynamic linker (/lib/ld-linux-x86-64.so.2) doesn't seem to like them. With the RHEL 7 ones it says it can't find a shared object file, but the name of the shared object is blank and I don't see anything missing with "ldd". With the Slackware ones, both /lib/ld-linux-x86-64.so.2 and ldd return "Bus error".
So the only option is compiling from source using pkgsrc's automated system, and so far that's proving slow and error-prone. Probably no easier than compiling manually and building TC extensions. Certainly not a quick way to try out newly discovered software and see if it's useful, which is what I was looking for.
So my only answer is static binaries, and when they're not an option it seems it's just tough luck!
-
CNK writes
I use them for pandoc and fsarchiver
I used to use Knoppix and partimage. Now for my Debian based distro I use fsarchiver with SystemRescue
formerly called SystemRescuecd
Saved my bacon more than once. So am free to make more mistakes ::)
-
Hi CNK. You made many good points. I'd like to share a few comments:
If an AppImage exists that you like, then that's great. So far that's yet to happen for me.
I agree that a lot of the time there is no AppImage available for the application I need. Sometimes when this happens I can build one myself. Depending on the application, building an AppImage can range from trivial to impossible.
Many AppImage developer's guides suggest building AppImages on older distros
Yes, because of the awful fact that glibc is not forwards compatible (i.e., AppImage built on a system with old glibc will work on systems with newer glibc, but AppImage built on system with new gllibc will not work on systems with older glibc).
So my only answer is static binaries, and when they're not an option it seems it's just tough luck!
At a high conceptual level, static binaries and AppImages are similar in that both consist of an application and the libraries it needs. Main difference is that in one case the libraries are statically linked, in the other they are bundled and use LD_LIBRARY_PATH hack. Both will save the user from being forced to upgrade an application because of dependency hell (i.e., a dependency no longer exists in the repository or has been upgraded to a new major version that breaks ABI compatibility with the application).
ABI-incompatible library upgrades create a situation with no ideal solution. Traditional (including TCL) solutions are a) the new library version replaces the old version in the repository and applications have to be rebuilt and b) old version and new version have to coexist in the repository under different names. Non-traditional solutions include nix-style packaging, AppImages, and static binaries. All the solutions come at a price--added complexity, added size, missed security updates, or all the above :(
-
@aus9: Yes I was using SystemRescueCD for fsarchiver too until they dropped x86 support. I still keep the last x86 release handy.
@GNUuser: Well said, and all agreed. Good to hear that AppImages sometimes can turn out to be trivial to create even when nobody's published one already.
Musl seems to be used sometimes instead of glibc for making better-compatible static binaries.
On AppImages vs static binaries. One difference is that a static binary only includes the parts from libraries that are actually used by the application. AppImages include the whole libraries, so you end up keeping a lot of code that will never be run. So I prefer static binaries on that principle. But both are indeed a dodgy work-around for the fundamental underlying problem of changing ABIs.
-
Hi all.
Just sharing some thoughts of mine.
I would probably do something like this:
For shared libraries...
Create a temporary directory, say /tmp/mypackage/usr/local/mylib
Then do something like copying all shared libraries (dependencies included) to that directory.
For executable...
Create 2 temporary directories, say /tmp/mypackage/usr/local/mybin, /tmp/mypackage/usr/local/myscript
Copy all the original bin files into mybin, then create a wrapper script for environmental variables in myscript.
#!/bin/sh
export LD_LIBRARY_PATH=/usr/local/mylib
exec /usr/local/mybin/myprogram "$@"Modify PATH in your ~/.ashrc or ~/.profile so you could call your programs more easily
For other entries, you can probably stick to the originals, in case the binaries had path hard-coded during compilation.
/tmp/mypackage/usr/local/share...
Make sure to edit your .desktop shortcut to the wrapper script if available.
Squash up everything (mksquashfs mypackage mypackage.tcz) and try it out in a clean environment.
If you missed something, add that into your package.
Remember to check the environmental variables, some programs need their custom variables set manually.
If no error occur, it should be good.
Kinda like a personal AppImage, more flexible in a way but can be tedious.
-
After seeing the symmetry of packages in alpinelinux, I had the idea of creating a script to automate tcz, for all tinycore architectures. I would emulate each architecture in qemu, compile and get the files over ssh.
however the "changelog" part in the tcz info would get in the way. this would not be a problem for creating new packages, but packages that are already present, with changelogs from other users, would be problematic. maybe put the changelog for all architectures? for example: "...
2012/06/06 (armv7) etc
2023/06/06 (x64) etc
..." that would be visible in the info of all architectures?
but deep down I wonder if this changelog value really matters. yes, it must have some use, but importance?
-
Hi CardealRusso
... but deep down I wonder if this changelog value really matters. yes, it must have some use, but importance?
The purpose of the change log is to provide some history
of what changed and when. Changing a package can effect
the behavior of other packages which depend on it. When
that happens, checking the change logs of dependencies
can help isolate which package caused that change.
-
The purpose of the change log is to provide some history
Hello Rich.
I see a bit of redundancy in having changelog in the .info, since updatelist.txt is supposed to do this job already. If updatelist.txt has a line limit, this could be solved by simply not having a limit.
A new changelog tab could be added in the Appbrowser and it would be very easy to parse updatelist.txt to get the changes for that package. This would also be interesting when showing package changes just for the tinycore version I'm using, as it doesn't seem to make as much sense to me to see tinycore 1.x changelog on 14.x
Another topic. I know very little to nothing about emulating arm. Looks like it needs kernel and device stuff. Could you provide some kernels for armv6, armv7 and aarch64 that I can try to emulate?
-
Hi CardealRusso
... This would also be interesting when showing package changes just for the tinycore version I'm using, ...
That's nice. Personally, I use the change logs for tracking information, not just
because they might be interesting.
... as it doesn't seem to make as much sense to me to see tinycore 1.x changelog on 14.x ...
Obviously. So next time I need to see the change history of a package, I can find
it by searching through 14 updatelist.txt files. Sounds like fun. And if I need to do
the same for dependencies, the fun just multiplies. ::)
A packages .info file provides its change history.
An updatelist.txt file provides a Tinycore releases change history.
I always use the .info file because what I'm looking for isn't necessarily from the
current release, or the prior release, or the one before that, ...
... Another topic. I know very little to nothing about emulating arm. Looks like it needs kernel and device stuff. Could you provide some kernels for armv6, armv7 and aarch64 that I can try to emulate?
Maybe one of these is of interest:
http://tinycorelinux.net/13.x/x86/tcz/qemu_arm.tcz.info
http://tinycorelinux.net/13.x/x86_64/tcz/qemu-arm.tcz.info
-
Obviously. So next time I need to see the change history of a package, I can find
it by searching through 14 updatelist.txt files. Sounds like fun. And if I need to do
the same for dependencies, the fun just multiplies. ::)
The point of programs and programming is NECESSARY and, from a certain point of view, exclusively to solve this type of problem.
If you can parse one updatelist.txt, you can parse 14 updatelists and get exactly what you want, the complete history. The biggest point is not removing the changelog, as I mentioned earlier, but separating it from the .info, as this will help with future package automation.
Actually I see this solution technically superior, since the changelog tab in separate, there would be less resource usage from both the client and the server (but it would have much more resource usage when clicking on the changelog tab, since it would parse 14 or more updatelists) but since viewing the changelog is not something very common among users, it seems like a good situation.
-
Even Debian couldn't set separate changelogs right - broken downloads for changelogs in Aptitude are my number one annoyance in Devuan and often with real Debian too. Yes I often look at them - I don't like to upgrade extensions/packages without an idea of what's changed, and it's nice to see how active and established the maintenance of the package is.
I don't understand how it's not possible to automate editing the existing changelogs. It seems to me that it should be easily done in any of many ways, and I can't see how it would be anything like the hardest part of designing the automated cross-compiling system that you describe. Your example before went completely over my head.
Technically it's also possible to use QEMU in a chroot environment ("chroot" is in Busybox and coreutils.tcz, there are also more advanced chroot systems available which would need to be compiled) so that you don't need to boot a separate system and deal with SSH (etc.). You copy the ARM/x86 distro root filesystem somewhere and then switch into that while still running the native system's kernel, and potentially still running its software, or ARM software in QEMU emulation. For building x86 binaries from x86_64 you don't actually need to use QEMU for this because standard x86_64 GCC can build x86 binaries with the right options.
Nothing's foolproof doing it that way though.
-
I don't understand how it's not possible to automate editing the existing changelogs
Please don't be flippant, this is a text file with no valid formatting. Yes, I know it's possible, but that this lack of standard is bad and makes things difficult
There is no validation, just the senders of the packages supposedly try to copy the same format and if you look at the updatelist.txt now from corepure 64 you will see that the inxi.tcz had the info misspelled
http://tinycorelinux.net/14.x/x86_64/tcz/inxi.tcz.info
and since it was a small error, it was probably not noticed by the sender, nor by the auditee.
Is this really YAML? If so, they should use arrays in the changelog. That would definitely make everything a little easier.
-
I don't understand how it's not possible to automate editing the existing changelogs
Please don't be flippant, this is a text file with no valid formatting. Yes, I know it's possible, but that this lack of standard is bad and makes things difficult
There is no validation, just the senders of the packages supposedly try to copy the same format and if you look at the updatelist.txt now from corepure 64 you will see that the inxi.tcz had the info misspelled
http://tinycorelinux.net/14.x/x86_64/tcz/inxi.tcz.info
and since it was a small error, it was probably not noticed by the sender, nor by the auditee.
So you're not talking about a practical problem with making your scripts, because with them you can use change-log formatting in the info file that your scripts will understand.
Making the change-log format of other extensions better standardised seems like an entirely separate matter to your automated cross-compiling idea. Personally I don't see a significant benefit.
Is this really YAML? If so, they should use arrays in the changelog. That would definitely make everything a little easier.
I don't think it's YAML. Or if it is then I haven't been doing it very well in the extensions that I've submitted because I've never looked into YAML at all. I think it's more like header fields (eg. in email) really - any line starting with "something:" indicates a new section, and anything else is part of the previous section. Personally I like it.
I don't know whether it's official, but the tcz.info format certainly seems to have been inspired by the Linux Software Map (http://www.ibiblio.org/pub/linux/LSM-TEMPLATE.html) format (also documented here (http://lsm.qqx.org/lsm/LSM.README)). The change-log is obviously one of the fields that's been added for the TC application. Actually it could be interesting to try adapting one of the old LSM index programs (http://www.ibiblio.org/pub/Linux/search/!INDEX.html) for generating a web front-end to the extension repo. I doubt that the change-log would be much of an issue even with that.