Tiny Core Linux
Tiny Core Extensions => TCE Corepure64 => Topic started by: GNUser on August 18, 2023, 01:33:59 PM
-
Hi, Juanito and jazzbiker. Messaging system is not letting me send PMs right now, so I will continue the discussion here.
Thank you both very much for your input. I think I have everything I need to move forward.
lua-5.4.tcz will contain executables called lua5.4 and luac5.4 and will happily coexist with lua-5.3.tcz
lua-5.4-lib.tcz and lua-5.4-dev.tcz I will just submit as-is. It seems too complicated to make any special provisions for concurrent different versions.
User can load lua-5.3-lib.tcz and lua-5.4-lib.tcz at same time if needed. The only clash will be liblua.so, which is just a link anyway. Without manual intervention, whichever version of the lib extension is loaded last will win with regard to where liblua.so is pointing.
Inside /usr/local/lib/pkgconfig/lua.pc there are variables for version and release as well as hardcoded paths to include/lua5.3 or include/lua5.4. I think having different versions of the dev extension loaded at the same time will not be a good idea.
-
Hi Bruno and Juanito,
Here's how I imagine the tce.installed for lua-x.y-lib.tcz:
LIBLUA_LATEST=$(ls -r /usr/local/lib/liblua.so.?.?.? | sed 1q )
ln -sf $(readlink $LIBLUA_LATEST) /usr/local/lib/liblua.so
What's Your opinion, is it correct code?
-
Inside /usr/local/lib/pkgconfig/lua.pc there are variables for version and release as well as hardcoded paths to include/lua5.3 or include/lua5.4. I think having different versions of the dev extension loaded at the same time will not be a good idea.
It would be nice for lua-5.4-dev.tcz to contain /usr/local/lib/pkgconfig/lua-5.4.pc instead of lua.pc. Such lua.pc and lua-5.4.pc files can coexist and contain differing paths.
In fact lua.pc is usually not necessary, still it would be nice to keep it existing.
-
What if version has two digits somewhere? I'd use * to cover for that. Also, sorting version numbers is tricky. Best to let "sort -V" handle it than rely on ls.
How about this?
LIBLUA_LATEST=$(ls /usr/local/lib/liblua.so.*.*.* | sort -V | tail -n1)
ln -sf $(readlink $LIBLUA_LATEST) /usr/local/lib/liblua.so
Regarding the .pc file, you are the lua expert. Let's go with lua-5.4.pc, then.
-
Okay, I submitted the following extensions:
* lua-5.4.tcz (with binaries renamed to lua5.4 and luac5.4)
* lua-5.4-lib.tcz (with startup script that points lualib.so to highest version of liblua.so.*.*.*)
* lua-5.4-dev.tcz (with lua.pc renamed to lua-5.4.pc)
Thanks for all your input, guys. If we discover this is not the very best solution, we can always tweak. But it seems really solid to me.
-
Great!
You can make some tests. For example install new lua-5.4.tcz and lua-5.4-lib.tcz aside with lua-5.3.tcz and lua-lib.tcz. Then load and install vis.tcz and see does vis editor will actually run.
-
You can make some tests. For example install new lua-5.4.tcz and lua-5.4-lib.tcz aside with lua-5.3.tcz and lua-lib.tcz.
I just did.
Then load and install vis.tcz and see does vis editor will actually run.
It sure does :)
-
It was compiled against the present lua-5.3. So liblua.so.5.3.6 is accessible, everything seems to be ok. Include files are separated by original lua installer and lua-5.4.pc is ready to use. Seems like now You have the choice of lua versions, even for development.
Thanks, great!
-
I think the only thing to be tested is
readlink /usr/local/lib/liblua.so
-
Hi, jazzbiker.
Loading old version of the library then loading new version works as expected:
$ tce-load -i lua-5.3-lib
lua-5.3-lib.tcz: OK
$ readlink /usr/local/lib/liblua.so
/tmp/tcloop/lua-5.3-lib/usr/local/lib/liblua.so
$ tce-load -i lua-5.4-lib
lua-5.4-lib.tcz: OK
$ readlink /usr/local/lib/liblua.so
/tmp/tcloop/lua-5.4-lib/usr/local/lib/liblua.so.5.4.6
Then I make sure onboot.lst has no lua-*-lib extensions, and reboot.
Next test: Load new version of the library, then load the old version:
$ tce-load -i lua-5.4-lib
lua-5.4-lib.tcz: OK
$ readlink /usr/local/lib/liblua.so
/tmp/tcloop/lua-5.4-lib/usr/local/lib/liblua.so.5.4.6
$ tce-load -i lua-5.3-lib
lua-5.3-lib.tcz: OK
$ readlink /usr/local/lib/liblua.so
/tmp/tcloop/lua-5.4-lib/usr/local/lib/liblua.so.5.4.6
This test also gives us the result we want (liblua.so pointing to newest version), but it is somewhat unexpected. I would have expected that loading lua-5.3.lib (which does not have a startup script for smart liblua.so) would have clobbered liblua.so with its own version (i.e., the one that points to /tmp/tcloop/lua-5.3-lib/usr/local/lib/liblua.so which in turns points to /tmp/tcloop/lua-5.3-lib/usr/local/lib/liblua.so.5.3.6).
-
Very nice!
If lua-lib is installed after lua-5.4-lib, /usr/local/lib/liblua.so link already exists, and is not overwritten.
-
Very nice!
If lua-lib is installed after lua-5.4-lib, /usr/local/lib/liblua.so link already exists, and is not overwritten.
Aha! I didn't realize that loading extensions did not overwrite existing files. I did some tests and can confirm this is true.
Rich, do you know where in the tce-load script is the logic that prevents existing files from being overwritten when extensions are loaded? I can't seem to find it.
-
I found it: Lines 11 and 143 in /usr/bin/tce-load:
FORCE="n" # Overwrite system files default to no. Use -f to force overwrite.
yes "$FORCE" | sudo /bin/cp -ais /tmp/tcloop/"$APPNAME"/* / 2>/dev/null
-
Hi GNUser,
What's Your opinion about adding tce.installed to lua-5.4.tcz?
LUA_LATEST=$(ls /usr/local/bin/lua*.* | sort -V | tail -n1)
ln -sf $(readlink $LUA_LATEST) /usr/local/bin/lua
LUAC_LATEST=$(ls /usr/local/bin/luac*.* | sort -V | tail -n1)
ln -sf $(readlink $LUAC_LATEST) /usr/local/bin/luac
and to lua-5.4-dev.tcz:
LUA_LATEST=$(ls /usr/local/lib/pkgconfig/lua-*.*.pc | sort -V | tail -n1)
ln -sf $(readlink $LUA_LATEST) /usr/local/lib/pkgconfig/lua.pc
I expect they will work nice.
EDIT: maybe the naming scheme needs to be changen in order for sort -V to work correctly. I mean
lua.5.4
luac.5.4
lua.5.4.pc
EDIT2: experiments show that it is not necessary.
-
Hi jazzbiker. These are very nice ideas but I already submitted the new set of extensions and am too burned out right now to create yet a third set of extensions.
If Juanito wants to swoop in and add the startup scripts to lua-5.4.tcz and lua-5.4-dev.tcz I would not object ;D
My only small objection is that lua.5.4 etc is a bit ugly and difficult to type. My vote is keep binary names as lua5.3, luac5.3, lua5.4, luac5.4, etc. Besides, sort -V works just fine with such names:
$ cd /tmp/dummy
$ touch lua5.3
$ touch lua5.4
$ touch lua10.2 # version from the future!
$ ls | sort -V
lua5.3
lua5.4
lua10.2
-
The logic would have to be refined, this approach does not give the expected results:
$ ls /usr/local/bin/lua*.*
/usr/local/bin/lua5.4 /usr/local/bin/luajit-2.0.5
/usr/local/bin/luac5.4
Something like this is what we need:
$ ls /usr/local/bin/lua* | grep -Eo '/usr/local/bin/lua[0-9]+\.[0-9]+' | sort -V
$ ls /usr/local/bin/lua* | grep -Eo '/usr/local/bin/luac[0-9]+\.[0-9]+' | sort -V
-
I'm going to submit a third set of extensions. Will do it quick, like ripping off a bandaid.
-
My only small objection is that lua.5.4 etc is a bit ugly and difficult to type. My vote is keep binary names as lua5.3, luac5.3, lua5.4, luac5.4, etc.
I think You don't need to bother about the names' ugliness because the tce.installed will hide all the ugliness behind the beautiful links.
I even see no trouble in lua-5.4 and luac-5.4 binaries. The one thing I want to ask is lua-5.4.pc, in order to the *.pc name will correspond with an extension name.
-
I think You don't need to bother about the names' ugliness because the tce.installed will hide all the ugliness behind the beautiful links.
Not if you have multiple versions installed and are trying to use one of the older versions ;D
Ok, I just submitted a third set of extensions of Lua 5.4 for TCL14 x86_64. Third time's the charm!
Note that in order for these new tce.installed scripts to work well, some changes are recommended in the Lua 5.3 extensions:
* lua-5.3.tcz: binaries should be renamed to lua5.3 and luac5.3
* lua-5.3-dev.tcz: /usr/local/lib/pkgconfig/lua.pc should be renamed /usr/local/lib/pkgconfig/lua-5.3.pc
P.S. Maybe in a future redo the hyphens could vanish from the extension names (to become lua5.4.tcz, for example--to match way python versions are packaged) and from the .pc file names (to lua5.3.pc, for example--to match the binary names). But that's something to consider some other time. I think we've done enough for now.
-
Hi GNUser
... * lua-5.3-dev.tcz: /usr/local/lib/pkgconfig/lua.pc should be renamed /usr/local/lib/pkgconfig/lua-5.3.pc ...
Doesn't that risk breaking any package that needs to call pkg-config for
lua-dev in order to build it? Possible examples:
14.x/x86/tcz/edje-dev.tcz.dep lua-dev.tcz
14.x/x86/tcz/gegl-dev.tcz.dep luajit-dev.tcz
14.x/x86/tcz/grilo-plugins-dev.tcz.dep lua-dev.tcz
14.x/x86/tcz/vlc3-dev.tcz.dep lua-dev.tcz
14.x/x86/tcz/wireshark-dev.tcz.dep lua-dev.tcz
14.x/x86_64/tcz/grilo-plugins-dev.tcz.dep lua-5.3-dev.tcz
14.x/x86_64/tcz/mpv-dev.tcz.dep luajit-dev.tcz
14.x/x86_64/tcz/vlc-dev.tcz.dep lua-5.3-dev.tcz
14.x/x86_64/tcz/wireshark-dev.tcz.dep lua-5.3-dev.tcz
-
Hi Rich. Yes it would. However, I think we should deprecate extensions called lua.tcz and lua-dev.tcz. Therefore, no packages should depend on lua.tcz or lua-dev.tcz.
The version number (for Lua x.y.z, version number is x.y and z is the bug fix release) should always be part of the extension name.
P.S. lua-x.y-dev.tcz would have a startup script that creates a link called lua.pc pointing to lua-x.y.pc. The lua-5.4-dev.tcz extension that I submitted today has a startup script that does just that.
-
Hi Rich. Sorry, I just realized that what I said above was an unsatisfactory, partial response to your very legitimate concerns. I will try to give a (hopefully) complete response below, in four bullet points. If you catch a missing detail, please let me know.
* lua-dev.tcz should be renamed lua-5.3-dev.tcz
* in lua-5.3-dev.tcz: lua.pc file should be renamed lua-5.3.pc
* in lua-5.3-dev.tcz: the extension should have a startup file that looks like this:
#!/bin/sh
LUAPC_LATEST=$(ls /usr/local/lib/pkgconfig/lua* | grep -Eo '/usr/local/lib/pkgconfig/lua-[0-9]+\.[0-9]+\.pc' | sort -V | tail -n1)
ln -sf $(readlink $LUAPC_LATEST) /usr/local/lib/pkgconfig/lua.pc* .dep files that contain lua-dev.tcz should be updated to contain lua-5.3-dev.tcz
These proposals are based on jazzbiker's ideas and I agree that they are the best (only?) way to support multiple versions of Lua on the machine. jazzbiker's smart idea is to include startup scripts in every lua-x.y.tcz, lua-x.y-lib.tcz, and lua-x.y-dev.tcz extension. The startup script's job is to ensure that those few files that don't normally have the Lua version in their name (namely: lua.pc, lua, luac, liblua.so) are just symlinks pointing the the most recent version of the relevant file that's available on the machine.
P.S. The Lua 5.4 submissions for the TCL14 x86_64 repo, which I sent today, incorporate jazzbiker's naming suggestions and include the startup scripts.
-
Hi GNUser
The following is not limited to lua but extensions in general.
What I've seen being done in many (but not all) instances is:
When Y changes in version X.Y of extension.tcz, it gets replaced with a new extension.tcz.
When Y changes, backward compatibility is supposed to be maintained.
When X changes in version X.Y and backward compatibility is needed:
The old extension.tcz becomes extensionX.tcz where X is the old version number.
Sometimes the lib.so->lib.so.X link gets removed from the old extension.tcz.
The old extension-dev.tcz gets removed so nothing gets compiled against the old version.
Extensions that depend on the old extension.tcz need their .dep files updated to extensionX.tcz.
The new version becomes extension.tcz.
My concern is what issues get introduced if version numbers get added to extension names.
Checking all of the build/run dependencies to see if their extension names
need to be changed in build scripts.
The same might hold true for configure scripts and make files.
I'm sure .dep files get carried over in many cases when extensions get
updated, but version numbers all over the place would complicate that.
I see Juanito and andyj maintain a lot of extensions. Maybe they could
provide some input.
-
Hi Rich and GNUser,
I want to remind one more scheme which was not yet mentioned.
lua-5.3.tcz includes lua-5.3, luac-5.3 and no tce.installed
lua-5.4.tcz includes lua-5.4, luac-5.4 and no tce.installed
lua.tcz has lua-5.4.tcz as the dependency and contains tce.installed linking lua -> lua-5.4 and luac -> luac-5.4.
-
Hi GNUser
The following is not limited to lua but extensions in general.
What I've seen being done in many (but not all) instances is:
When Y changes in version X.Y of extension.tcz, it gets replaced with a new extension.tcz.
When Y changes, backward compatibility is supposed to be maintained.
When X changes in version X.Y and backward compatibility is needed:
The old extension.tcz becomes extensionX.tcz where X is the old version number.
Hi Rich. Not in Lua, where x.y is the major version.
x.(y+1) in Lua is a very big, ABI-incompatible change. See https://www.lua.org/versions.html
I agree with everything you said, but in Lua you have to adjust your thinking by a decimal place.
-
Sometimes it helps to see how other projects have dealt with packaging conundrums.
In Devuan and Termux there is no binary named lua. In both projects, the lua interpreter is called lua5.3 or lua5.4 etc, depending on lua version(s) installed.
In Termux, interestingly, the package that provides Lua 5.3 is called lua53.deb (no decimal point in the package name). After installing the package, typing lua in the terminal results in "command not found" error. User must type lua5.3 to start the interactive interpreter.
In Devuan, /usr/bin/lua exists but is just a link:
Devuan$ which lua
/usr/bin/lua
Devuan$ ls -l /usr/bin/lua
lrwxrwxrwx 1 root root 33 Jul 17 2022 /usr/bin/lua -> /etc/alternatives/lua-interpreter
Devuan$ ls -l /etc/alternatives/lua-interpreter
lrwxrwxrwx 1 root root 15 Jul 17 2022 /etc/alternatives/lua-interpreter -> /usr/bin/lua5.4
-
Hi jazzbiker and friends. Two more questions to consider.
1) Regarding the startup scripts, wouldn't it be better for it to link the unversioned file to the version in that extension? Like this:
* lua-5.3.tcz forcibly links /usr/local/bin/lua to /tmp/tcloop/lua-5.3/usr/local/bin/lua5.3
* lua-5.4.tcz forcibly links /usr/local/bin/lua to /tmp/tcloop/lua-5.4/usr/local/bin/lua5.4
Same idea for lua.pc and liblua.so.
The advantage of this method is that the user could easily choose which lua version should be the default simply by the order in which extensions are loaded. Last one loaded wins.
2) Does everybody else also prefer the simpler names without hyphens (lua5.3.tcz, lua5.4.tcz, lua5.3.pc, etc)? If so, maybe the time to fix it is now while we are at it ;D
I don't mind recreating the extensions until we have a consensus and everybody is happy :)
-
Hi GNUser
... Not in Lua, where x.y is the major version. ...
Fair enough. :)
-
Hi GNUser,
Your extensions are in the x86_64 repo since yesterday morning, seems we are wasting time.
You can ask me, how I deal with Lua 5.4.
I built it in the separate directory, I use my own build scripts where I defined -I, -L and all necessary switches. I mean I don't use pkg-config for Lua. I link with static library. In the scripts I use shebang to point to the lua binary. My ~/.local/bin contains proper redirections to my lua and luac binaries.
All these preparations took quite small time. You know that Lua is built in a few moments.
I beg Your pardon for so much messing.
Happy hacking!
-
Hi jazzbiker.
Those extensions were my first attempt--vanilla build without external libs, without pkgconfig, without any thought to concurrent versions.
Yes, I feel like now I'm making a mess, creating noise and more work for everybody. I think I'm doing a disservice to Lua.
After I finish this post I will create and submit one last set of 5.4 extensions for the x86_64 repo, with today's timestamp (2023/08/19). The set will have the -lib extension, pkgconfig, and simple startup scripts that use the last-loaded-wins idea I mentioned in reply #26. I will keep hyphens in extension names.
If this final set is considered to add value to TCL, I would be happy to see it accepted. If not, no hurt feelings.
Rich, please mark thread as solved.
Happy hacking!
-
Final set submitted. Timestamp is 2023/08/19. GNUser out!
-
Hi GNUser
... Rich, please mark thread as solved. ...
Done. :)
-
1) Regarding the startup scripts, wouldn't it be better for it to link the unversioned file to the version in that extension? Like this:
* lua-5.3.tcz forcibly links /usr/local/bin/lua to /tmp/tcloop/lua-5.3/usr/local/bin/lua5.3
* lua-5.4.tcz forcibly links /usr/local/bin/lua to /tmp/tcloop/lua-5.4/usr/local/bin/lua5.4
Same idea for lua.pc and liblua.so.
I suggest we remove lua-5.3-dev and lua-5.3 so that only the latest version (5.4.x) will be built against
-
I suggest we remove lua-5.3-dev and lua-5.3
This is both convenient and dangerous, as many 5.3 libraries are incompatible with 5.4.
My only doubt is why they want to try to make it compatible to have 2 versions simultaneously loaded, a very inconvenient, problematic and unusual practice.
Having two separate versions without using tce.installed seems to be enough. Furthermore, it would be very annoying to have to type "lua5.4" instead of just "lua".
(I didn't follow the entire thread, so I may have missed some important information.)
-
I didn’t say that lua-5.3-lib would be removed.
-
In the 5.4 set of extensions I submitted for x86_64, I manually added version numbers to files that are normally unversioned (e.g., lua manually renamed lua5.4) and created startup scripts that create links (e.g., lua linked to lua5.4). If every version set were to do this, different versions could coexist in the repo and could even coexist loaded on the same machine.
The point of the thread was to pick your brains for ideas on how to make different Lua versions coexist, under the assumption that it is necessary or at least convenient for TCL to support more than one version at a time. If it is neither necessary nor convenient , I'm completely fine with scrapping the idea and just supporting vanilla build (with patch for shared libraries) of latest version. I 100% defer to Juanito on the question of necessity and convenience.
-
Maybe the simplest, most TCL way:
* keep complete vanilla set of current version (lua-5.4.tcz, lua-5.4-lib.tcz, and lua-5.4-dev.tcz)
* keep only *-lib.tcz of old version (lua-5.3-lib.tcz) because some repo applications need it
Juanito, if you decide to go with this approach, the lua-5.4.tcz already in x86_64 repo (timestamped 2023/08/17) should be fine. Please let me know if you want me to resubmit lua-5.4-lib.tcz and lua-5.4-dev.tcz built with the LFS patch, without any manually renamed files or startup scripts.
I guess the last detail would be to remove liblua.so from lua-5.3-lib.tcz, so that applications compiled in the future don't accidentally compile against the old version of the library.
-
Hi CardealRusso
... My only doubt is why they want to try to make it compatible to have 2 versions simultaneously loaded, a very inconvenient, problematic and unusual practice. ...
Sometimes when an extension gets updated it causes other extensions
to break. If a lot of extensions are affected, a renamed copy of the old
extension may be left in the repository to deal with the breakage. Both
copies can be installed and co-exist.
Some examples from TC10 x86 include:
ncurses5.tcz
ncurses.tcz
libffi5.tcz
libffi.tcz
libpng12.tcz
libpng.tcz
-
Hi Juanito.
I suggest we remove lua-5.3-dev and lua-5.3 so that only the latest version (5.4.x) will be built against
Sorry, I missed this. Okay, I understand that this is how you wish to proceed. In that case, I will send later today a vanilla set of 5.4 extensions for the TCL14 x86_64 repo (tagged 2023/08/20).
Do you agree that liblua.so should be removed from lua-5.3-lib.tcz? I learned recently that the extension manager does not overwrite existing files. Therefore, if lua-5.3-lib.tcz is loaded (e.g., because an application needs it) and user later loads lua-5.4-dev.tcz and lua-5.4-lib.tcz to build something, the old liblua.so will not be overwritten and will still be pointing to liblua.so.5.3.6
-
GNUser
Forgive my suggestions and comments below.
There may be an opportunity for you to submit a TCE with an install script that deletes ram drive files so that no matter which package is loaded first,
the "new" TCE populates the ram drive.
Let me give some examples from an idiot I know too well. In my early days, I would click in Apps GUI and leave the download option as "OnBoot"
and might end up with a boot list that is actioned from top to bottom
TCE-Z.tcz
TCE-A.tcz
such that the the shared object/executable/some config from Z populates the ram drive and A never gets a chance to grab those spots.
And....I can not stop a member from having accidently added TCE-Z to their boot list
but I can create an install script (for TCE-A) that deletes any current ram drive files and re-populates the ram drive with files from A.
so for other members who can "half" follow me, we have 2 examples
someone loades TCE-A then TCE-Z or TCE-Z then TCE-A
If TCE-Z is the unloved TCE and populates the system files, TCE-A does not grab those slots unless you have an install script for system file deletion
If TCE-A is the loved TCE and grabs the system files, as long as TCE-Z has no install script ..(that deletes any system files)..we are good to go.
I hope that makes sense?