RFC: unified extension naming convention

[maro]

I'd like to raise the question whether it would be desirable to unify the naming of our extensions by making them all .tcz (and eliminating .tczl, .tczlm, .tczml and .tczm).

Assuming that for a given extension ext.tcz there shall never exists an ext.tczl (or ext.tczm, or ext.tczlm, or ext.tczml), it would make the entries of the .dep files a bit more robust. I'd imagine that whenever an extension changes it's "flavor" (e.g. tcz -> tczl) this requires an update to all impacted .dep files. Not a big deal, but just another thing one could do without. A unified naming convention would also allow for those of us that use tce-load directly, not to having to remember the "flavor" of an extension.

I believe to know the prime motivation for having all those "flavors" of extensions: to guide tce-load in executing 'depmod' and 'ldconfig' respectively. Clearly, deriving the guiding information directly from the extension file name is hard to beat in terms of execution time.

I can see two alternatives to the current use of the file name for this determination. I concede they would potentially add a little time for loading extensions via appbrowser (or tce-load), but importantly not during boot time:

• fully automatic, by using 'find' to identify shared libraries and/or kernel modules within '/tmp/tcloop/ext' (or '/mnt/test' for ram installations),
• manually, by introducing additional (empty) files into an extension (e.g. 'usr/local/tce.installed/tczl/ext' and/or 'usr/local/tce.installed/tczm/ext') as flags to trigger 'depmod' or 'ldconfig' execution.

I imagine that the first alternative adds ca. 20-30 msec for each extension that tce-load handles. Whilst the second alternative should be almost as fast as the current solution, but for the "price" of requiring manual intervention during the extension packaging. And it is this "human factor" that could be eliminated all together with the first alternative. 

Many of you might see this questions as completely cosmetic and a total non-issue. I could see that some fear that it introduces too much risk for little gain. A proposal of such potential impact might have to wait for the TC 3.x cycle until it could be considered (if at all).

I've played around a bit with a "flavor"-tolerant version of tce-load, which might allow even it's introduction during the current 2.x cycle. But to go much further down this path I wanted to see first whether there are others that can see any benefit in this proposal in the first place.

[Jason W]

As you have seen, using extension.tczl in place of extension.tcz, or vice versa, is a common mistake in dep files.  I am very much a "keep it simple" person, and it would simplify extension names.  But it is a big change indeed that would require amending base functions.  If this was the approach to use, my thought would be to use a file in the extension, like /tmp/tctype/lib or /tmp/tctype/mod and "ls" for the file and then, say, export LIB="yes" and execute ldconfig when appropriate, then unset LIB.  Would be better to use a file that gets overwritten each extension load so files are not being scattered with each extension.

But, the way we do it now makes it easy to change the need for calling depmod/ldconfig by simply adding to the extension suffix.  And it is definitely familiar. 

Though we could agree to standardize on tczlm to save having to remember between tczlm and tczml.

I admit that having one single extension suffix is tempting.

[robc]

could the extension startup script be used to call ldconfig and/or depmod?
this would eliminate additional code to be put in the extension loading code.

[roberts]

All modules and libraries are processed before the startup scripts.
To move them into startup scripts would mean the order of processing startup scripts becomes paramount.

[robc]

All modules and libraries are processed before the startup scripts.
To move them into startup scripts would mean the order of processing startup scripts becomes paramount.

Couldn't this be left to the extension maker to make sure that ldconfig and/or depmod are called in the beginning of the startup script? Though this would also make almost all extensions require startup scripts.

Perhaps a flag would be the best approach. But wouldn't this break the extension loading for prior TC 2.x versions?

One way to simplfy this would be to always ldconfig and depmod after extension loading...

[althalus]

All modules and libraries are processed before the startup scripts.
To move them into startup scripts would mean the order of processing startup scripts becomes paramount.

Couldn't this be left to the extension maker to make sure that ldconfig and/or depmod are called in the beginning of the startup script? Though this would also make almost all extensions require startup scripts.

I think Rob was referring to the order each startup script runs in, not the order of commands in the script?

Perhaps a flag would be the best approach. But wouldn't this break the extension loading for prior TC 2.x versions?

I think any efforts down this path will result in a 2.x repo and a seperate 3.x repo.

One way to simplfy this would be to always ldconfig and depmod after extension loading...

That could result in a lot of unnecessary calls, particularly to depmod, given that most users are likely to have only a handful of tczm's if any.

Seems to me that a flag would be the best way to go if this were to be done.

But it is a big change indeed that would require amending base functions.  If this was the approach to use, my thought would be to use a file in the extension, like /tmp/tctype/lib or /tmp/tctype/mod and "ls" for the file and then, say, export LIB="yes" and execute ldconfig when appropriate, then unset LIB.  Would be better to use a file that gets overwritten each extension load so files are not being scattered with each extension.

I haven't looked at the code, so forgive my ignorance, but it would just be one small change in the module load script, wouldn't it? It would be a massive change from the users point of view(particularly with the repos) and definately a 3.0 sort of feature, but should be relatively trivial from a development stand point. It shouldn't even need setting and then subsequently unsetting something like export LIB="yes"

Code: [Select]

if [ -f /tmp/tce.type/lib ]; then
ldconfig
rm /tmp/tce.type/lib
fi

if [ -f /tmp/tce.type/mod ]; then
depmod
rm /tmp/tce.type/mod
fi


[roberts]

The code part is easy, the repo and backward compatible is not, unless we wait till 3.0 when we start
anew.

[Jason W]

As the benefit of one extension suffix would be to prevent breakage of dep files on extension name changes, now that the naming conventions are in place and the major changes to conform to it have been done, let's try not to be renaming extensions during upgrades unless there is a really good reason for it.

Any name change breaks existing dep files, so let's aim for the stabilizing of extension names.  By not including version numbers in the names of any extensions and following current naming conventions we should see decent stability.. 

[robc]

Instead of labeling a file as tczl or tczm or others, or even using a flag to show a ldconfig or depmod is needed, why not have tce-load check for *.so in /usr/local/lib or if there is a /usr/local/lib/modules directory within the extension?

For the mount type install the extensions are completely contained in /tmp/tcloop and for RAM installs the extension still needs to be mounted to a temporary location. Therefore these 2 checks can be performed rather quickly along with the extension loading process.

[gerald_clark]

Is the check for actual libraries or modules necessary?
The extension should not have any library directories or module directories if it has no libraries or modules.
Perhaps checking for the existence of lib, usr/lib, or usr/local/lib should be sufficient.
Similarly, the existence of lib/modules or usr/local/lib/modules should be sufficient.

[bmarkus]

It is more simple. Regarding .tczl a single unconditional ldconfig is enough after all extensions loaded and before startup scripts executed.

[robc]

Is the check for actual libraries or modules necessary?
The extension should not have any library directories or module directories if it has no libraries or modules.
Perhaps checking for the existence of lib, usr/lib, or usr/local/lib should be sufficient.
Similarly, the existence of lib/modules or usr/local/lib/modules should be sufficient.

For modules only a check for the /usr/local/lib/modules or /lib/modules would be needed, but for libraries I think the contents of /usr/local/lib:/usr/lib:/lib would need to be checked since other things (ie python modules) are loaded in /usr/local/lib

[danielibarnes]

It is more simple. Regarding .tczl a single unconditional ldconfig is enough after all extensions loaded and before startup scripts executed.

Throw in an unconditional depmod as well. I agree with this even though it may add to the boot time when no lib or module extensions are loaded.

I also agree that since it is not backwards compatible, 3.0 is only time to do this. Now is a good time to discuss the details.

[bmarkus]

I also agree that since it is not backwards compatible, 3.0 is only time to do this. Now is a good time to discuss the details.

Why? You can keep .tczl .tczm names but change initialization in next 2.x release for testing and in 3.0 change over to .tcz only

[danielibarnes]

That's what I mean. Wait until 3.0 to change the names. Changing ldconfig/depmod behavior should be fine anytime.