boot time extension processing

[Kingdomcome]

Without looking through the new code, it appears to me that in 2.4, an extension's post-install script is run immediately after the extension is loaded.  This would be a change from previous versions and possibly cause some post-install scripts to not run because the extensions dependencies are not yet loaded.  I believe some work has been done to process available dep files during loading, but the dep files of dependencies are not downloaded automatically causing some problems.  An example of this would be any extension that depends on dbus. dbus depends on expat2, but without its dep file, dbus gets loaded and post-install script runs but errors out before expat2 is loaded.

[roberts]

The actual requested extension is loaded after processing all of its dependencies.
Therefore is startup script is run after its dependencies are met.

It is a well know fact the nested or recursive dependencies are not supported.

Therefore the original rule, that being, if the extension's .dep file lists all of its required dependencies in order to function correctly by booting base norestore and then such extension will work in a normal runtime environment.

[bigpcman]

The actual requested extension is loaded after processing all of its dependencies.
Therefore is startup script is run after its dependencies are met.

It is a well know fact the nested or recursive dependencies are not supported.

Therefore the original rule, that being, if the extension's .dep file lists all of its required dependencies in order to function correctly by booting base norestore and then such extension will work in a normal runtime environment.

So just to make sure I understand this, if a dependency extension requires a startup script it should not be a dependency?

[Kingdomcome]

I try to do all my testing use base norestore, in this case I did boot base norestore wiped the hard drive and set up a fresh tce dir.  If the dep file has the dependencies of dependencies listed before the direct dependencies, then yes, the initial loading will complete sucessfully, but after a reboot extension loading order does not follow the dep order therefor deps of deps do not necessary get loaded first.  One possible fix would be to download the dep files for dependencies.   This has been tested and proved using hal.tczl "Installed" to filesystem.

[roberts]

The actual requested extension is loaded after processing all of its dependencies.
Therefore is startup script is run after its dependencies are met.

It is a well know fact the nested or recursive dependencies are not supported.

Therefore the original rule, that being, if the extension's .dep file lists all of its required dependencies in order to function correctly by booting base norestore and then such extension will work in a normal runtime environment.

An extension's startup script is not listed in the extenions .dep file.
It is embedded in the .tcz at /usr/local/tce.installed/extname   (sans the .tcz)

So just to make sure I understand this, if a dependency extension requires a startup script it should not be a dependency?

[roberts]

I try to do all my testing use base norestore, in this case I did boot base norestore wiped the hard drive and set up a fresh tce dir.  If the dep file has the dependencies of dependencies listed before the direct dependencies, then yes, the initial loading will complete sucessfully, but after a reboot extension loading order does not follow the dep order therefor deps of deps do not necessary get loaded first.  One possible fix would be to download the dep files for dependencies.   This has been tested and proved using hal.tczl "Installed" to filesystem.

Extension loading of its dependencies does follow the direct order of its .dep file. You seem to indicate that you want nested dependencies. That while loading an item from a .dep file, then also check if this file also has a dep file of its own, so push current processing onto stack, then start down the new dep list which it could possbly have a dep item that has its own dependency, so push onto stack and repeat. And hope for no "deadly embrace".

[combo3]

The actual requested extension is loaded after processing all of its dependencies.
Therefore is startup script is run after its dependencies are met.

It is a well know fact the nested or recursive dependencies are not supported.

Therefore the original rule, that being, if the extension's .dep file lists all of its required dependencies in order to function correctly by booting base norestore and then such extension will work in a normal runtime environment.

So in other words...

If:
C requires B, and
B requires A, then
C.dep should list both A and B as dependencies?

Is that correct?

[roberts]

Yes and the order is important and should be tested by boot base norestore and loading the application (C in this example) and not preloading non-applications such as dbus, hal, and other libraries and support extensions.

[combo3]

Thanks for the clarification.

[Kingdomcome]

Roberts, I understand the possible consequences of having TC try to load deps recursively and agree that it should not be done.
What I am trying to explain is that during boot of 2.4 post-install scripts run immediately after each extension is loaded, whereas in earlier versions all post-install scripts ran at the end of the loading process. This is causing problems because during boot, extensions are being loaded alphabetically.  In the above example C.dep includes both A and B, but if alphabetically B comes before A, and the B post-install script starts a daemon which needs A loaded first, the daemon will not get started on subsequent reboots, even if everything goes right during initial download and installation.
Conclusion: Because each extension's post-install script is run directly after loading at boot time, a simple test of booting base norestore and loading the extension is no longer a good enough test. A fresh PPR setup of each extension is required to fully test boot time loading.

[perthie]

Is there a difference between loading apps using tcz2ram.lst versus tcz2ram.flg? If I use the list, are apps loaded in list order? If I use the flag, are apps loaded alphabetically? If there is a dep file, is it handled differently depending on the method?

[roberts]

Roberts, I understand the possible consequences of having TC try to load deps recursively and agree that it should not be done.
What I am trying to explain is that during boot of 2.4 post-install scripts run immediately after each extension is loaded, whereas in earlier versions all post-install scripts ran at the end of the loading process. This is causing problems because during boot, extensions are being loaded alphabetically.  In the above example C.dep includes both A and B, but if alphabetically B comes before A, and the B post-install script starts a daemon which needs A loaded first, the daemon will not get started on subsequent reboots, even if everything goes right during initial download and installation.
Conclusion: Because each extension's post-install script is run directly after loading at boot time, a simple test of booting base norestore and loading the extension is no longer a good enough test. A fresh PPR setup of each extension is required to fully test boot time loading.

So you are saying that if, while booting, all startup scripts are delayed until the entire extension directory load process is completed, and then, even though the startup scripts are then processed in directory list order, there will not be an issue?

[Jason W]

My personal preference is to have the startup script do things such as copy files, add users, create directories, and so forth.  But have the /usr/local/etc/init.d scripts start and stop the daemons.  It seems folks are using the /usr/local/tce.installed startup script to start daemons, and that is much of Iwhat is creating the need for having to do full processing, deps first, on each extension during load on boot.  

Take nfs-utils for instance, it's /usr/local/etc/init.d/ script uses start-stop-daemon to start portmap if not started already and then start rpc.statd.  Portmap is a seperate extension, but the start function of nfs-utils will start portmap, but the stop function in nfs-utils will not stop portmap as it could be used by other processes.  I don't have to depend on portmap being started by the portmap extension for nfs-utils to work by invoking it's init.d script.

In other words, the previous way could work of loading all extensions on boot in /tce alphabetically, then depmod/ldconig once, after that alphabetically running the /usr/local/tce.installed startup scripts to manipulate files/add users/etc, and add to that the exectuting the /usr/local/etc/init.d scripts.  If the init.d scripts were made like the nfs-utils example, it would not too much matter the order they were executed.  Other distros start all needed dependency daemons in their init.d scripts, and it keeps it much more simple.  In Arch Linux, their hal init.d/rc.d script starts dbus and then hal making the hal startup script freestanding.  If Xfce needed hal, dbus, portmap, and whatever in it's init.d script, then those daemons could be started in xfce's init.d script.   If an init.d script needed to be started automatically with the loading of the extension, then it could of course made with executable perms.  Otherwise, say since portmap or dbus is not needed on it's own, it's init.d script could be non-executable.  But xfce would need it, so xfce's init.d script would be executable.

EDIT: They would all have to be executable, but some other means could determine which ones are to be automatically run with the loading of the extension.

In other words, put the responsibility of starting and stopping needed daemons in the extensions' init.d script rather than tax the boot/tce-setup process.

This is just my opinion.

[Kingdomcome]

Maybe its the way extension builders are trying to use the post-install scripts, maybe its the change in boot time processing of extension loading, but this problem has just started appearing, I am merely trying to bring attention to it and spark discussion.  Personally, Im liking the idea that Jason has suggested as a way to keep the extension loading process a little cleaner.  One question I have is how would the init.d scripts for only the newly installed extensions be processed during run time loading from appbrowser if this option was explored?

[Jason W]

One way to autostart daemons on boot is to automatically process those located in /usr/local/etc/init.d/boot, with the entries in that directory be symlinks to the main scripts in /usr/local/etc/init.d.  The tce.installed startup script could create the symlink if it is an extension that  needs it's daemon started automatically on boot or tce-load.

On boot/tce-setup, the entries that exist in /usr/local/etc/init.d/boot could be automatically started.  On tce-load a check for the existence of /usr/local/etc/init.d/boot/extensionname could execute it if it exists.

Of course this would limit what could be put in the tce.installed scripts, but at the benefit of effeciency.  Again, just throwing ideas in the air.