tce-load improvements

[cosmin_ap]

Hi,

I'd like to suggest a few improvements to tce-load:

1) ability to download a tcz extension from an arbitrary url (because I want to host extensions myself but don't want to make a mirror)

2) ability to download a tcz even if already downloaded (because it was updated on the server)

3) "require" flag, i.e. ability to download + install quietly even if extension was already downloaded and/ore installed; return exit code != 0 only if after all the extension is not installed (because I want to have tinycore automatically satisfy dependencies in my scripts)


Example combining all 3 features:

tce-load -W http://mysite.com/tinycore/myext.tcz   #force download
tce-load -W -i http://mysite.com/tinycore/myext.tcz   #force download, but otherwise just like -w -i
tce-load -r http://mysite.com/tinycore/myext.tcz   #like -i, but don't complain if the extension was already installed; exit with an error code only if the extension isn't installed after all
tce-load -w -r http://mysite.com/tinycore/myext.tcz   #like -w -i, but don't complain if the extension was already downloaded and/or installed; exit with an error code only if the extension isn't installed after all
tce-load -W -r http://mysite.com/tinycore/myext.tcz   #like -w -r, but force download -- however, don't force download if the extension was already installed

The last two combinations I find the most useful. Currently I have to manually delete the tcz and run tce-load -w -i &>/dev/null which hides errors and doesn't tell me if after all, the extension was installed (I have to check for that myself).


If you find these modifications useful enough to be included in tce-load, I could find some time to implement them myself.

[gerald_clark]

1. You can already choose mirrors from the system provided list plus your own list.

2. Run tce-update.

3. What is the point?  If it is already downloaded, there is no need to download again. Dependencies are already satisfied.

[cosmin_ap]

No offense, but what's the point of this reply?

1. You can already choose mirrors from the system provided list plus your own list.
  -- already said I don't want to do that. I want to make and host an extension an provide a link on my website to download it and install it, something like tce-load -wi http://my-website.com/myextension.tcz

2. Run tce-update.
  --tce-update can't re-download a single extension and it's extremely slow anyway to the point I never use it.

3. What is the point?  If it is already downloaded, there is no need to download again. Dependencies are already satisfied.
  --for example, I have a script which calls programX, which is a program included in extensionX.tcz. What I want to do in my script is this: tce-load -r extensionX || { echo "extensionX required but could not be installed"; exit 1; }

[gerald_clark]

The point of the reply is that none of your suggestions are necessary.

[danielibarnes]

1) Not unreasonable or difficult to implement.

2-3) The tce-load and tce-update tools are separate for a reason: they require separate environments. The tce-load tool downloads and installs extensions within a normal operating environment. The tce-update tool requires booting in "base norestore" mode for safe operation.

Your requirement, as I understand it, is to be able to update an extension and its dependencies without having to reboot into "base norestore" mode. Am I correct? If so, then what would it take to add an "update" option to tce-load?
1. Ensure specified extension and all its dependencies are not mounted or fail immediately.
2. Compare md5 sum of each extension with repository.
3. Download all updated extensions into a temporary directory.
4. After all extensions are downloaded successfully, copy them into the TCE directory in reverse dependency order.
Note: must handle download and copy failures gracefully.

This isn't a simple task, but if you have the time to implement your ideas then attach a patch to this topic so others can evaluate it. That's what the forum is all about.

[cosmin_ap]

@gerald_clark: necessary to whom? you?

@danielibarnes: my requirement for 2) is that I just want to download an extension that was updated on the server and reboot. that simple. I already stated why I don't want to use tce-update.

as requirement for 3) is this:

I have a script which calls programX, which is a program that needs (or is included in) extensionX.tcz. What I want to do as early as possible in my script is this:

tce-load -r extensionX || { echo "extensionX required to run programX but could not be installed"; exit 1; }

[danielibarnes]

So your requirements are:

• Allow specifying a URL for retrieval of an extension.
• Ability to update an extension.
• Return an exit code from tce-load in the event of failure.
• Supress all output in "quiet mode."

I am focusing on the "what" before the "how" this time.

[cosmin_ap]

Allow specifying a URL for retrieval of an extension.
--yes, so I can host an extension on my website and provide a simple tce-load -i URL command for users to download & install it.

Ability to update an extension.
-- yes, just to re-download it -- I can reboot just fine after that

Return an exit code from tce-load in the event of failure.
-- yes, but as opposed to -wi, -wr won't consider it a failure if the extension is already downloaded or installed; -r would consider it a failure if the extension is not in the local tce dir (just as -i does) but again, won't see it as a failure if it's already installed; dunno how much clearer I can say it, sorry.

Supress all output in "quiet mode."
-- that's not that important, but yes, if would be nice, in absence of a specific -q flag.



The semantics of -wr and especially -Wr is most useful to me. What these flags say is this: "I need this extension to continue, if it's not installed, try to download it and install it. In any case, tell me if it's installed or not." The difference between -Wr and -wr is that -Wr also downloads a possibly newer version if it has a chance to use it (i.e. the extension is not already installed).

[danielibarnes]

I personally have use cases for the things you mention, so I find the problem worth investigating.

Allow specifying a URL for retrieval of an extension.
--yes, so I can host an extension on my website ... for users to download & install it.

I wouldn't focus on the how too much. You don't want to restrict your solutions. For example, a small modification to /etc/init.d/tc-functions to not re-assign the MIRROR variable if it is already set would allow you to execute:
MIRROR=http://mywebsite.com/ tce-load -wi myext.tcz
and achieve the same effect (file would need to be at http://mywebsite.com/3.x/tcz/myext.tcz for example). This would satisfy your first requirement.

Ability to update an extension.

I looked at the tce-update script, and you can update individual extensions with:
tce-update update $(realpath myext.tcz)
The newly downloaded extension will be loaded after rebooting. This should satisfy your second requirement.

Return an exit code from tce-load in the event of failure.
-- yes, but as opposed to -wi, -wr won't consider it a failure if the extension is already downloaded or installed; -r would consider it a failure if the extension is not in the local tce dir (just as -i does) but again, won't see it as a failure if it's already installed; dunno how much clearer I can say it, sorry.

That's perfectly clear. I guess the best way to say it might be "return success if the extension is installed (-i) or downloaded (-w) upon exit, otherwise failure." As you noted, your request changes the semantics of tce-load. I like your proposal as it makes a stronger statement about the post-conditions vs. the actions of the script, but I'm not sure a new switch is the best way to achieve that.

[gerald_clark]

#1 can currently be accomplished by temporarily changing the contents of /opt/tcemirror.

[danielibarnes]

#1 can currently be accomplished by temporarily changing the contents of /opt/tcemirror.

It wasn't explicitly stated, but I assumed modifying /opt/tcemirror was not an option. Failure to restore the previous contents (due to SIGINT, etc.) would break the system.

[Guy]

http://wiki.tinycorelinux.net/wiki:mirror

[cosmin_ap]

Hm, guess I should have made one point clear from the start. Mine wasn't a "support question" or "how to do x or z in tinycore". Maybe I should have posted it under a different category but couldn't find a "feature requests" one. If I misplaced the message, I apologize. In any case I seem to have wasted your time because of this misunderstanding.

The fact is I know all these hacks, and none of them is a proper solution for my requirements, which I dare say, are not so specific to my case, hence the idea to post to the forums.

@danielibarnes
1) it's not the point that I can modify tc-functions to do what I want, already did that, the point of my post was to make a feature proposal. So anyone please stop with the mirrors thing.
2) tce-update update $(realpath myext.tcz) DOESN'T actually work, it updates all extensions (the parameter is the tce location not a file). This usage is undocumented anyway and it's operation is SLOW; so for the 3rd time, that's not what I was after.
3) "'m not sure a new switch is the best way to achieve that" -- what then?


The point of -r is to say "require" just like php's or Lua's require() function, which loads a module if not already loaded or raises an exception. It allows your script to automatically load extensions it depends on to run, and fail if they could not otherwise be satisfied. I'm running out of ways to rephrase this


The point of loading from a URL is so you can easily host extensions yourself and allow people to download and install them with a simple command. All package managers under the sun allow this, why all the fuss about it?

[danielibarnes]

Hm, guess I should have made one point clear from the start. Mine wasn't a "support question" or "how to do x or z in tinycore". Maybe I should have posted it under a different category but couldn't find a "feature requests" one. If I misplaced the message, I apologize. In any case I seem to have wasted your time because of this misunderstanding.

No, no time wasted. Like I said, I was interested in discussing it as well. I'd like to host extensions myself, but tce-load doesn't really support that. For example, I want to create a "meta-extension" which has no binaries, just a .dep file with dependencies. I only want to host the meta-extension, though, not the dependencies, and tce-load doesn't have a way to handle that. Trying to hack the mirrors feature won't work in my case.

2) tce-update update $(realpath myext.tcz) DOESN'T actually work

It does. I tested it before posting to be certain. As an example, boot with "base norestore" and execute:

tce-load -w ncurses.tcz
cd /tmp/tce/optional
sed s/0/1/ -i ncurses.tcz.md5.txt
sed s/0/1/ -i ncurses-common.tcz.md5.txt
tce-update update /tmp/tce/optional/ncurses-common.tcz

Note that only the ncurses-common extension is downloaded into the upgrade directory. You should be able to reproduce this.

I'm running out of ways to rephrase this

Yes, we've beat it to death. I'll admit, I've been working on requirements and system design at work for the last few weeks and I am stuck in that mindset.

The point of loading from a URL is so you can easily host extensions yourself and allow people to download and install them with a simple command.

It isn't quite so simple if you have dependencies. Right now, dependencies must exist in the same location as the extension being loaded. This is what breaks my "meta-extension" use case. Another use case is testing. When I create extensions, I put them in /tmp and load them with tce-load -i /tmp/myext.tcz. I get errors because the dependencies are not in /tmp also -- they are in my TCE directory, but tce-load doesn't even check that. It just fails. Maybe Robert can fix tce-load to at least failover to $(cat /opt/.tce_dir) on failure.

[gerald_clark]

How about multiple lines in /opt/tcemirrors that are searched in order to resolve missing deps.