Simple approach to version upgrade failing (sce-update -a)

[sm8ps]

Hi friends,

I am trying to upgrade from dCore-trusty to Xenial, preferably without having to re-import all extensions. My plan is to upgrade kernel and initrd and have ''sce-update'' take care of all the rest. I have set up a testing environment with a "naked" kernel and initrd (2016.10.19.20.58) together with persistent home, opt and tce directory. Then I copied some extensions to 'tce/sce' from Trusty (among others X11 meta-package, LXDE meta-package depending on X11 and Midori depending on LXDE) with all '.sce*'-files.

Now I would expect that ''sce-update -a'' recognizes newer versions and downloads them. However, it quickly states that the entire SCE directory is up to date. This even after having changed the content of 'X.sce.lst' to contain "graphics-4.2.9-tinycore" (instead of 3.6.16). I did also try ''sce-update -za'' just to be sure.

Yet if I run ''sce-update'' for individual extensions it does suggest to update them and successfully does so, even if more than one is specified. I could successfully boot to a desktop after having updated the necessary extensions. This behavior seems to point to a flaw in ''sce-update -a'' itself, independent of my peculiar set-up, doesn't it? Though, I wonder why nobody else has mentioned it yet. Can others verify? Or am I overlooking anything obvious?

Cheers!
sm

[nitram]

Hi sm8ps.

Glad to see you're still active with dCore!

To me best practice would be to start with a fresh TCE directory when setting up a different dCore flavor, which i've documented deep in the wiki:

Ensure the kernel and initrd files utilized are both from the same dCore port. Users upgrading to a newer dCore port will need to ensure the boot loader configuration file is appropriately updated. Updating to a new dCore port is essentially a new install, a fresh /tce directory should be utilized and any desired SCEs will need to be re-imported.

http://wiki.tinycorelinux.net/dcore:upgrading_kernel_and_initrd#dcore_kernel_updates

Pretty much all SCEs would require updates anyways, unless they are very small, with minimal dependencies from the dCore prebuilt repository. And if the user knows updates will be required then a fresh import takes less CPU cycles than completing an update check and then re-importing anyway.

However it seems you may have caught a bug when updating SCEs from another dCore flavour. If you all come up with a slick way to update to the next Debian/Ubuntu release i will gladly update the wiki. Thanks.

[sm8ps]

Yes, sure I stay active with dCore! Would not want to let go of this nifty little thing. Thanks for the continuous wiki updates, nitram! I hope to contribute something by myself, for example a simple upgrade strategy.

The point of upgrading over an old tce-directory is that I am too lazy to extract the information about which extensions are installed with what structure (lists, dependencies) in order to transfer it to a new installation and re-install everything. Sure I could do that but I have not yet understood why a more simplistic approach should not work. The test-installation does seem to work in principle but ''sce-update'' does not behave as expected.

I have not considered the meta-information possibly contained in SCE-files but AFAIC see they simply should be archives of Debian packages without any reference to the OS-version, or shouldn't they? Certainly, almost all packages of any full installation would probably need updating but the wait would not beat the comfort of having it all done automatically.

If this simplistic approach should definitely fail, I could of course write a script that would extract the necessary information from  the content of 'tce/sce/' but I think it should not be that complicated.

[sm8ps]

So here I am again, pursuing the same approach. I must confess that I had totally forgot about this thread and was feeling very bright when I devised the idea of doing a dist-upgrade in this fashion. I am feeling a bit old and quite forgetful now.

Nevertheless, the initial question persists. Namely, why does ''sce-update -a'' not show any updates available while ''sce-update <SCE>'' does update the extensions successfully? It would be simply great if the dist-upgrade process could be handled in a simpler fashion than restarting from scratch!

Here are the preliminary steps taken on the dCore-trusty system:
. Load Xenial kernel and initrd into /boot.
. Move content of '/opt/debextra/' to '/opt/debextra.trusty/'. (Will deal with the respective extensions later.)
. Boot with boot-code "base". (Do not load any extension.)
Effects:
. ''sce-update -a'' says no updates available after checking the whole list of extensions.*
. ''sce-update bash'' finds 4 new, 2 updated and 3 removed startup scripts and a long list of packages to be updated from the Xenial repos (81 files in total).

(*) One error is shown for sce-deb2sce.sce: "update check. isofs_fill_super: bread failed, dev=loop1, iso_blknum=16, block=32".

[Jason W]

I see what can cause the NEWDEBINX to be moved to OLDDEBINX which will cause "sce-update -a" to immediately show no updates available.  It is if you Ctrl-C during sce-update when the -a option is used, a loop is broken and the moving of NEWDEBINX to OLDDEBINX happens when it shouldn't.  I have fixed the bug and am testing it.  Also, I am having sce-import echo it's errors to a file /tmp/sceimporterror so one can more readily examine what went wrong during an import.  And sce-update will create a file /tmp/sceupdateerror that will contain a per package output of  sceimporterror so much easier to diagnose issues.   If sce-update has these errors, the user is prompted to review /tmp/sceupdateerror.

While I don't really promote the dist-upgrade that is being described here, I am doing that now with dCore-wheezy and using an existing dCore-jessie TCE directory.  And so far so good, things work, the main issue is packages that exist in SCEs in the old TCE directory that don't in the new, and how to deal with them.   But missing packages will just be missing, and all packages available will have it's deps filled as always.

This thread has spurred me to fix some bugs in sce-update and create better error reporting and review in sce-import, which is also made viewable by an sce-update session. 

[sm8ps]

I see your concern about obsolete extensions, Jason. At the moment, I consider my approach mostly a proof of concept. In the long run I hope to find a way of streamlining the process.

One word about NEWDEBINX being moved to OLDDEBINX: At the first attempt, ''sce-update'' did seem to walk through the full list of packages and only after some considerable time stated that there was nothing to do. At the second attempt however, it responded so immediately. So it seems that NEWDEBINX was not immediately moved to OLDDEBINX. I shall study the output of ''sh -x /usr/bin/sce-update -a'' and report back.

[sm8ps]

I have run ''sh -x /usr/bin/sce-update -a'' in a ''script''-session (cf. attachement 2). The set-up was a fresh dCore-xenial installation with its 'tce/' replaced by that of a dCore-trusty installation with only a few extenstions (cf. attachment 1). The state of the tce-directory after ''sce-update -a'' is captured in attachment 3.

''sce-update'' does seem to correctly recognize that there new versions are available as MDOLD and MDNEW differ. However, in the end it states "No updates available for main or any dependency SCEs." This seems kind of weird but I have not looked into the code. Is the behavior expected?

[Jason W]

MDOLD and MDNEW are the md5sums of OLDDEBINX and NEWDEBINX.  A difference in them does not mean there are updates to any SCEs. 

If when the new is moved to old after sce-update -a, a run of sce-update -a will exit quickly as there will not be a difference in the old and the new, meaning no changes have been made to Debian Packages files, md5sum or dependency differences, etc.  That is the reason for the old and new debinx so every extension does not have to be checked.  And each SCE has it's own .debinx so the same goes at the individual extension level also. So what you are seeing is expected. 

The bug I am testing the fix for involves a Ctrl-C during 'sce-update -a', as well as better error logging.  I hope to get it posted today.

[sm8ps]

Thanks for your explanations!