Building landing wheels while in flight

[mcdudeabides]

I've been studying the architecture http://distro.ibiblio.org/tinycorelinux/architecture.html diagram with an eye to understanding the possibilities of performing an in-situ update.  For my simple mind, it seems that the kernel could be updated by overwriting /boot/vmlinuz and the root file system by overwriting /boot/core.gz.  Depending on the application, all TCZ extensions might take a little more brute force.  There are some things in mydata.tgz that should probably be retained (hostname, ssid, wifi setup, etc.), but other that need to be overwritten. 

So before embarking on the fool's errand, I thought I might just ask why there is not a mechanism that would allow for a currently running instance of piCore to grab a new version and update itself?  Is there a GRUB (or something similar for the Pi oddities) that could bring the next version into a new boot area and reboot using the new download, yet fall back to the known good if it fails? 

I saw this discussion http://forum.tinycorelinux.net/index.php?topic=16464.0 and thought if it can be done for other targets. then it might just work for piCore itself.

Sorry for the rambling, but I'm between other obligations and only had a brief window to throw this out there.  If there are any online education resources or advice that can be offered, it will be much appreciated.

[sbp]

Hi Randy

Thanks for your help with the piCorePlayer scripts. As you know many users of the piCorePlayer have been asking whether it could be possible to update the piCorePlayer without physically removing the SD-card for burning a new image on the card. The reason is that several piCorePlayer users are hiding the raspberry inside a closure and sometimes behind walls and ceilings, so it might be difficult to get to them.

Therefore I have also been studying this and my conclusion was that it might be possible.
I was think of something like this:
To update the kernel and modules simply download and overwrite the piCore.img and root.gz files (like you suggested).
For update of  the piCorePlayer - download and update the complete TCE directory.
(I know that you download a lot of files that are unchanged, but still it is rather small, and for me it seems the simplest way.

The piCorePlayer is downloaded from Sourceforge, so I was thinking of making a new directory there with these separate files to download.

Off course there need to be some kind of fail-safe so that you are not deleting any files before you know that the new ones are ready.

Steen

[bmarkus]

So before embarking on the fool's errand, I thought I might just ask why there is not a mechanism that would allow for a currently running instance of piCore to grab a new version and update itself?  Is there a GRUB (or something similar for the Pi oddities) that could bring the next version into a new boot area and reboot using the new download, yet fall back to the known good if it fails? 

It is possible, but such mechanism not yet implemented. There are three areas for updata:

1) piCore base

It means update of kernel image and initrd file, and correction of config.txt and cmdline.txt

2) Kernel module tcz's

If kernel version changed, download matching kernel module tcz's if they are stored persistent on the SD card

3) Raspberry Pi firmware

They are available at github

As you see it is a complex task where you can separate 3) from 1)/2). For sure no sense to add it to base but as an optional extension. It must be a console tool which can get later an Fltk GUI frontend.

[mcdudeabides]

Hi Randy

Thanks for your help with the piCorePlayer scripts. As you know many users of the piCorePlayer have been asking whether it could be possible to update the piCorePlayer without physically removing the SD-card for burning a new image on the card. The reason is that several piCorePlayer users are hiding the raspberry inside a closure and sometimes behind walls and ceilings, so it might be difficult to get to them.

Therefore I have also been studying this and my conclusion was that it might be possible.
I was think of something like this:
To update the kernel and modules simply download and overwrite the piCore.img and root.gz files (like you suggested).
For update of  the piCorePlayer - download and update the complete TCE directory.
(I know that you download a lot of files that are unchanged, but still it is rather small, and for me it seems the simplest way.

The piCorePlayer is downloaded from Sourceforge, so I was thinking of making a new directory there with these separate files to download.

Off course there need to be some kind of fail-safe so that you are not deleting any files before you know that the new ones are ready.

Steen

Hello Steen.  Indeed piCorePlayer is one of the main drivers for my curiosity in this area.  I've dabbled with other small projects (a crude irrigation controller, IR/contact closure converter, one wire sensors) that seem to always have a curve where there is rapid adaption to the base platform to provide the intended functionality objective.  If the objective is valuable/desired enough, then the effort to adapt the base platform is tolerated.  When the objective is "close enough", the effort to adapt the platform is excessive and evolution of the project stalls. 

The combination of the Raspberry Pi and Tiny Core have opened up a new opportunity for some of my projects that have stalled.  If there is a way to avoid the physical SD card retrieval (for reasons you indicated as well as the case of physical distance) then it would seem to lower the effort to maintain the base platform and allow for continued innovation.  For piCorePlayer as well as others.

In the case of piCorePlayer, there are some specific opportunities.  Given the size of the target package, the ability to tolerate multiple images for failsafe is an option.  The particular customizations for wifi, squeezelite and alsa need to be configured once and allowed to be stored persistent as the various images are introduced.  The additional unique twists of the various DAC approaches (both USB and I2S) need to have an almost "tcz" like approach to insulate for conflicts amongst the various DACs without overcomplicating bootlocal.sh. 

I would be very interested in contributing/testing in this area for piCorePlayer.  I think the education will be most helpful in my other projects.  I will have windows of down time available in between caregiver duties for my parents.  I'm in the early stage of research now.  If there is a specific direction or architecture you wish for your project, please help me understand so that I can be helpful.

[mcdudeabides]

It is possible, but such mechanism not yet implemented. There are three areas for updata:

1) piCore base

It means update of kernel image and initrd file, and correction of config.txt and cmdline.txt

2) Kernel module tcz's

If kernel version changed, download matching kernel module tcz's if they are stored persistent on the SD card

This is the area that seems most "GRUB like" to me (almost a multi-boot approach maybe).  Most of the tools in this area seem to be x86 architecture and directed towards an interactive (monitor/keyboard connected) style.  Do you have an opinion on Berryboot? http://www.berryterminal.com/doku.php/berryboot

3) Raspberry Pi firmware

They are available at github

As you see it is a complex task where you can separate 3) from 1)/2). For sure no sense to add it to base but as an optional extension. It must be a console tool which can get later an Fltk GUI frontend.

An important consideration that I didn't see at first.  When targeting a specific application (like piCorePlayer), an accommodation could be made.  But for more generic projects this would seem an important consideration.

Thanks for your insights and advise.  I certainly appreciate the philosophy of Tiny Core.  The frugal nature makes it fast, efficient and resilient.  No wonder so many others are trying to adapt.  I don't want to bring in all the overhead of YUM, RPM, Anaconda and the like just to improve the deployment to SD card.  Hopefully there is a good balance.

[Paul_123]

The piCorePlayer is downloaded from Sourceforge, so I was thinking of making a new directory there with these separate files to download.

Off course there need to be some kind of fail-safe so that you are not deleting any files before you know that the new ones are ready.

I looked at this when the topic came up on the SB forums.   I've done a update inplace for several piCore updates.....

Updateing the initrd is pretty straight forward, updating the kernel is pretty straight forward.......the problem I ran into was updating the tcz extensions......specifically the extensions the ones that had kernel modules.....these have to be manually downloaded using wget (be sure to get the .tcz .dep .md5.txt). 

Here are the steps that I layed out.... and have done manually several times
1) Check partitions for adequate size to accept the new downloads.......the fat partition is a problem, as the resize partition is not always a clean process.   Maybe you should just make the partitions big enough to hold 2 images.
2) Make sure that modules are loaded with the "-KERNEL" filenames in onboot.lst....  (This should be the case)
3) Download new extensions for the new kernel.........
4) Download new Kernel and initrd
5) Download new Firmware
6) Change config.txt to point to the new kernel
7) Change cmdline.txt for the new initrd size
Make any other changes required to startup configuration....
9) Reboot
10) Cleanup old files.

[Paul_123]

This is the area that seems most "GRUB like" to me (almost a multi-boot approach maybe).

NOOBS uses some sort of a multiboot loader for the RPI.....i wonder if it is easily configurable for piCore type environments.

hmmm....maybe.

[mcdudeabides]

@Paul123 - you've gotten much further than me.  Not realizing the importance of the fat32 partition, I think I spent a lot of time banging my head on the door.  I wrote it off to "magic" and went looking for boot tools.

I've had a go a berry boot and initial experiments are rewarding.  There is not a Tiny Core image (but I think this could be created).  I've been experimenting with the headless approach (VNC spawned) and also attempting to add new image while one of the promoted images is running.  It's very early in my research and I haven't be able to devote much time.  Still trying to find my way around.

[bmarkus]

Added gzipped content of the boot partition (mnt/mmcblk0p1) to the download area for piCore-5.3alpha1 for easy update. For instructions see updated release announcement

[Paul_123]

Added gzipped content of the boot partition (mnt/mmcblk0p1) to the download area for piCore-5.3alpha1 for easy update.

I guess another reason that I update the way i did......my pi is in an enclosure.     With no keyboard access, so I update the modules manually, so wifi and such works upon the first reboot.

[sbp]

Added gzipped content of the boot partition (mnt/mmcblk0p1) to the download area for piCore-5.3alpha1 for easy update.

I guess another reason that I update the way i did......my pi is in an enclosure.     With no keyboard access, so I update the modules manually, so wifi and such works upon the first reboot.

Thats why I have made the config.cfg which should hold all the configuration settings for the piCorePlayer. So an in situ update could simply copy this file to the fat partition as newconfig.cfg, then after a reboot piCorePlayer will take this file as its configuration file and everything will be as before the update.

Steen

[bmarkus]

Added gzipped content of the boot partition (mnt/mmcblk0p1) to the download area for piCore-5.3alpha1 for easy update.

I guess another reason that I update the way i did......my pi is in an enclosure.     With no keyboard access, so I update the modules manually, so wifi and such works upon the first reboot.

Right, you must download the new kernel extension for wireless before rebooting to have link up after.

[mcdudeabides]

Here are the steps that I layed out.... and have done manually several times
1) Check partitions for adequate size to accept the new downloads.......the fat partition is a problem, as the resize partition is not always a clean process.   Maybe you should just make the partitions big enough to hold 2 images.
2) Make sure that modules are loaded with the "-KERNEL" filenames in onboot.lst....  (This should be the case)
3) Download new extensions for the new kernel.........
4) Download new Kernel and initrd
5) Download new Firmware
6) Change config.txt to point to the new kernel
7) Change cmdline.txt for the new initrd size
Make any other changes required to startup configuration....
9) Reboot
10) Cleanup old files.

Using this approach as basis, I took a slightly less elegant brute force approach.  I got one SD card image working as desired using version X of piCorePlayer.  Once working, I collected kernel img, initrd gz, (and for testing the elf file), config.txt and cmdline.txt to a separate directory.  I then got a second SD card image working as desired using version Y of piCorePlayer.  Again, collected img, gz, elf and txt files to separate directory.  Cloned version X SD card.  Placed SD card in reader and added version Y img, gz (elf was identical at this point) files to fat partition.  Then I added config.txt and cmdline.txt as config.new and cmdline.new.  Then booted and version X started as expected.  Mounted fat partition and renamed *.txt to *.old.  Then renamed *.new to *.txt and rebooted.  Version Y booted as expected.

Now if you have high confidence in the new image structure, this leap of faith can work.  But if anything goes wrong there is no failsafe fallback.  Looks like berryboot and NOOBS have a "boot once" logic that can start a different OS one time without changing the default startup OS.  But as you indicated, the FAT partition starts to get crowded if you add more stuff in.  Is there any way to perform some manipulations in the txt files while in bootup?  I guess that's the reason for GRUB and the like.

This almost feels like the snake eating it's own tail.  I'll have to percolate on this a while and see if anything better bubbles out.

[mcdudeabides]

Fooled myself.  I was using similar packages (piCorePlayer) to test this out.  I thought if I saw the different version boot, then it must be working.  Unfortunately I didn't realize that the onboot.lst (which Paul123) pointed out and the directly mounted things like Oliweb would not be there if using different packages.  Certainly more to learn.  I was once told that when the student is ready the teacher will appear.  Where are you teacher?

[Paul_123]

Now if you have high confidence in the new image structure, this leap of faith can work.  But if anything goes wrong there is no failsafe fallback. 

There is really no good fail safe for the upgrades we are talking about.    Fail Safe would be to have a backup of mydata.tcz........and a new image ready to burn....... of course pulling the SDcard is going to have to happen at this point.

In Steen's Case, I think the fail safe is less of an issue.   He is building a functional image, an then moving a configuration file over  (The config file will reside in the myData.tcz backup, which we do not over write).  He will just need to supply the individual archives for download.   This way we really only need to worry about verifying a MD5sum of the downloaded files before taking the leap.