Building landing wheels while in flight

[sbp]

Thanks for all your thoughts and suggestions.
So should we try to make such an update option for piCorePlayer?

I was thinking something like this.
Make an "Update piCorePlayer" option on the Main page.
When activated this will:
mount /dev/mmcblk0p1
copy the /usr/local/sbin/config.cfg file to /mnt/mmcblk0p1/newconfig.cfg

Then download the piCore.img and piCore.gz  - check their MD5sum, then delete the old piCore.img and the old piCore.gz files.
Download a new config.txt file containing the correct name and size of the piCore.img and piCore.gz files.
 
Next download the a zipped version of the TCE directory - check the MD5sum, then delete the old TCE directory and finally unzip the new TCE directory.

Then reboot, and when starting piCorePlayer will use the /mnt/mmcblk0p1/newconfig.cfg file to configure the new version of piCorePlayer with the settings from the old system (wifi, password, squeezelite settings, tweaks and overclock settings etc) - then delete the newconfig.cfg file. Then make a backup (to save the new config.cfg file) and we have an updated piCorePlayer using the old settings.

What do you think?

Steen

[mcdudeabides]

@Paul and Steen-
I agree with moving forward with some mechanism for upgrading without having to remove the SD card.  I do think that given the unique case that is piCorePlayer there is a way to accommodate upgrades with current session running shell script logic.  If there are serious architecture shifts for Tiny Core (or any dependent packages), then the upgrade process would become more disruptive (up to the point of full SD card rewrite).  There may need to be some deliberate change for the approach of someone starting new versus someone upgrading from prior start.  I would also anticipate a challenge in enabling/supporting the wide range of DAC cards.  Eventually there will be conflicts that prevent providing protocol stacks for all known DACs.  This would almost trigger a "TCE-like" environment where there are unique "packages" to load for the different DACs.  In any regard, I would be willing to volunteer efforts along the line of providing upgrades to the degree that we can control it.

I am also (quite selfishly) thinking of other embedded projects that would benefit from a remote upgrade approach.  The benefit of Tiny Core is that it is efficient, frugal and very embedded-friendly.  The downside is that it rapidly evolves and the upgrade method is now targeted to providing a new system image on a vanilla startup.  Thinking back to my process control days on Modcomp and Honeywell trigger a desire for a more robust deployment approach. 

Do you think it possible to include at least the "boot once" logic?  The goal being that a new image can be booted once, and if there is no way to confirm a positive execution (through user confirmation or script logic), then on power cycle the last "known good" image will be reconstituted and provide a recovery avenue?  The most elegant approach would provide the means at bootload (maybe like the berryboot VNC - and all it's hairy dependencies).  Perhaps there is something less elegant that will get most of the functionality.

In any event, I am willing to invest time in whatever approach is considered most viable and desired. 

Steen - this is really your creation.  Do you have any guidance?

[sbp]

@Paul and Steen-
I agree with moving forward with some mechanism for upgrading without having to remove the SD card.  I do think that given the unique case that is piCorePlayer there is a way to accommodate upgrades with current session running shell script logic.  If there are serious architecture shifts for Tiny Core (or any dependent packages), then the upgrade process would become more disruptive (up to the point of full SD card rewrite).  There may need to be some deliberate change for the approach of someone starting new versus someone upgrading from prior start.  I would also anticipate a challenge in enabling/supporting the wide range of DAC cards.  Eventually there will be conflicts that prevent providing protocol stacks for all known DACs.  This would almost trigger a "TCE-like" environment where there are unique "packages" to load for the different DACs.  In any regard, I would be willing to volunteer efforts along the line of providing upgrades to the degree that we can control it.

I agree that this will be a constant struggle to support all the new DACs that will be developed. However, for the near future I can't see that it will be possible to build "DAC packages". As it is now these new DACs require changes in the kernel and support for new DACs are added from time to time via a new kernel and modules.
For each major update of piCorePlayer I have build a new kernel and modules (in order to support more hardware and improve stability), so for piCorePlayer I think we often still need to supply our own kernel.img and module.gz. From time to time we are in sync with the official piCore development of bmarkus and sometime we are in front.
So we need to be able to download our own kernel.img and module.gz.
Therefore, we also need to be able to create and download the Kernel specific tcz packages (Alsa and wifi).   

I am also (quite selfishly) thinking of other embedded projects that would benefit from a remote upgrade approach.  The benefit of Tiny Core is that it is efficient, frugal and very embedded-friendly.  The downside is that it rapidly evolves and the upgrade method is now targeted to providing a new system image on a vanilla startup.  Thinking back to my process control days on Modcomp and Honeywell trigger a desire for a more robust deployment approach..

I'm not sure I completely understand.
But maybe piCorePlayer is not a good model for developing a generic in situ update system for piCore. The problem with piCorePlayer is that it often use its own kernel and module, therefore, the update system will be different compared to a system build on vanilla piCore and its modules.

Do you think it possible to include at least the "boot once" logic?  The goal being that a new image can be booted once, and if there is no way to confirm a positive execution (through user confirmation or script logic), then on power cycle the last "known good" image will be reconstituted and provide a recovery avenue?  The most elegant approach would provide the means at bootload (maybe like the berryboot VNC - and all it's hairy dependencies).  Perhaps there is something less elegant that will get most of the functionality.

In any event, I am willing to invest time in whatever approach is considered most viable and desired.

This sound good. However, I have no idea how to make it work.

Steen - this is really your creation.  Do you have any guidance?

Sorry - no, I think that I'm the one who knows the least when it comes to linux and programming.

[Paul_123]

Integrating something like berry boot would seem rather daunting.

I would think that we would assume that the kernel would boot.....as that would be tested.   The biggest concern would be if there was an incompatibility with the upgrade.   The process could go something like this.

-If we do a firmware update, that is pretty unlikely to cause a problem......so as long as these files are checksumed, I would not plan on a way to go back.
-Install the new Kernel and initrd  (Keep the last copies)
-Install the new config and cmdline   (Keep the last copies)
-After boot.....if there is no interaction.....after a specified time delay, the config and cmdline files from the last system would be put back.

The problem going totally back is if we update extensions that are not kernel dependant........but then I suppose you could create one large piCorePlayer extension that is kernel dependant.....and has the kernel number in the file name.

[mcdudeabides]

I'm willing to prototype something this weekend.  I should have a few periods of downtime while I'm monitoring parent's treatments.  Just to get feet wet we can use the 1.13 to 1.14 as a test case.  There should be all of the needed files to do something that can demonstrate the concept.

 @Paul123 - both berryboot and NOOBS were very easy to get going and surprisingly small footprint.  Unfortunately the TCL architecture is closer to something like berryboot than it is to a target OS package.  I wish I understood the bootloader better.  It just seems that there should be a way to point it to one partition to load one image, and then within that running image call the bootloader with directions to start a second, different, partition.  In any event, the berryboot and one of the frugal OS installs will help satisfy two of my other interests.

 @Steen - I misused the term package in relation to the DAC cards.  I was not speaking of a tce-like package, but rather keeping all of the modules and boot logic in their own representative script.  Then, much like the ALSA screen, the web page could select the correct card (Wolfson, IQaudiO, etc.) and the modules/startup/config for just that DAC card would be activated.  It just seems to me that as more DAC cards are pursued in the future that troubleshooting is easier if you keep things compartmentalized.

[sbp]

I'm willing to prototype something this weekend.  I should have a few periods of downtime while I'm monitoring parent's treatments.  Just to get feet wet we can use the 1.13 to 1.14 as a test case.  There should be all of the needed files to do something that can demonstrate the concept.

Sorry to hear about your parents condition. Are they both seriously ill?.
Regarding your prototype, I'm looking forward to see that. Please let me know what changes I need to apply in the piCorePlayer web-GUI.

@Steen - I misused the term package in relation to the DAC cards.  I was not speaking of a tce-like package, but rather keeping all of the modules and boot logic in their own representative script.  Then, much like the ALSA screen, the web page could select the correct card (Wolfson, IQaudiO, etc.) and the modules/startup/config for just that DAC card would be activated.  It just seems to me that as more DAC cards are pursued in the future that troubleshooting is easier if you keep things compartmentalized.

OK I see.
I already have some of the DAC switching logic in separate scripts, but some of the code is still present elsewhere as well  - so it might be a good idea to make a standard recipe for any new DAC.  I will start to look into that. Thanks for the suggestion.

Steen

[mcdudeabides]

Age is catching up with both my parents. One on dialysis and the other in therapy for mini-strokes. They still wish to be independent as possible, so I try to accommodate them as much as possible.

I was able to get some code snippets on my laptop. I'm getting a clean RPi up to do some integration and testing after dinner tonight. Hopeful to have something soon.

[mcdudeabides]

There are still some warts and hairs to remedy, but  I thought I would share where I currently progressed.  As it stands now, you can call the script  with the name of one of the images on SourceForge (insitu.sh piCorePlayer1.14d.img for example).  It still has some debugging cutouts, so be prepared to hit enter to get it to progress.  I haven't had time to put any comments so far, so some of the things will look strange at first blush.

In a nutshell it will -
1) create a "promote directory" to hold the upgrade image.
2) download image specified on command line from SourceForge
3) add a filesystem to hold a copy of all the important bits
4) create a local "revert" directory to hold a copy of the current config for potential fallback
5) mount the promoted image as two directories ready to be copied
=====at this point it doesn't actually copy over the new image files so that you can exercise it without too much danger

Code: [Select]

#!/bin/sh
set +x

imgname=$1
sudo mkdir -m 777 /tmp/promote
wget -P /tmp/promote http://sourceforge.net/projects/picoreplayer/files/$imgname/download

sudo /usr/local/etc/init.d/dropbear stop
sudo /usr/local/etc/init.d/oliweb_initd stop
sudo /usr/local/etc/init.d/squeezelite_initd stop
sleep 2
read -r sqlpid < /var/run/squeezelitehf.pid
sudo kill $sqlpid
read -p "wait"




if [ ! "`fdisk -l | grep 0p3`" ]
then
        par3="n\np\n3\n"$((`fdisk -l | grep 0p2 | awk '{print $3}'`+1))"\n+256M\nw\n"
        sudo umount -a
        sudo echo -e $par3 | fdisk /dev/mmcblk0
else
        echo "Existing work partition.  Nonstandard.  Exiting"
        exit 4
fi

sudo hdparm -z /dev/mmcblk0
sudo mount -a
sleep 2
read -p "wait"

sudo mkdir -m 777 /tmp/revert
sudo mkfs.ext4 -t ext4 /dev/mmcblk0p3
sudo mount /dev/mmcblk0p3 /tmp/revert

sudo mount -t vfat /dev/mmcblk0p1 /mnt/mmcblk0p1
sudo cp /mnt/mmcblk0p1/*.img /tmp/revert
sudo cp /mnt/mmcblk0p1/*.gz /tmp/revert
sudo cp /mnt/mmcblk0p1/*.txt /tmp/revert
sudo cp /mnt/mmcblk0p1/start.elf /tmp/revert
sudo cp /mnt/mmcblk0p1/fixup.dat /tmp/revert
sudo mount /dev/mmcblk0p2 /mnt/mmcblk0p2
sudo cp /mnt/mmcblk0p2/tce/mydata.tgz /tmp/revert
sudo cp /mnt/mmcblk0p2/tce/onboot.lst /tmp/revert

p1offset=$(fdisk -l -u -C 52 /tmp/promote/download | grep FAT32 | awk '{print $2*512}')
p2offset=$(fdisk -l -u -C 52 /tmp/promote/download | grep Linux | awk '{print $2*512}')

sudo mkdir /tmp/par1
par1dir=$(losetup -f)
sudo losetup -o $p1offset -f /tmp/promote/download
sudo mount $par1dir /tmp/par1

sudo mkdir /tmp/par2
par2dir=$(losetup -f)
sudo losetup -o $p2offset -f /tmp/promote/download
sudo mount $par2dir /tmp/par2

exit 0

There is still some research to do on getting the recover logic sorted out.  The squeezelite init.d is not stopping it now and I need to find a way to better prevent that.  I'm also hoping there is a better way to read in the partition table.  Trying to remove contention to allow the kernel to reread the table leaves everything in a shakey state.  Of course the next thing that would happen after this script is a reboot anyway, so it may be tolerable.

In any event, take a look and tell me what you find.

[Paul_123]

would it not be easier to just extend the partition and create a directory to hold the new image.

I agree deleting partitions get to be a problem.

[mcdudeabides]

would it not be easier to just extend the partition and create a directory to hold the new image.

I agree deleting partitions get to be a problem.

Hello Paul.  If I were doing this on a system that I intended to use every day, I would certainly agree with you.  In fact, when I was first trying to puzzle out the approach, I started with the idea to just resize the existing filesystem.  The primary objective for me was to use only what existed now (i.e. the .img file approach), respect the work that came before me, attempt to provide a known_good recovery and attempt to control it over one session.  I was unable to get the filesystem to grow in the current session.  I needed to reboot and the tricks to fool the system in to a onceboot kind of approach looked too painful.  It might have been a function of my prototyping environment, but the approach of adding a third partition seemed to allow me (with a few tricks) to do everything in one session. 

I am most certainly not married to this approach, but offer it as an early example of one way to accomplish the objective.  I think there are parts that could work in a number of variations on upgrading.  If you have some specific examples that would be improvements to offer up, I would be glad to incorporate or to support you in following the objective.

[mcdudeabides]

Here is a slightly updated version of insitu.sh with comments:

Code: [Select]

#!/bin/sh
set +x
#
# Upgrade in place a running piCorePlayer img from sourceforge
# Usage: insitu.sh file.img (i.e. piCorePlayer1.14d.img)
#

imgname=$1

# Grab the new image in RAM from sourceforge
sudo mkdir -m 777 /tmp/promote
if ! wget -P /tmp/promote http://sourceforge.net/projects/picoreplayer/files/$imgname/download
then
echo "Error retrieving" $imgname
exit 4
fi

# Stop the services that keep /dev/mmcblk0 busy
sudo /usr/local/etc/init.d/dropbear stop
sudo /usr/local/etc/init.d/oliweb_initd stop
sudo /usr/local/etc/init.d/squeezelite_initd stop
# Make sure they are gone (squeezelite down script currently throws error)
sudo kill $(ps|grep dropbear|grep root|awk '{print $1}')
sudo kill $(ps|grep oliweb|grep root|awk '{print $1}')
sudo kill $(ps|grep squeeze|grep root|awk '{print $1}')

sleep 2

read -p "wait"

# Create a new partition to store the existing piCorePlayer key files
if [ ! "`fdisk -l | grep 0p3`" ]
then
par3="n\np\n3\n"$((`fdisk -l | grep 0p2 | awk '{print $3}'`+1))"\n+256M\nw\n"
sudo umount -a
sudo echo -e $par3 | fdisk /dev/mmcblk0
else
echo "Existing work partition.  Nonstandard.  Exiting"
exit 4
fi

# Reread new partiion and make available for use
sudo hdparm -z /dev/mmcblk0
sudo mount -a
sleep 2
read -p "wait"
sudo mkdir /tmp/revert
sudo mkfs.ext4 -t ext4 /dev/mmcblk0p3
sudo mount /dev/mmcblk0p3 /tmp/revert

# Backup key existing files
sudo mount -t vfat /dev/mmcblk0p1 /mnt/mmcblk0p1
sudo cp /mnt/mmcblk0p1/*.img /tmp/revert
sudo cp /mnt/mmcblk0p1/*.gz /tmp/revert
sudo cp /mnt/mmcblk0p1/*.txt /tmp/revert
sudo cp /mnt/mmcblk0p1/start.elf /tmp/revert
sudo cp /mnt/mmcblk0p1/fixup.dat /tmp/revert
sudo mount -t ext4 /dev/mmcblk0p2 /mnt/mmcblk0p2
sudo cp /mnt/mmcblk0p2/tce/mydata.tgz /tmp/revert
sudo cp /mnt/mmcblk0p2/tce/onboot.lst /tmp/revert

# Calculate offsets in new img file to partitions
p1offset=$(fdisk -l -u -C 52 /tmp/promote/download | grep FAT32 | awk '{print $2*512}')
p2offset=$(fdisk -l -u -C 52 /tmp/promote/download | grep Linux | awk '{print $2*512}')

sudo mkdir /tmp/par1
par1dir=$(losetup -f)
sudo losetup -o $p1offset -f /tmp/promote/download
sudo mount $par1dir /tmp/par1

sudo mkdir /tmp/par2
par2dir=$(losetup -f)
sudo losetup -o $p2offset -f /tmp/promote/download
sudo mount $par2dir /tmp/par2

# Copy new key files to  mmcblk0
# Reboot

exit 0

I've used it with the half dozen older images of piCorePlayer that were on my machine (burned old .img to SD card, rebooted, loaded insitu.sh over and let it run with target of piCorePlayer1.14.img).  So far the only irritation is that squeezelite won't come down with the init.d script.  At this point the third partition is overkill at 256M.  If this is the way to go, then it may be better to scale it back.

So here is where I think this stands.  Steen has a successful and appealing approach for the burn&boot masses (no knowledge required, just burn SD card, boot your RPi and load URL).  If anybody grabs an IMG from sourceforge, their machine should boot standalone.  Keeps it simple and provides a worst case scenario disaster recovery.

There are those (like myself) that can't lay hands on their RPi to pull the SD and burn&boot.  The desire is to provide a means to upgrade a currently running version of piCorePlayer to a new version without touching the physical RPi.  Seems to me the approaches break down in to the following:

1) Provide failsafe upgrade -  (start known_good handler, backup old image, install new image, if new image fails to load then fallback known_good handler and revert old image)
Where it stands now - ugly berryboot proof of concept works, would be hairy for the masses.  Need a better idea (chain boot??)

2) Provide upgrade with revert - (backup old image, install new image, reboot and cross fingers)
Where it stands now - working prototype of insitu.sh here now.  Needs revert logic

3) Provide upgrade bare - (install new image, reboot and cross fingers)
Where it stands now - repurpose parts of insitu.sh

4) Rearchitect piCorePlayer - (develop a new image of piCorePlayer with new functionality/design/approach to allow for in place upgrades)
Where it stands now - need to develop and understand requirements

I'm ready to help in whatever direction you want to pursue.

PS - on a completely unrelated side note.  What in the !@#$@!# do I do to get scp working with putty/dropbear?  The old cut/paste a vi session is Neanderthal.

[sbp]

I've used it with the half dozen older images of piCorePlayer that were on my machine (burned old .img to SD card, rebooted, loaded insitu.sh over and let it run with target of piCorePlayer1.14.img).  So far the only irritation is that squeezelite won't come down with the init.d script.  At this point the third partition is overkill at 256M.  If this is the way to go, then it may be better to scale it back.

Thanks, I will give it a try. It will be a major advantage to be able to update piCorePlayer in situ.

Will it be easier if I make a special directory on sourceforge called update, where only the newest piCorePlayer version is?
Can there be some kind of check, so that it only tries to update if there is a newer version than what is currently running?

So here is where I think this stands.  Steen has a successful and appealing approach for the burn&boot masses (no knowledge required, just burn SD card, boot your RPi and load URL).  If anybody grabs an IMG from sourceforge, their machine should boot standalone.  Keeps it simple and provides a worst case scenario disaster recovery.

There are those (like myself) that can't lay hands on their RPi to pull the SD and burn&boot.  The desire is to provide a means to upgrade a currently running version of piCorePlayer to a new version without touching the physical RPi.  Seems to me the approaches break down in to the following:

1) Provide failsafe upgrade -  (start known_good handler, backup old image, install new image, if new image fails to load then fallback known_good handler and revert old image)
Where it stands now - ugly berryboot proof of concept works, would be hairy for the masses.  Need a better idea (chain boot??)

2) Provide upgrade with revert - (backup old image, install new image, reboot and cross fingers)
Where it stands now - working prototype of insitu.sh here now.  Needs revert logic

3) Provide upgrade bare - (install new image, reboot and cross fingers)
Where it stands now - repurpose parts of insitu.sh

4) Rearchitect piCorePlayer - (develop a new image of piCorePlayer with new functionality/design/approach to allow for in place upgrades)
Where it stands now - need to develop and understand requirements

I'm ready to help in whatever direction you want to pursue.

Thanks. I think we should continue (we I said - you have done all the work).
Option 2 would be OK. I was thinking that if we are able to boot after a update and can see squeezelite is running, then everything should be OK - and we could delete the old files. This kind of logic should be rather simple to make.
However, in the worst case scenario we are unable to boot, and then nothing can be done - we will then need to burn the SD-card with a new image? I don't know if this can be handled in any other way, but I doubt it.

Also piCorePlayer would need an overhaul in order to make this happen - but first we need to decide on the mechanism. 

PS - on a completely unrelated side note.  What in the !@#$@!# do I do to get scp working with putty/dropbear?  The old cut/paste a vi session is Neanderthal.

I use putty and WinSCP all the time without any problems?
It complains about saving is not permitted - but it saves the files anyway.

Steen

[mcdudeabides]

Hello Steen.  I'll take a first pass at the revert logic. 

Will it be easier if I make a special directory on sourceforge called update, where only the newest piCorePlayer version is?
Can there be some kind of check, so that it only tries to update if there is a newer version than what is currently running?

I would suggest that you put a file with a consistent name on sourceforge.  It can be called something like "ProductionVersion.img".  The file could contain at a minimum the latest version text string as is seen in the piversion.cfg in local/sbin.  A quick comparison of the downloaded version and the one in local piversion.cfg would allow to verify an upgrade exists.  Then the name would be fed to insitu to retrieve.  If needed there could also be an auxiliary script that could be executed before upgrade to allow for any "tweaks" that weren't thought about when the previous image delivered.  It should probably be something explicitly triggered from the web interface (probably button press on tweaks menu to look for upgrade).  I don't think sourceforge would want to be hammered every time a RPI restarts.

Thanks. I think we should continue (we I said - you have done all the work).
Option 2 would be OK. I was thinking that if we are able to boot after a update and can see squeezelite is running, then everything should be OK - and we could delete the old files. This kind of logic should be rather simple to make.
However, in the worst case scenario we are unable to boot, and then nothing can be done - we will then need to burn the SD-card with a new image? I don't know if this can be handled in any other way, but I doubt it.

Worst case scenario would be for the burn&boot crowd to load a new image fresh, then configure again (unless they saved config on USB).  The more adventurous could load the SD card and edit cmdline.txt external to try base or norestore bootcodes and see if they can get back to the revert logic.

Also piCorePlayer would need an overhaul in order to make this happen - but first we need to decide on the mechanism. 

Now would be the time to make any adjustments to the piCorePlayer layout.  It is still the same image file and can be loaded by burn&boot or insitu.

I use putty and WinSCP all the time without any problems?
It complains about saving is not permitted - but it saves the files anyway.

I must have something gorked on my windows machine.  Will try a different box.

[sbp]

I would suggest that you put a file with a consistent name on sourceforge.  It can be called something like "ProductionVersion.img".  The file could contain at a minimum the latest version text string as is seen in the piversion.cfg in local/sbin.  A quick comparison of the downloaded version and the one in local piversion.cfg would allow to verify an upgrade exists.  Then the name would be fed to insitu to retrieve.  If needed there could also be an auxiliary script that could be executed before upgrade to allow for any "tweaks" that weren't thought about when the previous image delivered.  It should probably be something explicitly triggered from the web interface (probably button press on tweaks menu to look for upgrade).  I don't think sourceforge would want to be hammered every time a RPI restarts.

That is fine.
There is already such a file. "piCorePlayer_last.txt" it contains only the name of the newest piCorePlayer and is used by SqueezePlug so that SqueezePlug users are able to download the newest version of piCorePlayer from inside SqueezePlug.
So this file could be used for our purpose as well - to check the name of the newest version and to get the name of the newest piCorePlayer which then could be downloaded.

Steen

[Paul_123]

making some good progress.....I just don't have as much free time to help out.   

As I looked through your script, I would make sure that you do a MD5 checksum on the downloaded image.    Much like the logic that is done in tce.   Sure the script should crash when trying to copy the image.....but that is a bit far in the process to catch the problem.