Tiny Core Linux

Tiny Core Base => Raspberry Pi => Topic started by: mcdudeabides on April 06, 2014, 05:54:25 PM

Title: Building landing wheels while in flight
Post by: mcdudeabides on April 06, 2014, 05:54:25 PM
I've been studying the architecture http://distro.ibiblio.org/tinycorelinux/architecture.html (http://distro.ibiblio.org/tinycorelinux/architecture.html) diagram with an eye to understanding the possibilities of performing an in-situ update.  For my simple mind, it seems that the kernel could be updated by overwriting /boot/vmlinuz and the root file system by overwriting /boot/core.gz.  Depending on the application, all TCZ extensions might take a little more brute force.  There are some things in mydata.tgz that should probably be retained (hostname, ssid, wifi setup, etc.), but other that need to be overwritten. 

So before embarking on the fool's errand, I thought I might just ask why there is not a mechanism that would allow for a currently running instance of piCore to grab a new version and update itself?  Is there a GRUB (or something similar for the Pi oddities) that could bring the next version into a new boot area and reboot using the new download, yet fall back to the known good if it fails? 

I saw this discussion http://forum.tinycorelinux.net/index.php?topic=16464.0 (http://forum.tinycorelinux.net/index.php?topic=16464.0) and thought if it can be done for other targets. then it might just work for piCore itself.

Sorry for the rambling, but I'm between other obligations and only had a brief window to throw this out there.  If there are any online education resources or advice that can be offered, it will be much appreciated.

Title: Re: Building landing wheels while in flight
Post by: sbp on April 06, 2014, 10:58:42 PM
Hi Randy

Thanks for your help with the piCorePlayer scripts. As you know many users of the piCorePlayer have been asking whether it could be possible to update the piCorePlayer without physically removing the SD-card for burning a new image on the card. The reason is that several piCorePlayer users are hiding the raspberry inside a closure and sometimes behind walls and ceilings, so it might be difficult to get to them.

Therefore I have also been studying this and my conclusion was that it might be possible.
I was think of something like this:
To update the kernel and modules simply download and overwrite the piCore.img and root.gz files (like you suggested).
For update of  the piCorePlayer - download and update the complete TCE directory.
(I know that you download a lot of files that are unchanged, but still it is rather small, and for me it seems the simplest way.

The piCorePlayer is downloaded from Sourceforge, so I was thinking of making a new directory there with these separate files to download.

Off course there need to be some kind of fail-safe so that you are not deleting any files before you know that the new ones are ready.

Steen
Title: Re: Building landing wheels while in flight
Post by: bmarkus on April 07, 2014, 12:47:29 AM

So before embarking on the fool's errand, I thought I might just ask why there is not a mechanism that would allow for a currently running instance of piCore to grab a new version and update itself?  Is there a GRUB (or something similar for the Pi oddities) that could bring the next version into a new boot area and reboot using the new download, yet fall back to the known good if it fails? 


It is possible, but such mechanism not yet implemented. There are three areas for updata:

1) piCore base

It means update of kernel image and initrd file, and correction of config.txt and cmdline.txt

2) Kernel module tcz's

If kernel version changed, download matching kernel module tcz's if they are stored persistent on the SD card

3) Raspberry Pi firmware

They are available at github

As you see it is a complex task where you can separate 3) from 1)/2). For sure no sense to add it to base but as an optional extension. It must be a console tool which can get later an Fltk GUI frontend.

Title: Re: Building landing wheels while in flight
Post by: mcdudeabides on April 07, 2014, 07:06:18 AM
Hi Randy

Thanks for your help with the piCorePlayer scripts. As you know many users of the piCorePlayer have been asking whether it could be possible to update the piCorePlayer without physically removing the SD-card for burning a new image on the card. The reason is that several piCorePlayer users are hiding the raspberry inside a closure and sometimes behind walls and ceilings, so it might be difficult to get to them.

Therefore I have also been studying this and my conclusion was that it might be possible.
I was think of something like this:
To update the kernel and modules simply download and overwrite the piCore.img and root.gz files (like you suggested).
For update of  the piCorePlayer - download and update the complete TCE directory.
(I know that you download a lot of files that are unchanged, but still it is rather small, and for me it seems the simplest way.

The piCorePlayer is downloaded from Sourceforge, so I was thinking of making a new directory there with these separate files to download.

Off course there need to be some kind of fail-safe so that you are not deleting any files before you know that the new ones are ready.

Steen

Hello Steen.  Indeed piCorePlayer is one of the main drivers for my curiosity in this area.  I've dabbled with other small projects (a crude irrigation controller, IR/contact closure converter, one wire sensors) that seem to always have a curve where there is rapid adaption to the base platform to provide the intended functionality objective.  If the objective is valuable/desired enough, then the effort to adapt the base platform is tolerated.  When the objective is "close enough", the effort to adapt the platform is excessive and evolution of the project stalls. 

The combination of the Raspberry Pi and Tiny Core have opened up a new opportunity for some of my projects that have stalled.  If there is a way to avoid the physical SD card retrieval (for reasons you indicated as well as the case of physical distance) then it would seem to lower the effort to maintain the base platform and allow for continued innovation.  For piCorePlayer as well as others.

In the case of piCorePlayer, there are some specific opportunities.  Given the size of the target package, the ability to tolerate multiple images for failsafe is an option.  The particular customizations for wifi, squeezelite and alsa need to be configured once and allowed to be stored persistent as the various images are introduced.  The additional unique twists of the various DAC approaches (both USB and I2S) need to have an almost "tcz" like approach to insulate for conflicts amongst the various DACs without overcomplicating bootlocal.sh. 

I would be very interested in contributing/testing in this area for piCorePlayer.  I think the education will be most helpful in my other projects.  I will have windows of down time available in between caregiver duties for my parents.  I'm in the early stage of research now.  If there is a specific direction or architecture you wish for your project, please help me understand so that I can be helpful.
Title: Re: Building landing wheels while in flight
Post by: mcdudeabides on April 07, 2014, 07:20:38 AM

It is possible, but such mechanism not yet implemented. There are three areas for updata:

1) piCore base

It means update of kernel image and initrd file, and correction of config.txt and cmdline.txt

2) Kernel module tcz's

If kernel version changed, download matching kernel module tcz's if they are stored persistent on the SD card

This is the area that seems most "GRUB like" to me (almost a multi-boot approach maybe).  Most of the tools in this area seem to be x86 architecture and directed towards an interactive (monitor/keyboard connected) style.  Do you have an opinion on Berryboot? http://www.berryterminal.com/doku.php/berryboot (http://www.berryterminal.com/doku.php/berryboot)

Quote
3) Raspberry Pi firmware

They are available at github

As you see it is a complex task where you can separate 3) from 1)/2). For sure no sense to add it to base but as an optional extension. It must be a console tool which can get later an Fltk GUI frontend.



An important consideration that I didn't see at first.  When targeting a specific application (like piCorePlayer), an accommodation could be made.  But for more generic projects this would seem an important consideration.

Thanks for your insights and advise.  I certainly appreciate the philosophy of Tiny Core.  The frugal nature makes it fast, efficient and resilient.  No wonder so many others are trying to adapt.  I don't want to bring in all the overhead of YUM, RPM, Anaconda and the like just to improve the deployment to SD card.  Hopefully there is a good balance.
Title: Re: Building landing wheels while in flight
Post by: Paul_123 on April 07, 2014, 01:26:00 PM
The piCorePlayer is downloaded from Sourceforge, so I was thinking of making a new directory there with these separate files to download.

Off course there need to be some kind of fail-safe so that you are not deleting any files before you know that the new ones are ready.

I looked at this when the topic came up on the SB forums.   I've done a update inplace for several piCore updates.....

Updateing the initrd is pretty straight forward, updating the kernel is pretty straight forward.......the problem I ran into was updating the tcz extensions......specifically the extensions the ones that had kernel modules.....these have to be manually downloaded using wget (be sure to get the .tcz .dep .md5.txt). 

Here are the steps that I layed out.... and have done manually several times
1) Check partitions for adequate size to accept the new downloads.......the fat partition is a problem, as the resize partition is not always a clean process.   Maybe you should just make the partitions big enough to hold 2 images.
2) Make sure that modules are loaded with the "-KERNEL" filenames in onboot.lst....  (This should be the case)
3) Download new extensions for the new kernel.........
4) Download new Kernel and initrd
5) Download new Firmware
6) Change config.txt to point to the new kernel
7) Change cmdline.txt for the new initrd size
8) Make any other changes required to startup configuration....
9) Reboot
10) Cleanup old files.
Title: Re: Building landing wheels while in flight
Post by: Paul_123 on April 07, 2014, 01:38:40 PM
This is the area that seems most "GRUB like" to me (almost a multi-boot approach maybe).

NOOBS uses some sort of a multiboot loader for the RPI.....i wonder if it is easily configurable for piCore type environments.

hmmm....maybe.
Title: Re: Building landing wheels while in flight
Post by: mcdudeabides on April 07, 2014, 10:15:57 PM
@Paul123 - you've gotten much further than me.  Not realizing the importance of the fat32 partition, I think I spent a lot of time banging my head on the door.  I wrote it off to "magic" and went looking for boot tools.

I've had a go a berry boot and initial experiments are rewarding.  There is not a Tiny Core image (but I think this could be created).  I've been experimenting with the headless approach (VNC spawned) and also attempting to add new image while one of the promoted images is running.  It's very early in my research and I haven't be able to devote much time.  Still trying to find my way around.
Title: Re: Building landing wheels while in flight
Post by: bmarkus on April 08, 2014, 12:45:25 AM
Added gzipped content of the boot partition (mnt/mmcblk0p1) to the download area for piCore-5.3alpha1 for easy update. For instructions see updated release announcement (http://forum.tinycorelinux.net/index.php?topic=16913.msg101323#msg101323)
Title: Re: Building landing wheels while in flight
Post by: Paul_123 on April 08, 2014, 04:50:32 AM
Added gzipped content of the boot partition (mnt/mmcblk0p1) to the download area for piCore-5.3alpha1 for easy update.

I guess another reason that I update the way i did......my pi is in an enclosure.     With no keyboard access, so I update the modules manually, so wifi and such works upon the first reboot.
Title: Re: Building landing wheels while in flight
Post by: sbp on April 08, 2014, 05:55:04 AM
Added gzipped content of the boot partition (mnt/mmcblk0p1) to the download area for piCore-5.3alpha1 for easy update.

I guess another reason that I update the way i did......my pi is in an enclosure.     With no keyboard access, so I update the modules manually, so wifi and such works upon the first reboot.

Thats why I have made the config.cfg which should hold all the configuration settings for the piCorePlayer. So an in situ update could simply copy this file to the fat partition as newconfig.cfg, then after a reboot piCorePlayer will take this file as its configuration file and everything will be as before the update.

Steen
Title: Re: Building landing wheels while in flight
Post by: bmarkus on April 08, 2014, 06:09:11 AM
Added gzipped content of the boot partition (mnt/mmcblk0p1) to the download area for piCore-5.3alpha1 for easy update.

I guess another reason that I update the way i did......my pi is in an enclosure.     With no keyboard access, so I update the modules manually, so wifi and such works upon the first reboot.

Right, you must download the new kernel extension for wireless before rebooting to have link up after.
Title: Re: Building landing wheels while in flight
Post by: mcdudeabides on April 09, 2014, 11:43:35 AM
Here are the steps that I layed out.... and have done manually several times
1) Check partitions for adequate size to accept the new downloads.......the fat partition is a problem, as the resize partition is not always a clean process.   Maybe you should just make the partitions big enough to hold 2 images.
2) Make sure that modules are loaded with the "-KERNEL" filenames in onboot.lst....  (This should be the case)
3) Download new extensions for the new kernel.........
4) Download new Kernel and initrd
5) Download new Firmware
6) Change config.txt to point to the new kernel
7) Change cmdline.txt for the new initrd size
8) Make any other changes required to startup configuration....
9) Reboot
10) Cleanup old files.

Using this approach as basis, I took a slightly less elegant brute force approach.  I got one SD card image working as desired using version X of piCorePlayer.  Once working, I collected kernel img, initrd gz, (and for testing the elf file), config.txt and cmdline.txt to a separate directory.  I then got a second SD card image working as desired using version Y of piCorePlayer.  Again, collected img, gz, elf and txt files to separate directory.  Cloned version X SD card.  Placed SD card in reader and added version Y img, gz (elf was identical at this point) files to fat partition.  Then I added config.txt and cmdline.txt as config.new and cmdline.new.  Then booted and version X started as expected.  Mounted fat partition and renamed *.txt to *.old.  Then renamed *.new to *.txt and rebooted.  Version Y booted as expected.

Now if you have high confidence in the new image structure, this leap of faith can work.  But if anything goes wrong there is no failsafe fallback.  Looks like berryboot and NOOBS have a "boot once" logic that can start a different OS one time without changing the default startup OS.  But as you indicated, the FAT partition starts to get crowded if you add more stuff in.  Is there any way to perform some manipulations in the txt files while in bootup?  I guess that's the reason for GRUB and the like.

This almost feels like the snake eating it's own tail.  I'll have to percolate on this a while and see if anything better bubbles out.
Title: Re: Building landing wheels while in flight
Post by: mcdudeabides on April 10, 2014, 05:29:04 AM
Fooled myself.  I was using similar packages (piCorePlayer) to test this out.  I thought if I saw the different version boot, then it must be working.  Unfortunately I didn't realize that the onboot.lst (which Paul123) pointed out and the directly mounted things like Oliweb would not be there if using different packages.  Certainly more to learn.  I was once told that when the student is ready the teacher will appear.  Where are you teacher?
Title: Re: Building landing wheels while in flight
Post by: Paul_123 on April 10, 2014, 07:59:28 AM
Now if you have high confidence in the new image structure, this leap of faith can work.  But if anything goes wrong there is no failsafe fallback. 

There is really no good fail safe for the upgrades we are talking about.    Fail Safe would be to have a backup of mydata.tcz........and a new image ready to burn....... of course pulling the SDcard is going to have to happen at this point.

In Steen's Case, I think the fail safe is less of an issue.   He is building a functional image, an then moving a configuration file over  (The config file will reside in the myData.tcz backup, which we do not over write).  He will just need to supply the individual archives for download.   This way we really only need to worry about verifying a MD5sum of the downloaded files before taking the leap.



Title: Re: Building landing wheels while in flight
Post by: sbp on April 10, 2014, 10:06:27 AM
Thanks for all your thoughts and suggestions.
So should we try to make such an update option for piCorePlayer?

I was thinking something like this.
Make an "Update piCorePlayer" option on the Main page.
When activated this will:
mount /dev/mmcblk0p1
copy the /usr/local/sbin/config.cfg file to /mnt/mmcblk0p1/newconfig.cfg

Then download the piCore.img and piCore.gz  - check their MD5sum, then delete the old piCore.img and the old piCore.gz files.
Download a new config.txt file containing the correct name and size of the piCore.img and piCore.gz files.
 
Next download the a zipped version of the TCE directory - check the MD5sum, then delete the old TCE directory and finally unzip the new TCE directory.

Then reboot, and when starting piCorePlayer will use the /mnt/mmcblk0p1/newconfig.cfg file to configure the new version of piCorePlayer with the settings from the old system (wifi, password, squeezelite settings, tweaks and overclock settings etc) - then delete the newconfig.cfg file. Then make a backup (to save the new config.cfg file) and we have an updated piCorePlayer using the old settings.

What do you think?

Steen
Title: Re: Building landing wheels while in flight
Post by: mcdudeabides on April 10, 2014, 07:45:43 PM
@Paul and Steen-
I agree with moving forward with some mechanism for upgrading without having to remove the SD card.  I do think that given the unique case that is piCorePlayer there is a way to accommodate upgrades with current session running shell script logic.  If there are serious architecture shifts for Tiny Core (or any dependent packages), then the upgrade process would become more disruptive (up to the point of full SD card rewrite).  There may need to be some deliberate change for the approach of someone starting new versus someone upgrading from prior start.  I would also anticipate a challenge in enabling/supporting the wide range of DAC cards.  Eventually there will be conflicts that prevent providing protocol stacks for all known DACs.  This would almost trigger a "TCE-like" environment where there are unique "packages" to load for the different DACs.  In any regard, I would be willing to volunteer efforts along the line of providing upgrades to the degree that we can control it.

I am also (quite selfishly) thinking of other embedded projects that would benefit from a remote upgrade approach.  The benefit of Tiny Core is that it is efficient, frugal and very embedded-friendly.  The downside is that it rapidly evolves and the upgrade method is now targeted to providing a new system image on a vanilla startup.  Thinking back to my process control days on Modcomp and Honeywell trigger a desire for a more robust deployment approach. 

Do you think it possible to include at least the "boot once" logic?  The goal being that a new image can be booted once, and if there is no way to confirm a positive execution (through user confirmation or script logic), then on power cycle the last "known good" image will be reconstituted and provide a recovery avenue?  The most elegant approach would provide the means at bootload (maybe like the berryboot VNC - and all it's hairy dependencies).  Perhaps there is something less elegant that will get most of the functionality.

In any event, I am willing to invest time in whatever approach is considered most viable and desired. 

Steen - this is really your creation.  Do you have any guidance?
Title: Re: Building landing wheels while in flight
Post by: sbp on April 10, 2014, 11:46:12 PM
@Paul and Steen-
I agree with moving forward with some mechanism for upgrading without having to remove the SD card.  I do think that given the unique case that is piCorePlayer there is a way to accommodate upgrades with current session running shell script logic.  If there are serious architecture shifts for Tiny Core (or any dependent packages), then the upgrade process would become more disruptive (up to the point of full SD card rewrite).  There may need to be some deliberate change for the approach of someone starting new versus someone upgrading from prior start.  I would also anticipate a challenge in enabling/supporting the wide range of DAC cards.  Eventually there will be conflicts that prevent providing protocol stacks for all known DACs.  This would almost trigger a "TCE-like" environment where there are unique "packages" to load for the different DACs.  In any regard, I would be willing to volunteer efforts along the line of providing upgrades to the degree that we can control it.

I agree that this will be a constant struggle to support all the new DACs that will be developed. However, for the near future I can't see that it will be possible to build "DAC packages". As it is now these new DACs require changes in the kernel and support for new DACs are added from time to time via a new kernel and modules.
For each major update of piCorePlayer I have build a new kernel and modules (in order to support more hardware and improve stability), so for piCorePlayer I think we often still need to supply our own kernel.img and module.gz. From time to time we are in sync with the official piCore development of bmarkus and sometime we are in front.
So we need to be able to download our own kernel.img and module.gz.
Therefore, we also need to be able to create and download the Kernel specific tcz packages (Alsa and wifi).   


I am also (quite selfishly) thinking of other embedded projects that would benefit from a remote upgrade approach.  The benefit of Tiny Core is that it is efficient, frugal and very embedded-friendly.  The downside is that it rapidly evolves and the upgrade method is now targeted to providing a new system image on a vanilla startup.  Thinking back to my process control days on Modcomp and Honeywell trigger a desire for a more robust deployment approach..

I'm not sure I completely understand.
But maybe piCorePlayer is not a good model for developing a generic in situ update system for piCore. The problem with piCorePlayer is that it often use its own kernel and module, therefore, the update system will be different compared to a system build on vanilla piCore and its modules.



Do you think it possible to include at least the "boot once" logic?  The goal being that a new image can be booted once, and if there is no way to confirm a positive execution (through user confirmation or script logic), then on power cycle the last "known good" image will be reconstituted and provide a recovery avenue?  The most elegant approach would provide the means at bootload (maybe like the berryboot VNC - and all it's hairy dependencies).  Perhaps there is something less elegant that will get most of the functionality.

In any event, I am willing to invest time in whatever approach is considered most viable and desired.

This sound good. However, I have no idea how to make it work.


Steen - this is really your creation.  Do you have any guidance?
Sorry - no, I think that I'm the one who knows the least when it comes to linux and programming.
Title: Re: Building landing wheels while in flight
Post by: Paul_123 on April 11, 2014, 09:09:59 AM
Integrating something like berry boot would seem rather daunting.

I would think that we would assume that the kernel would boot.....as that would be tested.   The biggest concern would be if there was an incompatibility with the upgrade.   The process could go something like this.

-If we do a firmware update, that is pretty unlikely to cause a problem......so as long as these files are checksumed, I would not plan on a way to go back.
-Install the new Kernel and initrd  (Keep the last copies)
-Install the new config and cmdline   (Keep the last copies)
-After boot.....if there is no interaction.....after a specified time delay, the config and cmdline files from the last system would be put back.

The problem going totally back is if we update extensions that are not kernel dependant........but then I suppose you could create one large piCorePlayer extension that is kernel dependant.....and has the kernel number in the file name.
Title: Re: Building landing wheels while in flight
Post by: mcdudeabides on April 11, 2014, 06:12:56 PM
I'm willing to prototype something this weekend.  I should have a few periods of downtime while I'm monitoring parent's treatments.  Just to get feet wet we can use the 1.13 to 1.14 as a test case.  There should be all of the needed files to do something that can demonstrate the concept.

 @Paul123 - both berryboot and NOOBS were very easy to get going and surprisingly small footprint.  Unfortunately the TCL architecture is closer to something like berryboot than it is to a target OS package.  I wish I understood the bootloader better.  It just seems that there should be a way to point it to one partition to load one image, and then within that running image call the bootloader with directions to start a second, different, partition.  In any event, the berryboot and one of the frugal OS installs will help satisfy two of my other interests.

 @Steen - I misused the term package in relation to the DAC cards.  I was not speaking of a tce-like package, but rather keeping all of the modules and boot logic in their own representative script.  Then, much like the ALSA screen, the web page could select the correct card (Wolfson, IQaudiO, etc.) and the modules/startup/config for just that DAC card would be activated.  It just seems to me that as more DAC cards are pursued in the future that troubleshooting is easier if you keep things compartmentalized.



Title: Re: Building landing wheels while in flight
Post by: sbp on April 12, 2014, 04:04:35 AM
I'm willing to prototype something this weekend.  I should have a few periods of downtime while I'm monitoring parent's treatments.  Just to get feet wet we can use the 1.13 to 1.14 as a test case.  There should be all of the needed files to do something that can demonstrate the concept.

Sorry to hear about your parents condition. Are they both seriously ill?.
Regarding your prototype, I'm looking forward to see that. Please let me know what changes I need to apply in the piCorePlayer web-GUI.



@Steen - I misused the term package in relation to the DAC cards.  I was not speaking of a tce-like package, but rather keeping all of the modules and boot logic in their own representative script.  Then, much like the ALSA screen, the web page could select the correct card (Wolfson, IQaudiO, etc.) and the modules/startup/config for just that DAC card would be activated.  It just seems to me that as more DAC cards are pursued in the future that troubleshooting is easier if you keep things compartmentalized.

OK I see.
I already have some of the DAC switching logic in separate scripts, but some of the code is still present elsewhere as well  - so it might be a good idea to make a standard recipe for any new DAC.  I will start to look into that. Thanks for the suggestion.

Steen
Title: Re: Building landing wheels while in flight
Post by: mcdudeabides on April 12, 2014, 05:50:33 PM
Age is catching up with both my parents. One on dialysis and the other in therapy for mini-strokes. They still wish to be independent as possible, so I try to accommodate them as much as possible.

I was able to get some code snippets on my laptop. I'm getting a clean RPi up to do some integration and testing after dinner tonight. Hopeful to have something soon.
Title: Re: Building landing wheels while in flight
Post by: mcdudeabides on April 13, 2014, 12:13:14 AM
There are still some warts and hairs to remedy, but  I thought I would share where I currently progressed.  As it stands now, you can call the script  with the name of one of the images on SourceForge (insitu.sh piCorePlayer1.14d.img for example).  It still has some debugging cutouts, so be prepared to hit enter to get it to progress.  I haven't had time to put any comments so far, so some of the things will look strange at first blush.

In a nutshell it will -
1) create a "promote directory" to hold the upgrade image.
2) download image specified on command line from SourceForge
3) add a filesystem to hold a copy of all the important bits
4) create a local "revert" directory to hold a copy of the current config for potential fallback
5) mount the promoted image as two directories ready to be copied
=====at this point it doesn't actually copy over the new image files so that you can exercise it without too much danger

Code: [Select]
#!/bin/sh
set +x

imgname=$1
sudo mkdir -m 777 /tmp/promote
wget -P /tmp/promote http://sourceforge.net/projects/picoreplayer/files/$imgname/download

sudo /usr/local/etc/init.d/dropbear stop
sudo /usr/local/etc/init.d/oliweb_initd stop
sudo /usr/local/etc/init.d/squeezelite_initd stop
sleep 2
read -r sqlpid < /var/run/squeezelitehf.pid
sudo kill $sqlpid
read -p "wait"




if [ ! "`fdisk -l | grep 0p3`" ]
then
        par3="n\np\n3\n"$((`fdisk -l | grep 0p2 | awk '{print $3}'`+1))"\n+256M\nw\n"
        sudo umount -a
        sudo echo -e $par3 | fdisk /dev/mmcblk0
else
        echo "Existing work partition.  Nonstandard.  Exiting"
        exit 4
fi

sudo hdparm -z /dev/mmcblk0
sudo mount -a
sleep 2
read -p "wait"

sudo mkdir -m 777 /tmp/revert
sudo mkfs.ext4 -t ext4 /dev/mmcblk0p3
sudo mount /dev/mmcblk0p3 /tmp/revert

sudo mount -t vfat /dev/mmcblk0p1 /mnt/mmcblk0p1
sudo cp /mnt/mmcblk0p1/*.img /tmp/revert
sudo cp /mnt/mmcblk0p1/*.gz /tmp/revert
sudo cp /mnt/mmcblk0p1/*.txt /tmp/revert
sudo cp /mnt/mmcblk0p1/start.elf /tmp/revert
sudo cp /mnt/mmcblk0p1/fixup.dat /tmp/revert
sudo mount /dev/mmcblk0p2 /mnt/mmcblk0p2
sudo cp /mnt/mmcblk0p2/tce/mydata.tgz /tmp/revert
sudo cp /mnt/mmcblk0p2/tce/onboot.lst /tmp/revert

p1offset=$(fdisk -l -u -C 52 /tmp/promote/download | grep FAT32 | awk '{print $2*512}')
p2offset=$(fdisk -l -u -C 52 /tmp/promote/download | grep Linux | awk '{print $2*512}')

sudo mkdir /tmp/par1
par1dir=$(losetup -f)
sudo losetup -o $p1offset -f /tmp/promote/download
sudo mount $par1dir /tmp/par1

sudo mkdir /tmp/par2
par2dir=$(losetup -f)
sudo losetup -o $p2offset -f /tmp/promote/download
sudo mount $par2dir /tmp/par2

exit 0

There is still some research to do on getting the recover logic sorted out.  The squeezelite init.d is not stopping it now and I need to find a way to better prevent that.  I'm also hoping there is a better way to read in the partition table.  Trying to remove contention to allow the kernel to reread the table leaves everything in a shakey state.  Of course the next thing that would happen after this script is a reboot anyway, so it may be tolerable.

In any event, take a look and tell me what you find.
Title: Re: Building landing wheels while in flight
Post by: Paul_123 on April 13, 2014, 07:15:34 AM
would it not be easier to just extend the partition and create a directory to hold the new image.

I agree deleting partitions get to be a problem.
Title: Re: Building landing wheels while in flight
Post by: mcdudeabides on April 13, 2014, 09:11:41 AM
would it not be easier to just extend the partition and create a directory to hold the new image.

I agree deleting partitions get to be a problem.
Hello Paul.  If I were doing this on a system that I intended to use every day, I would certainly agree with you.  In fact, when I was first trying to puzzle out the approach, I started with the idea to just resize the existing filesystem.  The primary objective for me was to use only what existed now (i.e. the .img file approach), respect the work that came before me, attempt to provide a known_good recovery and attempt to control it over one session.  I was unable to get the filesystem to grow in the current session.  I needed to reboot and the tricks to fool the system in to a onceboot kind of approach looked too painful.  It might have been a function of my prototyping environment, but the approach of adding a third partition seemed to allow me (with a few tricks) to do everything in one session. 

I am most certainly not married to this approach, but offer it as an early example of one way to accomplish the objective.  I think there are parts that could work in a number of variations on upgrading.  If you have some specific examples that would be improvements to offer up, I would be glad to incorporate or to support you in following the objective.
Title: Re: Building landing wheels while in flight
Post by: mcdudeabides on April 14, 2014, 05:42:27 AM
Here is a slightly updated version of insitu.sh with comments:
Code: [Select]
#!/bin/sh
set +x
#
# Upgrade in place a running piCorePlayer img from sourceforge
# Usage: insitu.sh file.img (i.e. piCorePlayer1.14d.img)
#

imgname=$1

# Grab the new image in RAM from sourceforge
sudo mkdir -m 777 /tmp/promote
if ! wget -P /tmp/promote http://sourceforge.net/projects/picoreplayer/files/$imgname/download
then
echo "Error retrieving" $imgname
exit 4
fi

# Stop the services that keep /dev/mmcblk0 busy
sudo /usr/local/etc/init.d/dropbear stop
sudo /usr/local/etc/init.d/oliweb_initd stop
sudo /usr/local/etc/init.d/squeezelite_initd stop
# Make sure they are gone (squeezelite down script currently throws error)
sudo kill $(ps|grep dropbear|grep root|awk '{print $1}')
sudo kill $(ps|grep oliweb|grep root|awk '{print $1}')
sudo kill $(ps|grep squeeze|grep root|awk '{print $1}')

sleep 2

read -p "wait"

# Create a new partition to store the existing piCorePlayer key files
if [ ! "`fdisk -l | grep 0p3`" ]
then
par3="n\np\n3\n"$((`fdisk -l | grep 0p2 | awk '{print $3}'`+1))"\n+256M\nw\n"
sudo umount -a
sudo echo -e $par3 | fdisk /dev/mmcblk0
else
echo "Existing work partition.  Nonstandard.  Exiting"
exit 4
fi

# Reread new partiion and make available for use
sudo hdparm -z /dev/mmcblk0
sudo mount -a
sleep 2
read -p "wait"
sudo mkdir /tmp/revert
sudo mkfs.ext4 -t ext4 /dev/mmcblk0p3
sudo mount /dev/mmcblk0p3 /tmp/revert

# Backup key existing files
sudo mount -t vfat /dev/mmcblk0p1 /mnt/mmcblk0p1
sudo cp /mnt/mmcblk0p1/*.img /tmp/revert
sudo cp /mnt/mmcblk0p1/*.gz /tmp/revert
sudo cp /mnt/mmcblk0p1/*.txt /tmp/revert
sudo cp /mnt/mmcblk0p1/start.elf /tmp/revert
sudo cp /mnt/mmcblk0p1/fixup.dat /tmp/revert
sudo mount -t ext4 /dev/mmcblk0p2 /mnt/mmcblk0p2
sudo cp /mnt/mmcblk0p2/tce/mydata.tgz /tmp/revert
sudo cp /mnt/mmcblk0p2/tce/onboot.lst /tmp/revert

# Calculate offsets in new img file to partitions
p1offset=$(fdisk -l -u -C 52 /tmp/promote/download | grep FAT32 | awk '{print $2*512}')
p2offset=$(fdisk -l -u -C 52 /tmp/promote/download | grep Linux | awk '{print $2*512}')

sudo mkdir /tmp/par1
par1dir=$(losetup -f)
sudo losetup -o $p1offset -f /tmp/promote/download
sudo mount $par1dir /tmp/par1

sudo mkdir /tmp/par2
par2dir=$(losetup -f)
sudo losetup -o $p2offset -f /tmp/promote/download
sudo mount $par2dir /tmp/par2

# Copy new key files to  mmcblk0
# Reboot

exit 0

I've used it with the half dozen older images of piCorePlayer that were on my machine (burned old .img to SD card, rebooted, loaded insitu.sh over and let it run with target of piCorePlayer1.14.img).  So far the only irritation is that squeezelite won't come down with the init.d script.  At this point the third partition is overkill at 256M.  If this is the way to go, then it may be better to scale it back.

So here is where I think this stands.  Steen has a successful and appealing approach for the burn&boot masses (no knowledge required, just burn SD card, boot your RPi and load URL).  If anybody grabs an IMG from sourceforge, their machine should boot standalone.  Keeps it simple and provides a worst case scenario disaster recovery.

There are those (like myself) that can't lay hands on their RPi to pull the SD and burn&boot.  The desire is to provide a means to upgrade a currently running version of piCorePlayer to a new version without touching the physical RPi.  Seems to me the approaches break down in to the following:

1) Provide failsafe upgrade -  (start known_good handler, backup old image, install new image, if new image fails to load then fallback known_good handler and revert old image)
Where it stands now - ugly berryboot proof of concept works, would be hairy for the masses.  Need a better idea (chain boot??)

2) Provide upgrade with revert - (backup old image, install new image, reboot and cross fingers)
Where it stands now - working prototype of insitu.sh here now.  Needs revert logic

3) Provide upgrade bare - (install new image, reboot and cross fingers)
Where it stands now - repurpose parts of insitu.sh

4) Rearchitect piCorePlayer - (develop a new image of piCorePlayer with new functionality/design/approach to allow for in place upgrades)
Where it stands now - need to develop and understand requirements

I'm ready to help in whatever direction you want to pursue.

PS - on a completely unrelated side note.  What in the !@#$@!# do I do to get scp working with putty/dropbear?  The old cut/paste a vi session is Neanderthal.

Title: Re: Building landing wheels while in flight
Post by: sbp on April 14, 2014, 08:11:23 AM
I've used it with the half dozen older images of piCorePlayer that were on my machine (burned old .img to SD card, rebooted, loaded insitu.sh over and let it run with target of piCorePlayer1.14.img).  So far the only irritation is that squeezelite won't come down with the init.d script.  At this point the third partition is overkill at 256M.  If this is the way to go, then it may be better to scale it back.

Thanks, I will give it a try. It will be a major advantage to be able to update piCorePlayer in situ.

Will it be easier if I make a special directory on sourceforge called update, where only the newest piCorePlayer version is?
Can there be some kind of check, so that it only tries to update if there is a newer version than what is currently running?


So here is where I think this stands.  Steen has a successful and appealing approach for the burn&boot masses (no knowledge required, just burn SD card, boot your RPi and load URL).  If anybody grabs an IMG from sourceforge, their machine should boot standalone.  Keeps it simple and provides a worst case scenario disaster recovery.

There are those (like myself) that can't lay hands on their RPi to pull the SD and burn&boot.  The desire is to provide a means to upgrade a currently running version of piCorePlayer to a new version without touching the physical RPi.  Seems to me the approaches break down in to the following:

1) Provide failsafe upgrade -  (start known_good handler, backup old image, install new image, if new image fails to load then fallback known_good handler and revert old image)
Where it stands now - ugly berryboot proof of concept works, would be hairy for the masses.  Need a better idea (chain boot??)

2) Provide upgrade with revert - (backup old image, install new image, reboot and cross fingers)
Where it stands now - working prototype of insitu.sh here now.  Needs revert logic

3) Provide upgrade bare - (install new image, reboot and cross fingers)
Where it stands now - repurpose parts of insitu.sh

4) Rearchitect piCorePlayer - (develop a new image of piCorePlayer with new functionality/design/approach to allow for in place upgrades)
Where it stands now - need to develop and understand requirements

I'm ready to help in whatever direction you want to pursue.

Thanks. I think we should continue (we I said - you have done all the work).
Option 2 would be OK. I was thinking that if we are able to boot after a update and can see squeezelite is running, then everything should be OK - and we could delete the old files. This kind of logic should be rather simple to make.
However, in the worst case scenario we are unable to boot, and then nothing can be done - we will then need to burn the SD-card with a new image? I don't know if this can be handled in any other way, but I doubt it.

Also piCorePlayer would need an overhaul in order to make this happen - but first we need to decide on the mechanism. 


PS - on a completely unrelated side note.  What in the !@#$@!# do I do to get scp working with putty/dropbear?  The old cut/paste a vi session is Neanderthal.

I use putty and WinSCP all the time without any problems?
It complains about saving is not permitted - but it saves the files anyway.

Steen
Title: Re: Building landing wheels while in flight
Post by: mcdudeabides on April 14, 2014, 07:02:07 PM
Hello Steen.  I'll take a first pass at the revert logic. 

Will it be easier if I make a special directory on sourceforge called update, where only the newest piCorePlayer version is?
Can there be some kind of check, so that it only tries to update if there is a newer version than what is currently running?

I would suggest that you put a file with a consistent name on sourceforge.  It can be called something like "ProductionVersion.img".  The file could contain at a minimum the latest version text string as is seen in the piversion.cfg in local/sbin.  A quick comparison of the downloaded version and the one in local piversion.cfg would allow to verify an upgrade exists.  Then the name would be fed to insitu to retrieve.  If needed there could also be an auxiliary script that could be executed before upgrade to allow for any "tweaks" that weren't thought about when the previous image delivered.  It should probably be something explicitly triggered from the web interface (probably button press on tweaks menu to look for upgrade).  I don't think sourceforge would want to be hammered every time a RPI restarts.


Quote
Thanks. I think we should continue (we I said - you have done all the work).
Option 2 would be OK. I was thinking that if we are able to boot after a update and can see squeezelite is running, then everything should be OK - and we could delete the old files. This kind of logic should be rather simple to make.
However, in the worst case scenario we are unable to boot, and then nothing can be done - we will then need to burn the SD-card with a new image? I don't know if this can be handled in any other way, but I doubt it.
Worst case scenario would be for the burn&boot crowd to load a new image fresh, then configure again (unless they saved config on USB).  The more adventurous could load the SD card and edit cmdline.txt external to try base or norestore bootcodes and see if they can get back to the revert logic.

Quote
Also piCorePlayer would need an overhaul in order to make this happen - but first we need to decide on the mechanism. 
Now would be the time to make any adjustments to the piCorePlayer layout.  It is still the same image file and can be loaded by burn&boot or insitu.

Quote
I use putty and WinSCP all the time without any problems?
It complains about saving is not permitted - but it saves the files anyway.
I must have something gorked on my windows machine.  Will try a different box.
Title: Re: Building landing wheels while in flight
Post by: sbp on April 15, 2014, 01:34:04 AM

I would suggest that you put a file with a consistent name on sourceforge.  It can be called something like "ProductionVersion.img".  The file could contain at a minimum the latest version text string as is seen in the piversion.cfg in local/sbin.  A quick comparison of the downloaded version and the one in local piversion.cfg would allow to verify an upgrade exists.  Then the name would be fed to insitu to retrieve.  If needed there could also be an auxiliary script that could be executed before upgrade to allow for any "tweaks" that weren't thought about when the previous image delivered.  It should probably be something explicitly triggered from the web interface (probably button press on tweaks menu to look for upgrade).  I don't think sourceforge would want to be hammered every time a RPI restarts.

That is fine.
There is already such a file. "piCorePlayer_last.txt" it contains only the name of the newest piCorePlayer and is used by SqueezePlug so that SqueezePlug users are able to download the newest version of piCorePlayer from inside SqueezePlug.
So this file could be used for our purpose as well - to check the name of the newest version and to get the name of the newest piCorePlayer which then could be downloaded.

Steen
Title: Re: Building landing wheels while in flight
Post by: Paul_123 on April 15, 2014, 06:45:12 AM
making some good progress.....I just don't have as much free time to help out.   

As I looked through your script, I would make sure that you do a MD5 checksum on the downloaded image.    Much like the logic that is done in tce.   Sure the script should crash when trying to copy the image.....but that is a bit far in the process to catch the problem.
Title: Re: Building landing wheels while in flight
Post by: bmarkus on April 15, 2014, 07:56:03 AM
Guys, really interesting topic. Just one thought. piCore has a roboust and well working update mechanism and tool set. Why not use it? Either we can add your components as standard tcz extensions to the repo or you can make your own online repo just with player components.  Maybe a private repo would fit better to your environment in case of custom kernel, but as RPi kernel is getting matured custom kernel can is not necessary. This would make life easier.
Title: Re: Building landing wheels while in flight
Post by: mcdudeabides on April 15, 2014, 09:34:12 AM
You had me at "make life easier".  I can search for making the tcz extensions and get some education.  How to do the private repo?  Hosting issues (space, bandwidth, etc.)?

PS - this may be OT, but is there a way with TC to boot a complete new instance one-time from a currently running instance without mucking up the existing cmdline.txt args?  I'm thinking something along the lines of "RPi GPU is loaded/initialized, I have a one-time kernel.img/initrd combo that I want to start with my own bootcode args (base, norestore, etc.), have my one-time bootlocal.sh do some specific things, then call for full reboot back to the initial starting point".  It's not as easy as just calling sbin/init again is it?
Title: Re: Building landing wheels while in flight
Post by: bmarkus on April 15, 2014, 10:09:39 AM
You had me at "make life easier".  I can search for making the tcz extensions and get some education.  How to do the private repo?  Hosting issues (space, bandwidth, etc.)?

Regarding extensions, see WIKI article http://wiki.tinycorelinux.net/wiki:creating_extensions

On repo, better to call it custom repo as private repo is discussed in the WIKI (obsolate BTW) with different goal. Such repo is a simple static WEB content with a specific directory structure. No PHP, no JAVA, ... It holds the tcz extensions with their support files like .info .list .md5.txt .tree .dep .zsync plus few repo support files holding tags, md5 and files provided by tcz's. Space depends on extensions but can't be a problem today for even the cheapest hosting site, bandwidth depends on users, update frequency, etc. But low.
Title: Re: Building landing wheels while in flight
Post by: bmarkus on April 15, 2014, 10:20:08 AM
PS - this may be OT, but is there a way with TC to boot a complete new instance one-time from a currently running instance without mucking up the existing cmdline.txt args?  I'm thinking something along the lines of "RPi GPU is loaded/initialized, I have a one-time kernel.img/initrd combo that I want to start with my own bootcode args (base, norestore, etc.), have my one-time bootlocal.sh do some specific things, then call for full reboot back to the initial starting point".  It's not as easy as just calling sbin/init again is it?

You can update all of your components on a running system, including adjusting initrd size in cmdline.txt keeping all of your sections but at the end you need a real physical reboot due to tcz's updated which are in use (loop mounted, see how extension update is done in TC) and kernel version change and changing RPi firmware. All can be scripted.
Title: Re: Building landing wheels while in flight
Post by: mcdudeabides on April 15, 2014, 10:39:58 AM
You can update all of your components on a running system, including adjusting initrd size in cmdline.txt keeping all of your sections but at the end you need a real physical reboot due to tcz's updated which are in use (loop mounted, see how extension update is done in TC) and kernel version change and changing RPi firmware. All can be scripted.

Thanks for quick reply and insight.  I was able to follow your logic and could manipulate startup as you describe when trying to puzzle out solution earlier.  I'm really looking for a way to revert failsafe if upgrade image hiccups (thinking as wifi,DACs, etc. become more intertwined, a good test image in lab could SNAFU in deployment).  Something along the lines of pseudo-logic:
1) start current image
2) download known-good image (a bare-bones, guaranteed to run, image)
3) download upgrade-image
4) save current image to revert-image
5) setup environment to boot known-good (can do so using your logic)
6) boot known-good with upgrade
7) perform upgrade-image
=== and this is where it gets tricky ====
8) boot once upgrade-image (don't change known-good startup)
9) if this boot is successful, change startup to always boot upgrade-image
otherwise reboot (probably power cycle) to known-good with auto revert to revert-image

This is probably a wish.  Most likely to do this would require puzzling out a custom bootloader that could make a decision branch on kernel/initrd combo to trigger.
 
Title: Re: Building landing wheels while in flight
Post by: mcdudeabides on April 15, 2014, 10:47:04 AM
Regarding extensions, see WIKI article http://wiki.tinycorelinux.net/wiki:creating_extensions

On repo, better to call it custom repo as private repo is discussed in the WIKI (obsolate BTW) with different goal. Such repo is a simple static WEB content with a specific directory structure. No PHP, no JAVA, ... It holds the tcz extensions with their support files like .info .list .md5.txt .tree .dep .zsync plus few repo support files holding tags, md5 and files provided by tcz's. Space depends on extensions but can't be a problem today for even the cheapest hosting site, bandwidth depends on users, update frequency, etc. But low.

Thanks for advice.  I'll defer to Steen on the approach he wants to pursue.  There is indeed elegance from the TCL world in using the extension system to provide all the builtin goodness of packages.  However there is a simplicity to the sourceforge SD card image approach.  The burn&boot masses have demonstrated ability.  It's kind of a chicken/egg dilemma.  There needs to be a working piCorePlayer image to be able to utilize the extension system for upgrades.  So far burning the SD card image seems to get the traction.  It's certainly worth puzzling over in my free time.
Title: Re: Building landing wheels while in flight
Post by: bmarkus on April 15, 2014, 11:22:31 AM
No chicken-egg paradigm. Simple .img is great to start system without complications nut once it is up and running you need a roboust and safe update mechanism. You can hide internals from users.
Title: Re: Building landing wheels while in flight
Post by: mcdudeabides on April 16, 2014, 09:46:29 AM
Hello again Bela.  Please forgive me if I'm being obtuse, but I am challenged in understanding how to apply the extensions concept.  If I were to put myself in Steen's shoes, then I would see piCorePlayer as the sum of the following parts:
1) the base TC kernel image (retrieved by direct download and burn)
2) the base support libraries (most likely the things found in a "full" Linux deploy, loaded by tcz- things like alsa, flac, ffmepeg, etc.)
3) support utilities (most likely things to extend Linux functionality, loaded by tcz- things like wireless, i2c tools, firmwares, etc.)
4) support applications (loaded by tcz, things like dropbear, perl, etc.)
5) third party applications (ready for ARM, not tcz packaged, requires custom integration - things like OliWeb, squeezelite)
5) end user support (the scripts, web pages, config stores that "are" piCorePlayer - custom created, integrated)

If I were the builder of the next flavor of the piCorePlayer image (assuming no kernel update), I could see the convenience of performing a tce-update (I guess this would have to be base norestore??) to refresh all the extensions.   The third party applications currently installed would be untouched.  Most likely I would then be making changes to the end user support environment (e.g. - troubleshooting new DAC shows new module and modprobe needed, base script modified).  I'm assuming at this point that all the changes for end user support would be packaged and become their own unique tcz? Is this package something that should be submitted to TC or is it in the "custom repo"?

Assuming the above, if I now try to think from the perspective of the end user.  I have a running instance of PiCorePlayer.  There is a button on the web page to "update" (assume no kernel update at this time).  Can I then call the tce-update and have it download and install the changes?  Is there a way in a running instance to drop all of the current extensions to allow a reload (could brute force all the LOOP devices)? I guess after this a reboot will be required.

I can see the application in going through minor iterations (e.g. 1.14a to 1.14b to 1.14c etc.) using tce-update.  I don't see it applying to kernel updates, or is that also something could work?

Forgive the ramblings, I had a quick window of time to ask the questions and wanted to get them out there.



Title: Re: Building landing wheels while in flight
Post by: bmarkus on April 17, 2014, 12:48:23 AM
Assuming the above, if I now try to think from the perspective of the end user.  I have a running instance of PiCorePlayer.  There is a button on the web page to "update" (assume no kernel update at this time).  Can I then call the tce-update and have it download and install the changes?  Is there a way in a running instance to drop all of the current extensions to allow a reload (could brute force all the LOOP devices)? I guess after this a reboot will be required.

You can run the tce-update script from the WEB application similarly as the factory GUi front-end does, check its source code. Update is placing new files to tce/optional/backup direcotry. It is checked by the system during startup and moves them to tce/optional before mounting them. You can pack your custom apps not in the main piCore repo to TCZ and submit to there. You can use a secondary repo also, just change the URL to your own online repo and do update in two steps. TC is a toolkit, you have a freedom to setup your system.

What is not covered yet is the base system update (kernel, initrd and firmware). It is in progress at my side. Hope to share in the near future for testing.
Title: Re: Building landing wheels while in flight
Post by: mcdudeabides on April 17, 2014, 09:25:53 AM
Thanks Bela.  I appreciate your tolerance for my continued education.  I've been looking inside the tce* scripts to attempt to grasp the concepts better.  Slowly it is sinking in.  Most of my puzzling in done offline with simulation in my laptop.  I'll need to find a way to set up in such a way that it can be exercised in isolation.

@Steen - To me, it seems there would be value in packing the OliWeb and Squeezelite applications for tcz install.  Squeezelite is in git already.  From scratching the surface, it looks like the tce tools provide for going all the way to source compile if needed (or just distribution/config of binary).  I have a sense that there may be some delay to the next iteration of piCorePlayer while exploring this philosophy.  It is not my intent to hinder progress.  To that extent, do you have advice on areas that would advance the project in which I could be useful?
Title: Re: Building landing wheels while in flight
Post by: Paul_123 on April 17, 2014, 09:59:44 AM
Are you using the direct binary from triode for squeezelite, or do you have a self compiled version?   What are the compile flags?    Seems the extension route is the way to go.    It may be best to just plan this route.   If Bela is working on the kernel/base upgrade system, then perhaps that may be a better to use what he develops.
Title: Re: Building landing wheels while in flight
Post by: mcdudeabides on April 17, 2014, 11:34:14 AM
Are you using the direct binary from triode for squeezelite, or do you have a self compiled version?   What are the compile flags?    Seems the extension route is the way to go.    It may be best to just plan this route.   If Bela is working on the kernel/base upgrade system, then perhaps that may be a better to use what he develops.

Hello Paul.  You seem to have valuable experience and opinion in this area.  Consider me your willing student and let us use squeezelite as the tutorial.  Assuming that I have a clean install of piCore, with tcztools installed, what are the initial steps I should follow to start this process of developing an extension for squeezelite?
Title: Re: Building landing wheels while in flight
Post by: Paul_123 on April 18, 2014, 09:48:27 AM
Hello Paul.  You seem to have valuable experience and opinion in this area.  Consider me your willing student and let us use squeezelite as the tutorial.  Assuming that I have a clean install of piCore, with tcztools installed, what are the initial steps I should follow to start this process of developing an extension for squeezelite?

The process of packing a extension is laid out in steens picore thread.    http://forum.tinycorelinux.net/index.php/topic,14634.msg89051.html#msg89051

That is for a kernel specific extension, but if you follow that as an example, extract an extension and look at the layout.   From that you can see how to create your own extension.

squeezelite itself is a single executable.   But it has dependencies to specific versions of libraries depending on who compliled it, so you will want to make sure to build a .dep file.    Dependencies can be an issue, for example.......libavcodec is one that is a big issue, as the dependencies for this on the tce repo are a mile long, when all we really want is this single library.   So we may want to consider including all of the required libraries in the squeezelite extension.   Or we can build each library into it's own extension and then just use a private Repo (I think the private Repo issue needs to be solved sooner, rather than later)

I use a version of squeezelite that I compiled on piCore, using the versions of libraries that are available on piCore.    My compile linker options are -DFFMPEG -DRESAMPLE   (I don't use -DDSD or -DVIS EXPORT)
Title: Re: Building landing wheels while in flight
Post by: bmarkus on April 18, 2014, 10:35:53 AM
Dependencies can be an issue, for example.......libavcodec is one that is a big issue, as the dependencies for this on the tce repo are a mile long, when all we really want is this single library.   

There is nothing against to add a minimal version of libavcodec to the official repo. squeezelite is also welcome in the repo as well as oliweb as a kind contibution to the piCore community :)
Title: Re: Building landing wheels while in flight
Post by: mcdudeabides on April 18, 2014, 06:07:15 PM
Thanks both for your advice. I will seek out Steen's posts that Paul indicated. Slowly I'm beginning to grasp the idea.

Sent from my Nexus 7 using Tapatalk

Title: Re: Building landing wheels while in flight
Post by: Paul_123 on April 19, 2014, 02:07:35 PM
Dependencies can be an issue, for example.......libavcodec is one that is a big issue, as the dependencies for this on the tce repo are a mile long, when all we really want is this single library.   

There is nothing against to add a minimal version of libavcodec to the official repo. squeezelite is also welcome in the repo as well as oliweb as a kind contibution to the piCore community :)

It is the identical version, libavformat itself has no dependencies.   It is just that it is a part of ffmpeg.   I assume that when you build the ffmpeg extension, libavformat gets the same dependencies.   This also happens with libavcodec and libavutil
Title: Re: Building landing wheels while in flight
Post by: bmarkus on April 19, 2014, 02:30:47 PM
Check current ffmpeg2 components. libavformat2, libavcodec2, ... each have its own real dependency list. For piCorePlayer video codecs are not needed I guess. This is where you can reduce dep list.
Title: Re: Building landing wheels while in flight
Post by: mcdudeabides on April 19, 2014, 05:50:50 PM
Slow going for me. I need to set up an environment outside the RPi to work with.

Aren't the AV libs used for FLAC decoding? 


Much to learn.
Title: Re: Building landing wheels while in flight
Post by: Paul_123 on April 19, 2014, 06:15:54 PM
Slow going for me. I need to set up an environment outside the RPi to work with.

Aren't the AV libs used for FLAC decoding? 


Much to learn.

They are for windows media decoding.

Why not develop on a RPi? 
Title: Re: Building landing wheels while in flight
Post by: mcdudeabides on April 19, 2014, 06:41:26 PM
I have primary care responsibilities for my parents. Most of my down time is during dialysis or other treatments. I usually take a laptop and use that time to puzzle things out.

I do have a spare RPi. I can try that approach tomorrow.

Sent from my Nexus 7 using Tapatalk

Title: Re: Building landing wheels while in flight
Post by: sbp on April 20, 2014, 03:14:55 AM
Hi Poul, Randy and Bela.

Sorry for not being around for some time - but I have read this thread with interest, and I think we should try to see if we could make a piCorePlayer which can be upgraded in situ.

As I see it there are a few issues/decisions we have to make before we continue.
As it is now Oliweb, Squeezelite and the piCorePlayer web pages are placed at mnt/mmcblk0p2/tce... When we build tcz packages we should agree on a correct location.
I would suggest the following, however I'm open for a discussion:

For OliWeb /usr/local/sbin/Oliweb/
For Squeezelite /usr/local/sbin/squeezelite/
For piCorePlayer web pages /usr/local/sbin/Oliweb/ piCorePlayer/
For piCorePlayer SHH scripts /usr/local/sbin/piCorePlayer

In addition, piCorePlayer can use two versions of squeezelite - the one provided by Triode, which does not allow playing wma and the use of sox resampling. It is simply downloaded from Triodes google site.
The other is provided by Ralphy from the Squeezebox forum, it depend upon ffmpeg (which also is provided by Ralphy, and as you can see it is already packed into a tcz) also he provided libfaad.tcz and libsoxr.tcz which is used by this special version of piCorePlayer. I know he tested several versions of ffmpeg before finding a good one for our purpose.
We need to decide whether we should make these tcz packages as well, or continue just downloading the two different squeezelite versions (Trodes or Ralphys)?

Finally, I think we should wait and see what Bela have been working on in regard to the update mechanism of the kernel and base, and hopefully we can adapt this to piCorePlayer.

Steen

EDIT: We will need to be able to build and provide our own alsa-modules-<KERNEL NAME>.tcz
and wireless-<KERNEL-NAME>.tcz
for two reasons:
1. Sometimes we will be ahead of the piCore kernel.
2. I need to change the bcm2835-pcm.c before building the sound modules in order to allow 192 khz via HDMI.
Title: Re: Building landing wheels while in flight
Post by: bmarkus on April 20, 2014, 05:02:29 AM
EDIT: We will need to be able to build and provide our own alsa-modules-<KERNEL NAME>.tcz
and wireless-<KERNEL-NAME>.tcz
for two reasons:
1. Sometimes we will be ahead of the piCore kernel.
2. I need to change the bcm2835-pcm.c before building the sound modules in order to allow 192 khz via HDMI.

@Steen

Probably we can include your patches in the mainstream piCore kernel. In such case you do not have to create another version and also users of piCore can enjoy benefits. Lets discuss it in PM.

Title: Re: Building landing wheels while in flight
Post by: Paul_123 on April 20, 2014, 07:49:22 AM
We need to decide whether we should make these tcz packages as well, or continue just downloading the two different squeezelite versions (Trodes or Ralphys)?

No reason to keep 2 versions, there is not much changing right now.  I would just use Ralphy's version.   For those not resampling or wma, it does not get in the way.

I should probably get a Pi Running piCoreplayer.......so I can see all of the software versions.     I just like to do things myself.  plus I have a modded squeezelite to turn my amp off and on using GPIO......  I also have a realtime clock chip and digital clock display for an alarm clock.   Then there is push button control of the player using a python program and the LMS command line interface.   The extra modules I need means I have to stick with official piCore, or build my own kernel.
Title: Re: Building landing wheels while in flight
Post by: mcdudeabides on April 20, 2014, 11:01:37 PM
I see I'm late to the party:-)  And still trying to learn the right dance moves.

IF you can still spare the time to point in the right way, I could use help with some answers.  And if there is a tutorial out there or if you would be willing to share your approach and how you set it up, that would be most appreciated as well.

I've set up a RPi at Paul's suggestion to use in this exercise.  I download the tcetools.tcz from the google code site (is this the correct tool for the RPi environment).  Following their instructions, I began the exercise of setting up a working directory that would be processed to create the tcz for squeezelite.  I started with the idea of just downloading the binary (didn't see the conversation on Ralpy's version as default, but would certiainly agree).  I was trying to put the environment together manually to run squeezelite and thus create the package from it. 

Getting the binary with wget was no problem.  I could also see where to put things like the init.d script.  In working on the deps, I was able to see alsa.tcz, flac.tca, labmad.tca and libvorbix.tcz to satisfy most of the dependancies.  From Steen's picoreplayer image, it seemed that libffmpeg, libfaad, libsoxr and libmad are also required.  I couldn't find them in tce.  How do you know they are required and how did you find them?

Just trying to list the output device in squeezelite shows that I've most likely overlooked something for alsa as even the sysdefault is absent.  I know I need to do some investigation.  Is there something special about amixer in this environemnt?  I notice that Steen uses amixer set PCM 400 unmute.  I would have expected alsamixer to be available to me, but it also is not working right.  Most likely my own cause.

I think I'm slowly understanding the convolutions to restore squash images back on loop devices for the persistence.  I can see the uphill climb to get squeezelite working.  What did your environment look like when you first put this pack together?  Did you have a working directory that held all the files before rolling up the tcz?  Did you do a "squeezelite-get" script or any of the build scripts for the package?
Title: Re: Building landing wheels while in flight
Post by: mcdudeabides on April 20, 2014, 11:16:51 PM
As it is now Oliweb, Squeezelite and the piCorePlayer web pages are placed at mnt/mmcblk0p2/tce... When we build tcz packages we should agree on a correct location.
I would suggest the following, however I'm open for a discussion:

For OliWeb /usr/local/sbin/Oliweb/
For Squeezelite /usr/local/sbin/squeezelite/
For piCorePlayer web pages /usr/local/sbin/Oliweb/ piCorePlayer/
For piCorePlayer SHH scripts /usr/local/sbin/piCorePlayer

It would follow the conventions that seem to be in place so far.  I might suggest that you keep the user configuration information separate from the script directory.  Much as wifi.sh keeps its wifi.db in home, there might be a similar approach to keep config.cfg there as well. 

Quote
In addition, piCorePlayer can use two versions of squeezelite - the one provided by Triode, which does not allow playing wma and the use of sox resampling. It is simply downloaded from Triodes google site.
The other is provided by Ralphy from the Squeezebox forum, it depend upon ffmpeg (which also is provided by Ralphy, and as you can see it is already packed into a tcz) also he provided libfaad.tcz and libsoxr.tcz which is used by this special version of piCorePlayer. I know he tested several versions of ffmpeg before finding a good one for our purpose.

Now I see where the extra deps come from :-)  Seems that a good approach would be to select Ralphy's image as default and add the deps in to the repository as well.  Is it good practice to get the binaries or to develop the build logic to compile locally and keep in repository maintenance (if/when kernel updates drive recompile)?

Quote
We need to decide whether we should make these tcz packages as well, or continue just downloading the two different squeezelite versions (Trodes or Ralphys)?
I would agree that Ralphy's is best suited.

Quote
Finally, I think we should wait and see what Bela have been working on in regard to the update mechanism of the kernel and base, and hopefully we can adapt this to piCorePlayer.
Certainly agree. 

How can I be helpful/useful in this objective?  (and "stay out of the way" is a viable answer)
Title: Re: Building landing wheels while in flight
Post by: sbp on April 22, 2014, 12:21:35 PM
Quote
How can I be helpful/useful in this objective?  (and "stay out of the way" is a viable answer)

Hi Randy - I will be very happy if you would help with this development of piCorePlayer.

I just noticed that bmarkus posted about the in situ update of the firmware and kernel - have you tried it?

I really don't know what would be the best way to go - maybe take it step by step.

First incorporate the kernel and firmware update by bmarksu into piCorePlayer - an Update button on the "Main" web-page.
Then maybe do a "crude and dirty" update of piCorePlayer and all its dependencies, by simply downloading a new TCE directory (which contain all the tcz packages , OliWeb and piCorePlayer web-pages)

When we have this working, start to make the individual tcz packages in order to do the update the "proper" way?

Steen
Title: Re: Building landing wheels while in flight
Post by: Paul_123 on April 22, 2014, 01:43:56 PM
I just noticed that bmarkus posted about the in situ update of the firmware and kernel - have you tried it?

All there is public at this point is a tarball of the boot partition......but it does not update kernel modules that are in extensions.   SO it is not yet ready for a test.    The theory of a tarball of the boot partition is fine, as that is how I update anyway.

Again, you have to download the new kernel modules extensions first, before update the kernel and reboot.

Title: Re: Building landing wheels while in flight
Post by: Paul_123 on April 22, 2014, 01:52:58 PM
Getting the binary with wget was no problem.  I could also see where to put things like the init.d script.  In working on the deps, I was able to see alsa.tcz, flac.tca, labmad.tca and libvorbix.tcz to satisfy most of the dependancies.  From Steen's picoreplayer image, it seemed that libffmpeg, libfaad, libsoxr and libmad are also required.  I couldn't find them in tce.  How do you know they are required and how did you find them?

We know the dependencies base on the compile options.   If you run the squeezelite -help and it will show you the compile time options.

Quote
I think I'm slowly understanding the convolutions to restore squash images back on loop devices for the persistence.  I can see the uphill climb to get squeezelite working.  What did your environment look like when you first put this pack together?  Did you have a working directory that held all the files before rolling up the tcz?  Did you do a "squeezelite-get" script or any of the build scripts for the package?

I've only built a few extensions, so I have always done it manually.    And yes, I just keep a working directory.
Title: Re: Building landing wheels while in flight
Post by: sbp on June 28, 2014, 02:34:31 AM
Hi

Thanks for your suggestions, and we have now been able to build such a in-situ update system into piCorePlayer. (I got help from Randy and Greg for testing and developing this - so thank you very much)

You are now able to update the whole microcore and piCorePlayer via the web-GUI in piCorePlayer. You choose which version you want to use via a drop-down list of available versions. This is found on the "Main" web-GUI web page.

The system will then download and update your system, and after a reboot you will have a new version running, and using your old settings, which have been saved in your config file.

Steen

Please test version piCorePlayer1.16:  https://sites.google.com/site/picoreplayer/home/news
Title: Re: Building landing wheels while in flight
Post by: Paul_123 on June 30, 2014, 06:52:40 AM
Hi

Thanks for your suggestions, and we have now been able to build such a in-situ update system into piCorePlayer. (I got help from Randy and Greg for testing and developing this - so thank you very much)


Very cool.