Concatenate initramfs on Windows

[sodface]

I was goofing around tonight with picore 6.0 and made a small initramfs with just the w1 kernel module tcz in it (~62k).  As an experiment, on my windows laptop I wrote the picore 6.0 image to sd card as usual.  I moved the 6.0.gz file off the card into a directory on my laptop where I had my extra initramfs file (my_initramfs.gz).  Then I renamed 6.0.gz to 6.0.gz.orig and tacked the two together to create a new 6.0.gz:

Code: [Select]

copy /b 6.0.gz.orig + my_initramfs.gz 6.0.gz
Then I moved the 6.0.gz back onto the sd card and inserted into pi and booted... it booted fine and the contents of my_initramfs.gz were in the root file system where expected (I built my_initramfs.gz to extract to /tmp/builtin/optional).

I didn't have to edit any boot files or anything else other than combine the two initramfs files and replace the original.  Not sure what the use case is but I was pleasantly surprised that it worked first try.

[sodface]

Maybe this is kind of a pointless thread but anyway, just to see if I could get it working, I built another version of my_initramfs.gz with the following extensions for wifi and ssh:

libiw.tcz
libnl.tcz
ncurses.tcz
openssh.tcz
openssl-1.0.1.tcz
readline.tcz
wifi.tcz
wireless-3.12.36-piCore+.tcz
wireless_tools.tcz
wpa_supplicant.tcz

Connected once to my WLAN to create the wifi.db, updated bootlocal.sh to start wifi on boot etc. and then did filetool.sh -b to get a good mydata.tgz.

Then formatted an SD card with FAT32, used 7zip to copy all the boot files out of the piCore 6.0 .img file and onto the sd card.  Combined the 6.0.gz and my_initramfs.gz into a new 6.0.gz as in previous post and replaced the original on the sd card.  Placed a copy of mydata.tgz on the sd card also.

Pi booted fine, loaded extensions, restored mydata backup, connected to wifi and I was able to ssh to it.  SD card has only the one fat partition that gets unmounted after boot.  I guess this is some form of a "remaster", not sure if there's anything new or helpful here.

[bmarkus]

Well done, thanks for sharing. For sure it helps other remastering piCore.

[sodface]

I picked up a pi zero w at Micro Center while we were camping near Atlanta.  I didn't mess with it until now.  I knew that the USB and HDMI ports were a different form factor but I didn't buy any adapters at Micro Center (even though I knew I didn't have any at home) so I decided to try and get up and running on wifi without the benefit of video, keyboard, or serial access on the pi, but rather just customizing the sd card on my windows laptop.

I came close, but I didn't know until just now.

Continuing what I did previously in this thread, I downloaded picore 9.0.3 and:

- used 7zip to extract the picore img file.
- downloaded extensions needed for wifi
- installed cpio for windows
- created the directory structure /tmp/builtin/optional with extensions in the optional dir
- created onboot.lst and put it one dir up in /tmp/builtin
- created a bare format file list with:

Code: [Select]

dir /s /b > dir.txt- edited the dir.txt to change it from absolute paths to relative path starting with tmp
- fed the path list into cpio and then into 7z with gzip compression via command line:

Code: [Select]

type dir.txt | cpio -H newc -o | 7z a -tgzip -si my_initramfs.gz- tacked the original picore initramfs and my extra one together to create the new one:

Code: [Select]

copy /b 9.0.3_orig.gz + my_initramfs.gz 9.0.3.gz- formatted an sd card with a 100mb fat32 partition and copied all the extracted picore files there but used my custom 9.0.3.gz
- I had a mydata.gz from before that had my wifi.db and stuff in it so I put that in the root of the sd card also

I put the sd card in and powered up the pi zero w.  I had a green activity light but no idea what was really happening.  I was hoping to see a new dhcp client address show up in my router but alas, nothing.

So I gave up and ordered an adapter kit from Amazon and also a serial cable,  I got the adapter kit tonight (serial cable tomorrow) so I hooked it up to the TV to see what was going on.  The only mistake I made was the direction of the slashes in my dir.txt file (rookie!)  I could see right away that my extensions were being extracted into the root and their filenames were the full paths with windows style back slashes in them.  So I went back to my laptop and just did a replace in dir.txt changing the direction of the slashes to be linux style forward slashes.  The cpio command line above worked fine still and then I moved the card back to the pi and it came right up on wifi.

So I was close 

Sounds kind of convoluted I know but I think it's kind of cool to be able to prep the sd card completely from Windows and have it actually work!

[curaga]

You should never create Linux archives on Windows, as the permissions and ownership will be all wrong.

[sodface]

There's another best practice that goes "never say never"!

The only archive I created on Windows was the extra initramfs archive which only contained extensions.  The mydata.gz was created previously on piCore.

In this case it seems to work ok.  Here's how the additions look from piCore after boot:

Code: [Select]

/tmp/builtin/:
total 4
drwxrwxrwx    3 root     root          80 Jan  1  1970 ./
drwxrwxrwt    5 root     root         180 Jan  1  1970 ../
-rw-rw-rw-    1 root     root         189 Oct  9 16:20 onboot.lst
drwxrwxrwx    3 root     root         980 Jan  1  1970 optional/

/tmp/builtin/optional:
total 11768
drwxrwxrwx    3 root     root         980 Jan  1  1970 ./
drwxrwxrwx    3 root     root          80 Jan  1  1970 ../
-rw-rw-rw-    1 root     root       32.0K Dec 28  2016 bzip2-lib.tcz
-rw-rw-rw-    1 root     root          48 Dec 28  2016 bzip2-lib.tcz.md5.txt
-rw-rw-rw-    1 root     root      616.0K Dec 28  2016 ca-certificates.tcz
-rw-rw-rw-    1 root     root          54 Dec 28  2016 ca-certificates.tcz.md5.txt
-rw-rw-rw-    1 root     root      248.0K Oct  9 16:20 firmware-rpi3-wireless.tcz
-rw-rw-rw-    1 root     root       16.0K Dec 28  2016 gamin.tcz
-rw-rw-rw-    1 root     root          44 Dec 28  2016 gamin.tcz.md5.txt
-rw-rw-rw-    1 root     root      944.0K Jan 12  2017 gcc_libs.tcz
-rw-rw-rw-    1 root     root          47 Jan 12  2017 gcc_libs.tcz.md5.txt
-rw-rw-rw-    1 root     root        1.2M Mar 10  2017 glib2.tcz
-rw-rw-rw-    1 root     root          30 Apr  2  2017 glib2.tcz.dep
-rw-rw-rw-    1 root     root          44 Apr  2  2017 glib2.tcz.md5.txt
-rw-rw-rw-    1 root     root       76.0K Dec 28  2016 libedit.tcz
-rw-rw-rw-    1 root     root          12 Dec 28  2016 libedit.tcz.dep
-rw-rw-rw-    1 root     root          46 Dec 28  2016 libedit.tcz.md5.txt
-rw-rw-rw-    1 root     root       32.0K Dec 28  2016 libelf.tcz
-rw-rw-rw-    1 root     root          45 Dec 28  2016 libelf.tcz.md5.txt
-rw-rw-rw-    1 root     root      384.0K Dec 28  2016 libgcrypt.tcz
-rw-rw-rw-    1 root     root          17 Dec 28  2016 libgcrypt.tcz.dep
-rw-rw-rw-    1 root     root          48 Dec 28  2016 libgcrypt.tcz.md5.txt
-rw-rw-rw-    1 root     root       36.0K Dec 28  2016 libgpg-error.tcz
-rw-rw-rw-    1 root     root          51 Dec 28  2016 libgpg-error.tcz.md5.txt
-rw-rw-rw-    1 root     root       16.0K Oct  9 16:25 libiw.tcz
-rw-rw-rw-    1 root     root      240.0K Oct  9 16:22 libnl.tcz
-rw-rw-rw-    1 root     root       68.0K Dec 28  2016 libssh2.tcz
-rw-rw-rw-    1 root     root          14 Dec 28  2016 libssh2.tcz.dep
-rw-rw-rw-    1 root     root          46 Dec 28  2016 libssh2.tcz.md5.txt
-rw-rw-rw-    1 root     root        1.0M Mar  7  2017 mc.tcz
-rw-rw-rw-    1 root     root          34 Apr  2  2017 mc.tcz.dep
-rw-rw-rw-    1 root     root          41 Apr  2  2017 mc.tcz.md5.txt
-rw-rw-rw-    1 root     root      188.0K Dec 28  2016 ncurses.tcz
-rw-rw-rw-    1 root     root          46 Dec 28  2016 ncurses.tcz.md5.txt
-rw-rw-rw-    1 root     root        1.8M Dec 28  2016 openssh.tcz
-rw-rw-rw-    1 root     root          24 Dec 28  2016 openssh.tcz.dep
-rw-rw-rw-    1 root     root          46 Dec 28  2016 openssh.tcz.md5.txt
-rw-rw-rw-    1 root     root        1.1M Dec 28  2016 openssl.tcz
-rw-rw-rw-    1 root     root          20 Dec 28  2016 openssl.tcz.dep
-rw-rw-rw-    1 root     root          46 Dec 28  2016 openssl.tcz.md5.txt
-rw-rw-rw-    1 root     root      380.0K Dec 28  2016 pcre.tcz
-rw-rw-rw-    1 root     root          27 Dec 28  2016 pcre.tcz.dep
-rw-rw-rw-    1 root     root          43 Dec 28  2016 pcre.tcz.md5.txt
-rw-rw-rw-    1 root     root      116.0K Oct  9 16:21 readline.tcz
drwxrwxrwx    2 root     root          40 Dec 28  2016 upgrade/
-rw-rw-rw-    1 root     root        8.0K Oct  9 16:26 wifi.tcz
-rw-rw-rw-    1 root     root        2.5M Oct  9 16:25 wireless-4.9.22-piCore.tcz
-rw-rw-rw-    1 root     root       40.0K Oct  9 16:26 wireless_tools.tcz
-rw-rw-rw-    1 root     root      364.0K Oct  9 16:24 wpa_supplicant.tcz

/tmp/builtin/optional/upgrade:
total 0
drwxrwxrwx    2 root     root          40 Dec 28  2016 ./
drwxrwxrwx    3 root     root         980 Jan  1  1970 ../


[DrRob]

I am wondering if it might be possible to create a tool that takes all the packages that have been added into piCore and generate a custom initramfs with everything already in it so that extensions don't need to be loaded separately at boot time, and if so would that speed up the boot process?  What do you think?

[curaga]

Sure, I think there's a script for that posted here somewhere.

[polikuo]

I am wondering if it might be possible to create a tool that takes all the packages that have been added into piCore and generate a custom initramfs with everything already in it so that extensions don't need to be loaded separately at boot time, and if so would that speed up the boot process?  What do you think?

Try this script.
It merges all extensions you want into a single tcz, that should reduce your boot time without touching initramfs.

[DrRob]

Try this script.
It merges all extensions you want into a single tcz, that should reduce your boot time without touching initramfs.

Thanks, I'll take a look at that.

[sodface]

To go along with the extra initramfs above, I wanted to try creating a mydata.tgz from scratch since I used one I already had for my other tests which was kind of cheating.  Idea being, if you had a pi zero w for example, you could prep the sd card totally on Windows and boot the pi all the way up to wifi with it.

I'm not advocating it, I just wanted to see if I could get it to work, which I eventually did.  As curaga notes above, the permissions and ownerships aren't right particularly for the files within the mydata.tgz archive, I think the extra initramfs archives are less of an issue in that regard.

So basically, you just create your file structure in a working directory, I tested tonight with just a custom bootlocal.sh and wifi.db so the pi would get on the network at boot:

{working dir}\opt\bootlocal.sh
{working dir}\home\tc\wifi.db

added this line to the standard text of the bootlocal.sh file:
/usr/local/bin/wifi.sh -a 2>&1 > /tmp/wifi.log

and one line in the wifi.db with tabs separating the fields, in the form:
ssid password type

for example:
mywifinetwork P@ssw0rd WPA

The I used powershell to convert crlf to nl and convert to ASCII to avoid undesired characters at the start of the files, for example:

Code: [Select]

(Get-Content bootlocal.sh) -join "`n" | Out-File -Encoding ASCII bootlocal.sh
Then from command line in the working dir, use 7z to create the tar gzip file:

Code: [Select]

7z a -ttar -so mydata.tar * | 7z a -si mydata.tgz
Format the micro sd card with a single FAT32 partition of whatever size you need, extract the picore files from the stock image (all files from 0.fat as seen from 7zip gui) to the micro sd card and then replace the initramfs files with your custom ones with added extensions and drop the mydata.tgz into the root of the card and boot!

[sodface]

Oh, and I meant to note that I somehow killed my A+ that I was testing with. 

No idea what happened, at one point it was booting up and trying to start wifi and then it would just reboot again.  I figured I might have a power supply problem so I switched to a different one.  Seemed like the Pi was unusually warm.  Then eventually I just wasn't getting anything via the serial console and the Pi light was just on solid green.

I think it's dead.  I'll hook it up to a monitor tomorrow.  I guess it's possible that the power wire from the serial cable which I wasn't using and just had dangling may have landed on one of the other 40 pins at some point as I was moving things around, plugging and unplugging power, inserting and removing the sd card etc.  Not really sure what happened.

[sodface]

I'm a little annoyed that I didn't realize that none of this extra initramfs stuff is necessary.  For the little projects I've been working on, I thought it was best to setup piCore to load everything in RAM so it's like an appliance that can just be unplugged without fear of corrupting anything.  I liked the idea of not having to mess around with the extra partition that needed to be resized and thought that the custom initramfs detailed above was a good way of getting around it, basically:

- make a single fat32 partition of whatever size
- dump the main piCore files on it
- replace the initramfs with your custom one with added extensions
- optionally place a mydata.tgz in the root of the fat32 partition

Done!  Everything is loaded in RAM on boot, SD card is umounted, unplug at will.

Tonight I figured out you can just do the first two steps above, create a big fat32 partition and copy the piCore main files to it, but then instead of messing with the initramfs, just make the following dirs in the root of the fat32 partition:

tce
tce/optional

Copy all the extensions you want to load into tce/optional and create tce/onboot.lst to list them all.  Touch tce/copy2fs.flg so now you have in the root of the SD card along with all the boot files:

tce/optional/{all your .tcz files}
tce/onboot.lst
tce/copy2fs.flg

And again, you can also put a copy of mydata.tgz right in the root of the SD card fat32 partition.

This seems to accomplish the same thing as above with reduced complexity, you still end up with a single fat32 partition, all extensions being loaded to RAM and the option of restoring a mydata.tgz.  (you can also still run filetool.sh -b to update the mydata.tgz in the root of the sd card).

The only difference between the two methods that I've noticed so far is, with the custom initramfs, the fat32 partition gets unmounted after boot is complete and extensions are loaded.  The only time it gets remounted is if you do a backup with filetool.sh.  With the second method of having the tce directory in the root of the SD card, the fat32 partition stays mounted after boot, even with the copy2fs.flg set.  Need to look into that as I'd prefer it to be unmounted, but other than that method 2 seems just a bit simpler.

[curaga]

You need to unmount it yourself in bootlocal.sh, in that setup.

[Greg Erskine]

hi sodface,

IMHO the "standard" piCore image is already extremely robust.

We have been using piCore as the basis of piCorePlayer (music streamer) for years and I can't remember a SD corruption. We have hundreds of users just pulling the plug.

I have also tested the copy2fs.flg and unmount second partition method because it seems more eligant. I even removed the SD card from the pi and it continued to work as expected. In the end it just wasn't necessary.

It does depend on your application, if you are writing to SD card alot then one would expect SD card corruption would be a possibility.

regards
Greg