WelcomeWelcome | FAQFAQ | DownloadsDownloads | WikiWiki

Author Topic: Zero USB booting and remastering  (Read 3547 times)

Offline gavinmc42

  • Sr. Member
  • ****
  • Posts: 301
Zero USB booting and remastering
« on: September 09, 2017, 08:48:08 AM »
Got a Clusterhat months ago and been playing with USB booting Zeros with no SD card.

Mostly been using baremetal to make a kernel.img, it's quicker
https://ultibo.org/forum/viewtopic.php?f=10&t=333&hilit=zero
https://www.raspberrypi.org/forums/viewtopic.php?f=72&t=184840

Ultibo will make small kernels but I cannot do camera stuff with it yet.
The Pi camera does work on piCore :P

I did get picore usb booting but I don' t get the tce folder stuff to work.
Looking at the files I noticed 9.0.3.gz.

Trying to figure this out I remembered something about remastering.
Ch 11 page 63 of the corebook covers remastering,  making mods to the *.gz file.
I followed the steps and made a simple change to the bootsync file changing the box name.
it worked 8)

It will take me some time to master a new image but a Zero usb booting usb camera system with 4 Zero's and one B+ is possible.
One advantage is the Zero's usb boot from the B+, so they don't need SDcards.
With no files on the sdcards it also means easier mods done remotely.

I guess the best way to do it is do a standard SD card install, add all the camera stuff and usb comms.
Then make a master of the working SDcard and pack it into a new 9.0.3.gz.

What would be the steps to doing that?


Offline doctoranonymous

  • Newbie
  • *
  • Posts: 22
Re: Zero USB booting and remastering
« Reply #1 on: December 07, 2019, 11:55:54 AM »
I know this is an old post, but I am a new brain, or fairly new. I am a biologist (marine biologist, officially, but evolution biologist in heart, and head), And, while studying for an undergrad degree at the University of Rhode Island (conferred, August 2002), I encountered the book "Darwin's Dangerous Idea". It was a time of intellectual ferment (and foment) and, long story short, after my four year degree I spent the next five years on basic electrical and optical engineering, and computer science, assembly language (minimal x86, but much 6502). Then ten more years trying to match the basic physics and history of computer science, to the "actual market", of expectations, and real goods. Long story short, I find Linux in 2016 (Congratulations "automatic-update-Windiows-10", your crap-@$$ business tactics, and frozen-slow-bulky-OS just lost you one of your smarter customers), the FOSS movement, and the-not-quite-pure-as-the-driven-snow, Raspberry pi (zero). It IS the right price, for the processing power I want and need, and for its ability to ask and answer my questions about the nature of Natural Intelligence, and Artificial Intelligence (Strong AI).

So, why post a long-ish monologue/diatribe here? Seemed like an interesting thing to do? But not completely off-topic, as I am intensely interested in R-pi cluster computers (and, as floundering and hopeless as I am in the field of computer science, I ALWAYS NEED HELP!). So, after dancing around the idea of "ethernet gadget" for about 6 mos. to 1 yr., I think I might NOT do that. As I understand it, ether-gadg. involves powering the "slave/device/client" pis, through its "data+power" micro-USB port, and leaving the power micro-USB port empty. For my purposes, this seems inadequate, as the "master/host/server" pi then must push about 150 mA to each of the "slave-device-client" pis. I have 5 or 6 such pi zeros, already, and this could add up, especially if I want to do something with them other than boot them and let them sit idle.

I have a wall power supply (12-ish USB power-only ports, of varying amperages; 1.0A to 3.5A; will too many amps be bad? probably not? The pi will draw what it wants/needs, right? Basic EE; Ohms Law!?), though ""wall tethered", should be adequate for now. So, assume I acquire about 5 more pi zeros, boot them all with a single SD card (piCore, at 50 MB, "bare metal"), powered and wired with the flotsam and jetsam of the "wireless society" (curious the wireless revolution should product such prodigious VOLUMES of e-junk wires, i.e. "USB chargers"; the salvation army/goodwill stores are buried in USB wires! And none of them properly marked, as to "data" or "data+power"!), and all with their "data-USB" hooked to a 10-USB-port "strip-hub". One pi to rule them all... right? ... And in their pi-ness bind them? Or free them (from slavish weak-AI, to freedom of Strong-AI?! Are not my cells and tissues slaves to physics and chemistry, yet my mind ... marginally, nominally "free"?). The master-pi, also hooks to the 10-USB-port "strip hub", and he/she/it is "The Boss" (sorry Bruce, the 80's were great, and we were "Born to Run", but there's a new Boss in Town!).

I get ahead of myself. True Strong-AI must have "smellivison" (and associated "taste-o-vision") first, and I see no IC chips of that sort, yet (though perhaps, some kind of multi-sensor, SAW sensors, SAW-array sensor? I posit it, and hypothesize it, but the task is/would-be monumental! How many unique chemicals must we design such an array for? And is the "sensation process" consumptive of the sensation medium?) Unfortunately, much of that SAW-chemical research is military, and only sniffs weapons, exposives, war-chemicals, etc. Anyhoo, I've digressed.

Back to the subject at hand. Sd-cardless, cluster-pis! I will follow some more of what gavinmc42 has said, and trace some of his other trains of thought. I found my way to raspberry pi official github page, and dl-ed / cloned usbboot-master.zip ; and I got one pi zero booted (through CLI, through a single USB wire connected to the pi-zero "data+power" micro-USB, hooked to one of my kali linux laptop's USB-A ports) past the rainbow screen, to the black-screen-plus-top-left-corner-large-rapsberry-logo, but I can't find the website I read (about a year back) that explains "where" in the boot process that puts me? I would like CLI, if possible? Any suggestions? (I will post the Kali CLI output that results when I use this boot method, below, as this message is cluttered enough with "big ideas"; minutiae would bog it down irreparably!).

So, I guess, I am admitting to farting around  (due to my general ignorance and incompetence) with the wrong process for about a year! Ethernet gadget is not what I want. I found the raspberry pi foundation official pages on "boot processes", yesterday (about two years too late!), and there are some in depth discussions of "USB Host Boot" and "USB Device Boot". These are still inchoate in my mind,  but the language is more similar to the things I already know about USB (I read books, sometimes; had several Jan Axelson books on USB; but landlord/troglodytic-philistine "stole" them.). Whenever I read about "ethernet gadget" I was always perplexed by the complete lack of "Host" and "Device"  terminology? As if programmers do not need to know what they are doing?

So, I also read, somewhere, that raspberry pis have trouble enumerating more than 8 devices? More than 8 "gadget" or "device' pis, for the one "master/Host" pi? Is this just the not-so-great advice of an inexperienced user? Or is the pi-zero limited as a USB master/host, in some way I am unaware of? USB standards do provide for a ridiculously large number of devices, I think?

Also, do note, the Kali-forced, USB-boot, only sent some of piCore 9.x.? ; the parts I deemed essential; the parts I placed in the boot folder. Perhaps, my first mistake? If I put the whole 50 MB in, maybe I'll get past the big-raspberry-screen? Maybe, for completions sake, the "minutiae post" below this "big idea" post, could include this information; i.e. which parts I put in the boot folder. I'll probably do that.

Also, looking forward, into the future, I have downloaded TC.tcz, separately, using the FetchExt.sh script designed by Rich (Moderator here at tinycore forum), so that I can run all piCore, for every pi zero. Right now, the one I intend to be "master" is running raspbian-buster, for ease of use, and desktop GUI. So, looking forward, can I have one tcz folder; one central location of extensions, from which, all pi zeros, either master or slave, will draw from, when booting, or when needed, and thus, save space? Can I have the "master" pi core dish out the extensions, when requested by the "slaves"? (Can I co-opt the shell script already written by Rich, for this purpose? I'll bet I can!)

One more aspect I have neglected to mention; for strong-AI, one must have a "human-like" body, to match and facilitate a "human-like" mind; thus, the rpi zeros will be distributed, across the 12 "nodal points", namely, the major musculo-skelatal joints of the anthropoid body (wrists, elbows, shoulders, hips, knees, ankles.).

In the future, with SSH (and one-time pads) I will be comfortable enough to buy 12 r-pi zero Ws, but at present, I am not master hacker enough to keep secure, and I want a PERFECTLY controlled experiment (or as close as possible). Using the zero-Ws would save on weight (wires), in the final robot. However, wireless might have some latency issues? Have to carefully balance these concerns, moving forward, and ultimately, it will be a question of bandwidth, A.I. or N.I., bandwidth. ("human bandwidth"),

Sight? Sound? Touch? These problems, largely solved, elsewhere, by a little thing we call "humanity", (or "science"; see the previous 100 to 500 years, for proof positive of this statement!).

So, to summarize the relevant questions, and chart a path forward (1) can a pi cluster master/host enumerate more than 8 devices?, (2) USB Host Boot, Device Boot, or SD card-picore remove and reinsert, method, (3) how to make one r-pi-zero a master/host, and the rest slave/devices, (4) Are IPs needed for local networking of pis through the USB? Static IPs, in particular?, (5)How to distribute the relevant/requisite .tcz files throughout the "robot frame", i.e. musculo-skeletal system (one pi-zero at each "nodal point", for "distributed computing"), (6) "where" is my usb-boot, when it is at the large-raspberry screen? How to go forward in this "cardless" boot process, to get to CLI, and eventually, a TC.tcz,  GUI?. I guess that's everything? See post below, if I have any minutiae data to add to this essay/book.

Offline doctoranonymous

  • Newbie
  • *
  • Posts: 22
Re: Zero USB booting and remastering
« Reply #2 on: December 07, 2019, 03:58:10 PM »
OK, not the "minutiae" post yet. That will be one more below (I think). I guess things "roll out", as they develop (evolve?), as some of these boot methods are not very old (about a year and a half). R-pi.org has this documentation for possible boot sequences, but it still seems a bit abstruse and arcane to me. About six months ago I was sniffing around some githubs that were working (5 years ago) on freeing up the VPU/QPU/GPU stuff. I am not sure if it ever became FOSS (to Stallman's standards!). Luckily, as a biologist, nobody holds me to the standards of a FOSS purist, so I can "feel" my way through the ethical morass. (I'll probably end up close to the "purists" perspective, anyway!). Anyway, these guys were "bare metal" and I was feeling bare metal just reading them. I guess I forgot most of it, though, and I certainly didn't get to read all the official documentation on boot modes, since I have just found those, yesterday.

Anyhoo, https://www.raspberrypi.org/documentation/hardware/raspberrypi/bootmodes/ and https://www.raspberrypi.org/documentation/hardware/raspberrypi/bootmodes/bootflow.md are useful, here.

So, here is an interesting notion : https://www.raspberrypi.org/documentation/hardware/raspberrypi/bootmodes/device.md" USB device boot mode : When this boot mode is activated (usually after a failure to boot from the SD card), the Raspberry Pi puts its USB port into device mode and awaits a USB reset from the host.".

So, all my r-pi zeros (slave/devices) could be "sd-boot failed", then wait for the one that is host, to USB-reset them?

USB Host Bootmode, is unavailable to pi zero, as far as I know? https://www.raspberrypi.org/documentation/hardware/raspberrypi/bootmodes/host.md .

So, since I ruled out one official method, and identified a possible suitability for another official method, let us focus on the latter:

"Example code showing how the host needs to talk to the Pi can be found here." . Here goes here => https://github.com/raspberrypi/usbboot , which is, exactly where I went yesterday, when I got my pi booted to the large-raspberry-black-screen.

Not quite CLI, which, if I am smart enough, I might find today!

So, in the read-me file of usb-boot utility, there is a section "Running your own (not MSD) build[/u]". I think this is what I want to do. Now, how much do I remember about tinycore initramfs, or for that matter, any initramfs? 6 months ago I read, but definitely forgot by now. Can't remember if arms have an init?

Anyway, perhaps this website will help; seems to deal with the issue of setting up a filesystem for the usb-boot device? https://dev.webonomic.nl/how-to-run-or-boot-raspbian-on-a-raspberry-pi-zero-without-an-sd-card

"March 22nd, 2018 : Then you could burn the image to an USB flash drive, but we gonna do something clever, we gonna setup a loop device.

Code: [Select]
cd ~
 sudo losetup -P /dev/loop0 Downloads/2018-03-13-raspbian-stretch-lite.img

Then we need two mountpoints for the boot and rootfs directories. Let’s create them.

Code: [Select]
sudo mkdir -p /pi/{boot,root}  "

I don't know. I guess I have to read the rest of that post? Unless anybody else out there knows more about what I am doing than I do? If so, please let me know, so I don't break anything.
« Last Edit: December 07, 2019, 04:10:11 PM by doctoranonymous »

Offline doctoranonymous

  • Newbie
  • *
  • Posts: 22
Re: Zero USB booting and remastering
« Reply #3 on: December 07, 2019, 05:38:10 PM »
OK, minutiae Number 1 (of 2). USB Device Boot (wherein Kali-Linux CLI commands, with usbboot utility, "squirts" the bootcode.bin into the r-pi-zero "data+power" micro-USB port).

Code: [Select]
root@localhost:~# cd /usbboot
root@localhost:/usbboot# sudo ./rpiboot -d boot
Waiting for BCM2835/6/7
Sending bootcode.bin
Successful read 4 bytes
Waiting for BCM2835/6/7
Second stage boot server
File read: config.txt
File read: start.elf
File read: fixup.dat
File read: config.txt
File read: config.txt
File read: cmdline.txt
File read: kernel4922.img
File read: bcm2708-rpi-0-w.dtb
File read: config.txt
Second stage boot server done
root@localhost:/usbboot#

So, this gets me past rainbow screen, and then to "black-screen-with-giant-raspberry-logo". As I said, in one of the two marathon posts, above, it might be my fault because I didn't put enough files in the "/boot/" folder. (no reason to skimp, as 50 MB isn't even a drop in most buckets, nowadays).

Offline doctoranonymous

  • Newbie
  • *
  • Posts: 22
Re: Zero USB booting and remastering
« Reply #4 on: December 07, 2019, 05:50:57 PM »
And, minutiae number 2 (of 2); list of files I placed in "/usbboot/boot/" : bcm2708-rpi-0-w.dtb , bootcode.bin , cmdline.txt , config.txt , fixup.dat , kernel4922.img , kernel4922v7.img , LICENCE.brioadcom , README , RELEASE , start.elf . These 11 QTY files, are all the files that I put in the boot folder. Give a shout if you see something else that I should have put in. But, I will probably put the whole piCore in, at some point this evening, as it seems I am, lately, motivated to get this thing working exactly how I want it to work.


Offline doctoranonymous

  • Newbie
  • *
  • Posts: 22
Re: Zero USB booting and remastering
« Reply #5 on: December 07, 2019, 06:17:32 PM »
Then, the README file says to do this: "To provide a filesystem there are many options,
you can build an initramfs into the kernel, add an initramfs to the boot directory or provide some
other interface to the filesystem.". This is probably--definitely--"out of my wheelhouse".

https://www.jootamam.net/howto-initramfs-image.htm , if, someday, I'm feeling ambitious? Anybody got a better plan?

Offline doctoranonymous

  • Newbie
  • *
  • Posts: 22
Re: Zero USB booting and remastering
« Reply #6 on: December 07, 2019, 06:42:42 PM »
OK, my bad! All I had to do was use ALL the piCore files from the FAT partition. It works now; booted to CLI, command prompt. Now, I guess I have to SSH in? Or, what? Can't unplug the usb can I? Can I plug a power wire into the power-usb port? And then unplug the data usb? and then plug in a keyboard? There has got to be a way to run the pi through my Kali-laptop? I dunno, I think my brain might be spent for today? Here is the CLI readout for my successful usbboot (USB Device Boot).

Code: [Select]
root@localhost:/# cd /usbboot/
root@localhost:/usbboot# ls
bin2c    boot    LICENSE  Makefile  Readme.md  win32
bin2c.c  debian  main.c   msd       rpiboot
root@localhost:/usbboot# sudo ./rpiboot -d boot
Waiting for BCM2835/6/7
Sending bootcode.bin
Successful read 4 bytes
Waiting for BCM2835/6/7
Second stage boot server
File read: config.txt
File read: start.elf
File read: fixup.dat
File read: config.txt
File read: config.txt
File read: cmdline.txt
File read: kernel4922.img
File read: 9.0.3.gz
File read: bcm2708-rpi-0-w.dtb
File read: config.txt
Second stage boot server done
root@localhost:/usbboot#

Sorry so verbose, but I feel I accomplished something? Maybe someone will stumble across this string someday, and thank me for it? No sense re-inventing the wheel, right!?