Transferring USB build images over the internet

[Tuftec]

Hi Guys,

I am a bit of novice with Linux and a bit new here so be gentle.

I have a need to transfer an image of my small built TC system (12.5MB) to a friend in another region.

I have built an image on USB that I have managed to get up and going and tested on a USB stick but I do not want to have to rely on the post to get the stick to him. I also need to send him the data quickly.

The image has been built so that it will run on a small embedded machine with no monitor, keyboard etc. The user interface will be via SSH from another computer (using Drop Bear on the TC box).

Unfortunately my friend has very limited resources at his end. He has a laptop running Windows an the box that he wants TC on but there is no spare monitor, CD drive, etc for him to fire up the TC box in the traditional way and load up the code.

What I want to be able to do, is somehow create an image of my machine here that I can email to my friend. He can then load this onto a USB stick, plug it into the TC box and he is then up an running with TC and an SSH connection.

Being a bit of a novice, I do not know the process involved in getting the data off my device in an appropriate format and then having him be able to put this on a stick. This part will all have to be done on troublesome Windows stuff.

Can anyone help with some advice?

Cheers

Peter.

[maro]

First off to get a better idea of your own setup, e.g. what boot options are you using. This could be answered if you'd share with us the output of: cat /proc/cmdline

Furthermore am I correct to assume that all of your TC system runs off the stick (i.e. there are no persistent "home" or "opt" on any hard disk partition)?

Furthermore the "real challenge" in my view is that we can not assume that your friend owns an identical USB stick. Because otherwise you could try to simply send him an image of your stick and he would be able to produce a bit-wise identical clone. But as we need to assume that the two sticks in question are of at least different size we need to do it in a more complicated way. It would help (at least for my own thinking about it) if we would know the size of your and your friends sticks, and also what kind of file system you've used on your stick. Therefore I'd suggest to let us know the result of: blkid /dev/sd* ; fdisk -lu /dev/sd*

[Rich]

Hi Tuftec
You mention that the target does not have a CD drive, how about the laptop? If it has a CD drive and
he can burn a TC CD he can boot into TC and make a bootable stick. At the very least, he will have
an environment that he can use to set a stick up with any files you send him. You use the term image
but I presume you mean you have directories with apps and bzImage etc. on the stick. You probably
want to tar the contents of the stick, gzip it and send that to him. With the laptop running TC he will
have access to the tools he needs to transfer the files to his stick.

[Tuftec]

Hi Guys,

the boot options are fairly simple
ro waitusb=5

You are coorect. We can not make any assumptions about his memory stick, although I do not believe that is much of an issue. As lond as his mem stick is bigger we can just put a bit image on it and waste the rest of the stick.

All code runs off the stick. There is no persistence. We need a clean boot every time we get a power up.

I had originally hoped that he could creat his own CD, then run this in his laptop and then set up the necessary options (drop bear) etc to make his own stick for the TC box. Unfortunately this seems to be beyond him at present or he just has not got the time. He is a bit of a Ubuntu fan and I am trying to convince him that TC is a better option for waht we are trying to do.

He really would like a cut down Ubuntu on the box but the standard kernel build is missing the appropriate drivers for the box and I do not want to create a special kernel build. TC already has the right drivers.

In my view, the real challenge is in passing a binary image over the internet and successfully installing it on a stick to use in another  machine. It sounds simple in principle, but beyond me at present.

Thanks.

Peter.

[Tuftec]

A quick thought.
Maybe I could massage the MBR from the stick into a file (relatively small) and then put the rest of the files on the drive into a zip file.

Then the only real hassle would be to reload the MBR at the other end. There must be a tool under Windows to do this. Under Linux it would be so much easier.

What do you think?

Peter

[maro]

Off the top of my head (i.e. without the usual testing that I apply to my suggestions here): If you are not worried about wastage and you furthermore assume that your friends stick is larger than yours, why not use a Windows version of dd?

So at your end do a dd.exe --list and figure out (e.g. by using the size information) which one is your USB stick (e.g. "\\?\Device\Harddisk2\Partition0"). Note: this one should be the entire disk, there should be another one (e.g. "\\?\Device\Harddisk2\Partition1") which would be the included (primary) partition. I see this pretty much as the same as '/dev/sda' vs. '/dev/sda1', so the size should be different and the larger one should be the entire disk.

With that information you could create the full disk image (e.g. via dd.exe if=\\?\Device\Harddisk2\Partition0 of=usb_stick.img bs=1M --progress). Note: There is likely to be an error message at the end of reading like "Error reading file: 27 The drive cannot find the sector requested", but I don't consider it a real problem as you are basically instructing it to read the entire USB stick until the end. Furthermore the size of the image file should exactly match the size of the disk as reported by 'dd.exe --list'.

Note: I've now come across a situation where when using 'bs=2M' I ended up with an image file that was 1 MBytes short. If using 'bs=1M' does not create a file of identical size to the disk I'd suggest to fall back to not specifying a block size at all. In this case the default of a single block (i.e. 512 bytes) will be used, and the reading goes much slower. But at least this way the entire image should be read.

Then zip it up with the compression tool of your choice and send it to your friend. He'll have to extract it again and then comes the "real fun part" of writing it to his stick (e.g. via dd.exe if=usb_stick.img of=\\?\Device\Harddisk2\Partition0 bs=1M --progress). Needless to say if he mistypes the destination device, or is wrong in his analysis of which one would be the correct representation of his target, he stands a good chance to screw up his Windows installation big time (or whatever other false target he might hit). So as they say here: "No pressure!!"


EDIT 1: As a consequence of my own testing I've made minor adjustments to the suggested command samples above (i.e. made it more obvious that we are dealing with a Windows program, added a blocksize parameter (which increased the execution speed quite significantly) and added the useful (but non-default) option of '--progress').

EDIT 2: I had to change the block size option from 2M to 1M, and added an additional suggestion to compare image file size with the disk size (i.e. partition0)

[Rich]

Hi Tuftec
To copy the MBR of off of the stick

     dd if=/dev/sda of=mbr.img bs=512 count=1

To copy the MBR onto another stick

    dd if=mbr.img of=/dev/sdb bs=512 count=1

You would have to replace sda and sdb with the correct values on your system. As I said, you could
then tar and gzip the files from the stick. If your friend has access to an Ubuntu machine he can use
that to make the stick from the files you send him.

[Tuftec]

Hi Guys,

I think maybe the Windows version of dd would the easiest.

If I format the stick as FAT then I can simply dd the MBR and then copy the rest.
This would enable the email transfer to be broken up into smaller managable chunks.

I will test this method and report back.

Cheers

Peter

[tinypoodle]

Off the top of my head (i.e. without the usual testing that I apply to my suggestions here): If you are not worried about wastage and you furthermore assume that your friends stick is larger than yours, why not use a Windows version of dd?

So at your end do a dd --list and figure out (e.g. by using the size information) which one is your USB stick (e.g. "\\?\Device\Harddisk2\Partition0"). Note: this one should be the entire disk, there should be another one (e.g. "\\?\Device\Harddisk2\Partition1") which would be the included (primary) partition. I see this pretty much as the same as '/dev/sda' vs. '/dev/sda1', so the size should be different and the larger one should be the entire disk.

With that information you could create the full disk image (e.g. via dd if=\\?\Device\Harddisk2\Partition0 of=usb_stick.img). Note: There is likely to be an error at the end of reading like "Error reading file: 27 The drive cannot find the sector requested", but I don't consider it a real problem as you are basically instructing it to read the entire USB stick until the end.

Then zip it up with the compression tool of your choice and send it to your friend. He'll have to extract it again and then comes the "real fun part" of writing it to his stick (e.g. via dd if=usb_stick.img of=\\?\Device\Harddisk2\Partition0). Needless to say if he mistypes the destination device, or is wrong in his analysis of which one would be the correct representation of his target, he stands a good chance to screw up his Windows installation big time (or whatever other false target he might hit). So as they say here: "No pressure!!"

Additionally when following such an approach of dd'ing and then compressing a whole filesystem, a potentially significant gain in compression could be achieved by zeroing out unused sectors first.

e.g. under Linux something along the lines of:

Code: [Select]

cat /dev/zero >/mnt/sda1/zerofileuntil an error message is output, e.g. "No space left on device", then

Code: [Select]

rm /mnt/sda1/zerofile

[maro]

All right, I've now done my own test and can state with some more confidence that the steps as described in reply #5 worked for me just fine.

I started with creating an image from an older 128 MBytes  USB stick on which TC had been installed previously. For obvious reasons I skipped over the compression, transfer and de-compression part of the process, and then wrote the image file onto a 8 GBytes USB stick. With this stick I was able to boot a QEMU VM as well as "real" hardware. So far so good, and I've made minor changes to the commands in my suggestion above to reflect some findings of my experiment.

I also had to consider a good way to make my 8 GB stick useful again, as I did not want to leave it limited to a single 128 MB VFAT partition. Luckily I had taken a snapshot of its MBR before overwriting it (via dd.exe if=\\.\physicaldrive2 of=usb_stick.mbr count=1) and could now "repair the damage" quite easily (via dd.exe if=usb_stick.mbr of=\\.\physicaldrive2).

As you can see there is an alternative way of specifying the USB stick in question. I just did not want to use this form in my original suggestion, as it does not appear in the 'dd.exe --list' output and therefore might lead to confusion with some readers.


EDIT: Another point worth sharing seems to be that I had to properly disconnect and the reconnect the target USB stick after each image write process. Otherwise Windows was quite "confused": e.g. even though the "repaired" MBR showed a 8 GB FAT32, a simple format.com f: lead to the re-creation of a FAT16B partition and it reported only to have processed 122 MB. So I had to overwrite the MBR a second time, disconnect and reconnect the 8 GB USB stick, and could then execute a format.com f: /fs:fat32 to get me back to where I had started from.

[Tuftec]

I am still having a bit of trouble with dd under Windows.
I am running Windows 7.

I loaded up dd and opened a commend prompt window to type the dd commands.

To copy to the stick I used:
dd if=c:\mbr.img of=\\?\HarddiskVolume8 bs=512 count=1

It gave an error
 Error native opening file: 0. The operation completed successfully.

Not sure what this means but the MBR was not copied.

What did I do wrong?

[maro]

To copy to the stick I used:
dd if=c:\mbr.img of=\\?\HarddiskVolume8 bs=512 count=1

It gave an error
 Error native opening file: 0. The operation completed successfully.

Well, for starters of all the many listed Windows device names you picked one that can't be used. If I understand the story correctly those HarddiskVolumes should correspond to a drive letter or a file system partition. So by all means not the right target for a MBR. BTW, I believe it should be more like '\\?\Device\HarddiskVolume8' and not '\\?\HarddiskVolume8', so just as well that it did not work as you could have put your system at risk.

I suggest to go back and have a good read over the DD for Windows page. A couple of points I spotted, which seem to confirm the advice I've given previously:

... Partition0 is the entire disk. Under Windows XP, some partitions may not have a Volume device. In this case you can still use the Harddisk<n>\Partition<n> name ...

or about how to

Read the entire USB memory device
dd if=\\?\Device\Harddisk1\Partition0 of=c:\temp\usb2.img bs=1M --size --progress

I had not read this second quote myself beforehand, but it matches pretty much what I've written earlier.

So in a way to repeat what I believe to be correct:

• Either use \\?\Device\Harddisk<n>\Partition0 or \\.\physicaldrive<n> where <n> is the "disk number". The former one gets reported in the 'dd.exe --list' output and I use my knowledge about my disk sizes to identify the correct <n>. The latter one is AFAIK just a different naming convention and I personally prefer to use it, but you won't find it mentioned in the 'dd.exe --list' output.
• Don't use any other device names (like \\.\f: or \\?\HarddiskVolume8) as they appear to be referencing partitions and not the entire disk. Remember: the MBR is the first block of a disk and the VBR (i.e. volume boot record) is the first block of a partition.
• If you are not certain about the target which you are trying to write to, read it out first and have a look. A "trained eye" like myself will have no difficulty to spot the difference between a MBR and a VBR. But one can alway read up about these matters at place like here or possibly even more details here.

If you are still unsure you'd better attach here the output of dd.exe --list and let us know the size of your USB stick. If your friend provides us (e.g. via yourself) with the same information we should be able to give you the commands that should work.

Also note that I've make another small amendment to reply #5, so better read that one again.

[Guy]

It is not clear how you have Tiny Core set up.

It may be easier to get your friend to download Tiny Core, and install it. He will then have the bootloader set up.

You could then send over a copy of anything particular to your setup, and he could copy it.

[Tuftec]

Right! Problem solved.

Here is the SIMPLE solution

Set up a stick under TC and build a system on it with what you want as a frugal install (no persistence)
[using the standard TC instal GUI tools]

Take the stick out and slap it in a Windows PC.

Pull off all the main folders and files (boot directory, tce directory and syslinux.cfg).
Zip these all up and put them somewhere safe.

Email the zipped files to my friend.

Now what was tricky becomes real easy.
Tell me friend to get hold of a copy of syslinux.exe.

He then run's on his PC with his formatted stick  -  syslinux.exe -a -m z:  (where z: is his drive)
This sets up the MBR and loads the syslinux bits correctly.

Unzip the received files and copy to the stick.

Put the stick in the TC box and boot.

Bingo.

All done.


I hope this helps someone else.

Cheers

Peter.
 

[Tuftec]

I forgot to add that I also removed the UUID specific bits that are in the syslinux.cfg file as I figure this would be meaningless on another device.


Thanks for all your help and making me think about the problem a bit more.

Cheers


Peter.