Running custom kernel with qemu-kvm

[michelp]

Hello!

TCL looks pretty amazing.  It was really wild to see qemu-kvm start up a micro instance in less than a second!  Pretty impressive and small than all the other "micro" distros that I've seen by 10-100x.

So, here's my question.  I'm hacking a new feature into the linux kernel.  This is an implementation of an in-kernel messaging system similar to ZeroMQ but optimized at kernel speeds.  I'd like to be able to, right from my kernel tree, do something like "make vm_test" and have it build the kernel, drop it or remaster it somehow into a runnable image, and then fire up that image with qemu-kvm.  The image would contain my test harness that run continuously until an assert fails or the kernel panics, which hopefully doesn't happen.

This seems like it's possible but I'm not sure about how to put it together, the structure of TCL is definitely unique.  Does this sound like something that is appropriate for TCL?  Can I build a bleeding edge kernel and bundle it with TCL and an extension that contains my userland test code?  Any helpful hints or direction in this regard would be awesome!

Thanks,

-Michel

[curaga]

Absolutely possible. I would start by remastering those test scripts into the main initrd, and every time you build a kernel, storing those modules in a second initrd. Look under the contrib dir for an example of how such a module initrd looks like:
ftp://ftp.nluug.nl/pub/metalab/distributions/tinycorelinux/4.x/x86/contrib/rt-kernel/

Then launching your Qemu test instance becomes
cat core.gz my_modules.gz > new.gz
qemu -kernel /path/to/new/bzImage -initrd new.gz

No need for persistent storage, etc.

[michelp]

Awesome!  Your advice has gotten me very close.  I do have a question about the kernel configs that I need to use.  I'm copying the kernel config from the released files area and appending the config specific to my patch to it, that works and builds fine, but I see in the wiki page "custom kernel" in the remastering section that I also need to apply a bunch of other kernel patches.  Are all those really necessary?  I quick glance looks like many of them are simply boot time optimizations.  Are any of those patches absolutely critical to running Micro Core or I can I get away with a basically stock kernel clone of the git master repo with my patches applies and work from there?

Thanks so much for your help, I was showing off MicroCore to my coworkers today, I'm hoping to convince them that TCL+kvm is a better solution than vagrant+virtualbox for what we need, which is to quickly spin up lots of customized headless vms for development and deployment.

-Michel

[curaga]

Are any of those patches absolutely critical to running Micro Core or I can I get away with a basically stock kernel clone of the git master repo with my patches applies and work from there?

You can run Core with vanilla (stock) kernels, yes.

Some of the patches are boot time optimizations, some are cosmetic, a couple bugfixes, the advdef one creates a smaller kernel. Only the tmpfs root and zram patches are functionality, but as long as you have enough RAM, Core will run without either of them.

[michelp]

So, TCL is really kicking butt for me on this project, I have a bash script that starts with a kernel checkout, downloads and extracts core64.gz, installs the modules and firmware, and runs qemu with the new kernel, and it works great!  I love automation and automated vm tests are really going to work well here.  The instant boot times of TCL make it almost painless.

The only last hurdle I have is that my script builds and copies some test binaries into the /bin of the initrd.  When the vm boots, the files are there in bin, they are the right size, and they are +x, but when I try to run them, I get a shell error about no such file.

https://gist.github.com/3033531

Any clues here?  Running these test programs is my last step to kernel testing glory!

-Michel

[michelp]

Looks like the problem is, the binaries I compiled on my host platform are not compatible with TCL.  ldd complains that they are the wrong ELF class.  There is likely library version mismatch as well, so I guess my options are to include a compiler suite in my vm and build them in the guest, or rewrite my tests in Python.  I think I might try the second option.

[curaga]

They're probably 64-bit; even in core64 we're so far 32-bit, core64 only has a 64-bit kernel (it's advantageous to run it on capable platforms, even when using 32-bit userspace).

If they don't change often, you could precompile them and just copy the binaries. Alternatively, compile them statically on your host - since then they won't depend on any 64-bit libs, TC's 64-bit kernel can run them without userspace help.

[michelp]

Static compiling!  You're a genius!  It works!

I'm so happy right now I'm dancing a jig.

Thank you so much!

-Michel