Tiny Core Linux

Tiny Core Base => TCB Q&A Forum => Topic started by: xnix on November 03, 2010, 02:39:35 PM

Title: Custom Distro for Linode
Post by: xnix on November 03, 2010, 02:39:35 PM
Well, I'm going to do my best to think this through before just writing and hoping someone will answer my musings. I want to ask questions that bear some meaning, and not just random and pointless 'can I do this?'

I just moved my VPS over to Linode, since my previous VPS provider was overkill for my needs and Linode offers better control for my experience level, and have been reading a bit about deploying a custom distro on their servers. I thought about taking a run at deploying TinyCore, considering it is already minimal and taking into consideration that on a desktop it is blazing fast, I do think that TC might be a nice project to put together and eventually use on a production server.

So my first question: would TC be a decent option to use as a production server's distro? Taking into consideration that it is built on the same kernel as those other commonly excepted distro's, why would TC fit into this category as a power house? I mean, you would have to be meticulous about what you run in RAM and mount to disk, considering the limitations of RAM, but having the web server running in the RAM might be an extremely fast server. Maybe I am not considering how much RAM it actually takes to run, say, httpd, but perhaps nginx ot lighttpd, or even cherokee might be lihgt enough to run in this matter.

My intention is to have a web server, sql server, and mta (mail) server so that a custom TC distro could focus its efforts on those tasks, thus freeing up resources. I believe this would allow me to load, say, the web server into RAM and not have to worry about using too much RAM since this would be its primary purpose. Of course, I'm not taking into consideration how much RAM my applications would use, but I don't intend on using resource-heavy software either. Let's put that RAM to work for me, making Apache (definitely not my first choice for server if I go this route, but using it as an example here), or whatever flavor of web server I choose, and get it loading and serving that much faster. Am I dreaming here?

I know there is more for me to research, and I need to buy a dev VPS so I can play with it, but would this be feasible, worthwhile, and efficient to pursue?
Title: Re: Custom Distro for Linode
Post by: curaga on November 04, 2010, 01:03:16 AM
Considering the usual VPS is ram-constrained, and many of the virt/isolation techniques do not allow swap, TC would be fairly ideal in that environment. Or better yet, MC with possibly Xlibs.gz, no need for an X server there.

Security-wise TC isn't quite on the level of the enterprise distros, what we have is speed and features.
Title: Re: Custom Distro for Linode
Post by: xnix on November 04, 2010, 09:48:32 AM
I had already intended on MC, since I don't need or want GUI or X11, I was just posting TC for the sake of simplicity. I can understand that TC/MC doesn't have the security features and robustness of an enterprise distro, but how hard could it be (coming from a Linux noob) to enhance it's security? Harden the shell, throw in a firewall, and whatever else that would be absolutely necessary. I definitely need to take security into consideration, but this shouldn't be a hurdle which is exceptionally difficult to overcome. I'm hoping to add a low-resource node to my Linode account soon and start playing with this possibility, I just wanted some advice in what might make this process more painless and usable.
Title: Re: Custom Distro for Linode
Post by: curaga on November 04, 2010, 10:32:43 AM
If you have a properly set up firewall and keep your servers up to date, it should work just fine.
Title: Re: Custom Distro for Linode
Post by: danielibarnes on November 04, 2010, 01:45:09 PM
Linode is a interesting option. For $20/mo you get 512MB RAM, 16GB HDD, and 200GB data transfer. Their plans scale linearly: 2x the price gets you 2x RAM, HDD, and data transfer, so no discount for higher usage.

Linode has a blog post which describes Custom kernels with pv-grub (http://blog.linode.com/2008/12/23/custom-kernels-with-pv-grub/). When you read the wiki article (http://www.linode.com/wiki/index.php/PV-GRUB) referenced in the post, there are some important details:

The host must either be running xen-sources or another kernel that supports pv_ops. In order to work with Xen, a number of options that must be selected:
CONFIG_PARAVIRT_GUEST=y
CONFIG_XEN=y
CONFIG_PARAVIRT=y
CONFIG_PARAVIRT_CLOCK=y
CONFIG_XEN_BLKDEV_FRONTEND=y
CONFIG_XEN_NETDEV_FRONTEND=y
CONFIG_HVC_XEN=y
CONFIG_XEN_BALLOON=y
CONFIG_XEN_SCRUB_PAGES=y

The default 3.x Tiny Core kernel has:
CONFIG_PARAVIRT_GUEST=y
CONFIG_PARAVIRT=y
CONFIG_PARAVIRT_CLOCK=y

Note that the Xen-specific flags are not present. The LKDDB entry (http://cateee.net/lkddb/web-lkddb/XEN.html) for "CONFIG_XEN" states: "Enabling this will allow the kernel to boot in a paravirtualized environment under the Xen hypervisor." This appears to imply that Tiny Core will not boot in Xen without this option. I know little of Xen, but I believe this to be the case. My initial conclusion is you will need to build a custom kernel based on the Tiny Core kernel, but that isn't too difficult.

Whatever you do, my advice is to script everything you build. This makes it easy to save your work for reference and re-use later. I find that invaluable when building extensions and remastering.
Title: Re: Custom Distro for Linode
Post by: xnix on November 04, 2010, 09:13:43 PM
My initial conclusion is you will need to build a custom kernel based on the Tiny Core kernel, but that isn't too difficult.

Mwuahahaha, so we'll find out. You did notice my comment about being a Linux noob, yes? I'm not afraid to explore this, and actually am eager to see what may come, so really if all I need to do is add those options and variables to a custom kernel of micro's already custom kernel, then hey, it shouldn't be too hard. Now if those added variables require more than simply a 'y' then I might be in for an adventure.

Whatever you do, my advice is to script everything you build. This makes it easy to save your work for reference and re-use later. I find that invaluable when building extensions and remastering.

Scripting? Meh, I just encountered my first experience with shell scripting, and while it didn't not work, it didn't quite work as planned. How about I start by simply documenting every little detail I can come up with, then come hunt you down for that said scripting? Kidding, really!

I'm going to purchase a dev VPS from Linode December 1, when the next billing cycle rolls around for my account, so hopefully I will be able to start looking at MC as a distro for Linode.
Title: Re: Custom Distro for Linode
Post by: danielibarnes on November 07, 2010, 08:07:16 PM
Quote
Now if those added variables require more than simply a 'y' then I might be in for an adventure.
A small adventure, perhaps, but I can help:

1) Load the compiletc, bash, perl5, and coreutils extensions.
2) Get the patched Tiny Core kernel source (http://distro.ibiblio.org/pub/linux/distributions/tinycorelinux/3.x/release/src/kernel/linux-2.6.33.3-patched.tbz2).
3) Extract it
4) make mrproper
5) Copy the Tiny Core kernel config (http://distro.ibiblio.org/pub/linux/distributions/tinycorelinux/3.x/release/src/kernel/config-2.6.33.3-tinycore) into the directory as .config
6) make menuconfig
7) Select the following features (in order):

Processor type and features --> Processor family --> Pentium-Classic (or later of your choosing)
Processor type and features --> High Memory Support --> 64GB
Processor type and features --> PAE Support
Processor type and features --> Paravirtualized quest support --> Xen guest support

8) make bzImage

I don't know if these kernel options are significant enough to require re-compiling all the modules. If so, I could probably script a solution without too much effort.
Title: Re: Custom Distro for Linode
Post by: Juanito on November 07, 2010, 08:37:15 PM
I could be mistaken, but doesn't "make mrproper" have to come before copying the .config file across (it'll be deleted otherwise)?
Title: Re: Custom Distro for Linode
Post by: danielibarnes on November 07, 2010, 09:03:23 PM
Oops. Post edited for correctness. :)
Title: Re: Custom Distro for Linode
Post by: curaga on November 08, 2010, 06:43:12 AM
CPU type change makes existing modules incompatible, and I think so does the mem change.
Title: Re: Custom Distro for Linode
Post by: tc@tinycore:#/ on November 08, 2010, 04:17:02 PM
Howdy, xnix here. Requested for xnix to be deleted, and registered this username instead, so there is no confusion as to who I am and why I'm continuing the topic.

I appreciate the offer about the script, and while I would definitely like that, I would still like to explore doing it myself. If you feel up to the task, having a script would give me something to refer to as I learn the process myself.

I'm curious as to more specifics to curaga's post, about this making existing modules incompatible. If all I'm doing is recompiling the kernel, then what modules are we talking about? I hear modules, and think extensions, so if this is something more specific to the kernel, then I'll read up on it.

edit - I'm guessing that these modules are the LKM's and not extensions. I suppose I should have made a quick search first. Where can I read about these said modules that apparently TC makes use of?

I'm going to try this out on my VM, see what I get from it. I'll keep this topic up to date on what happens, and I would much appreciate some more feedback and direction as well. Of course, if I don't do it myself, I'll never learn myself!
Title: Re: Custom Distro for Linode
Post by: danielibarnes on November 08, 2010, 04:31:55 PM
Quote
I'm guessing that these modules are the LKM's
Yes, you'll need to recompile all of the .ko.gz files under /lib/modules/2.6.33.3-tinycore/kernel. A simple "make" should build everything. You'll need to create a new "tinycore.gz" or "microcore.gz" with the new files, then boot your new kernel and core.gz in Xen.
Title: Re: Custom Distro for Linode
Post by: tinypoodle on November 08, 2010, 05:14:15 PM
http://tldp.org/HOWTO/Module-HOWTO/
Title: Re: Custom Distro for Linode
Post by: tc@tinycore:#/ on November 08, 2010, 06:24:38 PM
Thanks for the link. It all sounds pretty straight forward, but since this will be my first time messing with it, we'll see.

make bzImage
make modules
make modules_install

Although I don't get how making the modules will find themselves in the bzImage, I'll figure it all out when I actually do it. Perhaps I would be all the wiser not even bothering to make another post until I actually do a 'test,' that way I can come back with something worth posting.
Title: Re: Custom Distro for Linode
Post by: danielibarnes on November 08, 2010, 08:29:14 PM
The bzImage is the kernel. In Tiny Core, the modules are contained in microcore.gz. You can see this for yourself:

$ mkdir /tmp/rootfs
$ cd /tmp/rootfs
$ zcat /mnt/hdc/boot/microcore.gz | cpio -idv
$ ls lib/modules/2.6.33.3-tinycore/kernel

Note this assumes you have microcore on a cd mounted at /mnt/hdc.
Title: Re: Custom Distro for Linode
Post by: tc@tinycore:#/ on November 08, 2010, 09:59:17 PM
Oh! Makes sense. Learning something new about TC right now. I appreciate your patience. I might be getting over my head, being a novice Linux user, wanting to attempt a custom kernel, but this thread has definitely made sense of things. I think I can compile the kernel and modules now, especially since I know the difference between bzImage (which I had a feeling was the kernel itself) and tinycore.gz/microcore.gz (which I didn't have a clue concerning). Heck, even thinking about it, extlinux even lists the kernel image as, well, bzImage. I suppose I'm an idiot for not figuring this out sooner.
Title: Re: Custom Distro for Linode
Post by: majbthrd on December 16, 2010, 02:56:00 PM
I was also curious to see if TCL (specifically microcore) could be run as a paravirtualized Xen guest VM.

I have it working somewhat... the remaining problem I have is that the console output stops at:

Loading Tiny Core Extensions...Done
login[213]: root login on 'tty1'

TCL appears to still be running, and the console starts outputting text again when the VM is shutdown.

I've tried boot options like "text" and whatnot, but to no avail.

So, I assume that all that remains is to figure out how to coax TCL to use whatever virtual tty that Xen expects.

I'm documenting the process I've used in the hopes that it helps others, and perhaps someone much more familiar than I could suggest what remains to be done.

I'm going to divide this into several posts so it isn't one huge post.
Title: compile a XEN-enabled TCL kernel
Post by: majbthrd on December 16, 2010, 02:57:10 PM
Rather than try to patch the standard code, I took the easy way out, and used the patched code and config file available here:

http://distro.ibiblio.org/pub/linux/distributions/tinycorelinux/3.x/release/src/kernel/

# extract linux kernel source code

tar xjf linux-2.6.33.3-patched.tbz2

# go to created subdirectory

cd linux-2.6.33.3

# prep

make mrproper

# copy in default .config

cp ../config-2.6.33.3-tinycore .config

make oldconfig
make menuconfig

# alter configuration with menuconfig in this order:
# Processor type and features --> Processor family --> Pentium-Classic
# Processor type and features --> High Memory Support --> 64GB
# Processor type and features --> PAE Support
# Processor type and features --> Paravirtualized quest support --> Xen guest support
#
# Note: Pentium-Classic or higher is needed for XEN support
# this means a change from the existing TCL modules (which are compiled for 486)

# build kernel

make bzImage

# build modules

make modules

# copy modules (in this case, to ~/newmodules/, but you can pick your own directory of choice)

mkdir ~/newmodules/
make INSTALL_MOD_PATH=~/newmodules modules_install firmware_install
Title: Make a .XVA file system for Xen
Post by: majbthrd on December 16, 2010, 03:02:40 PM
# create raw image file (64 * (512 * 63 * 16 = 516096) = ~32MBytes)
# adjust the size to your tastes

dd if=/dev/zero of=image.bin count=64 bs=516096c

# create partition (-C64 means 64 cylinders, the size we decided upon in the previous step)

fdisk -cu -C64 -S63 -H16 image.bin

n (new partition)
p (primary partition)
1 (partition #1)
63 as start sector
default as last sector
a (toggle bootable)
1 (partition #1)
p (confirm it shows as bootable and that the start sector is 63)
w (write changes)

# mount image for formatting (32256 = 63 * 512)

sudo losetup -o32256 /dev/loop0 image.bin

# format

sudo mkfs.ext3 -b1024 /dev/loop0

# mount filesystem

sudo mount /dev/loop0 /mnt

# create directories

sudo mkdir -p /mnt/boot
sudo mkdir -p /mnt/tce

OR

sudo mkdir -p /mnt/boot
sudo mkdir -p /mnt/tce

# copy files

sudo cp ~/microcore.gz /mnt/boot/
sudo cp ~/linux-2.6.33.3/arch/x86/boot/bzImage /mnt/boot/

# unmount

sudo umount /mnt
sudo losetup -d /dev/loop0

# create XVA
# this requires python and xva.py; the latter is available here:
# http://www.xen.org/files/xva/xva.py

python xva.py -n my_test_image --is-pv --disk image.bin --filename=output.xva
Title: Loading XVA image into XEN
Post by: majbthrd on December 16, 2010, 03:11:59 PM
Copy the .xva file that you've created over to the XEN server.  Then:

xe vm-import filename=output.xva

IMPORTANT: note the UUID that it spits back out
if for some reason, you miss that UUID, you can figure it out with the command:

xe vm-list

note that we didn't bother putting a /boot/grub/menu.lst (or a syslinux.cfg)
This is because XEN uses pygrub

In theory, pygrub reads /boot/grub/menu.lst.

I originally tried putting a /boot/grub/menu.lst file in the partition, but I could NOT get pygrub in XEN to properly read it.

The workaround seems to set some additional XEN parameters.

All parameters for a particular VM (specified by the UUID which you should have saved earlier) can be listed with:

xe vm-param-list uuid=yourUUID

We need to change two of them:

xe vm-param-set uuid=yourUUID PV-bootloader-args="--kernel /boot/bzImage --ramdisk /boot/microcore.gz"

xe vm-param-set uuid=yourUUID PV-args="quiet console=hvc0 text initrd=/boot/microcore.gz"

Note that you can change the second to suit (and I'm not sure that what I have shown is right).

I am seeing a difference in behavior (for example, "quiet" versus no "quiet"), so I know it is being passed to the kernel.
Title: Re: compile a XEN-enabled TCL kernel
Post by: majbthrd on December 16, 2010, 03:16:50 PM
# Note: Pentium-Classic or higher is needed for XEN support
# this means a change from the existing TCL modules (which are compiled for 486)

Note that I left out replacing the modules in the microcore.gz/tinycore.gz image in my previous posts.

This is well documented in the wiki: http://wiki.tinycorelinux.com/Remastering

In my testing, I tried both replacing the modules and not, and neither changed the behavior of the VM (apart from the warnings when the kernel was compiled for Pentium and the modules were for 486).

Also note that all the modules take an obscene amount of disk space, which goes against the principle of TCL.  So, some sort of intelligent subset of all the modules should be copied over.
Title: Re: Custom Distro for Linode
Post by: maro on December 16, 2010, 03:30:43 PM
Regarding your "obscene" disk space issue: Have a look at this script (http://distro.ibiblio.org/pub/linux/distributions/tinycorelinux/3.x/release/src/kernel/sorter.sh). AFAIK it is used by the Core team to "slim down" the output of the kernel build process.
Title: Re: Custom Distro for Linode
Post by: majbthrd on December 16, 2010, 06:28:59 PM
Thanks, maro, for the pointer on paring down the size of the modules.

Answering my own question, the solution to the halt of console activity after:

login[213]: root login on 'tty1'

is covered in a previous forum thread:

http://forum.tinycorelinux.net/index.php?topic=5809.0

For running TCL as a Xen guest, I commented out the tty1 line in /etc/inittab and replaced it with:

hvc0::respawn:/sbin/rungetty hvc0

So, yes, it is perfectly viable to customize TCL to run as a Xen guest (aka domU).
Title: Re: Custom Distro for Linode
Post by: slamet on January 16, 2011, 04:16:26 AM
Not being knowledgeable enough, this looks a bit out of my league at present.  However, linode is the main reason I am very interested in micro core.  If this can be done successfully, then we may see uptake on micro / tiny's adoption by more users.

Suggestion: Depending on the final size of the build, can it be considered either to turn this into the base micro core or have another core (we can call it server / VPS core maybe).  This is up to the group obviously - just a suggestion.

We are not deviating from the "core" and toolkit concept.  Just to make the core suitable for running in VPS set up which is the way to go for many internet servers.  And there is a lot of demand out there to have something that runs frugally, the way MC does.  Once linode works, then someone else can try AWS and slicehost and then MC will rule the VPS market.

I understand about adding security and what not, but that is step 2.
Title: Re: Custom Distro for Linode
Post by: curaga on January 16, 2011, 04:43:45 AM
Depends on the VPS ;) We run fine in KVM ones and probably OpenVZ (that one depends on the host kernel). Don't know about VMWare ones, but I don't think they're that popular for VPS uses.
Title: Re: Custom Distro for Linode
Post by: slamet on January 16, 2011, 09:29:14 PM
Interesting.  A quick search shows there are options.  KVM seems to cost more than Open VZ.

Can you (or anyone) please share which vps hosts has been used / proven to run MC?  Many thanks.