Tiny Core Linux

General TC => Programming & Scripting - Unofficial => Topic started by: danielibarnes on February 24, 2009, 10:40:42 AM

Title: Remaster Script
Post by: danielibarnes on February 24, 2009, 10:40:42 AM
I use Tiny Core Linux to create virtual appliances, and since I like to keep things current I frequently remaster it. I created a script to make this easier, and I am sharing it so that others may benefit (and hopefully improve upon).

Features
The script will optionally do any or all of the following:
1) Rebuild the kernel according to a specific .config
2) Rebuild the tinycore.gz root filesystem, including:
3) Create a new ISO image which includes:

Instructions
1) Load the following extensions into Tiny Core:
Not all of these are necessary (especially if the kernel isn't compiled).

2) Copy the following files into a src/ directory in the same directory as the remaster.sh script:
Here are some helpful links for these files:
ftp://distro.ibiblio.org/pub/linux/distributions/tinycorelinux/release/src/ (http://ftp://distro.ibiblio.org/pub/linux/distributions/tinycorelinux/release/src/)
ftp://distro.ibiblio.org/pub/linux/distributions/tinycorelinux/release (http://ftp://distro.ibiblio.org/pub/linux/distributions/tinycorelinux/release)

3) Run the remaster script. It takes two parameters:

Operation
Here is what the script does, step by step:
1) Loop mount the image (tinycore_VERSION.iso) and copy the files into a temporary directory.
2) Look for a tce/ directory in the project directory and copy files from it into the temporary directory.
3) If a rootfs/ directory or "modlist" file exists, then:
4) If a "config" file exists and a kernel/ directory does not exist, then:
5) If the kernel/linux-2.6.26/arch/x86/boot/bzImage file exists, copy it.
6) If a "modlist" file exists and the kernel/linux-2.6.26 directory exists, copy the specified kernel modules
7) Rebuild the tinycore.gz root filesystem if a new one was created in steps 4 or 6.
8) If an isolinux/ directory exists, then copy the custom isolinux files.
9) Create a new customized ISO image.

The output is placed in the out/ directory within the project directory. If the kernel is built, it will be placed in the project directory unless a kernel/ directory already exists. This will avoid building the kernel next time.

Summary
1) To build a new kernel, create a "config" file and place it in the project directory.
2) To use files (bzImage and/or modules) from a previously built kernel, place the kernel tree in a kernel/ directory.
3) To copy kernel modules into tinycore.gz, create a "modlist" file containing pathnames of the kernel modules.
4) To copy custom files into tinycore.gz, create a rootfs/ directory and place files there.
5) To create a custom isolinux boot configuration, create a isolinux/ directory and place files there.

Attached is a file containing a script and an example remaster project based on my VMware customizations. It does not contain a tce/ directory to save space. Extract the file and run the script with "sh remaster.sh example 1.1" to remaster against version 1.1.

The root filesystem customization in the example simply adds a modified /etc/inittab. The kernel is rebuilt to include support for LSI Logic and BusLogic SCSI adapters (emulated by VMware), remove support for the pcnet32 network adapter (so the VMware virtual adapter driver can load), and add support for VMI paravirtualization. The custom isolinux.cfg is almost exactly like the original; I just added a few boot parameters to the default and changed the timeout.

I tested the script quite a bit, but let me know if any improvements or fixes are necessary.

I compiled VMware Tools and created an extension for my personal use, but I am uncertain if distributution is allowed by VMware.
Title: Re: Remaster Script
Post by: ^thehatsrule^ on March 01, 2009, 08:57:02 PM
Took a quick glance... looks pretty good.

Just wondering why the `trap` is there, and why there is the check for some of the programs when they were already found by `which`
Title: Re: Remaster Script
Post by: mikshaw on March 02, 2009, 05:30:16 AM
Quote
Just wondering why the `trap` is there
It looks like it's to clean up the temp files in the event that the script is cancelled before it completes.

Quote
why there is the check for some of the programs when they were already found by `which`
We see similar things in other scripts, where a variable is set and then the value of the variable is checked.  It seems redundant to me too, but it's not uncommon.   On another note, I'd probably also be more specific about a fail, passing an error message to it:
Code: [Select]
fail() {
echo $*  #not sure what would be most useful here: $1, $@, $*
exit 1
}

for i in advdef make mkisofs ; do
echo -n "Finding $i: "
which $i || fail "Cannot find $i"
done
Title: Re: Remaster Script
Post by: danielibarnes on March 02, 2009, 10:16:41 AM
Thanks for the comments.

The trap is for cleanup. It's just personal script style, as is the "which". I have a template I use when writing a new script, and it contains ideas and other stuff I pick up from other scripts. It looks like overkill for a small script like this, but it also serves as a base for others I have which are more complicated.
Title: Re: Remaster Script
Post by: helander on March 10, 2009, 01:39:46 AM
Hi,

I am new to TinyCore and I believe that this script is precisely what I want/need.
Being new I have not found my ways around yet, so I have no clue on from where to download this script :(

Any help on how I can get this script is highly appreciated.

Kind Regards

/Lars
Title: Re: Remaster Script
Post by: mikshaw on March 10, 2009, 03:39:40 AM
It is attached to the first post....look for the paperclip at the end of the post.

I'm not convinced that remastering is what you need as a new user.  Considering TC was built from scratch it is a bit different than your typical distro.  If a problem occurs during a remaster, it may not be as easy to debug.  I'm not saying don't do it, just suggesting that you check out TC's abilities as-is for a start.
Title: Re: Remaster Script
Post by: helander on March 10, 2009, 04:59:22 AM
It is attached to the first post....look for the paperclip at the end of the post.

I'm not convinced that remastering is what you need as a new user.  Considering TC was built from scratch it is a bit different than your typical distro.  If a problem occurs during a remaster, it may not be as easy to debug.  I'm not saying don't do it, just suggesting that you check out TC's abilities as-is for a start.

Thanks. Found it and downloaded it :)

One thing I will need to do is to build my own version of the kernel, since the current kernel does not match all my needs. I have built kernels for a multitude of distros and having a "framework" to that in helps a lot. Since building the kernel is done within the remaster script I saw that as useful, either to use fully or just to look at a process that works :)

Primarily I will build a kernel configured with CONFIGURE_PREEMPT and secondly one with CONFIGURE_PREEMPT_RT. The latter requires the RT patches to be applied so I will start out with the former, which only requires a small modification of the current TC kernel config.

Further along I might want to build an ISO with a specific set of packages installed - and for creating that the remaster script seemed useful.

Kind Regards

/Lars
Title: Re: Remaster Script
Post by: mikshaw on March 10, 2009, 05:14:38 PM
ahh...so as a "new user" you meant specifically to TC, and definitely not to linux =o)

I have very little experience or success with compiling the kernel myself, so on that subject you are already way ahead of me, so I won't do anything more to dissuade you from your quest.

On the subject of packages, though, i will suggest you consider sticking with extensions rather than adding them to a remaster as installed applications/  Tis will allow you to more easily upgrade specific apps, and might save you some ram by using tcz if that is a concern for you.
Title: Re: Remaster Script
Post by: florian on March 11, 2009, 10:25:33 AM
I'm in similar position that mikshaw in that I don't  have much experience with kernel tuning and even recompiling. I admire people who attempt such things and often promise myself that someday I will  ;)

Meanwhile I've also found that on the forums, some are are quick to think they need to remaster. Of course there are cases where you really need a remaster (for modified kernel for instance), but in most cases this is not necessary thanks to the mechanisms offered by tc with the boot codes and its fantastic extension system
Title: Re: Remaster Script
Post by: rhombic on April 15, 2009, 08:33:09 AM
 ;D  This has been a most informative conversation and I am certiain it will help my efforts.

I do have a special need.  I need to install an Kontron JIDA I/O package library and it would be handy if someone could show me how to "sneek" this in without a total rebuild.

Otherwise I am about to  boldly go where .... and follow the script thanks

bobo
Title: Re: Remaster Script
Post by: danielibarnes on April 15, 2009, 10:10:15 AM
To add to florian's comment, I, too, notice there is often confusion about remastering vs. extensions. There are very few cases where you need to remaster, namely to:

In short, you only need to remaster for something that is needed before extensions are loaded. Everything else can go into an extension. Keep in mind that an extension in its simplest case (.tce) is just a tar.gz which is extracted to the root filesystem at bootup. If you want to add programs or modify the filesystem, follow the wiki guidelines for creating an extension. That is the real strength of TC, and it will serve you well.

Daniel
Title: Re: Remaster Script
Post by: mwhit95 on May 11, 2009, 12:00:56 PM
I just run into a problem with extensions so I was looking at a remaster.  I am using about 20 or so tcz type extensions and I noticed some of them didn't load because it was too many loops.  I was thinking that it might be good to take all of the smaller extensions and combine into one but I am not sure how to do that.
Title: Re: Remaster Script
Post by: curaga on May 11, 2009, 12:10:47 PM
You can increase the default up to 255 (max_loop=xx as a boot argument). The default has been upped to 80 in tc2.
Title: Remaster File Disappeared
Post by: JoXo009 on August 27, 2009, 02:51:03 AM
Quote
I created a script to make this easier
Sonds great.

But I fear the remaster script file disappeared in the meantime.

Unfortunately the so-called helpful links don't link to specific files but to web folders only.

(The content of these foldes becomes visible in your browser it you copy the link into the address bar and exchange 'http' for 'ftp'.)

And in fact the four files listed under 'Instructions top2' do exist in that folder. See my posting
http://forum.tinycorelinux.net/index.php?topic=2794.msg14191#msg14191

But just the main file, the remaster script which (see: 'Instructions top2') should bear the name 'remaster.sh' doesn't exist within the linked web-folders.

As this thread got over 2,500 hits and earlier postings discuss the content of the remaster file I suppose that the linked web-folder contained that file at least till May.

But in the meantime that file seems to have disappeared.

Any help appreciated.
Title: File Disappeared
Post by: JoXo009 on August 27, 2009, 03:06:22 AM
By the way.

Using my FileTransferClient and the original link:
ftp://distro.ibiblio.org/pub/linux/distributions/tinycorelinux/release/src/

the result is the same.

There is no remaster script with the name 'remaster.sh' anymore.    :(
Title: Sorry my fault
Post by: JoXo009 on August 27, 2009, 03:18:51 AM
Until now I didn't know that the software of this forum makes attachemts invisilbe if your aren't logged in.

After another user gave me that hint, I can see and download the remaster script now.

Would be nice if the forum software would present the same content no matter if you are logged in or not.

It could ask for login, if you try to download. That would be a far better solution than hiding important content.
Title: Forum Software Bug
Post by: JoXo009 on August 27, 2009, 04:32:51 AM

My above error, how could it happen?

Have a look at the following screenshot:

(http://i30.tinypic.com/34hyzau.jpg)
Title: Re: Remaster Script
Post by: tclfan on August 27, 2009, 06:52:11 AM
I compiled VMware Tools and created an extension for my personal use, but I am uncertain if distributution is allowed by VMware.

Hi Danielibarnes,

Vmware Tools ready for TC would be a tremendous asset if you can post it.  This is usually an enourmous pain for an end user (not developer) to install VMware Tools in Linux. Although I did not research the terms of free distribution, but I know that compiling VMware Tools and distributing VM virtual machines with VMware Tools included is quite popular and such virtual machines can be published on VMware 'Virtual Appliance' site as third party products. Bagvapp (bagside.com) is probably the biggest contributor compiling and publishing large number of free virtual appliances, all of them with VMware Tools installed in them by default. Tiny Core Linux is still missing from the list... Installed VMware Tools are very important in virtual machines, as this is key to overall integration of such virtual machine with the host. I myself would greatly appreciate if VMware Tools could be made available as tc extension or a ready to go TC virtual machine including VMware Tools (basic configuration) is published. This could lead to a variety of virtual appliances based in Tiny Core, along with the efficiency, speed  and small size coming with it...
Title: Re: Remaster Script
Post by: tclfan on August 27, 2009, 07:27:09 AM
I compiled VMware Tools and created an extension for my personal use, but I am uncertain if distributution is allowed by VMware.
Just a quick question: If you publish VMware Tools as tc extension (tcz?) in the repository, will VMware Tools be installed in the TC virtual appliance automatically upon loading tc extension, or it will still require manual installation? I read that it is so complex that for some Linux distro's (I think Puppy comes to my mind as example) has not been done yet... This even more emphasizes that if such task has been accomplished for TC, this is a significant asset, that could make TC virtual appliances popular. I myself have been playing with various virtual appliances for the past several years and most are too heavy. Bagvapp has not posted Slitaz 2.0 yet, just 'cooking', which means beta version, being busy with making virtual machines containing Ubuntus (various) and Windows 7. If TC virtual appliance is posted, first thing I see it will do is replace the original secure browser appliance, which is based on Ubuntu, originally posted by VMware for secure browsing on Windows machines...
If you decide to not post TC virtual machine on the VMware Appliances site, can we have it available on ibiblio or some place else for download?
Thanks very much. I have been waiting for this to happen...
Title: virtual TC
Post by: JoXo009 on August 27, 2009, 07:38:32 AM
Hi TCLFAN,

did you give VirtualBox a try too?

I think to use TC as guest within a virtual machine that's just what Danielibarnes has in mind the same as I do.

But using VMware for public extensions could be difficult for license reasons.

Fortunately in the meantime there is Sun's VirtualBox, leading in performance and not only available as a PULP licensed (free personal usage) version but also as a GNU version.

So to my opinion TC and vBox fit perfectly together.

By courtesy of KingDomcome there is already a script how to integrate the Guest Additions into TC:
http://forum.tinycorelinux.net/index.php?topic=2755.0

And perhaps in the near future w'll see more to come.

JoXo009
Title: Re: Remaster Script
Post by: danielibarnes on August 27, 2009, 07:41:02 AM
The repository has my open-vm-tools extension, which is the open source equivalent of VMware Tools. I think that appliances are licensed in such a way that allow for VMware Tools to be included. I preferred to bypass the whole issue by packaging open-vm-tools. I just built an extension based on the 8/24 release. I will get it to Jason today.

Performance is also improved by compiling the kernel and SCSI modules with VMI support and using them with a VMware product which supports paravirtualization (like ESX/ESXi). I remastered 1.4.3, 2.2, and 2.3rc2 with this support and will be posting it very soon.
Title: Re: Remaster Script
Post by: JoXo009 on August 27, 2009, 07:58:15 AM

Great, the VM Tools are open source and can be distributed.

But VMware itself does there exist of an open version too, can it be included into an extension of its own?
Title: Re: Remaster Script
Post by: tclfan on August 27, 2009, 10:21:51 AM
The repository has my open-vm-tools extension, which is the open source equivalent of VMware Tools. I think that appliances are licensed in such a way that allow for VMware Tools to be included. I preferred to bypass the whole issue by packaging open-vm-tools. I just built an extension based on the 8/24 release. I will get it to Jason today.

Performance is also improved by compiling the kernel and SCSI modules with VMI support and using them with a VMware product which supports paravirtualization (like ESX/ESXi). I remastered 1.4.3, 2.2, and 2.3rc2 with this support and will be posting it very soon.

Do open-vm-tools provide the same functionality and performance as the VMware Tools? This means video support, functionality and performance, seamless mouse integration, hibernate on VM close, Resume on restart, etc? In a word, should I expect equal, high quality experience as using VMware Tools? I have not researched open-vm-tools, so pardon my question...
On VMI support/paravirtualization, I understand the current kernel does not include such support and needed to be re-compiled? I sort of mistakenly assumed all modern kernels have such support by default...
If this is the case, then would this not be beneficial to use such kernel in TC by default, which would bring this capability to all TC users?
E.g. if I am an end user, not a developer, and I want to create a TC VM, I could use a stock TC release along with vm-tools, not having to re-compile the kernel, since recompiling is not what comes to end-user's mind as realistic.
Looking forward to the TC VM posted... Thank You!
Title: Re: virtual TC
Post by: tclfan on August 27, 2009, 11:09:09 AM
did you give VirtualBox a try too?
I tried several VM solutions in the past (not recently), but at that time nothing was coming close to VMware. Perhaps things changed in the meantime.
I read somewhat about VirtualBox recently, but it is not clear to me if it has some key features, like video support and performance, dynamic resolution, audio transparent support, seamless mouse integration and hibernating on VM close? These are sometimes overlooked also among VMware appliances. On VMware, these features are implemented by installing VMware Tools, apparently as part of integration with the host system and hardware on the host.
I will try VirtualBox as soon I have the time, though. From my reading it seems to resemble more like experience of running a LiveCD ISO in a WMware VM. I may be wrong on this, since I did not test myself the recent versions.  Optimal use of VM is complete integration of VMs with the host.
Please correct me if I am wrong on my expectations...
Title: Re: Remaster Script
Post by: JoXo009 on August 27, 2009, 11:32:50 AM

> I tried several VM solutions in the past ...

Yea, that was in the past.

VirtualBox 3.0.4 was released August 4, 2009

DeDoimedo reports:

VirtualBox 3.0.0 is amazing!

First, let's take a look at a short comparison between VirtualBox and VMware Server, just so you know what my motives are and why I think VirtualBox 3 is phenomenal ....

... read it yourself on
http://www.dedoimedo.com/computers/virtualbox-3.html

and give it a try, switch on the guest additions and you get seamless mouse and much more.




Title: Re: Remaster Script
Post by: tclfan on August 27, 2009, 11:52:10 AM
VirtualBox 3.0.0 is amazing!
First, let's take a look at a short comparison between VirtualBox and VMware Server, just so you know what my motives are and why I think VirtualBox 3 is phenomenal ....
I was hoping for something here, until I saw the above is comparison to VMware Server.  Fact is VMware Server is a server hosted virtualization and does not have the quoted features... I have tested VMware server, including the latest version 4 and as much as it is OK to run VMs and applications inside with remote admnistration, it is to me virtually useless as a desktop virtualization solution and it is not meant to be used as such. A proper comparison would be to VMware Workstation 6.5 and VMware Player 2.5, but any version of these VMware products would do.
But as I said, I will test VirtualBox at some point and perhaps discover there is more in it than just comparing it to VMware Server...
Title: Re: Remaster Script
Post by: danielibarnes on August 27, 2009, 04:14:24 PM
Quote
Do open-vm-tools provide the same functionality and performance as the VMware Tools?
Probably not exactly the same. If someone has the time to compare them after I submit the updated extensions that would be great as I have not had time yet.

Quote
On VMI support/paravirtualization, I understand the current kernel does not include such support and needed to be re-compiled?
Yes, the stock TC kernel is not compiled with VMI support. It is a feature used by a relatively small percentage of users, although it seems more and more people every day are finding TC (especially microcore) makes an ideal base for a virtual appliance. I started posting my remastered versions to serve that minority, but perhaps the option could be considered for TC 3.0 if it does not impact kernel/rootfs size.
Title: Re: Remaster Script
Post by: curaga on August 28, 2009, 01:14:41 PM
It is against VMWare's license to post benchmarks including their products without their acceptance :P

@danielibarnes: Please PM me with the desired config options, and I'll look into them.
Title: Re: Remaster Script
Post by: danielibarnes on August 28, 2009, 01:39:21 PM
Quote
It is against VMWare's license to post benchmarks including their products without their acceptance.
To paraphrase an earlier quote,

VMware ESXi 4.0.0 is amazing!

:)
Title: Re: Remaster Script
Post by: JoXo009 on August 28, 2009, 11:58:17 PM
Quote
VMware ESXi 4.0.0 is amazing!
True.

Certainly true for companies who don't need to care about $ 1,000 for the ESXi basis pack with three licenses.

For other useres VirtualBox and its free OSE version might be a better choice.

So it would be great if TC could support both VMware and VirtualBox.

Title: Re: Remaster Script
Post by: Kingdomcome on August 29, 2009, 01:48:45 PM
I have been working on making a VirtualBox-OSE extension that I hope to wrap it up this weekend.  In the meantime I just submitted the OSE Guest Additions as a tcz for 2.x.  Although it does require Xorg for the X related functions, shared folders will work from a terminal.
Title: Re: Remaster Script
Post by: danielibarnes on August 29, 2009, 04:41:03 PM
I had the same dependency on X when I built open-vm-tools, so I built two versions: open-vm-tools and open-vm-tools-withx. There was a 2x difference in size and number of dependencies when including X, but I also wanted a small extension which would include the kernel modules and simple stuff. Basically, an extension that works for microcore and one that will work for tinycore.

Title: Re: Remaster Script
Post by: TooFar on September 05, 2009, 02:03:32 PM
I used the script to simply test the size of the resultant tinycore.gz compared to the downloadable one, the file size increased by 4MB, does anyone have the config file that was used to build the downloadable tinycore.gz, I would like to see the options that are used by this config compared to config-... that is used by the script.
Title: Re: Remaster Script
Post by: sonu on September 11, 2009, 01:50:05 AM
I reckon this script should be added to the desktop menu....just like in DSL... "Make Remaster CD" and prompt the user where they want to save the iso...

God I love this distro...i always though DSL was the best you could get for under 50Mb...but tinycore blew me away....i was banging my head trying to mount windows partition in DSL using Qemu...i tried smbclient and this and that....and when i found tinycore i seen the FAQ and i was accessing windows in less than 5 minute of downloading tinycore...

but i am after all a n00b...so i will be spamming the forum with stupid questions pretty soon.
 ;D
Title: Re: Remaster Script
Post by: danielibarnes on December 08, 2009, 11:48:04 AM
I split building the kernel into a separate script. I attached it to help anyone trying to rebuild the kernel.  It references a kernel/ directory in the same directory as the script. This directory should have: linux-2.6.29.1.tar.bz2 (http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.29.1.tar.bz2), config-2.6.29.1-tinycore (http://distro.ibiblio.org/pub/linux/distributions/tinycorelinux/2.x/release/src/config-2.6.29.1-tinycore), and 2.6.29.1-tinycore.patch (http://distro.ibiblio.org/pub/linux/distributions/tinycorelinux/2.x/release/src/2.6.29.1-tinycore.patch.gz) (uncompress the .gz first!) files in it. It also compiles the 1.x kernel if you have the appropriate source files in the kernel/ directory.

It takes one parameter, the name of your specialized config file, which it copies into the build directory as .config at the appropriate time. You can also add a "-n" switch to skip building the kernel. This leaves the files in /tmp/kernel so you can make menuconfig and customize your .config.

This script builds in /tmp/kernel, so make sure you have about 1GB of RAM when you build.
Title: Re: Remaster Script
Post by: Ulysses_ on April 01, 2011, 11:20:07 AM
I use Tiny Core Linux to create virtual appliances, and since I like to keep things current I frequently remaster it.

Do you have a prebuilt virtual appliance that I can download by any chance?  With official vmware tools installed?
Title: Re: Remaster Script
Post by: danielibarnes on April 01, 2011, 11:32:48 AM
I don't have a pre-built virtual appliance. I usually just remaster an .iso image with the extensions I need. There are a couple extensions I've neglected to finish but are almost complete: an updated open-vm-tools and vm-scsi-2.6.33.3-tinycore.

I don't redistribute the official VMware Tools, but I can help you compile them for yourself after I get these two tasks out of the way. It's been a long time since I've used the official Tools.
Title: Re: Remaster Script
Post by: Ulysses_ on April 01, 2011, 11:44:00 AM
There are a couple extensions I've neglected to finish but are almost complete: an updated open-vm-tools and vm-scsi-2.6.33.3-tinycore.

Once you're done with those, will adding the open-vm-tools extension to an ordinary VM support arbitrary screen resize, copy-paste from VM to host and time synchronization between VM and host?

Or do these features need a complete remaster?
Title: Re: Remaster Script
Post by: danielibarnes on April 01, 2011, 11:51:31 AM
Quote
will adding the open-vm-tools extension to an ordinary VM support arbitrary screen resize, copy-paste from VM to host and time synchronization between VM and host?

I will test each of those things and let you know.
Title: Re: Remaster Script
Post by: Ulysses_ on April 14, 2011, 02:05:24 PM
Quote
will adding the open-vm-tools extension to an ordinary VM support arbitrary screen resize, copy-paste from VM to host and time synchronization between VM and host?

Just tested and they all seem to work.  Except copy-paste with files, rather than text.  Is a file manager other than the default FileMgr required?
Title: Re: Remaster Script
Post by: danielibarnes on April 14, 2011, 02:14:52 PM
From http://mentoliptus.blogspot.com/2006/06/vmware-player-drag-n-drop-howto.html:
"In the case of a Linux based virtual machine and a Windows host, VMware Player only supports textual copy and paste between them"

Better to use Shared Folders if you want to transfer files.