A Script to Simplify Remastering Micro Core

[vitex]

I am interested in using Micro Core Linux as the foundation for building special-purpose virtual machines that can execute without persistent storage. For such applications, a custom ISO is the ideal way to define the file system of a virtual machine. I have built a script that I call mcx (Micro Core eXtended) that I use to remaster Micro Core Linux. The script makes it easy to unpack an ISO, to add extensions and modifications to its file system, and to pack a new ISO.

Execution of the script mcx can be interleaved with regular shell commands that modify the file system that will be packed into the new ISO.  For example, here is a script that I use to remaster Micro Core to add the components needed for X and to change some boot parameters.

Code: [Select]

#!/bin/sh

[ $# -lt 1 ] && echo "Usage: $(basename $0) /path/to/MicroCoreIso" && exit 0

# Add the directory of this script to PATH.
PATH=$(dirname $0):$PATH

# Specify the directory in which to work on the ISO.
TMP=/tmp/mcx

# Unpack the ISO into /tmp/mcx/iso and /tmp/mcx/microcore.
mcx -unpack $1

# Add the components to implement X.
mcx -install ../release/core_elements/Xlibs.core.tczl
mcx -install ../release/core_elements/Xprogs.core.tczl
mcx -install ../release/core_elements/Xserver/Xfbdev.core.tcz
mcx -install flwm.tcz

# Add/change the boot parameters waitusb, vga, and timeout.
sed -i -e 's/microcore.gz quiet/microcore.gz waitusb=5 vga=788 quiet/' \
       -e 's/timeout 300/timeout 20/' \
    /tmp/mcx/iso/boot/isolinux/isolinux.cfg

# Copy mcx into /usr/local/bin on the ISO so it can be used on Micro Core.
cp $(which mcx) /tmp/mcx/microcore/usr/local/bin/

# Pack the files into the ISO /tmp/mcX.iso.
mcx -pack /tmp/mcX.iso

The script mcx can be downloaded from http://dl.dropbox.com/u/244822/mcx. Execute that script with no arguments to see more detailed usage information.

[beetle]

Hi,

I was having difficulties with your script then I realised that - being new to tc & mc and using the latest version - some things have recently changed that affects your script.

In the changelog I see "* New /opt/tcemirror replaces /opt/.tcrc"
..And I see your script depends on .tcrc

Do you plan into updating your script and posting it for us to use?

Thanks I advance!

[vitex]

Do you plan into updating your script and posting it for us to use?

Your message is the first indication that I have had that anyone is actually using the script.

I have been waiting for the 2.9 release to make a new release of the script.  I have replaced the old version with a new one that works for me with 2.8.1 and 2.9rc4.  The new version also supports remastering of Tiny Core as well as Micro Core.

Let me know if you run into problems or have questions or suggestions.

[beetle]

The script rocks! thanks a lot..
Does make my life easier!

[beetle]

Hi Vitex,

I have another question,

How can I tell mcx to get packages from a local storage instead of getting it always from the web server? I've looked around in your script where you define a Cache() function, but I could not figure out how to do what I want to do.

Like.. copying /tmp/mcx/cache somewhere, then retrieve it at some point when I build a new image to prevent depending on the internet, or using updated packages that I haven't  confirmed to work alongside my apps...

Thanks in advance...

[vitex]

How can I tell mcx to get packages from a local storage instead of getting it always from the web server? I've looked around in your script where you define a Cache() function, but I could not figure out how to do what I want to do.

The script mcx maintains a cache (default /tmp/mcx/cache) so it can avoid downloading extensions multiple times.  Each time mcx is commanded to load an extension, it fetches the corresponding .md5.txt file from the repository.  If the extension exits in the cache and its calculated MD5 sum matches the latest .md5.txt file from the repository, then mcx uses the extension from the cache.  Otherwise, mcx downloads the extension from the repository and saves a copy in the cache.  This algorithm always gives precedence to an extension in the repository, so putting a local extension in the cache will not give the behavior that you want.

If you want to save the cached extensions across reboots, one option is to use the "-tmp" option to change the location of the directory that contains temporary files (including the cache) so that directory is allocated in persistent storage.   (I have not used or tested the "-tmp" option.)

Another option is to use rsync to move the cache:

Code: [Select]

rsync -av /tmp/mcx/cache/ /path/to/my/storage/cache/   # Save the cache
...
rsync -av /path/to/my/storage/cache/ /tmp/mcx/cache/   # Restore the cache

Rsync has the advantage over cp that it does not copy files that are unchanged in the destination directory.

Like.. copying /tmp/mcx/cache somewhere, then retrieve it at some point when I build a new image to prevent depending on the internet, or using updated packages that I haven't  confirmed to work alongside my apps...

Below is an mcx script that I use to remaster Micro Core when I am testing a new extension.  It accepts a source ISO, a destination ISO, and a list of extensions.  If an extension is specified by a path containing "/", then that extension and any of its dependencies that are found in the same directory are loaded from that directory instead of the repository.   For example, to remaster Micro Core to contain the dropbear extension from the repository and a local extension that I am testing, I use something similar to:

Code: [Select]

./remaster /path/to/MicroCoreIso /path/to/RemasteredIso dropbear /path/to/my/extension.tcz


Code: [Select]

#!/bin/sh

[ $# -lt 2 ] && echo "Usage: $(basename $0) /path/to/MicroCoreIso /path/to/RemasteredIso [extension ...]" && exit 0
 
#=====================================================================

# If $EXT is a local file, copy it to /opt/tce/optional, add its name
# to /opt/tce/onboot.lst, and load its dependencies.  An attempt will
# be made to load those dependencies from the same directory as $EXT.
# If $EXT or one of its dependencies is not found in the same
# directory as $EXT, call mcx to fetch the extension from the repository.

Load () {
  local DIR=$(dirname "$EXT")
  local EXT=$(basename "$EXT")
  if [ -r "$EXT" ]
  then
    echo "Getting $DIR/$EXT"
    cp "$DIR/$EXT"* /tmp/mcx/microcore/opt/tce/optional/
    echo "$EXT" >>/tmp/mcx/microcore/opt/tce/onboot.lst
    for EXT in $(cat /tmp/mcx/microcore/opt/tce/optional/$EXT.dep 2>/dev/null)
    do
      EXT="$DIR/$EXT"
      Load
    done
  else
    mcx -install "$EXT"
  fi
}

#=====================================================================

# Remember the first two parameters.
SRC=$1 ; DST=$2 ; shift 2

# Add the directory of this script to PATH.
PATH=$PATH:$(dirname $0)

# Unpack the ISO.
mcx -unpack $SRC

# Install the user-specified extensions.
for EXT in "$@"
do

  # Add ".tcz" to the extension if needed.
  case "$EXT" in
    *.tcz)                ;;
    *.gz)                 ;;
    *)     EXT="$EXT.tcz" ;;
  esac

  # Load the extension with special processing if it contains "/".
  case "$EXT" in
    */*) Load "$EXT"         ;;
    *)   mcx -install "$EXT" ;;
  esac
 
done

# Pack the files into the ISO $DST.
mcx -pack $DST


This script or something similar will work with a local subset of the repository if you execute the script from within the local repository and list each extension that you want to load with a "./" prefix.

I will consider adding the capability to load local extensions to the next version of mcx.

[danielibarnes]

I have a remaster script I wrote which uses files in a "project" directory to remaster the .iso, but I like your approach better. Breaking it down into discrete, scripted steps simplifies using it, especially for novice users. I think I will try this method with my extension building script.

Why not use the directory in /opt/.tce_dir for your extension cache? I do that for my extension rebuilding script.

[vitex]

I have a remaster script I wrote which uses files in a "project" directory to remaster the .iso, but I like your approach better. Breaking it down into discrete, scripted steps simplifies using it, especially for novice users. I think I will try this method with my extension building script.

I like this approach because it relaxes the pressure to keep adding features to a single script.  The subcommands of mcx are essentially macros that that one can call in  higher-level shell scripts that implement custom features.  For example, I had not started building extensions when I did the original version of mcx, so I did not anticipate the need to remaster with experimental extensions.  A little shell scripting makes it possible to copy experimental extensions into /opt/tce/optional and add their names to /opt/tce/onboot.lst.  Another example is the special processing that I perform when I remaster with dropbear or openssh:  I copy my public keys into .ssh/authorized_keys of users tc and root, change configuration files as needed to permit root logins using public keys, and modify /opt/bootlocal.sh to start the ssh server.

Why not use the directory in /opt/.tce_dir for your extension cache? I do that for my extension rebuilding script.

I usually do not do my remastering on a Micro or Tiny Core system.

[vitex]

As suggested by jls_legalize, I have attached a copy of my last script, "remaster".

That script uses "/" to indicate that an item should be loaded from a local path instead of  the repository.  That interpretation of "/" is in conflict with using a repository-relative path to download a core element or a prerelease component.  In those items, it is probably best to use wget to download into a local directory from which the item can be loaded by the script.  Another approach is to remaster once with mcx to include such items in an ISO and to then use that ISO as the source ISO for further remastering.

[vitex]

I have upgraded the remastering script mcx to work with both the 2.x (x >= 9) and the 3.x extension repositories.  It determines which repository to use by examining the file /usr/share/doc/tc/release.txt of the ISO that it receives as its input.

[Jock]

This works for me, a nice piece of work.
Many thanks for sharing it with us.

Best regards
Jock