Local initramfs generation script (stripping unneeded modules)

[gutmensch]

Heya,

just wanted to share with you all a script I'm using for generating local initramfs' for thin clients - as they don't need all modules, neither from base nor from the kernel extensions... and it can save quite some MB of space (e.g. tinycore.gz + extension modules about 10 MB, after creation mycore.gz about 6 MB).

The script builds up a new initramfs (let's say mycore.gz), which only includes all modules that you have loaded at this moment! It also incorporates all modules coming from extensions (e.g. filesystems-3.0.21-tinycore.tcz) and puts a marker for that extension into /usr/local/tce.installed (that means you cannot easily change the modules you generated with this type of local initramfs, be aware of that!)

Usage: Put script and core.gz in one folder and run

Code: [Select]

sudo ./local_initramfs.sh core.gz mycore.gz
It will also generate .modlist files for an easy overview about included modules.

Comments are very welcome!

Remember: No support for scripting action placed in this forum, just a proof of concept - use at your own risk! ;-)

Code: (bash) [Select]

#!/bin/sh

# Help
[ "$1" = "--help" ] && { echo Usage: $0 core.gz mycore.gz ; exit 0 ; }

# Tests
[ ! -e "$1" ] && { echo need existing core.gz path as first parameter. ; exit 1 ; }
[ ! -n "$2" ] && { echo need output gz name as parameter \(e.g. mycore.gz\). ; exit 1 ; }
[ `id -u` -ne 0 ] && { echo must be root. ; exit 1 ; }
[ -z "`which advdef`" ] && { echo install advcomp. ; exit 1 ; }
[ "`which find`" != "/usr/local/bin/find" ] && { echo install findutils. ; exit 1 ; }

# Source tc functions
. /etc/init.d/tc-functions

# Vars
TCKERNEL=`uname -r`
WORKDIR=`pwd`
TCEDIR="/etc/sysconfig/tcedir"
OLDSIZE=0
SRCFSNAME=`basename $1`
SRCFSSIZE=`ls -l $1 | awk '{print $5}'`
DESTFSNAME=`basename $2`

# Starting
echo "${YELLOW}Using initramfs ${WHITE}$SRCFSNAME [ `dc $SRCFSSIZE 1000000 div p | sed 's%\([0-9]*\.[0-9]\{2\}\).*%\1%'` MB ].${YELLOW}"
[ -f "$2" ] && { echo Removing old target file. ; rm -f "$2" ; }
tgztemp=`mktemp -d -p $WORKDIR`
chmod -R ugo+rwx $tgztemp
cp $1 $tgztemp/
cd $tgztemp
zcat $SRCFSNAME | cpio -i -H newc -d 2>/dev/null
find . -name *.ko.gz | sed 's%^\./%%g' > $WORKDIR/$SRCFSNAME.modlist
rm $SRCFSNAME

# Copying extension modules
for mod_ext_dir in `find /tmp/tcloop -maxdepth 1 -type d -name "*$TCKERNEL*"` ; do
  if [ -d "$mod_ext_dir/usr/local/lib/modules/$TCKERNEL" ] || [ -d "$mod_ext_dir/lib/modules/$TCKERNEL" ]; then
    short_dir=`basename $mod_ext_dir`
    mod_ext_size=`ls -l $TCEDIR/optional/$short_dir.tcz | awk '{print $5}'`
    OLDSIZE=`echo "$OLDSIZE $mod_ext_size add p" | dc`
    mkdir -p usr/local/tce.installed 2>/dev/null
    echo "Including modules from: ${WHITE}`basename $mod_ext_dir`${YELLOW}."
    touch usr/local/tce.installed/`basename $mod_ext_dir` && chown tc:staff usr/local/tce.installed/`basename $mod_ext_dir`
    cp -apfr $mod_ext_dir/usr/local/lib/modules/$TCKERNEL/* lib/modules/$TCKERNEL/ 2>/dev/null
    cp -apfr $mod_ext_dir/lib/modules/$TCKERNEL/* lib/modules/$TCKERNEL/ 2>/dev/null
  fi
done

# Removing unneeded modules
for mod_file in `find lib/modules/$TCKERNEL -name *.ko.gz` ; do
  sstr="$(basename $mod_file | sed -e 's%.ko.gz%%' -e 's%-%_%g' )"
  loaded=$(lsmod | grep -e "^$sstr[[:space:]]\{1\}")
  [ -z "$loaded" ] && rm -f $mod_file || echo "Saving ${RED}`basename $mod_file`${YELLOW}."
done

# Remove empty directories
find lib/modules/$TCKERNEL -type d -empty | xargs rm -rf

# Running depmod
/sbin/depmod -a -b $tgztemp $TCKERNEL || echo depmod failed.

# Packing new gz
[ -d "usr/local/tce.installed" ] && find usr/local/tce.installed/ -name "*$TCKERNEL" | sed 's%^\./%%g' > $WORKDIR/$DESTFSNAME.modlist
find . -name *.ko.gz | sed 's%^\./%%g' >> $WORKDIR/$DESTFSNAME.modlist
find | cpio -o -H newc | gzip -2 > $WORKDIR/temp.gz.$$
cd $WORKDIR
advdef -z4 temp.gz.$$ 1>/dev/null 2>&1
mv temp.gz.$$ $2
rm -rf $tgztemp

# Summary
OLDSIZE=$(echo "$OLDSIZE `ls -l $1 | awk '{print $5}'` add 1000000 div p" | dc | sed 's%\([0-9]*\.[0-9]\{2\}\).*$%\1%')
NEWSIZE=$(echo "`ls -l $2 | awk '{print $5}'` 1000000 div p" | dc | sed 's%\([0-9]*\.[0-9]\{2\}\).*$%\1%')
echo "Summary of used and saved space:"
echo "-------------------------------------------"
echo "${WHITE}Before :   $SRCFSNAME + mod exts   :  $OLDSIZE MB"
echo "${GREEN}After${WHITE}  :   ${GREEN}$DESTFSNAME${WHITE}            :  ${GREEN}$NEWSIZE MB"
echo "${YELLOW}-------------------------------------------${NORMAL}"

Updated for TC 4.x.

[jls]

Code: [Select]

Saving loop.ko.gz...
Saving ppdev.ko.gz...
find: unrecognized: -empty
BusyBox v1.18.3 (2011-02-11 12:08:19 PST) multi-call binary.

Usage: find [PATH]... [EXPRESSION]

Search for files. The default PATH is the current directory,
default EXPRESSION is '-print'

after installing findutils I don't see the error anymore
but I see this mesage:

Code: [Select]

Summary of used and saved space:
---------------------------------------------
Before :   tinycore.gz + mod exts   :  1 MB
After  :   scenic-microcore.gz            :  2 MB
---------------------------------------------

Notes: I used microcore.gz instead of tinycore.gz

trying the new microcore alsa doesn't work anymore.
 I have

Code: [Select]

modprobe snd-opl3sa2
in /opt/bootlocal.sh and I got:

Code: [Select]

modprobe: can't load module soundcore (kernel.tclocal/sound/soundcore.ko.gz): No such file or directory

modprobe: can't load module soundcore (kernel.tclocal/sound/soundcore.ko.gz): No such file or directory

[gutmensch]

Alright, updated in the original post - works now for me with stock microcore.gz and tinycore.gz, thanks!

Don't know what's going wrong with your special microcore.gz - it doesn't seem to be the original one?

[edit]
Are you using copy2fs mode for extensions? I think this script doesn't support this atm but it should only be a matter of switching from kernel to kernel.tclocal dir in the script (if I'm correct).
[/edit]

[jls]

I don't use copy2fs and also I use the official microcore.gz.
When I boot the new microcore I see an empty dir: /lib/modules/2.6.33.3-tinycore/kernel.tclocal/

I don't know if this can be a problem but I use persistent home & opt

[gutmensch]

please post
1. complete output of script
2. the content of microcore.gz.modlist
3. the content of yourmicrocore.gz.modlist
4. output of lsmod
5. output of ls /tmp/tcloop

[jls]

your script output:

Code: [Select]

tc@fisso:/tmp$ sudo local_initramfs.sh microcore.gz scenic_microcore.gz
Including modules from: alsa-modules-2.6.33.3-tinycore
Including modules from: input-joystick-2.6.33.3-tinycore
Saving soundcore.ko.gz...
Saving snd-wss-lib.ko.gz...
Saving snd-opl3sa2.ko.gz...
Saving snd-opl3-lib.ko.gz...
Saving snd-mpu401-uart.ko.gz...
Saving snd.ko.gz...
Saving snd-timer.ko.gz...
Saving snd-rawmidi.ko.gz...
Saving snd-pcm.ko.gz...
Saving snd-page-alloc.ko.gz...
Saving snd-hwdep.ko.gz...
Saving snd-seq-device.ko.gz...
Saving ramzswap.ko.gz...
Saving squashfs.ko.gz...
Saving 3c509.ko.gz...
Saving scsi_wait_scan.ko.gz...
Saving parport.ko.gz...
Saving parport_pc.ko.gz...
Saving loop.ko.gz...
Saving ppdev.ko.gz...
Summary of used and saved space:
---------------------------------------------
Before :   microcore.gz + mod exts   :  6 MB
After  :   scenic_microcore.gz            :  2 MB
---------------------------------------------
tc@fisso:/tmp$


Code: [Select]

tc@fisso:/tmp$ lsmod
Module                  Size  Used by    Not tainted
snd_opl3sa2             7320  0
snd_wss_lib            14212  1 snd_opl3sa2
snd_pcm                37628  1 snd_wss_lib
snd_page_alloc          4016  2 snd_wss_lib,snd_pcm
snd_opl3_lib            4888  1 snd_opl3sa2
snd_timer              10564  3 snd_wss_lib,snd_pcm,snd_opl3_lib
snd_hwdep               3004  1 snd_opl3_lib
snd_mpu401_uart         2780  1 snd_opl3sa2
snd_rawmidi            10004  1 snd_mpu401_uart
snd_seq_device          2808  2 snd_opl3_lib,snd_rawmidi
snd                    26200  9 snd_opl3sa2,snd_wss_lib,snd_pcm,snd_opl3_lib,snd_timer,snd_hwdep,snd_mpu401_uart,snd_rawmidi,snd_seq_device
soundcore               2640  1 snd
3c509                   5968  0
ppdev                   3732  0
squashfs               14728 23
parport_pc             18656  0
parport                18560  2 ppdev,parport_pc
scsi_wait_scan           276  0
ramzswap               10240  1
loop                    8068 46
tc@fisso:/tmp$ ls /tmp/tcloop/
Mplayer-nodeps/                   input-joystick-2.6.33.3-tinycore/
Xlibs/                            kmaps/
Xprogs/                           libasound/
Xvesa/                            libpci/
actkbd/                           ncurses/
advcomp/                          ncurses-common/
alsa/                             opera/
alsa-modules-2.6.33.3-tinycore/   pci-utils/
alsa-oss/                         tar/
findutils/                        vim/
flit/                             wbar/
flwm_topside/
tc@fisso:/tmp$



[gutmensch]

that looks ok, mine is also "2 MB" (the script is rounding to the lower value) with alsa modules included, real size is about 2600000 bytes...

can he still not load the soundcore module after booting with it?

[jls]

Code: [Select]

tc@fisso:~$ sudo modprobe snd-opl3sa2
modprobe: can't load module soundcore (kernel.tclocal/sound/soundcore.ko.gz): No such file or directory
tc@fisso:~$
infact the module is in kernel not in kernel.tclocal

[curaga]

Then it looks to me like depmod has not run properly. If you run "sudo depmod -a" on the running, after-script system, does the modprobe then work?

[jls]

yes

[gutmensch]

alright, got the same issue like you@jls and now added "-a" to depmod... for my stuff I'm also removing kernel.tclocal but that's up to everyone I think

[edit]
now more colors and precise calculations with bc, yeehaww.
[/edit]

[tinypoodle]

Very interesting approach which I think could be of great benefit to many users.

Few observations:
The script seems to ignore OSS modules.
Instead of using 'bc' which creates a dependency, perhaps 'dc' which is in base could be used?

[jls]

now it works without depmod

[gutmensch]

@tiny: thanks for testing! right now I'm not seeing any reason why it should ignore oss modules, I just modprobed snd-pcm-oss and snd-mixer-oss and it saved them like expected (of course it only saves the modules, which are loaded at runtime, so be sure to have them loaded when you want to have them in your image).

@dc: pheeewww. great idea, really! I love stack computing of numbers, it's so "logical" but a good idea anyway, will try to get it done!

[edit]
Alright, removed bc dependency, thanks for the hint@tiny!
[/edit]

[curaga]

@OSS, IMHO the script should also check for modules at $mod_ext_dir/lib/modules. It's not the standard place, but it's not always possible to place the modules in the usr/local path.