Tiny Core Linux
General TC => Programming & Scripting - Unofficial => Topic started by: helander on February 14, 2010, 02:59:11 PM
-
The script below, which I call tcz-unload, allows you to unload an installed extension. By unload I mean that it:
- removes all symlinks to the extension
- unmounts and removes it from /tmp/tcloop
- removes the entry in /usr/local/tce.installed
But before it allows you to do this it checks, using the lsof program (i.e. means that you have to have the lsof.tcz extension installed), that no program is currently using any of the files that are to be "removed".
NOTE! This does not guarantee that any program will not try to access these files "in the future" and if so happens, unexpected things might happen.
All programs that currently is using files (= files opened) within the extension is listed, so that you can take actions to terminate these programs if so required.
This tool is very useful to me when I create/modify extensions and need to repeatedly re-install new "versions".
/Lars
#!/bin/sh
listlines() {
for i in $*; do
echo $i
done
}
package=$1
if [ -z "$package" ]; then
echo " [Error]: No package name specified"
exit 1
fi
if [ -d /tmp/tcloop/$package ]; then
usedby=`lsof +D /tmp/tcloop/$package -Fp | sed -e "s/^p//"`
if [ "$usedby" ]; then
echo "Files in extension $package are currently in use by the following programs:"
usedby=`listlines $usedby | sort -ug`
for p in $usedby; do
/bin/ps | awk "/^$p/ "'{$1="";$2="";print $0}'
done
echo " [Error]: Extension $package has not been unloaded."
exit 1
fi
f=`cd /tmp/tcloop/$package; find ! -type d | sed -e "s%^./%/%"`
for l in $f; do
if [ -s $l ]; then
linkto=`readlink $l`
if [ "$linkto" == "/tmp/tcloop/$package$l" ]; then
echo " Unlink $l from $linkto"
sudo rm $l
else
echo " [Warning]: $l links to $linkto"
fi
else
echo " [Warning]: $l is not a link"
fi
done
sudo umount /tmp/tcloop/$package
sudo rmdir /tmp/tcloop/$package
else
echo " [Warning]: /tmp/tcloop/$package does not exist"
fi
sudo rm -f /usr/local/tce.installed/$package
-
The script below, which I call tcz-unload, allows you to unload an installed extension. By unload I mean that it:
- removes all symlinks to the extension
- unmounts and removes it from /tmp/tcloop
- removes the entry in /usr/local/tce.installed
But before it allows you to do this it checks, using the lsof program (i.e. means that you have to have the lsof.tcz extension installed), that no program is currently using any of the files that are to be "removed".
NOTE! This does not guarantee that any program will not try to access these files "in the future" and if so happens, unexpected things might happen.
All programs that currently is using files (= files opened) within the extension is listed, so that you can take actions to terminate these programs if so required.
This tool is very useful to me when I create/modify extensions and need to repeatedly re-install new "versions".
/Lars
#!/bin/sh
listlines() {
for i in $*; do
echo $i
done
}
package=$1
if [ -z "$package" ]; then
echo " [Error]: No package name specified"
exit 1
fi
if [ -d /tmp/tcloop/$package ]; then
usedby=`lsof +D /tmp/tcloop/$package -Fp | sed -e "s/^p//"`
if [ "$usedby" ]; then
echo "Files in extension $package are currently in use by the following programs:"
usedby=`listlines $usedby | sort -ug`
for p in $usedby; do
/bin/ps | awk "/^$p/ "'{$1="";$2="";print $0}'
done
echo " [Error]: Extension $package has not been unloaded."
exit 1
fi
f=`cd /tmp/tcloop/$package; find ! -type d | sed -e "s%^./%/%"`
for l in $f; do
if [ -s $l ]; then
linkto=`readlink $l`
if [ "$linkto" == "/tmp/tcloop/$package$l" ]; then
echo " Unlink $l from $linkto"
sudo rm $l
else
echo " [Warning]: $l links to $linkto"
fi
else
echo " [Warning]: $l is not a link"
fi
done
sudo umount /tmp/tcloop/$package
sudo rmdir /tmp/tcloop/$package
else
echo " [Warning]: /tmp/tcloop/$package does not exist"
fi
sudo rm -f /usr/local/tce.installed/$package
It's good to see someone has taken up the extension uninstall challenge again. There has been extensive discussion on this subject for quite some time. See:
http://forum.tinycorelinux.net/index.php?PHPSESSID=649fac8d1ce81103810539df92f9ac58&topic=2081.0
Have you considered all the issues that have been discussed regarding uninstalling extensions without performing a "base norestore" reboot before the uninstall and a reboot after the uninstall? Have you used Jason's uninstall script as a guide?
eidit: sorry for not paying closer attention, unload is obviously different than uninstall. However, if I'm not mistaken some of the issues discussed in the uninstall thread apply to doing a "live" unload. Anyway, I hope we can get a reliable unload to work in conjunction with Ondemand "tce-load".
-
Prior to your reply I had not read the discussion you pointed to nor I had I seen Jason's script. Now I have :) at least read the discussion.
I fully support Jason's view and I think that a tool like this could only be used with great caution and if TC should support removal of installed extensions arbitrarily, this would add complexity that I think none would be willing to pay the "cost" for.
As I stated, and others in the discussion you pointed to also did, was that in some contexts a tool like this is very helpful.
Since I can not access Jason's script I can not tell how much different it is from mine, this said you probably understand that I did not base my design on anything from Jason's work.
From the discussion I conclude that in principle the solutions are similar, but the "novelty" of my approach is the use of lsof to pro-actively not allow removal in case it will not be possible to complete immediately. The approach of using "umount -l" is simple but it leaves the system in a state where you do not know when the operation will complete, if ever. My solutions provides the user with information what could be done in order to complete the operation immediately and then it is up to the user to decide on taking these actions, or defer this to a later point, e.g. wait until the applications no longer are used or reboot the system in case you need to uninstall something you can not be without, e.g. an extension used by your window manager.
/Lars
-
Prior to your reply I had not read the discussion you pointed to nor I had I seen Jason's script. Now I have :) at least read the discussion.
I fully support Jason's view and I think that a tool like this could only be used with great caution and if TC should support removal of installed extensions arbitrarily, this would add complexity that I think none would be willing to pay the "cost" for.
As I stated, and others in the discussion you pointed to also did, was that in some contexts a tool like this is very helpful.
Since I can not access Jason's script I can not tell how much different it is from mine, this said you probably understand that I did not base my design on anything from Jason's work.
From the discussion I conclude that in principle the solutions are similar, but the "novelty" of my approach is the use of lsof to pro-actively not allow removal in case it will not be possible to complete immediately. The approach of using "umount -l" is simple but it leaves the system in a state where you do not know when the operation will complete, if ever. My solutions provides the user with information what could be done in order to complete the operation immediately and then it is up to the user to decide on taking these actions, or defer this to a later point, e.g. wait until the applications no longer are used or reboot the system in case you need to uninstall something you can not be without, e.g. an extension used by your window manager.
/Lars
Jason could you provide the last version of your uninstall script for general reference?
-
Honestly I don't a copy of it in it's final state. I kind of wish I kept a copy for code reference.
-
Wondering if the needed functionality of 'lsof' could be replaced by using 'fuser' instead, as latter is part of base.
-
What about changing
sudo umount /tmp/tcloop/$package
to
sudo umount -d /tmp/tcloop/$package
?
See http://forum.tinycorelinux.net/index.php?topic=7998.0 (http://forum.tinycorelinux.net/index.php?topic=7998.0)
-
What about changing
sudo umount /tmp/tcloop/$package
to
sudo umount -d /tmp/tcloop/$package
?
See http://forum.tinycorelinux.net/index.php?topic=7998.0 (http://forum.tinycorelinux.net/index.php?topic=7998.0)
+1
-
I also made a very similar script as part of tcztools package, for helping me with extension building.
You can see it here: http://code.google.com/p/tcztools/source/browse/tcz-umount
The only difference with mine is that it shows you any files that were changed/removed since extension was loaded:
Files that are not symlinks to the extension loop mount anymore are shown
with a '?' in front, and those which are missing with '!'. In dry-run mode
it also shows the symlinks that would have been deleted.
I'm thinking of also adding a reverse-dependencies check by consulting the .dep files of currently loaded extensions to see if any one contains the extension you want to umount and maybe warn you about it.
-
I devoted some time to the issue, and produced my version, called untcz.
tcz-unload is not very error-tolerant: just pass it the argument
$package/ (trailing slash) and it umounts extension without unlinking.
tcztools is even worse, I don't like its approach at all.
Plus, lsof complexity can be avoided.
In my script, this is done by directly umounting, and giving up on umount error.
To do this, I have to postpone unlinking after umounting.
This requires saving a list of to-do unlinks somewhere.
The drawback is that I need temp space on filesystem, and it can go up to megs
for biggest extensions (texlive.tcz, a custom extension I took as testbench for its size).
The highligts are:
- more robust fault tolerance: you shouldn't get umount without
unlinking done
- no relevant speed loss with this approach: taking my texlive.tcz as a testbench,
I got 1m30s with tcz-unload, and 1m36s with untcz
- the user can decide to remove all directories emptied in the process.
For some private-directories-rich extensions, like texlive, this is important.
In this case untcz-ing time increases.
- less complexity (smaller than tcz-unload, and it can be reduced pruning
comments and debug lines)
- corrected umount -d bug reported above
Anyway, from a time complexity point of view, the problem seems hard
if confined to scripting.
I have another approach in mind, if I have time I will give it a shot.
Performance aside, the following issues remain:
- - While tinycore provides the /usr/local/tce.installed/script to do load management tasks, there is no mechanism to do the symmetrical tasks when unloading.
- - No check if there are extensions depending on the one being unloaded.
This is would be best done via an external wrapper, in my opinion.
Hoping it helps, here is my untcz, comments and reports welcome:
#!/bin/sh
# Copyright 2011 Marco Caminati. Released under GPLv3.
# Pass any content to $keepdir to prevent deleting the directories emptied in the process
set -e
set -u
set -x
b=busybox
name=`$b echo "$1" | $b sed -e 's_/*$__' | $b egrep -o "[^/]*$"`
tmpd=/tmp/untcz
mntdir=/tmp/tcloop/$name
keepdir="${keepdir:-}"
cd $tmpd 2>/dev/null && exit 1
$b mkdir -p "$tmpd"
cd "$tmpd"
find "${mntdir}" ! -type d | while read i
do
i=/${i#*$mntdir}
readlink "$i" | egrep -q "^/*${mntdir}/" && echo "$i" >> ${tmpd}/links
done || exit 4
[ "$keepdir" ] || find "${mntdir}" -type d | sed = | sed 'N;s/\n/ /' | sort -nr | cut -f 2- -d " " > dirs
#sort -nr is due to the way rmdir -p works
$b umount -d $mntdir || exit 5
#starting with umount, it is safer to fallback to busybox to avoid using what we are untczing
$b cat links | while read i do
do
sudo $b rm "$i"
done || exit 6
$b rm links
$b cat dirs | while read i
do
i=/${i#*$mntdir}
sudo $b rmdir -p "$i" || true
done || exit 8
$b rm dirs || true
cd /usr/local/tce.installed || exit 2
$b find -maxdepth 1 -size +0c | $b egrep -q "${name}" && exit 7
# Debugline for inner consistency check
$b rm "${name}" || true
$b rmdir "${tmpd}"
$b rmdir "${mntdir}"
Please note that this is an alpha version. Try it at your risk.
-
Ok, gave a shot at the other approach I was mentioning, and it is way faster.
Maybe not fast enough, though: it unloads (averaged) in almost 14 seconds (to be compared with 1m36s in my last post), while tce-load -i that extension takes 1.5 seconds.
So we have roughly a x10 factor when untcz-ing, on a large tcz.
To make it faster, I had to do few assumptions, commented in script.
The remaining considerations in my last post should still apply. Moreover, this one needs no tempfiles.
Again, feedback welcome, and again, very alpha.
#!/bin/sh
# Copyright 2011 Marco Caminati. Released under GPLv3.
# Pass any content to $keepdir to prevent deleting the directories emptied in the process
set -e
set -u
set -x
# To speed up scripts, I assumed that:
# 1) tce-load -i links each /tmp/tcloop/filepath to /filepath
# 2) there is no path in an extension containing " character
# 3) No abmnormously long paths in extension (xargs -n 100 places a very high, but finite, limit on it). Oddly, the increase in execution time got setting -n 1 is unexpectedly low, about +10% (on a very big tcz).
b=busybox
name=`$b echo "$1" | $b sed -e 's_/*$__' | $b egrep -o "[^/]*$"`
mntdir=/tmp/tcloop/$name
keepdir="${keepdir:-}"
mntlist="/proc/mounts"
echo $mntdir | grep -q ! && exit 6
loopdev="$(egrep "${mntdir}" "${mntlist}" | sed "s_\(.*\) squashfs .*\$_\1_" | egrep "${mntdir} *$" | cut -f 1 -d ' ')"
squashfile="$($b losetup | egrep "^ *${loopdev} *:" | cut -f 3- -d " ")"
[ -s "${squashfile}" ] || exit 2
$b umount "${mntdir}" || exit 3
sudo $b mount "${squashfile}" "${mntdir}" -t squashfs -o loop,ro,bs=4096 || exit 4
# Error 4 is big trouble!
cd "${mntdir}"
find ! -type d | cut -f 2- -d "." |sed 's/.*/"&"/' | $b xargs -r -n 1 readlink | grep "^${mntdir}/" |sed "s!\(^${mntdir}\)\(.*\)!\"\2\"!" | sudo $b xargs -x -n 100 rm
[ "$keepdir" ] || $b find -type d | $b cut -f 2- -d . | $b sed = | $b sed 'N;s/\n/ /' | $b sort -nr | $b cut -f 2- -d " " | $b sed 's/.*/"&"/'| sudo $b xargs -x -n 100 rmdir -p || true
#sort -nr is due to the way rmdir -p works
cd ..
$b umount -d "${mntdir}" || exit 5
cd /usr/local/tce.installed || exit 2
$b find -maxdepth 1 -size +0c | $b egrep -q "${name}" && exit 7
# Debugline for inner consistency check
$b rm "${name}" || true
$b rmdir "${mntdir}"
-
Ok, gave a shot at the other approach I was mentioning, and it is way faster.
Maybe not fast enough, though: it unloads (averaged) in almost 14 seconds (to be compared with 1m36s in my last post), while tce-load -i that extension takes 1.5 seconds.
So we have roughly a x10 factor when untcz-ing, on a large tcz.
Latest development: replaced readlink with its poorman version via ls+grep+sed: the 14 seconds above are now 2.70 seconds. Things are getting reasonably fast, I think.
Run time could increase without findutils.tcz, grep.tcz, sed.tcz and/or coreutils.tcz loaded, I haven't time to test now.
#!/bin/sh
# Copyright 2011 Marco Caminati. Released under GPLv3.
# Pass any content to $keepdir to prevent deleting the directories emptied in the process
set -e
set -u
set -x
# To speed up script, I assumed that:
# 1) tce-load -i links each /tmp/tcloop/filepath to /filepath
# 2) there is no path in an extension containing " character
# 3) No abmnormously long paths in extension (xargs -n 100 places a very high, but finite, limit on it). Oddly, the increase in execution time got setting -n 1 is unexpectedly low, about +10% (on a very big tcz).
b=busybox
name=`$b echo "$1" | $b sed -e 's_/*$__' | $b egrep -o "[^/]*$"`
mntdir=/tmp/tcloop/$name
keepdir="${keepdir:-}"
mntlist="/proc/mounts"
echo $mntdir | grep -q ! && exit 6
loopdev="$(egrep "${mntdir}" "${mntlist}" | sed "s_\(.*\) squashfs .*\$_\1_" | egrep "${mntdir} *$" | cut -f 1 -d ' ')"
squashfile="$($b losetup | egrep "^ *${loopdev} *:" | cut -f 3- -d " ")"
[ -s "${squashfile}" ] || exit 2
$b umount "${mntdir}" || exit 3
sudo $b mount "${squashfile}" "${mntdir}" -t squashfs -o loop,ro,bs=4096 || exit 4
# Error 4 is big trouble!
cd "${mntdir}"
find ! -type d | cut -f 2- -d . | sed 's_.*_"&"_'| xargs -r -n 100 $b ls -l | grep -o -- "-> ${mntdir}/.*$"| sed "s_^-> ${mntdir}\(.*$\)_\"\1\"_" | sudo $b xargs -r -n 100 rm
[ "$keepdir" ] || $b find -type d | $b cut -f 2- -d . | $b sed = | $b sed 'N;s/\n/ /' | $b sort -nr | $b cut -f 2- -d " " | $b sed 's/.*/"&"/'| sudo $b xargs -r -n 100 rmdir -p || true
#sort -nr is due to the way rmdir -p works
cd ..
$b umount -d "${mntdir}" || exit 5
cd /usr/local/tce.installed || exit 2
$b find -maxdepth 1 -size +0c | $b egrep -q "${name}" && exit 7
# Debugline for inner consistency check
$b rm "${name}" || true
$b rmdir "${mntdir}"
-
Good work Caminati. Your unload script is very useful.
It would be much more beneficial if extension usage detection could be included.
"- No check if there are extensions depending on the one being unloaded.
This is would be best done via an external wrapper, in my opinion."
I wonder if some of the code in apps audit could be used?
-
Good work Caminati. Your unload script is very useful.
It would be much more beneficial if extension usage detection could be included.
"- No check if there are extensions depending on the one being unloaded.
This is would be best done via an external wrapper, in my opinion."
I wonder if some of the code in apps audit could be used?
Glad you like it. I put some effort in it.
By the way, I tried it on the usual big testbench of mine, texlive.tcz, after a base norestore reboot, i.e. using only busybox, and running time did not increase significantly.
This somehow surprised me, because busybox's grep is surely much slower than gnu's.
Good news, anyway.
Regarding dependencies check, I stick to the unix tenet of "do one thing, and do it well", so I insist that this should done somewhere else.
Moreover, there is no unique goal in dep.check: some might like just to be warned, others to have a recursive untczing, others to force untcz anyway.
Finally, dep.check is the trivial part, compared with untczing, which is the speed-sensitive core.
Anyway, yes, one could base on tce-audit requiredby to do that, or proceed in another zillion alternative ways.
Finally, I probably should modify script so that sudo busybox mount is done on a fresh mountpoint, other than /tmp/tcloop; that would be saner (that's kinda memo for myself :) )
-
Good work Caminati. Your unload script is very useful.
It would be much more beneficial if extension usage detection could be included.
"- No check if there are extensions depending on the one being unloaded.
This is would be best done via an external wrapper, in my opinion."
I wonder if some of the code in apps audit could be used?
Glad you like it. I put some effort in it.
By the way, I tried it on the usual big testbench of mine, texlive.tcz, after a base norestore reboot, i.e. using only busybox, and running time did not increase significantly.
This somehow surprised me, because busybox's grep is surely much slower than gnu's.
Good news, anyway.
Regarding dependencies check, I stick to the unix tenet of "do one thing, and do it well", so I insist that this should done somewhere else.
Moreover, there is no unique goal in dep.check: some might like just to be warned, others to have a recursive untczing, others to force untcz anyway.
Finally, dep.check is the trivial part, compared with untczing, which is the speed-sensitive core.
Anyway, yes, one could base on tce-audit requiredby to do that, or proceed in another zillion alternative ways.
Finally, I probably should modify script so that sudo busybox mount is done on a fresh mountpoint, other than /tmp/tcloop; that would be saner (that's kinda memo for myself :) )
The biggest problem for me would be uninstalling dependencies that are used by other applications. The app audit tool provides all the necessary information. I suppose one way to solve this particular problem would be to use an exclusion/skip dependency list that is created manually. Just a thought.
-
It's good to see someone has taken up the extension uninstall challenge again. There has been extensive discussion on this subject for quite some time. See:
http://forum.tinycorelinux.net/index.php?PHPSESSID=649fac8d1ce81103810539df92f9ac58&topic=2081.0
Seems like the link is no longer available
An Error Has Occurred!
The topic or board you are looking for appears to be either missing or off limits to you.
-
Remove the PHPSESSID.
http://forum.tinycorelinux.net/index.php?topic=2081.0
-
Hi curaga
That gives the same error message.
-
Oh, indeed. It was deleted, and only admins can see deleted posts.