WelcomeWelcome | FAQFAQ | DownloadsDownloads | WikiWiki

Author Topic: Unloading extensions  (Read 22341 times)

Offline helander

  • Full Member
  • ***
  • Posts: 183
Unloading extensions
« on: February 14, 2010, 02:59:11 PM »
The script below, which I call tcz-unload, allows you to unload an installed extension. By unload I mean that it:
  • removes all symlinks to the extension
  • unmounts and removes it from /tmp/tcloop
  • removes the entry in /usr/local/tce.installed

But before it allows you to do this it checks, using the lsof program (i.e. means that you have to have the lsof.tcz extension installed), that no program is currently using any of the files that are to be "removed".

NOTE! This does not guarantee that any program will not try to access these files "in the future" and if so happens, unexpected things might happen.

All programs that currently is using files (= files opened) within the extension is listed, so that you can take actions to terminate these programs if so required.

This tool is very useful to me when I create/modify extensions and need to repeatedly re-install new "versions".

/Lars


Code: [Select]
#!/bin/sh

listlines() {
  for i in $*; do
    echo $i
  done
}
         
package=$1
if [ -z "$package" ]; then
  echo "  [Error]: No package name specified"
  exit 1
fi
if [ -d /tmp/tcloop/$package ]; then

  usedby=`lsof +D /tmp/tcloop/$package -Fp | sed -e "s/^p//"`

  if [ "$usedby" ]; then
    echo "Files in extension $package are currently in use by the following programs:"
    usedby=`listlines $usedby | sort -ug`
    for p in $usedby; do
      /bin/ps | awk "/^$p/ "'{$1="";$2="";print $0}'
    done 
    echo "  [Error]: Extension $package has not been unloaded."
    exit 1
  fi
  f=`cd /tmp/tcloop/$package; find ! -type d | sed -e "s%^./%/%"`
  for l in $f; do
     if [ -s $l ]; then
       linkto=`readlink $l`
       if [ "$linkto" == "/tmp/tcloop/$package$l" ]; then
          echo "    Unlink $l from $linkto"
          sudo rm $l
       else
         echo "  [Warning]: $l links to $linkto"
       fi
     else
       echo "  [Warning]: $l is not a link"
     fi
  done
  sudo umount /tmp/tcloop/$package
  sudo rmdir /tmp/tcloop/$package
else
  echo "  [Warning]: /tmp/tcloop/$package does not exist"
fi
sudo rm -f /usr/local/tce.installed/$package

Offline bigpcman

  • Hero Member
  • *****
  • Posts: 719
Re: Unloading extensions
« Reply #1 on: February 14, 2010, 03:25:54 PM »
The script below, which I call tcz-unload, allows you to unload an installed extension. By unload I mean that it:
  • removes all symlinks to the extension
  • unmounts and removes it from /tmp/tcloop
  • removes the entry in /usr/local/tce.installed

But before it allows you to do this it checks, using the lsof program (i.e. means that you have to have the lsof.tcz extension installed), that no program is currently using any of the files that are to be "removed".

NOTE! This does not guarantee that any program will not try to access these files "in the future" and if so happens, unexpected things might happen.

All programs that currently is using files (= files opened) within the extension is listed, so that you can take actions to terminate these programs if so required.

This tool is very useful to me when I create/modify extensions and need to repeatedly re-install new "versions".

/Lars


Code: [Select]
#!/bin/sh

listlines() {
  for i in $*; do
    echo $i
  done
}
        
package=$1
if [ -z "$package" ]; then
  echo "  [Error]: No package name specified"
  exit 1
fi
if [ -d /tmp/tcloop/$package ]; then

  usedby=`lsof +D /tmp/tcloop/$package -Fp | sed -e "s/^p//"`

  if [ "$usedby" ]; then
    echo "Files in extension $package are currently in use by the following programs:"
    usedby=`listlines $usedby | sort -ug`
    for p in $usedby; do
      /bin/ps | awk "/^$p/ "'{$1="";$2="";print $0}'
    done  
    echo "  [Error]: Extension $package has not been unloaded."
    exit 1
  fi
  f=`cd /tmp/tcloop/$package; find ! -type d | sed -e "s%^./%/%"`
  for l in $f; do
     if [ -s $l ]; then
       linkto=`readlink $l`
       if [ "$linkto" == "/tmp/tcloop/$package$l" ]; then
          echo "    Unlink $l from $linkto"
          sudo rm $l
       else
         echo "  [Warning]: $l links to $linkto"
       fi
     else
       echo "  [Warning]: $l is not a link"
     fi
  done
  sudo umount /tmp/tcloop/$package
  sudo rmdir /tmp/tcloop/$package
else
  echo "  [Warning]: /tmp/tcloop/$package does not exist"
fi
sudo rm -f /usr/local/tce.installed/$package

It's good to see someone has taken up the extension uninstall challenge again. There has been extensive discussion on this subject for quite some time. See:
 http://forum.tinycorelinux.net/index.php?PHPSESSID=649fac8d1ce81103810539df92f9ac58&topic=2081.0

Have you considered all the issues that have been discussed regarding uninstalling extensions without performing a "base norestore" reboot before the uninstall and a reboot after the uninstall? Have you used  Jason's uninstall script as a guide?

eidit: sorry for not paying closer attention, unload is obviously different than uninstall. However, if I'm not mistaken some of the issues discussed in the uninstall thread apply to doing a "live" unload. Anyway, I hope we can get a reliable unload to work in conjunction with Ondemand "tce-load".
« Last Edit: February 14, 2010, 03:38:50 PM by bigpcman »
big pc man

Offline helander

  • Full Member
  • ***
  • Posts: 183
Re: Unloading extensions
« Reply #2 on: February 14, 2010, 03:55:52 PM »
Prior to your reply  I had not read the discussion you pointed to nor I had I seen Jason's script. Now I have :) at least read the discussion.

I fully support Jason's view and I think that a tool like this could only be used with great caution and if TC should support removal of installed extensions arbitrarily, this would add complexity that I think none would be willing to pay the "cost" for.

As I stated, and others in the discussion you pointed to also did, was that in some contexts a tool like this is very helpful.

Since I can not access Jason's script I can not tell how much different it is from mine, this said you probably understand that I did not base my design on anything from Jason's work.

From the discussion I conclude that in principle the solutions are similar, but the "novelty" of my approach is the use of lsof to pro-actively not allow removal in case it will not be possible to complete immediately. The approach of using "umount -l" is simple but it leaves the system in a state where you do not know when the operation will complete, if ever. My solutions provides the user with information what could be done in order to complete the operation immediately and then it is up to the user to decide on taking these actions, or defer this to a later point, e.g. wait until the applications no longer are used or reboot the system in case you need to uninstall something you can not be without, e.g. an extension used by your window manager.

/Lars
« Last Edit: February 14, 2010, 03:58:50 PM by helander »

Offline bigpcman

  • Hero Member
  • *****
  • Posts: 719
Re: Unloading extensions
« Reply #3 on: February 14, 2010, 05:56:01 PM »
Prior to your reply  I had not read the discussion you pointed to nor I had I seen Jason's script. Now I have :) at least read the discussion.

I fully support Jason's view and I think that a tool like this could only be used with great caution and if TC should support removal of installed extensions arbitrarily, this would add complexity that I think none would be willing to pay the "cost" for.

As I stated, and others in the discussion you pointed to also did, was that in some contexts a tool like this is very helpful.

Since I can not access Jason's script I can not tell how much different it is from mine, this said you probably understand that I did not base my design on anything from Jason's work.

From the discussion I conclude that in principle the solutions are similar, but the "novelty" of my approach is the use of lsof to pro-actively not allow removal in case it will not be possible to complete immediately. The approach of using "umount -l" is simple but it leaves the system in a state where you do not know when the operation will complete, if ever. My solutions provides the user with information what could be done in order to complete the operation immediately and then it is up to the user to decide on taking these actions, or defer this to a later point, e.g. wait until the applications no longer are used or reboot the system in case you need to uninstall something you can not be without, e.g. an extension used by your window manager.

/Lars

Jason could you provide the last version of your uninstall script for general reference?
big pc man

Offline Jason W

  • Retired Admins
  • Hero Member
  • *****
  • Posts: 9730
Re: Unloading extensions
« Reply #4 on: February 14, 2010, 09:24:22 PM »
Honestly I don't a copy of it in it's final state.  I kind of wish I kept a copy for code reference.
« Last Edit: February 14, 2010, 09:26:10 PM by Jason W »

Offline tinypoodle

  • Hero Member
  • *****
  • Posts: 3857
Re: Unloading extensions
« Reply #5 on: August 27, 2010, 08:46:07 AM »
Wondering if the needed functionality of 'lsof' could be replaced by using 'fuser' instead, as latter is part of base.
"Software gets slower faster than hardware gets faster." Niklaus Wirth - A Plea for Lean Software (1995)

Offline caminati

  • Full Member
  • ***
  • Posts: 184
    • Homepage
Re: Unloading extensions
« Reply #6 on: December 03, 2010, 07:44:40 AM »
What about changing

Code: [Select]
sudo umount /tmp/tcloop/$package
to
Code: [Select]
sudo umount -d /tmp/tcloop/$package
?

See http://forum.tinycorelinux.net/index.php?topic=7998.0

Offline tinypoodle

  • Hero Member
  • *****
  • Posts: 3857
Re: Unloading extensions
« Reply #7 on: December 03, 2010, 11:12:31 AM »
What about changing

Code: [Select]
sudo umount /tmp/tcloop/$package
to
Code: [Select]
sudo umount -d /tmp/tcloop/$package
?

See http://forum.tinycorelinux.net/index.php?topic=7998.0

+1
"Software gets slower faster than hardware gets faster." Niklaus Wirth - A Plea for Lean Software (1995)

Offline cosmin_ap

  • Newbie
  • *
  • Posts: 48
Re: Unloading extensions
« Reply #8 on: June 12, 2011, 07:51:56 AM »
I also made a very similar script as part of tcztools package, for helping me with extension building.

You can see it here: http://code.google.com/p/tcztools/source/browse/tcz-umount

The only difference with mine is that it shows you any files that were changed/removed since extension was loaded:

Files that are not symlinks to the extension loop mount anymore are shown
with a '?' in front, and those which are missing with '!'. In dry-run mode
it also shows the symlinks that would have been deleted.

I'm thinking of also adding a reverse-dependencies check by consulting the .dep files of currently loaded extensions to see if any one contains the extension you want to umount and maybe warn you about it.

Offline caminati

  • Full Member
  • ***
  • Posts: 184
    • Homepage
Re: Unloading extensions
« Reply #9 on: September 21, 2011, 12:10:20 PM »
I devoted some time to the issue, and produced my version, called untcz.
tcz-unload is not very error-tolerant: just pass it the argument
$package/ (trailing slash) and it umounts extension without unlinking.
tcztools is even worse, I don't like its approach at all.
Plus, lsof complexity can be avoided.
In my script, this is done by directly umounting, and giving up on umount error.
To do this, I have to postpone unlinking after umounting.
This requires saving a list of to-do unlinks somewhere.
The drawback is that I need temp space on filesystem, and it can go up to megs
for biggest extensions (texlive.tcz, a custom extension I took as testbench for its size).
The highligts are:
  • more robust fault tolerance: you shouldn't get umount without
    unlinking done
  • no relevant speed loss with this approach: taking my texlive.tcz as a testbench,
    I got 1m30s with tcz-unload, and 1m36s with untcz
  • the user can decide to remove all directories emptied in the process.
    For some private-directories-rich extensions, like texlive, this is important.
    In this case untcz-ing time increases.
  • less complexity (smaller than tcz-unload, and it can be reduced pruning
    comments and debug lines)
  • corrected umount -d bug reported above

Anyway, from a time complexity point of view, the problem seems hard
if confined to scripting.
I have another approach in mind, if I have time I will give it a shot.
Performance aside, the following issues remain:
  • - While tinycore provides the /usr/local/tce.installed/script to do load management tasks, there is no mechanism to do the symmetrical tasks when unloading.
  • - No check if there are extensions depending on the one being unloaded.
    This is would be best done via an external wrapper, in my opinion.
Hoping it helps, here is my untcz, comments and reports welcome:

Code: [Select]
#!/bin/sh
# Copyright 2011 Marco Caminati. Released under GPLv3.
# Pass any content to $keepdir to prevent deleting the directories emptied in the process
set -e
set -u
set -x

b=busybox
name=`$b echo "$1" | $b sed -e 's_/*$__' | $b egrep -o "[^/]*$"`
tmpd=/tmp/untcz
mntdir=/tmp/tcloop/$name
keepdir="${keepdir:-}"

cd $tmpd 2>/dev/null && exit 1
$b mkdir -p "$tmpd"
cd "$tmpd"
find "${mntdir}" ! -type d | while read i
do
 i=/${i#*$mntdir}
 readlink "$i" | egrep -q "^/*${mntdir}/" && echo "$i" >> ${tmpd}/links
done || exit 4

[ "$keepdir" ] || find "${mntdir}" -type d | sed = | sed 'N;s/\n/ /' | sort -nr | cut -f 2- -d " " > dirs
#sort -nr is due to the way rmdir -p works

$b umount -d $mntdir || exit 5
#starting with umount, it is safer to fallback to busybox to avoid using what we are untczing
$b cat links | while read i do
do
 sudo $b rm "$i"
done || exit 6

$b rm links

$b cat dirs | while read i
do
  i=/${i#*$mntdir}
  sudo $b rmdir -p "$i" || true
done || exit 8
$b rm dirs || true
cd /usr/local/tce.installed || exit 2
$b find -maxdepth 1 -size +0c | $b egrep -q "${name}" && exit 7
# Debugline for inner consistency check
$b rm "${name}" || true
$b rmdir "${tmpd}"
$b rmdir "${mntdir}"

Please note that this is an alpha version. Try it at your risk.
« Last Edit: September 23, 2011, 05:15:58 AM by caminati »

Offline caminati

  • Full Member
  • ***
  • Posts: 184
    • Homepage
Re: Unloading extensions
« Reply #10 on: September 21, 2011, 08:56:59 PM »
Ok, gave a shot at the other approach I was mentioning, and it is way faster.
Maybe not fast enough, though: it unloads (averaged) in almost 14 seconds (to be compared with 1m36s in my last post), while tce-load -i that extension takes 1.5 seconds.
So we have roughly a x10 factor when untcz-ing, on a large tcz.
To make it faster, I had to do few assumptions, commented in script.
The remaining considerations in my last post should still apply. Moreover, this one needs no tempfiles.
Again, feedback welcome, and again, very alpha.

Code: [Select]
#!/bin/sh
# Copyright 2011 Marco Caminati. Released under GPLv3.
# Pass any content to $keepdir to prevent deleting the directories emptied in the process
set -e
set -u
set -x

# To speed up scripts, I assumed that:
# 1) tce-load -i links each /tmp/tcloop/filepath to /filepath
# 2) there is no path in an extension containing " character
# 3) No abmnormously long paths in extension (xargs -n 100 places a very high, but finite, limit on it). Oddly, the increase in execution time got setting -n 1 is unexpectedly low, about +10% (on a very big tcz).

b=busybox
name=`$b echo "$1" | $b sed -e 's_/*$__' | $b egrep -o "[^/]*$"`
mntdir=/tmp/tcloop/$name
keepdir="${keepdir:-}"
mntlist="/proc/mounts"
echo $mntdir | grep -q ! && exit 6
loopdev="$(egrep "${mntdir}" "${mntlist}" | sed "s_\(.*\) squashfs .*\$_\1_" | egrep "${mntdir} *$" | cut -f 1 -d ' ')"
squashfile="$($b losetup | egrep "^ *${loopdev} *:" | cut -f 3- -d " ")"
[ -s "${squashfile}" ] || exit 2
$b umount "${mntdir}" || exit 3
sudo $b mount "${squashfile}" "${mntdir}" -t squashfs -o loop,ro,bs=4096 || exit 4
# Error 4 is big trouble!
cd "${mntdir}"
find ! -type d | cut -f 2- -d "." |sed 's/.*/"&"/' | $b xargs -r -n 1 readlink | grep "^${mntdir}/" |sed "s!\(^${mntdir}\)\(.*\)!\"\2\"!" | sudo $b xargs -x -n 100 rm
[ "$keepdir" ] || $b find -type d | $b cut -f 2- -d . | $b sed = | $b sed 'N;s/\n/ /' | $b sort -nr | $b cut -f 2- -d " " | $b sed 's/.*/"&"/'| sudo $b xargs -x -n 100 rmdir -p || true
#sort -nr is due to the way rmdir -p works
cd ..
$b umount -d "${mntdir}" || exit 5
cd /usr/local/tce.installed || exit 2
$b find -maxdepth 1 -size +0c | $b egrep -q "${name}" && exit 7
# Debugline for inner consistency check
$b rm "${name}" || true
$b rmdir "${mntdir}"
« Last Edit: September 23, 2011, 05:15:30 AM by caminati »

Offline caminati

  • Full Member
  • ***
  • Posts: 184
    • Homepage
Re: Unloading extensions
« Reply #11 on: September 22, 2011, 08:16:28 AM »
Ok, gave a shot at the other approach I was mentioning, and it is way faster.
Maybe not fast enough, though: it unloads (averaged) in almost 14 seconds (to be compared with 1m36s in my last post), while tce-load -i that extension takes 1.5 seconds.
So we have roughly a x10 factor when untcz-ing, on a large tcz.

Latest development: replaced readlink with its poorman version via ls+grep+sed: the 14 seconds above are now 2.70 seconds. Things are getting reasonably fast, I think.
Run time could increase without findutils.tcz, grep.tcz, sed.tcz and/or coreutils.tcz loaded, I haven't time to test now.

Code: [Select]
#!/bin/sh
# Copyright 2011 Marco Caminati. Released under GPLv3.
# Pass any content to $keepdir to prevent deleting the directories emptied in the process
set -e
set -u
set -x

# To speed up script, I assumed that:
# 1) tce-load -i links each /tmp/tcloop/filepath to /filepath
# 2) there is no path in an extension containing " character
# 3) No abmnormously long paths in extension (xargs -n 100 places a very high, but finite, limit on it). Oddly, the increase in execution time got setting -n 1 is unexpectedly low, about +10% (on a very big tcz).

b=busybox
name=`$b echo "$1" | $b sed -e 's_/*$__' | $b egrep -o "[^/]*$"`
mntdir=/tmp/tcloop/$name
keepdir="${keepdir:-}"
mntlist="/proc/mounts"
echo $mntdir | grep -q ! && exit 6
loopdev="$(egrep "${mntdir}" "${mntlist}" | sed "s_\(.*\) squashfs .*\$_\1_" | egrep "${mntdir} *$" | cut -f 1 -d ' ')"
squashfile="$($b losetup | egrep "^ *${loopdev} *:" | cut -f 3- -d " ")"
[ -s "${squashfile}" ] || exit 2
$b umount "${mntdir}" || exit 3
sudo $b mount "${squashfile}" "${mntdir}" -t squashfs -o loop,ro,bs=4096 || exit 4
# Error 4 is big trouble!
cd "${mntdir}"
find ! -type d | cut -f 2- -d . | sed 's_.*_"&"_'| xargs -r -n 100 $b ls -l | grep -o -- "-> ${mntdir}/.*$"| sed "s_^-> ${mntdir}\(.*$\)_\"\1\"_" | sudo $b xargs -r -n 100 rm
[ "$keepdir" ] || $b find -type d | $b cut -f 2- -d . | $b sed = | $b sed 'N;s/\n/ /' | $b sort -nr | $b cut -f 2- -d " " | $b sed 's/.*/"&"/'| sudo $b xargs -r -n 100 rmdir -p || true
#sort -nr is due to the way rmdir -p works
cd ..
$b umount -d "${mntdir}" || exit 5
cd /usr/local/tce.installed || exit 2
$b find -maxdepth 1 -size +0c | $b egrep -q "${name}" && exit 7
# Debugline for inner consistency check
$b rm "${name}" || true
$b rmdir "${mntdir}"
« Last Edit: September 23, 2011, 05:14:51 AM by caminati »

Offline bigpcman

  • Hero Member
  • *****
  • Posts: 719
Re: Unloading extensions
« Reply #12 on: September 22, 2011, 08:55:14 AM »
Good work Caminati. Your unload script is very useful.

It would be much more beneficial if extension usage detection could be included.

"- No check if there are extensions depending on the one being unloaded.
This is would be best done via an external wrapper, in my opinion."

I wonder if some of the code in apps audit could be used?
big pc man

Offline caminati

  • Full Member
  • ***
  • Posts: 184
    • Homepage
Re: Unloading extensions
« Reply #13 on: September 22, 2011, 09:16:37 AM »
Good work Caminati. Your unload script is very useful.

It would be much more beneficial if extension usage detection could be included.

"- No check if there are extensions depending on the one being unloaded.
This is would be best done via an external wrapper, in my opinion."

I wonder if some of the code in apps audit could be used?

Glad you like it. I put some effort in it.
By the way, I tried it on the usual big testbench of mine, texlive.tcz, after a base norestore reboot, i.e. using only busybox, and running time did not increase significantly.
This somehow surprised me, because busybox's grep is surely much slower than gnu's.
Good news, anyway.

Regarding dependencies check, I stick to the unix tenet of "do one thing, and do it well", so I insist that this should done somewhere else.
Moreover, there is no unique goal in dep.check: some might like just to be warned, others to have a recursive untczing, others to force untcz anyway.
Finally, dep.check is the trivial part, compared with untczing, which is the speed-sensitive core.
Anyway, yes, one could base on tce-audit requiredby to do that, or proceed in another zillion alternative ways.

Finally, I probably should modify script so that sudo busybox mount is done on a fresh mountpoint, other than /tmp/tcloop; that would be saner (that's kinda memo for myself :) )

Offline bigpcman

  • Hero Member
  • *****
  • Posts: 719
Re: Unloading extensions
« Reply #14 on: September 22, 2011, 09:41:25 PM »
Good work Caminati. Your unload script is very useful.

It would be much more beneficial if extension usage detection could be included.

"- No check if there are extensions depending on the one being unloaded.
This is would be best done via an external wrapper, in my opinion."

I wonder if some of the code in apps audit could be used?

Glad you like it. I put some effort in it.
By the way, I tried it on the usual big testbench of mine, texlive.tcz, after a base norestore reboot, i.e. using only busybox, and running time did not increase significantly.
This somehow surprised me, because busybox's grep is surely much slower than gnu's.
Good news, anyway.

Regarding dependencies check, I stick to the unix tenet of "do one thing, and do it well", so I insist that this should done somewhere else.
Moreover, there is no unique goal in dep.check: some might like just to be warned, others to have a recursive untczing, others to force untcz anyway.
Finally, dep.check is the trivial part, compared with untczing, which is the speed-sensitive core.
Anyway, yes, one could base on tce-audit requiredby to do that, or proceed in another zillion alternative ways.

Finally, I probably should modify script so that sudo busybox mount is done on a fresh mountpoint, other than /tmp/tcloop; that would be saner (that's kinda memo for myself :) )

The biggest problem for me would be uninstalling dependencies that are used by other applications. The app audit tool provides all the necessary information. I suppose one way to solve this particular problem would be to use an exclusion/skip dependency list that is created manually.  Just a thought.
big pc man