[Solved] Hybrid $HOME : traditional backup/restore + persistence

[jazzbiker]

Hi, Core people!

I use multiple TC versions residing on a nomadic dual-boot USB flash drive, intended to work everywhere. Mostly these are 3 boxes with their internal HDDs and I want to see some dirs, located on them, to be accessible through my $HOME. I have the solution, but encounter it too verbose. Let me describe it with the hope You will give me some advices which will help to improve and maybe shrink the code or reorganize it.

So, I create $HOME/.persistent, each line of which contains

Code: [Select]

<acquired device UUID> <Target directory name> <Link name>

And have 2 pieces of code. One in /opt/bootlocal.sh:

Code: [Select]

# Function mount_present_in recieves filenames as the input parameters.
# Each file is expected to contain the lines, describing directories, located on
# external storage devices, which need to be linked into the $HOME.
# Each line of the list cantains 3 data fields:
# <device UUID> <Target dir name> <Link name>
# Link name is relative to $HOME.
# The task of the function is to detect availability of storage devices which can
# be acquired by any user and mount those present, preparing for the linking, which
# will be done by individual .profile. Another task is to check, whether the
# requested link is present in .xfiletool.lst, and in case of its absence add
# it.

mount_present_in () {
  while [ $# -gt 0 ]
  do
    while read PERSISTENT_UUID PERSISTENT_TARGET PERSISTENT_LINK
    do
      [ -n $PERSISTENT_LINK ] || continue # Are all 3 vars defined ?
      PERSISTENT_DEV="$(blkid -U $PERSISTENT_UUID)" || continue # Is requested storage device present ?
      mount $PERSISTENT_DEV
      cat /opt/.xfiletool.lst | grep -x "$PERSISTENT_LINK" 2>&1 >/dev/null || echo "$PERSISTENT_LINK"  >> /opt/.xfiletool.lst
    done < $1

    shift
  done
}

PERSISTENT_LISTS="$(find /home -mindepth 2 -maxdepth 2 -name '.persistent')"

mount_present_in $PERSISTENT_LISTS


and the other in $HOME/.profile:

Code: [Select]

PERSISTENT_LIST="$HOME/.persistent"

if [ -f "$PERSISTENT_LIST" ]
then
  while read PERSISTENT_UUID PERSISTENT_TARGET PERSISTENT_LINK
  do
    [ -n "$PERSISTENT_LINK" ] || continue # All 3 variables are defined ?
    PERSISTENT_DEV=$(blkid -U "$PERSISTENT_UUID") || continue # Is device present ?
    MOUNT_DELAY=10
    while [ $MOUNT_DELAY -ne 0 ]
    do
      PERSISTENT_MNT="$(mount | grep "$PERSISTENT_DEV")" && break # Is device mounted ?
      MOUNT_DELAY=$(( $MOUNT_DELAY - 1 ))
      sleep 1
    done

    if [ $MOUNT_DELAY -eq 0 ]
    then
      echo "Can not find $PERSISTENT_DEV mounted. Skipping $PERSISTENT_LINK."
      continue
    fi


    PERSISTENT_MNT="$(echo "$PERSISTENT_MNT" | awk '{print $3}')" # get mount point
    PERSISTENT_TARGET_PATH="$PERSISTENT_MNT/$PERSISTENT_TARGET"
    [ -d "$PERSISTENT_TARGET_PATH" ] || continue # Does requested dir really exist ?
    PERSISTENT_LINK_PATH="$HOME/$PERSISTENT_LINK"
    [ -e "$PERSISTENT_LINK_PATH" ] || ln -s "$PERSISTENT_TARGET_PATH" "$PERSISTENT_LINK_PATH"
  done < "$PERSISTENT_LIST"
fi

The task seems to be quite simple, but the solution looks quite verbose. I will be grateful for any help for shrinking the stuff.

Best regards!

[Rich]

Hi jazzbiker
It looks overly complex. Here's what I came up with:

Code: [Select]

#!/bin/sh

# List of External Storage Devices. <UUID> <Target dir name> <Link name>
ESDs=".ExternalDrives"

# Create a list of UUIDs of visible drives, remove any loop devices, remove quotation marks.
blkid -s UUID | grep -v loop | tr -d '"' > DRIVES

exec 3<"$ESDs"
while true
do
read -r -u3 esd # Read an entry from the list of external drives.
[ -z "$esd" ] && break # End of file.
UUID="$(echo $esd | cut -d ' ' -f1)" # Grab UUID.

drive="$(grep $UUID DRIVES)" # See if the UUID is in the list of visible drives.
[ -z "$drive" ] && continue # If not, read the next entry.

DEVICE="/mnt${drive:4:5}" # Grab /dev/sd??, convert to /mnt/sd??.

grep "$DEVICE" /proc/mounts
[ "$?" -ne 0 ] && echo "mount $DEVICE"

TARGET="$DEVICE/$(echo $esd | cut -d ' ' -f2)"
LINK="$HOME/$(echo $esd | cut -d ' ' -f3)"

# Create link.
echo "ln -s $TARGET $LINK"

# See if $LINK is already in /opt/.xfiletool.lst.
grep -q "$LINK" /opt/.xfiletool.lst
[ "$?" -ne 0 ] && echo "echo $LINK >> /opt/.xfiletool.lst"

done
exec 3>&-
This is the list of UUIDs on my system:

Code: [Select]

tc@E310:~/hybridhome$ cat DRIVES
/dev/sda2: UUID=22559ae7-7e12-4a3b-9efa-6f8c8a9a8a6a
/dev/sda5: UUID=81c9827f-5952-4bb2-b1b2-298be789abee
/dev/sda3: UUID=bedfc958-a562-4225-ad9d-cf150e7cdb4e
/dev/sda1: UUID=543cac60-3224-4cab-b1d5-008407dd9ce8
/dev/sda6: UUID=7b839431-514c-44a7-a1d3-9baeffb595e0
/dev/sdg1: UUID=9d76e84d-61a9-4959-a93e-21d77d45122f
tc@E310:~/hybridhome$
This is the test file of eternal devices I used:

Code: [Select]

tc@E310:~/hybridhome$ cat .ExternalDrives
bedfc958-a562-4225-ad9d-cf150e7cdb4e PDFs/manuals docs
tc@E310:~/hybridhome$
Rather than execute the commands I echoed them on my system:

Code: [Select]

tc@E310:~/hybridhome$ ./hybridhome.sh
mount /mnt/sda3
ln -s /mnt/sda3/PDFs/manuals /home/tc/docs
echo /home/tc/docs >> /opt/.xfiletool.lst
tc@E310:~/hybridhome$
If you need to link multiple directories from the same drive, you can list the same UUID on multiple lines.

[jazzbiker]

Hi, Rich!

Thanks a lot for Your aid! I have some questions before testing.

1. As I understand, Your main message is, that I don't need to split the actions between "/etc/bootlocal.sh" and "$HOME/.profile". Right?

It looks overly complex.

I've did in such a way in order to avoid "sudo" in .profile (mount will need it).

2. Has the use of "sudo" in user scripts some negative connotation?

3. BB's "read" builtin has the -u option, oooeee! Where from can I got this other than from BB sources?

4. You've used

Code: [Select]

exec 3<"$ESDs"
while true
do
read -r -u3 esd # Read an entry from the list of external drives.
[ -z "$esd" ] && break # End of file.

# loop body

done
exec 3>&-
What are the benefits of this technique over dumb

Code: [Select]

while read -r esd
do
# loop body
done < "$ESDs"
?

[jazzbiker]

Oh, I've missed the hottest question!

This is the test file of eternal devices I used:

Where have You bought them??

[Rich]

Hi jazzbiker

... I've did in such a way in order to avoid "sudo" in .profile (mount will need it). ...

When the system boots, it adds every partition that it finds to  /etc/fstab.  If it is listed in  fstab,  you don't need  sudo
to mount it.

... 2. Has the use of "sudo" in user scripts some negative connotation? ...

Not as far as I'm concerned. Sometimes a command needs elevated privileges but the rest of the script does not.

... 3. BB's "read" builtin has the -u option, oooeee! Where from can I got this other than from BB sources? ...

Possibly the BB config file. I used this man page for possible options for the  read  command:
https://linuxcommand.org/lc3_man_pages/readh.html
and then verify that BB read accepts them.

... What are the benefits of this technique over dumb ...

I was having a syntax problem with the "dumb" version and did not feel like fighting with it.

... Where have You bought them??

I made it up. I used one of the UUIDs my system recognized (DRIVES file), added an arbitrary path and link name. Since
I wasn't executing the actual commands, the directory path didn't need to exist. I just needed to verify the commands
being echoed were correct.

Place the script in ~/.local/bin. That way it's persistent and included in $PATH.
Call the script from  ~/.X.d/  or  ~/.profile  so that  $HOME  has been set when the script gets called.
Change this:

Code: [Select]

# List of External Storage Devices. <UUID> <Target dir name> <Link name>
ESDs=".ExternalDrives"so it points to your file of persistent external drives.

At this point, just enter the script name in a terminal:

Code: [Select]

hybridhome.shSee if the commands it echos out look correct. If they do, change the lines with echo so the commands actually
get executed.

[jazzbiker]

Hi, Rich!

Thanks for a plenty of useful information! Some making me feel a kind of fool, like sudoless mount 

Please, take a look at the new hybridhome.sh:

Code: [Select]

tc@box:~$ cat ~/.local/bin/hybridhome.sh
#!/bin/sh

PERSISTENT_LIST="$HOME/.persistent"

[ -f "$PERSISTENT_LIST" ] || exit 0

while read PERSISTENT_UUID PERSISTENT_TARGET PERSISTENT_LINK
do
        PERSISTENT_DEV=$(blkid -U "$PERSISTENT_UUID") || continue
        grep "$PERSISTENT_DEV" /proc/mounts || mount "$PERSISTENT_DEV"
        ln -s "/mnt/${PERSISTENT_DEV:5}/$PERSISTENT_TARGET" "$HOME/$PERSISTENT_LINK" || continue
        grep -x -q "$PERSISTENT_LINK" /opt/.xfiletool.lst || echo "$PERSISTENT_LINK" >> /opt/.xfiletool.lst

done < "$PERSISTENT_LIST"

tc@box:~$

Looks slim enough! Can You criticize it a little if You will have a spare minute?

PS: Those "Where have You bought them?" was the joke, concerning Your mistyped "external" as "eternal". Eternal devices - just a sweet dream, what can be better!

[Rich]

Hi jazzbiker
It's certainly slim. I don't see any problems with it.

... Your mistyped "external" as "eternal". Eternal devices ...

Ah, I missed that when I was proofreading my post.

[jazzbiker]

Thanks for Your guidance! I've got exactly what I wanted. Is it the right moment for marking the thread as solved?

[Rich]

Hi jazzbiker

Thanks for Your guidance! ...

You are welcome.

... Is it the right moment for marking the thread as solved?

It sure sounds like it is.

[jazzbiker]

Unfortunately script posted above can misbehave in some situations. I've added some tests and script parameters availability. This version started without parameters will process all .persistent* files in $HOME, while getting any filenames will process only the named ones.

Code: [Select]

#!/bin/sh

PERSISTENT_LIST="${*:-.persistent*}"

cd $HOME ; cat $PERSISTENT_LIST |

while read PERSISTENT_UUID PERSISTENT_TARGET PERSISTENT_LINK THE_REST
do
        [ -n "$PERSISTENT_LINK" ] || continue

        PERSISTENT_DEV=$(blkid -U "$PERSISTENT_UUID") || continue
        grep -q "$PERSISTENT_DEV" /proc/mounts || mount "$PERSISTENT_DEV"

        LINK_PATH="$HOME/$PERSISTENT_LINK" ; [ -e "$LINK_PATH" ] && continue

        LINK_DIR="${LINK_PATH%/*}" ; [ -d "$LINK_DIR" ] || mkdir -p "$LINK_DIR"

        ln -s "/mnt/${PERSISTENT_DEV:5}/$PERSISTENT_TARGET" "$LINK_PATH" || continue
        grep -q -x "$PERSISTENT_LINK" /opt/.xfiletool.lst || echo "$PERSISTENT_LINK" >> /opt/.xfiletool.lst

done


[mocore]

thanks for the nomadic scripts ! 

Unfortunately script posted above can misbehave in some situations.

i wander what kind of misbehavior did you encounter !

[jazzbiker]

i wander what kind of misbehavior did you encounter !

Hi, mocore!

I'm glad, that You find this script useful. Thanks to Rich for pushing into right direction and overall guidance and useful tips. I use this script almost every day without any issues.

if I am not mistaken, the previous version failed in case requested link location is inside nonexistent directory in Your $HOME. The current version tests directory presence and creates it if necessary:

Code: [Select]

        LINK_DIR="${LINK_PATH%/*}" ; [ -d "$LINK_DIR" ] || mkdir -p "$LINK_DIR"

I've made some non-sufficient changes, just rearrangement to fit my taste, and now the script looks in that way:

Code: [Select]

#!/bin/sh

PERSISTENT_LIST="${*:-.persistent*}"

cd $HOME ; cat $PERSISTENT_LIST |

while read PERSISTENT_UUID PERSISTENT_TARGET PERSISTENT_LINK THE_REST
do
        [ -n "$PERSISTENT_LINK" ] || continue

        LINK_PATH="$HOME/$PERSISTENT_LINK" ; [ -e "$LINK_PATH" ] && continue

        PERSISTENT_DEV=$(blkid -U "$PERSISTENT_UUID") || continue
        grep -q "$PERSISTENT_DEV" /proc/mounts || mount "$PERSISTENT_DEV"

        LINK_DIR="${LINK_PATH%/*}" ; [ -d "$LINK_DIR" ] || mkdir -p "$LINK_DIR"

        grep -q -x "$PERSISTENT_LINK" /opt/.xfiletool.lst ||
        echo "$PERSISTENT_LINK" >> /opt/.xfiletool.lst &&
        ln -s "/mnt/${PERSISTENT_DEV:5}/$PERSISTENT_TARGET" "$LINK_PATH"

done

Best regards!

[jazzbiker]

Hi Core nomads,

Here is the next script for hybrid $_HOME treatment. The previous short shell script worked well, but I was not very happy with it. What I wanted to  improve:
1. All config editing is manual. I wanted to make the script a little bit smarter.
2. Add some flexibility
3. Make exclusion globs in .xfiletool.lst more aimed.

Current version is in Lua, because it is easier to maintain and expand if it will be necessary. The current version doesn't deals with configuration file only, it takes into consideration all the changes You've made manually. So if You create the link and then run hybridhome.lua, the link will be captured and reproduced any time in the future in appropriate conditions (being run at the same host). If You remove the target, then corresponding link is removed from the config file too. Now every node has its own "keep" parameter, influencing the .xfiletool.lst mode for the node. So now the node may be excluded unconditionally or depending on the link was created or no. Node names are written to .xfiletool.lst with $_HOME prefix, which allows to avoid some troubles caused by the short or common patterns. Garbage is wiped out without the need of human intervention.

If this script will appear to be useful for someone I will be happy to answer the questions and apply proposed improvements.

Best regards!

[jazzbiker]

Though nobody is interested in, here is the version with parent-child ambiguities resolved.

Have a nice Core!