persistent data

[wysiwyg]

I was looking through the TC book under the "bootcodes explained" and the "persistent storage" chapters to see about preserving configuration changes between bootups.  The latter chapter says "The backup is on by default as long as you have set up a tce directory. It will save all your personal files in your home directory, and the system config files under /opt, excluding common browser caches."  I have a "tce=" declaration in my bootloaders config file (which is working correctly), unfortunately I don't see a mydata.tcz in that tce directory.  It should be noted that I'm not using a graphical UI, but rather the cli (corepure64.gz).  Any ideas?

Thanks,
Dave

[Rich]

Hi wysiwyg
You should be able to run a backup using  filetool.sh.

[wysiwyg]

@Rich so is this something that gets run automatically *if* a GUI is installed, but not just with a cli install?

Thanks,
Dave

[curaga]

Yes, it is run by default from the "Exit" GUI. In a CLI env you're responsible for doing a backup.

[wysiwyg]

Thanks for the response curaga.  Is there a reason why this isn't integrated with the system instead of being dependent upon having an GUI installed?  Otherwise you rely on having the user perform this manually before shutting down the system (which can be very problematic).  I have added a small section in the /etc/init.d/rc.shutdown file to accomplish this:


#!/bin/busybox ash
# /etc/init.d/rc.shutdown - used by /etc/inittab to shutdown the system.
#
. /etc/init.d/tc-functions
useBusybox

clear

# backup the system files
if [ $(cat /etc/sysconfig/backup) == 1 ]; then
        filetool.sh -b
fi

...snip the rest of the existing script...


This way everything is performed automatically no matter if you have a GUI installed or just using the console.  Plus it uses TC's current configuration on indicating if a backup should be performed or not via the /etc/sysconfig/backup file value.

Dave

[coreplayer2]

Many folks like myself specifically do not want a backup performed on every shutdown and have to modify the ~/.profile to ensure this does not occur.   I make a backup of a few specific files once only, disabling BACKUP in the process.  Thereafter on each reboot a restore from backup preserves my selected files and keeps any new files in persistent storage.

Perhaps "export BACKUP=0" could be moved to another location but why would core.gz need a routine back up without gui apps?

[Rich]

Hi coreplayer2

... but why would core.gz need a routine back up without gui apps?

Because even non-GUI apps can have config files that someone may wish to modify, so they need to be backed up.

[CentralWare]

1) exitcheck.sh is already implemented and if memory serves, it tends to backups and then passes along what ever command (poweroff/reboot) as needed.  The odds of everyone remembering to run exitcheck.sh shutdown aren't all that good, though.

2) the poweroff/reboot commands, if memory serves, are nothing more than links to busybox.  It wouldn't be difficult or insensible to replace them with dinky scripts:

Example /sbin/poweroff:
#!/bin/sh
if [ -f /opt/.autobackup ]; then     ## Conditional...  file missing, no backup
     filetool.sh -b
fi
/bin/busybox poweroff

(Repeat for /sbin/reboot)


EDIT: It just dawned on me there may be a permission issue.  @Rich/Curaga/etc. when rc.shutdown is launched, who is launching it?  TC or Root?

[core-user]

I find automatic backups annoying, it's not easy to find out how to disable this feature for a newbie, so rather than enable it by default, maybe a small utility/script to set it to whatever the user wants might be a better solution.

[wysiwyg]

@coreplayer2 the beautiful thing about the solution I presented is that if you don't want backups to automatically occur during shutdown, then a simple "echo 0 > /etc/sysconfig/backup" will prevent that from happening.  According to the TC book, this file is used to indicate if the backup should be run or not - so this fits perfectly.

@rich I agree, I'm not using a GUI but I would still like to make sure the system gets backed up automatically without having to manually run some command before I shutdown.

@centralware I agree with *not* having people run a script manually to backup before shutting down.  People are forgetful creatures.  As mentioned to coreplayer2, I don't think there is any need to adjust the symlinks or rearrange how that is currently working.  My solution works without any other changes to the system and is a simple edit that works with other currently implemented methods of the OS - no permissions issues either!

@core-user then my solution is still perfect, a simple "echo 0 > /etc/sysconfig/backup" will prevent that from happening and you don't have to do anything else!  Regarding the comment about "it's not easy to find out how to disable this feature for a newbie", I can say the same thing about if they want it turned on.  Currently I had to do some investigation on what was being used to indicate backups should be run and what could be done to make this happen automatically (something very difficult to explain to a newbie).  Also, I would say most people would rather have a backup by default and worry about how to turn it off than the other way around.  Again, this is a simple solution that if you wanted to script it to turn it off or on would be highly trivial.

Dave

[CentralWare]

Dave: The only problem with sysconfig/backup is persistence.  (You'll need to add that file to /opt/.filetool.lst otherwise it gets regenerated by init.d/tc-config at each boot if the boot code of norestore is not found.)  The same applies to my direction I mentioned as well.

What we ended up doing on our end tends to suit both your desires as well as remote administration needs without the need for "backups" but does not (our version) tend to compression as is the case with the MyData.gz file which filetool works with.

On install, we persist /opt - regardless of reboots, it always remains intact without the filetool necessity.  We then create a directory of /opt/.root which is an overlay intended for persisting everything outside of /opt (with the exception of the /home directory, of course, since it has its own persistence boot code.)  As such, if I want to make permanent changes to /etc/hosts (for example) the file I change is /opt/.root/etc/hosts and if I need to make the changes immediate, I have another script to tend to that which does nothing more than:

#!/bin/sh
cd /opt/.root        # Move into the root storage directory
sudo cp -a * /       # Copy all files/dirs found onto the root file system

This usually maintains all permissions, etc. and never requires a "backup" which is something I don't want to do in the first place, especially on flash-based drives, as it requires writing to the drive every time a user reboots or shuts down, killing the life expectancy of a flash drive.

Example: Let's pretend an entire flash drive allowed for 1,000 writes before crashing and the computer in question is rebooted 3x daily and that a single file was updated once per day:

Your method:
*) The update to the given file is done in RAM -- no write to flash
*) Running filetool.sh 3x daily will kill the flash drive in 333 days

My method:
*) The update to the given file is done in FLASH
*) The flash drive will fail in 1,000 days

Mind you, that's my way of thinking and doesn't apply to everyone out there so it's based on our specific needs/wants, not necessarily those globally.  The method mentioned also prevents "To err is human" with end users as there's nothing to be "backed up" so to speak and managing every file within the system (with one exception) can easily be implemented in both directions.  For example, if I wanted to preserve user/pass information:
sudo cp -a /etc/shadow /opt/.root/etc/shadow
viola'!  The next time she boots...  passwords are restored.  (More files are required for multiple accounts above/beyond tc and root, but you get the idea.)  Our TC image itself has already been remastered to tend to all of this, but to implement it into /opt/bootsync.sh is just as viable a solution.  Mind you, when you add users or change passwords, you have to copy the updated /etc/shadow back into /opt/.root but we created an automation system for that as well to some degree.

** The exception for /opt/.root is the file /etc/sudoers as for some reason or another it doesn't like being copied by root.staff (must be root.root) and requires permission updates (0440) which cannot easily be done sudo-ing, so instead we create a new file in /tmp/sudoers, once it's ready to implement we reset permissions/ownership and then sudo mv /tmp/sudoers /etc/sudoers which tends to the problem of modifying the sudo settings while USING sudo.

[wysiwyg]

@centralware you are correct, that is something I didn't consider (persistence of /etc/sysconfig/backup).  I guess another point I would make is that if /etc/sysconfig/backup indicates whether backups should occur automatically during shutdown/reboot, why should it make a difference if I have a GUI or not.  Shouldn't that be the case no matter what?  Dealing with persistence, file placement, file naming, etc is another matter entirely (see below).


The problem that I see with adding another file (/opt/.autobackup) is that now we have multiple files controlling the backups when shutting down or rebooting since the documentation in the TC book says to manipulate the /etc/sysconfig/backup file.  If there was to be another file added, I would suggest (for conformity) to create a file in the users home directory ~/[.etc/]backup.  This way, the user directory file can take preference over the global /etc/sysconfig file easily:

if [ $(cat ~/.etc/backup) == 1 ]; then
        filetool.sh -b
elif [ $(cat /etc/sysconfig/backup) == 1 ]; then
        filetool.sh -b
fi

This way simply doing an "echo 0 > ~/.etc/backup" would easily allow no bacukps by those who don't want it and simply deleting that home file would ensure that the system backs up by default (which again, I would assume would be the desired route to take for end users).

As a side question, is there a reason why /etc was not included as part of the persistent backup like ~/ and /opt?  Wouldn't doing this prevent the /opt/.root overlay issue, which in my opinion is yet another "oh yeah and remember this admins if you're working with the TC distro".

Dave

[CentralWare]

sysconfig/backup is intended for X for the most part, but also is set to zero in the event the boot code norestore exists if I'm speculating correctly.  (Why make a backup if you're not going to restore it?)

Your argument as to using sysconfig/backup as the determining factor, though, has benefits and drawbacks.  backup=1 is set in the event boot codes don't say otherwise.  When doing an install, this will always be "1" unless you specify norestore regardless of there being X or not.  Now, remote administration (ssh, for example) it's been customary here NEVER to update boot codes remotely unless direly necessary (as you can't SEE the results from the user's machine) so boot/loader is off limits for the most part.  Having items in the file system tends to be safer and also makes admin as well as user modifications simpler as well.

Our "general" TC boot codes are:
quiet syslog norestore nodhcp tce=something opt=something
...and if ~/ is needed, home=something is added as well.

This allows me to boot TC and implement the "merge" in /opt/.root making all machines virtually identical, we then check in with a server for updates (kernel, init image, /opt/.root updates, etc.) before any extensions are tended to, making administration quite a bit more automated.  In a good number of cases, /home, local and /tce are mounted remotely (NFS, AoE, etc) so the "backup" system really doesn't work too well in these cases, especially for users with a TON of files (or simply large files) in their homes.

Imagine you have a user who likes surfing multimedia sites online (example using firefox).  The caches are done in /home/user which on a flash drive, is hell on the drive AND tends to waste a large amount of space.  If they're tab-mongers like I am (many tabs open at once) this impacts performance tremendously.  If you were to back up ~/ ...  God help you!

If you have users with persistent homes, be sure to link caching and other similar activities to /tmp or similar, thus your backup should only tag the link file as opposed to everything within it.  (I never looked, but I have to assume filetool.sh doesn't follow symlinks - or at least I'd hope not!)

As for why /etc isn't mounted like /opt or /home...  I can't speak for Robert and/or any others who decided this, but it's my guess that it's the goal of the backup system so that you have a "clean" install at every boot --- including passwords (or the lack thereof) as people forget those things all the time.  Since it's basically a single user environment from install, I imagine they weighed the thought and saw it being more problem than purposeful.  It CAN easily be done, though:

Create a static directory on /mnt/sdXX/etc
in bootsync.sh as the first line of the file, add the line
mount --bind /mnt/sdXX/etc /etc
This probably is not recommended by the Tiny Core Team, but it's do-able if needed.
(I'd likely make it a bit more elaborate than that since sdXX isn't guaranteed on every machine, but it gives the concept.)  Actually, this should work rather well with a static /tce:

Code: [Select]

tce=`readlink /etc/sysconfig/tcedir`
mnt=${tce:1:3}
dev=${tce:5:4}
if [ "$mnt" == "mnt" ];
  if [ -d /mnt/$dev/etc ]; then
    mount --bind /mnt/$dev/etc /etc
  fi
fi
tce= gets the /mnt/sdXX/tce directory
mnt= tests to make sure we're using a device (/mnt/sdXX) instead of /tmp/tce
dev= gets just the "sdXX" portion of the device
if /mnt/sdXX/etc exists, we mount it on top of /etc (the old /etc is then "hidden" so to speak)

EDIT: The persistent sdXX/etc directory will have to be populated first otherwise you'll end up mounting an empty /etc which will cause all kinds of havoc.  As such, you'd likely want to test /mnt/sdXX/etc for a file (such as sudoers) to make sure it already exists on sdXX/etc otherwise the system will fail to do much of anything.  Due to specific permissions of /etc/sudoers I recommend running tar -zcf on /etc and extracting the zip file onto the persistent.

[wysiwyg]

Good info centralware!  Thankfully I haven't gotten to the point where I need to do any root overlays, so that really isn't affecting me at this point (although I still think a better way can be found to handle that aspect).

And just for the record, the TC book does not say that the /etc/sysconfig/backup file is for backing up the /opt and /home directories - not sure how I got that information.  The book says to add 'backup=0' to the ~/.profile file.  However, the /etc/init.d/skel/.profile looks like it does use it for this purpose.  I'm a little confused myself now...

Dave

[CentralWare]

No, the /opt being backed up is (by default) in /opt/.filetool.lst which /etc can be placed there as well (though I discourage doing so with flash drives as you write to the drive EVERY time you reboot/shutdown which is relatively taxing on the drive over time.  I'm testing the concept right this moment and just using firefox with a persistent /home directory is almost constantly writing to the flash drive with every character typed.  This is why I suggest the /opt/.root concept as the /home directory can otherwise stay in memory except for files needing to be saved which normally exclude a large number of the /home/user/.* directories.

There are flash drives claiming to be "commercial grade" which I haven't a clue as to how they can "withstand" more write cycles (I doubt they can, I imagine they just have more "backup" memory cells than the average flash drive) but these drives will likely be best suited for long term writing (in the case of TC, backups included) but I'm unsure of the cost difference being worthwhile.