Tiny Core Linux
Tiny Core Base => TCB Talk => Topic started by: soundcheck on July 14, 2009, 07:59:11 AM
-
Hi folks.
I just had an idea, while waiting for the backup to be finished. ;)
The backup takes quite some for obvious reasons.
The main reason (as far as I can see): you cp/tar the entire persistent stuff from scratch all the time.
It might be worth to spend a thought on rsync.
This way you'd only save the deltas. And the backups where done in a second.
Please apologize if this is a nonsene idea. (I am still getting into TC) :D
-
I see two potential problems with the proposal compared to the current solution:
- Can you rsync to FAT volumes and retain all file properties correctly (ownership, permissions, etc)?
- The current solution compresses the data. This might be of benefit for some users. For one it speeds up the restore.
/Lars
-
If your backup is taking a long time to "backup"
you should check your trash can(if using XFE), clear your firefox cache, etc
and also I noticed that firefox sooner or later makes a 30-50mb "database"
or something, which can only be resolved by uninstalling firefox and re-installing anew.
I no longer use Firefox because of this:)
Opera with flash10 works and is light.
My backup takes about 5-8sec at reboot,etc
that's with opera, flash10, xfe, xfw, mtpaint, and OSS
a few wallpapers and my dragon jwmthemes
-
Good tip on ff. Opera on tc is amazingly fast and renders pages well.
5-8 secs is an eternity in tc though!
Fat or ntfs can't handle symlinks which rules out rsync for me.
Best solution at present would be persistent home.
Andrew
-
Yes, to clarify, that's on USB without home or opt.
I have alot of wallpapers and themes, which slows it a little too.
Opera is the ticket though.
I discovered the firefox thing when trying to make a small tinycore cd
after doing bookmarks, etc the damn thing would somehow build a database or something in /home/tc/.mozilla and my little cd would balloon to over 100mb.
so, no-go firefox
I checked and without my wallpapers and themes the backup is like 2 sec
-
Minefield / Firefox /... and those pesky sqlite databases have been discussed. They can really slow down your backup and use up much space. See: http://forum.tinycorelinux.net/index.php?topic=492.0
Other tips to reduce your backup time, is to factor out static data into personal tce.
Just use tar command to make, for example, mybackgrounds.tce, which could be the wallpapers and themes.
As for rsync, I use it as the primary tool to backup all the development files for TC and MC. But for reasons stated it is not the default. However, rsync is included in the base system.
-
I didn't realize I could make a tce of my wallpapers and themes?
what is the process to do this Roberts?
simply tar the opt and jwmthemes folders and rename it .tce or what?
I have alot of themes
( http://multidistro.com/tinycore-shots/tc-scrnshots.html )
and each wallpaper, jwmtray, jwmtheme, etc are all different for each theme
so how would I make sure when I want blackdragon theme+wallpaper that it would work?
would I have to seperate each theme/wallpaper into a different tce?
right now I use grub menu entry to boot into theme I want
this is for my 4gb ext3 kingston usb tc entry
title <<<<---Back to Main Menu
root (hd0,0)
configfile /boot/grub/menu.lst
title TinyCore_2.1-Amethyst Dragon
root (hd0,0)
kernel /boot/bzImage quiet desktop=jwm restore=sda1/colors/amethyst tce=sda1/tce waitusb=5 max_loop=255 noicons embed
initrd /boot/tinycore.gz
title TinyCore_2.1-Black Dragon
root (hd0,0)
kernel /boot/bzImage quiet desktop=jwm restore=sda1/colors/black tce=sda1/tce waitusb=5 max_loop=255 noicons embed
initrd /boot/tinycore.gz
title TinyCore_2.1-Blue Dragon
root (hd0,0)
kernel /boot/bzImage quiet desktop=jwm restore=sda1/colors/blue tce=sda1/tce waitusb=5 max_loop=255 noicons embed
initrd /boot/tinycore.gz
title TinyCore_2.1-Gold Dragon
root (hd0,0)
kernel /boot/bzImage quiet desktop=jwm restore=sda1/colors/gold tce=sda1/tce waitusb=5 max_loop=255 noicons embed
initrd /boot/tinycore.gz
title TinyCore_2.1-Red Dragon
root (hd0,0)
kernel /boot/bzImage quiet desktop=jwm restore=sda1/colors/red tce=sda1/tce waitusb=5 max_loop=255 noicons embed
initrd /boot/tinycore.gz
title TinyCore_2.1-Silver Dragon
root (hd0,0)
kernel /boot/bzImage quiet desktop=jwm restore=sda1/colors/silver tce=sda1/tce waitusb=5 max_loop=255 noicons embed
initrd /boot/tinycore.gz
title TinyCore_2.1-White Dragon
root (hd0,0)
kernel /boot/bzImage quiet desktop=jwm restore=sda1/colors/white tce=sda1/tce waitusb=5 max_loop=255 noicons embed
initrd /boot/tinycore.gz
-
Yikes, so many backups, how do you keep track of your real data?
In concept based on your grub menu.lst, I would make each wallpaper/theme a tce with a startup script to copy them to their proper locations. I would store them in the optional directory under the tce directory. I would use a custom boot option of say, theme=BlackDragon. Then in .xsession test for the contents of theme boot code and tce-load the result.Finally, adjust your grub with the new theme boot option. Now you will once again have one common backup file and it will be much faster. There are obviously other ways to approach this, but this is one. Have fun with it.
-
I didn't realize I could make a tce of my wallpapers and themes?
what is the process to do this..
Load the advcomp extension, then something like
$ tar -czvf my.tce --numeric-owner -T files
or
$ tar -czvf my.tce --numeric-owner file1 file2 ...
then
$ advdef -z4 my.tce
..where "files" is a file containing the names of the files you would like to include in your tce
Edit: I'm assuming the files are owned by tc:staff otherwise you need to use "sudo"
-
OK, thanks very much for that info Roberts and Juanito! ::)
I'm gonna try it now.
Yikes, so many backups, how do you keep track of your real data?
lol, there actually is no "real" data there, I make multidistro stuff so I try to keep everything real clean.
meaning, I just installed opera, flash, oss, xfe and xfw and my wallpaper/theme stuff.
I use it mostly for jammin down some heavy tunes, and web-surfing ::)
I will say that tinycore with just OSS is louder than Puppy, or my Ultimate-edition 2.2 (ubuntu 9.04), on my 300w speakers, and that's good too ;D
I'm still a little confused on all you said there Roberts, but I will try to muddle thru.
-
I see two potential problems with the proposal compared to the current solution:
- Can you rsync to FAT volumes and retain all file properties correctly (ownership, permissions, etc)?
- The current solution compresses the data. This might be of benefit for some users. For one it speeds up the restore.
/Lars
Hi.
1.
For fat32 you need to use the --modify-window option
man-page:
--modify-window
When comparing two timestamps, rsync treats the timestamps as being equal if they differ by no more than the modify-window value. This is normally 0 (for an exact match), but you may find it useful to set this to a larger value in some situations. In particular, when transferring to or from an MS Windows FAT filesystem (which represents times with a 2-second resolution), --modify-window=1 is useful (allowing times to differ by up to 1 second).
rsync -av --modify-window=2 /srcdir /destdir
2. Compression
Since I work with different builds and storage space is no issue I could easily live with uncompressed trees on my media. I could also move data
from one tree to another.
With rsync I can even easily implement incremental backups, in case I messed something up I can easily rollback to an earlier stage.
The incremental backups just include the delta data. space is no subject.
-------------
The above comments "Take Opera" -- keep your system lean. Come on. That's not the way it works.
I have a build environment , with quite some sources and kernel-headers, etc. Backup gets quite annoying.
-
Backup will be quicker if you add
home/tc/.xfe/trash
home/tc/.mozilla
to .xfiletool.lst
If you use those programs.
Also save any large files, such as downloads, in a folder other than /home/tc
If you use home you can save files in /home rather than /home/tc
If you use encrypted home, you can save files in /mnt/sda1
-
just tried to use rsync in order to backup folders and data from tc to my home server with:
rsync -a -e ssh /mnt/sda1/MyfolderonTC/ ssh me@myhomeserver:/home/me/folder/backupdirectoryonmyserver
impossible to perform the backup even if i can connect to my server; rsync ends up saying:
rsync: connection unexpectedly closed (0 bytes read so far)
rsync error: error in rsync protocol data stream ...
-
Use
rsync -av /mnt/sda1/MyfolderonTC/ me@myhomeserver:/home/me/folder/backupdirectoryonmyserver
-
I use Tiny Core with persistent home and opt, and make extensions for anything with personal settings, such as printer setup, and don't use backup at all.
People are asking, how can we improve backup?
I think a better question is, how can we set up Tiny Core to not use backup at all?
The only situation where backup may be an advantage is when running from a USB drive, as it will result in the USB drive lasting longer.
If you run from a USB drive and browse the internet, backup takes too long, so it is easier not to use it (or don't backup internet browser cache). It is probably easier to buy a new USB drive when it fails, than to waste too much time backing up.
I would only use backup, is if I ran Tiny Core from a USB drive and did not use it for browsing the internet.
The other place where backup is useful (when set up this way) is to backup documents (not Tiny Core files and settings).
These are personal preferences. I know others have different opinions.
-
I do a lot of websurfing, and I clear the (Opera) browser cache before I exit, but I do notice a gradual increase in backup time. I think it's mostly cookies (which of course keep increasing as you websurf). Unfortunately a lot of sites expect you to have them. And I'm using a USB so I don't have persistent Opt or Home. I generally uncheck backup when I shutdown unless I've added a browser favorite or updated a setting I want to keep. Any mp3s or whatever that I download a put in a personal directory that TC doesn't save.
-
I recommend on occasion to run filetool.sh backup to see what is being included in your backup.
Usually this results in adding more files or directories to .xfiletool.lst. Maintaining a lean backup requires some file management, i.e., do you know what your backup includes?
-
I confess to not even being aware of this utility. Backup has been something of a black box to me. I'll certainly give it a try.
-
I don't think Opera removes all the files. IIRC there are many empty and other small files, such as icons, that are left.
-
I ran filetool.sh backup and got an intimidating list of files.
I wonder if this is something that needs to be looked at more carefully by application developers. It seems like most (other than TCL ones) expect that everyone has a huge hard disk and is running a standard installation, so that backup is not a major consideration. It is in TCL though.
-
Especially true for browsers!
But even those with huge hard drives and traditional installations should perform a backup!
Placing your home directory onto a persistent media is easy enough by using the bootcode home=xxx (see http://www.tinycorelinux.com/faq.html)
-
Hi everyone
I have just use tiny core for few days and I hated backups taking so long, so I wrote this. Now, I dont tell anyone to use it, but it works find for me. Configuring tiny core the way backups are not needed is good idea tough.
filetool.sh:
#!/bin/sh
FILETLIST="/opt/.filetool.lst"
XFILELIST="/opt/.xfiletool.lst"
BACKUPDIR="/mnt/sda1/backup"
MOUNT_DEV="/dev/sda1"
abort(){
echo "Usage: filetool.sh options device"
echo "Where options are:"
echo "-b backup"
echo "-r restore"
exit 1
}
[ -z $1 ] && abort
while getopts br OPTION
do
case ${OPTION} in
b) BACKUP=TRUE ;;
r) RESTORE=TRUE ;;
*) abort ;;
esac
done
# $MOUNT_DEV needs /etc/fstab line
if [ ! "$(cat /proc/mounts | grep $MOUNT_DEV)" ]; then
echo "$MOUNT_DEV not mounted ..mounting"
mount $MOUNT_DEV
if [ "$?" != 0 ]; then
echo "Unable to mount $MOUNT_DEV"
exit 1
fi
fi
if [ ! -d $BACKUPDIR ]; then
echo "Creating nonexistent backup directory ($BACKUPDIR)"
mkdir -p $BACKUPDIR
if [ "$?" != 0 ]; then
echo "Unable to create $BACKUPDIR"
exit 1
fi
fi
if [ "$BACKUP" ]; then
echo "Running rsync"
rsync -ar --delete --delete-excluded --exclude-from "$XFILELIST" --files-from "$FILETLIST" / "$BACKUPDIR"
fi
if [ "$RESTORE" ]; then
echo "Running rsync"
rsync -ar "$BACKUPDIR/" /
fi
echo "done."
exit 0
-
Rsync transfers deltas to save bandwidth over the network but it does NOT store deltas to disk. It stores an entire file even when just one byte changes in the source. In this case rsync is slower than cp because:
1. the file is copied mostly from target to target instead of source to target (backup target is usually slower)
2. calculation of checksums adds a little work to the cpu, albeit not much
The only benefit for TC backups is when the backup filesystem is ext2 or anything that supports symbolic links, so duplicate files can be stored as symbolic links.
FAT can be used too but the lack of symbolic links means a lot of duplication occurs. And it is worse than cp because duplication is from target to target.
-
Instead of having symbolic links in FAT, I wonder if a .tar file can be mounted as a writeable filesystem, with any changes appended to the .tar. Then symbolic links would effectively be available on any filesystem, even FAT. Then rsync could greatly speed up the backup.
-
tar is an archive file format, not a filesystem.
A loop file with a UNIX filesystem could be mounted.
The UMSDOS filesystem driver would provide for symlinks on FAT* filesystems, but it is no longer maintained, AFAIK.
-
tar is an archive file format, not a filesystem.
Thanks, I know what tar is, the attractive attribute for TC is that it stores symbolic links therefore it can store what rsync generates - if only the intermediate software layer is available to mount it. Someone must have written such software, if not, what is to stop you guys from writing it? AFAIK extensions are .tar files, how much work is it to make similar mounts read-write?
Even primitive windows shortcuts plus a layer to make them look like symbolic links might do.
-
TC extensions are squash filesystems, inherently read-only.
Windows shortcuts have technically nothing in common with UNIX symbolic links.
-
Windows shortcuts have technically nothing in common with UNIX symbolic links.
For backup purposes we do not need pointers to file nodes but just paths and file names. Because we are not going to rename anything in the backup media. So a layer that makes rsync write duplicate files as windows shortcuts (minus the C: part) makes sense. It won't break the backup, but will save a lot of space and time.
-
How to mount a .tar file:
http://code.google.com/p/freebsd-tarfs/
-
Windows shortcuts have technically nothing in common with UNIX symbolic links.
For backup purposes we do not need pointers to file nodes but just paths and file names. Because we are not going to rename anything in the backup media. So a layer that makes rsync write duplicate files as windows shortcuts (minus the C: part) makes sense. It won't break the backup, but will save a lot of space and time.
::)
I rest my case...
-
Just make sure you don't look at everything from the computer's point of view. :) Human purposes and abstractions are what matters. Otherwise we'd still be programming in assembly.
-
Linux file copy benchmark rsync vs cpio vs cp vs tar
http://blog.mudy.info/2010/07/linux-file-copy-benchmark-cp-vs-cpio-vs-tar-vs-rsync/ (http://blog.mudy.info/2010/07/linux-file-copy-benchmark-cp-vs-cpio-vs-tar-vs-rsync/)
-
Linux file copy benchmark rsync vs cpio vs cp vs tar
http://blog.mudy.info/2010/07/linux-file-copy-benchmark-cp-vs-cpio-vs-tar-vs-rsync/ (http://blog.mudy.info/2010/07/linux-file-copy-benchmark-cp-vs-cpio-vs-tar-vs-rsync/)
Interesting per se, but not of much relevance for the TC backup mechanism for more than one reason:
1. The biggest part of time used for the TC backup is due to compression - as opposed to simple copying or uncompressed archiving.
2. Several options used in this test are not applicable to the busybox version of these apps.
3. rsync seems to be no longer in base - as opposed to mentioned in an earlier post in this thread.
What could be of more relevance for the backup would be benchmarks for compression (time versus size trade-off). But then there also the choices within TC base would be limited.