WelcomeWelcome | FAQFAQ | DownloadsDownloads | WikiWiki

Author Topic: filetool.sh: Why not using rsync as backup utility?  (Read 11759 times)

Offline thane

  • Hero Member
  • *****
  • Posts: 688
Re: filetool.sh: Why not using rsync as backup utility?
« Reply #15 on: December 27, 2009, 08:51:07 PM »
I do a lot of websurfing, and I clear the (Opera) browser cache before I exit, but I do notice a gradual increase in backup time. I think it's mostly cookies (which of course keep increasing as you websurf). Unfortunately a lot of sites expect you to have them. And I'm using a USB so I don't have persistent Opt or Home. I generally uncheck backup when I shutdown unless I've added a browser favorite or updated a setting I want to keep. Any mp3s or whatever that I download a put in a personal directory that TC doesn't save.

Offline roberts

  • Administrator
  • Hero Member
  • *****
  • Posts: 7361
  • Founder Emeritus
Re: filetool.sh: Why not using rsync as backup utility?
« Reply #16 on: December 28, 2009, 10:10:11 AM »
I recommend on occasion to run filetool.sh backup to see what is being included in your backup.
Usually this results in adding more files or directories to .xfiletool.lst. Maintaining a lean backup requires some file management, i.e., do you know what your backup includes?
10+ Years Contributing to Linux Open Source Projects.

Offline thane

  • Hero Member
  • *****
  • Posts: 688
Re: filetool.sh: Why not using rsync as backup utility?
« Reply #17 on: December 28, 2009, 10:17:17 AM »
I confess to not even being aware of this utility. Backup has been something of a black box to me. I'll certainly give it a try.

Offline ^thehatsrule^

  • Administrator
  • Hero Member
  • *****
  • Posts: 1726
Re: filetool.sh: Why not using rsync as backup utility?
« Reply #18 on: January 02, 2010, 12:53:47 PM »
I don't think Opera removes all the files.  IIRC there are many empty and other small files, such as icons, that are left.

Offline thane

  • Hero Member
  • *****
  • Posts: 688
Re: filetool.sh: Why not using rsync as backup utility?
« Reply #19 on: January 02, 2010, 01:26:17 PM »
I ran filetool.sh backup and got an intimidating list of files.

I wonder if this is something that needs to be looked at more carefully by application developers. It seems like most (other than TCL ones) expect that everyone has a huge hard disk and is running a standard installation, so that backup is not a major consideration. It is in TCL though.

Offline roberts

  • Administrator
  • Hero Member
  • *****
  • Posts: 7361
  • Founder Emeritus
Re: filetool.sh: Why not using rsync as backup utility?
« Reply #20 on: January 02, 2010, 01:43:23 PM »
Especially true for browsers!

But even those with huge hard drives and traditional installations should perform a backup!

Placing your home directory onto a persistent media is easy enough by using the bootcode home=xxx  (see http://www.tinycorelinux.com/faq.html)


10+ Years Contributing to Linux Open Source Projects.

Offline henrixd

  • Newbie
  • *
  • Posts: 2
Re: filetool.sh: Why not using rsync as backup utility?
« Reply #21 on: December 25, 2010, 10:10:09 AM »
Hi everyone

I have just use tiny core for few days and I hated backups taking so long, so I wrote this. Now, I dont tell anyone to use it, but it works find for me. Configuring tiny core the way backups are not needed is good idea tough.

filetool.sh:
Code: [Select]
#!/bin/sh

FILETLIST="/opt/.filetool.lst"
XFILELIST="/opt/.xfiletool.lst"
BACKUPDIR="/mnt/sda1/backup"
MOUNT_DEV="/dev/sda1"

abort(){
  echo "Usage: filetool.sh options device"
  echo "Where options are:"
  echo "-b backup"
  echo "-r restore"
  exit 1
}

[ -z $1 ] && abort

while getopts br OPTION
do
  case ${OPTION} in
   b) BACKUP=TRUE ;;
   r) RESTORE=TRUE ;;
   *) abort ;;
  esac
done

# $MOUNT_DEV needs /etc/fstab line
if [ ! "$(cat /proc/mounts | grep $MOUNT_DEV)" ]; then
        echo "$MOUNT_DEV not mounted ..mounting"
        mount $MOUNT_DEV
        if [ "$?" != 0 ]; then
                echo "Unable to mount $MOUNT_DEV"
                exit 1
        fi
fi

if [ ! -d $BACKUPDIR ]; then
        echo "Creating nonexistent backup directory ($BACKUPDIR)"
        mkdir -p $BACKUPDIR
        if [ "$?" != 0 ]; then
                echo "Unable to create $BACKUPDIR"
                exit 1
        fi
fi

if [ "$BACKUP" ]; then
        echo "Running rsync"
        rsync -ar --delete --delete-excluded --exclude-from "$XFILELIST" --files-from "$FILETLIST" / "$BACKUPDIR"
fi
if [ "$RESTORE" ]; then
        echo "Running rsync"
        rsync -ar "$BACKUPDIR/" /
fi

echo "done."
exit 0

Offline Ulysses_

  • Full Member
  • ***
  • Posts: 232
Re: filetool.sh: Why not using rsync as backup utility?
« Reply #22 on: December 27, 2010, 10:46:13 AM »
Rsync transfers deltas to save bandwidth over the network but it does NOT store deltas to disk.  It stores an entire file even when just one byte changes in the source.  In this case rsync is slower than cp because:

1. the file is copied mostly from target to target instead of source to target (backup target is usually slower)

2. calculation of checksums adds a little work to the cpu, albeit not much

The only benefit for TC backups is when the backup filesystem is ext2 or anything that supports symbolic links, so duplicate files can be stored as symbolic links.

FAT can be used too but the lack of symbolic links means a lot of duplication occurs.  And it is worse than cp because duplication is from target to target.
« Last Edit: December 27, 2010, 11:04:46 AM by Ulysses_ »

Offline Ulysses_

  • Full Member
  • ***
  • Posts: 232
Re: filetool.sh: Why not using rsync as backup utility?
« Reply #23 on: December 27, 2010, 10:57:06 AM »
Instead of having symbolic links in FAT, I wonder if a .tar file can be mounted as a writeable filesystem, with any changes appended to the .tar.  Then symbolic links would effectively be available on any filesystem, even FAT.  Then rsync could greatly speed up the backup.
« Last Edit: December 27, 2010, 11:02:53 AM by Ulysses_ »

Offline tinypoodle

  • Hero Member
  • *****
  • Posts: 3857
Re: filetool.sh: Why not using rsync as backup utility?
« Reply #24 on: December 27, 2010, 01:41:04 PM »
tar is an archive file format, not a filesystem.

A loop file with a UNIX filesystem could be mounted.

The UMSDOS filesystem driver would provide for symlinks on FAT* filesystems, but it is no longer maintained, AFAIK.
"Software gets slower faster than hardware gets faster." Niklaus Wirth - A Plea for Lean Software (1995)

Offline Ulysses_

  • Full Member
  • ***
  • Posts: 232
Re: filetool.sh: Why not using rsync as backup utility?
« Reply #25 on: December 27, 2010, 02:02:44 PM »
tar is an archive file format, not a filesystem.

Thanks, I know what tar is, the attractive attribute for TC is that it stores symbolic links therefore it can store what rsync generates - if only the intermediate software layer is available to mount it.  Someone must have written such software, if not, what is to stop you guys from writing it?  AFAIK extensions are .tar files, how much work is it to make similar mounts read-write?

Even primitive windows shortcuts plus a layer to make them look like symbolic links might do.
« Last Edit: December 27, 2010, 02:05:08 PM by Ulysses_ »

Offline tinypoodle

  • Hero Member
  • *****
  • Posts: 3857
Re: filetool.sh: Why not using rsync as backup utility?
« Reply #26 on: December 27, 2010, 03:10:22 PM »
TC extensions are squash filesystems, inherently read-only.

Windows shortcuts have technically nothing in common with UNIX symbolic links.
"Software gets slower faster than hardware gets faster." Niklaus Wirth - A Plea for Lean Software (1995)

Offline Ulysses_

  • Full Member
  • ***
  • Posts: 232
Re: filetool.sh: Why not using rsync as backup utility?
« Reply #27 on: December 27, 2010, 04:16:12 PM »
Windows shortcuts have technically nothing in common with UNIX symbolic links.

For backup purposes we do not need pointers to file nodes but just paths and file names.  Because we are not going to rename anything in the backup media.  So a layer that makes rsync write duplicate files as windows shortcuts (minus the C: part) makes sense.  It won't break the backup, but will save a lot of space and time.
« Last Edit: December 27, 2010, 04:21:00 PM by Ulysses_ »

Offline Ulysses_

  • Full Member
  • ***
  • Posts: 232
Re: filetool.sh: Why not using rsync as backup utility?
« Reply #28 on: December 27, 2010, 04:26:20 PM »

Offline tinypoodle

  • Hero Member
  • *****
  • Posts: 3857
Re: filetool.sh: Why not using rsync as backup utility?
« Reply #29 on: December 27, 2010, 06:06:17 PM »
Windows shortcuts have technically nothing in common with UNIX symbolic links.

For backup purposes we do not need pointers to file nodes but just paths and file names.  Because we are not going to rename anything in the backup media.  So a layer that makes rsync write duplicate files as windows shortcuts (minus the C: part) makes sense.  It won't break the backup, but will save a lot of space and time.

 ::)
I rest my case...  
"Software gets slower faster than hardware gets faster." Niklaus Wirth - A Plea for Lean Software (1995)