Better implementation of restore feature?

[Spix]

I've been taking a second look at Tiny Core Linux, and am scratching my head over the restore feature.  It seems that when the restore feature is enabled, an entirely new gzip'ed tar file is created with all applicable files, every time a backup is made before shutdown/restart (when selected).  This means it takes the same amount of time, and disk writing, whether you changed no files at all, or changed all files.  Would it not be WAY better to instead mount a file as a loopback volume, then use rsync or unison (or similar tool) to backup only files that have changed (or are newer)?  It seems to me that this would be WAY faster (not to mention saving life time of flash storage devices).

[mocore]

just ftr
"rbackup : script to perform backup and restore via rsync to a remote server" ~ https://forum.tinycorelinux.net/index.php/topic,9508.msg51782.html#msg51782

+
"zsync is like rsync, except that it's over http and doesn't need server support" ~ https://forum.tinycorelinux.net/index.php/topic,5013.msg26647.html#msg26647

[CentralWare]

My two methods (depending on whether using persistent opt/home or not using):

• Persistent /opt stops me from using mydata.tgz entirely; I have a directory /opt/.root which contains all of the files that have changed (manually) and backup is turned off
• WITH MyData: Completely shut off automated backups and run them ONLY when something new is needed inside the backup

• No wasted time or resources
• Flash Friendly(er)

For RasPi, the Micro-SD card is isolated and dismounted on boot (here in my shop, not necessarily a world-wide suggestion) thus keeping it mostly corruption-safe by:

• Using an almost-flush-mounted USB device for /tce (extensions) and persistent /opt, /home, etc.
• Dismount the Micro-SD once the above usb device is found and activated
• In the event of any possible corruption, it's usually on the usb device - unplug it and the unit still boots piCore, just not with backed up items and extra extensions; it's an easy fix "live"

NOTE: The virtually fool-proof method is to have a RasPi or similar, an old PC or a NAS box set up as a tiny file server and using NFS or other similar remote file system, use IT for persistence completely; the Micro SD card then becomes just the means to get basic TC kernel running, the remote files do everything else.  TinyCore supports a number of file systems in the boot process (nfs, aoe, nbd, etc.) so whether on a Pi or PC, TinyCore can be expanded at boot time to endless possibilities when managing multiple devices.

[mocore]

TinyCore supports a number of file systems in the boot process (nfs, aoe, nbd, etc.) so whether on a Pi or PC,
TinyCore can be expanded at boot time to endless possibilities when managing multiple devices.

wrt *endless possibilities* hear is another similar thread with a tweaked filetool.sh

" filetool.sh: Why not using rsync as backup utility?" https://forum.tinycorelinux.net/index.php/topic,2345.msg44137.html#msg44137
just for ref
https://github.com/tinycorelinux/Core-scripts/blob/master/usr/bin/filetool.sh

this interesting quote from the above thread

Rsync transfers deltas to save bandwidth over the network but it does NOT store deltas to disk.  It stores an entire file even when just one byte changes in the source.  In this case rsync is slower than cp because:

1. the file is copied mostly from target to target instead of source to target (backup target is usually slower)

2. calculation of checksums adds a little work to the cpu, albeit not much

The only benefit for TC backups is when the backup filesystem is ext2 or anything that supports symbolic links, so duplicate files can be stored as symbolic links.

FAT can be used too but the lack of symbolic links means a lot of duplication occurs.  And it is worse than cp because duplication is from target to target.

+ a working link from another post in that thead
+ https://d1vz4y16krebbd.cloudfront.net/2010/07/linux-file-copy-benchmark-cp-vs-cpio-vs-tar-vs-rsync/index.html

...
https://unix.stackexchange.com/questions/747626/why-is-uncompressed-tar-much-faster-than-rsync
 
all the *endless possibilities* bring many considerations !

depending on where rsync is its pointed over *whoevers* network  this might be one to consider
" issue  in rsync pre 3.2.5 that allows malicious remote servers to write arbitrary files inside the directories of connecting peers. " - https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-29154 ... .tinycorelinux.net/14.x/x86/tcz/rsync.tcz.info

which reminded me of some bsd system calls (pledge() and unveil()) mentioned in "Using Landlock to Sandbox GNU Make" - https://justine.lol/make/ & https://justine.lol/pledge/

researching alittle on that tangent  i discovered  there is BSD-licensed implementation of rsync  @ .github.com kristapsdz/openrsync
who's readme mentions

The actual work of porting is matching the security features provided by OpenBSD's pledge(2) and unveil(2).
These are critical elements to the functionality of the system.
Without them,
your system
accepts arbitrary data
from the public network.

[mocore]

TinyCore supports a number of file systems in the boot process (nfs, aoe, nbd, etc.) so whether on a Pi or PC, TinyCore can be expanded at boot time to endless possibilities when managing multiple devices.

  wrt file systems / restore ect

* some interesting / noteworthy observations in the below quote

make me wander (wonder!!! @rich  ;-) && || ponder;
how

Code: [Select]

btrfs send my_snapshot > my_backup.btrfs.datfunctionality might potentially be intertwingled with core restore

...

Folks still using btrfs owe it to themselves to understand the peculiarities of btrfs send and receive, which can be described, at best, as "quirky". However, sends do not necessarily require an existing btrfs partition to receive it. If one plans to upload their snapshot to the cloud, for example, one should be able to send a snapshot to a file, "btrfs send my_snapshot > my_backup.btrfs.dat", and then store that .dat file as a single archive of data. It will be unusable in that state until it's received back into a functional btrfs partition, but it's a single file which can be stored anywhere, on any file system.

via
https://forum.mxlinux.org/viewtopic.php?p=763097#p763097 #  MX Linux with fs btrfs and timeshift with @root and @home label partitions'

....
++ the next reply in the thread mentions https://github.com/Antynea/grub-btrfs 

grub-btrfs improves the grub bootloader by adding a btrfs snapshots sub-menu, allowing the user to boot into snapshots.

grub-btrfs supports manual snapshots as well as snapper and timeshift created snapshots.

Warning: booting read-only snapshots can be tricky

... which presumably just processes *somehow* a list of 'snapshots' into grub menu format

make me wander if any one has tested/considered btrfs snapshots (in some other topics?) or using some other method to achieve similar results?
... i think iv seen scrips in the past which rotated 5-10 backup's afair allowing restoring from different version


*imho