Tiny Core Linux
Tiny Core Base => Raspberry Pi => Topic started by: onelife on June 03, 2020, 03:19:51 AM
-
Hi there Forum,
I seem to be going in circles with making a custom TCZ that will place a boot script in the /usr/local/tce.installed/ directory that runs md5sum check on the mydata.tgz backup file.
The issue is not that the tcz isn't loading, but for some reason I am getting a 2nd copy in a duplicate foler called /usr_1/ - seems very odd.
Example :
snb@snb:~$ md5sum /usr_1/local/tce.installed/md5mydata
baffab92405c6f437de2e7653dc523b4 /usr_1/local/tce.installed/md5mydata
snb@snb:~$ md5sum /usr/local/tce.installed/md5mydata
baffab92405c6f437de2e7653dc523b4 /usr/local/tce.installed/md5mydata
Both folders have my exact script and md5 shows the files are identical - I don't know how maybe I have told it to load twice?
By squashfs folder path is all in order from what I know as :
/squash-root/usr/local/tce.installed/md5mydata
Any suggestions on how I fix this issue?
BIG Thanks.
-
This issue happens when you create a squashfs package on top of itself.
If you create a squashfs package and wish to change something, you need to delete it before re-creating a squashfs package with the same name.
-
Hi Juanito
... If you create a squashfs package and wish to change something, you need to delete it before re-creating a squashfs package with the same name.
Or you can tell mksquashfs to handle it for you:
mksquashfs SOURCE DESTINATION -noappend
-
:) THANK YOU THANK YOU!!!! I would never have thought this was based on the mksquashfs command. All sorted now. Apprecaite your help.
-
I thought perhaps another user might be trying to also make a md5sum check of their mydata.tgz - I don't know if this will help anyone but FYI, after getting the squashfs correctly working it seems to help 100%. FYI, my md5data bash script is similar to this :
#!/bin/sh
if [ -f /mnt/mmcblk0p2/tce/md5mydata ] ; then
sudo cat /mnt/mmcblk0p2/tce/md5mydata | md5sum -c > /tmp/md5sum
if grep "FAILED" /tmp/md5sum ; then
echo "MD5SUM FAILED"
if [ -f /mnt/mmcblk0p2/tce/mydata.corrupt ] ; then
echo "NO BACKUP AVAILABLE - GOING INTO RESCUE"
sudo /usr/local/etc/init.d/openssh start
sleep 20
sudo sed -i -e "s|#PermitEmptyPasswords no|PermitEmptyPasswords yes|" /usr/local/etc/ssh/sshd_config
sudo /usr/local/etc/init.d/openssh restart
sleep 1
exit
else
sudo mv -f /mnt/mmcblk0p2/tce/mydata.tgz /mnt/mmcblk0p2/tce/mydata.corrupt
sleep 1
sudo mv -f /mnt/mmcblk0p2/tce/mydatabk.tgz /mnt/mmcblk0p2/tce/mydata.tgz
sleep 1
sync
sleep 2
sudo reboot
fi
else
echo "MD5SUM PASSED"
fi
else
echo "NO MD5SUM - CHECKING SYSTEM"
fi
exit
And essentially the file /tce/md5mydata file is created whenever I call filetool.sh by my own script :
#!/bin/sh
TCRASPBERRY=/home/snb/update_skynetwork/log/tcraspberry
DAILYFILETOOL=/home/snb/update_skynetwork/log/dailyfiletool
if [ -f $DAILYFILETOOL ] ;then
echo "Waiting Until 01h00 to run filetool.sh"
exit
fi
if [ -f $TCRASPBERRY ] ;then
filetool.sh -b
sleep 2
sync
sleep 2
sudo chmod 777 /mnt/mmcblk0p2/tce/md5mydata
sudo md5sum /mnt/mmcblk0p2/tce/mydata.tgz > /mnt/mmcblk0p2/tce/md5mydata
echo "Backup Complete"
else
echo "System NOT Tiny Core"
fi
exit
So on boot the md5mydata script checks the md5sum of the backup and if it doesn't match, then move bk backup into position and reboot.
There are a few pieces I've removed from my script as once in "rescue mode" I have a few things run such as ssh start etc.
But as I say, I hope maybe this can help someone elese if they too experience corrupt mydata.tgz files and are looking for a way to auto recover to the mydatabk.tgz file.
Thanks
-
Hi onelife
Just a few comments. You need to learn proper indentation. If I had to apply a name to your formatting, it would probably be called
SRI (Semi Random Indentation) ::). What you have makes it difficult to follow which lines of code belong to which if clause. Here
is my suggestion of how to make it more readable:
#!/bin/sh
if [ -f /mnt/mmcblk0p2/tce/md5mydata ] ; then
sudo cat /mnt/mmcblk0p2/tce/md5mydata | md5sum -c > /tmp/md5sum
if grep "FAILED" /tmp/md5sum ; then
echo "MD5SUM FAILED"
if [ -f /mnt/mmcblk0p2/tce/mydata.corrupt ] ; then
echo "NO BACKUP AVAILABLE - GOING INTO RESCUE"
sudo /usr/local/etc/init.d/openssh start
sleep 20
sudo sed -i -e "s|#PermitEmptyPasswords no|PermitEmptyPasswords yes|" /usr/local/etc/ssh/sshd_config
sudo /usr/local/etc/init.d/openssh restart
sleep 1
exit
else
sudo mv -f /mnt/mmcblk0p2/tce/mydata.tgz /mnt/mmcblk0p2/tce/mydata.corrupt
sleep 1
sudo mv -f /mnt/mmcblk0p2/tce/mydatabk.tgz /mnt/mmcblk0p2/tce/mydata.tgz
sleep 1
sync
sleep 2
sudo reboot
fi
else
echo "MD5SUM PASSED"
fi
else
echo "NO MD5SUM - CHECKING SYSTEM"
fi
exit
Now you can see where each if clause ends and which else belongs to which if. And don't use the spacebar for indentation, that's
what the Tab key is for.
What's with all the sleep commands here:
else
sudo mv -f /mnt/mmcblk0p2/tce/mydata.tgz /mnt/mmcblk0p2/tce/mydata.corrupt
sleep 1
sudo mv -f /mnt/mmcblk0p2/tce/mydatabk.tgz /mnt/mmcblk0p2/tce/mydata.tgz
sleep 1
sync
sleep 2
sudo reboot
fi
The kernel should handle completing the first move before the second one starts.
Does it not work correctly like this:
else
sudo mv -f /mnt/mmcblk0p2/tce/mydata.tgz /mnt/mmcblk0p2/tce/mydata.corrupt
sudo mv -f /mnt/mmcblk0p2/tce/mydatabk.tgz /mnt/mmcblk0p2/tce/mydata.tgz
sync
sudo reboot
fi
The same basic comments apply to your other script.
Will it not work if you change this:
filetool.sh -b
sleep 2
sync
sleep 2
sudo chmod 777 /mnt/mmcblk0p2/tce/md5mydata
sudo md5sum /mnt/mmcblk0p2/tce/mydata.tgz > /mnt/mmcblk0p2/tce/md5mydata
to this:
filetool.sh -b
sudo chmod 777 /mnt/mmcblk0p2/tce/md5mydata
sync
sudo md5sum /mnt/mmcblk0p2/tce/mydata.tgz > /mnt/mmcblk0p2/tce/md5mydata
-
Hi, Rich!
Maybe my 2 cents are misplaced, but for slow flash drives sync doesn't guarantee that data have found their place, and sometimes those sleeps are the only way to success.
For example here are citations from tc-istall.sh:
partition_setup_common(){
echo "Writing zero's to beginning of /dev/$DEVICE"
dd if=/dev/zero of=/dev/$DEVICE bs=1k count=1 > /dev/null 2>&1 &
sync; sleep 1
echo "Partitioning /dev/$DEVICE"
if [ "$FORMAT" == "vfat" ]; then
echo -e "n\np\n1\n\n\na\n1\nt\nc\nw" | fdisk /dev/$DEVICE > /dev/null 2>&1
else
echo -e "n\np\n1\n\n\na\n1\nt\n83\nw" | fdisk /dev/$DEVICE > /dev/null 2>&1
fi
sync; sleep 1
}
and then after calling partition_setup_common:
hdd_setup(){
partition_setup_common
sync; sleep 1
rebuildfstab
...
My own TC lives on the flash drives, local hard disks are used as additional storage and I can confirm, that for scripts those sleeps, seeming extraneous, sometimes are unavoidable.
-
Hey onelife,
I was just wondering what problem you are having that requires you to md5sum mydata.tgz
I been using piCore since it was invented, do many backups and reboots a day and I have lots of RPi's. I never had a problem with mydata.tgz getting corrupted, except when it is very big and you run out of space on your SD card.
So I check the size of mydata.tgz and give a message if it is getting large.
regards
Greg
-
Hi there Greg,
Thank you for checking in on this - Essentially I have caught on more than one occasion the backup fails as sadly here in SA we have regular power cuts. If the power goes off while the system happens to be running a filetool.sh, the created backup is then corrupt.
I actually made a video of this in 2014 and posted it here on the forum. I suspect though this is unique to our situation here in SA. Yes, the backup process is super quick and the chance is very very slim BUT it has happened on several occasions.
Out of interest so we can compare a "ratio" as such - How many pi's are you running TC on?
Chat soon - Thank you.
-
lol, I have about 10 running at any given time. And another 5 or so used during development cycles. The ones used during development get many many backups and power cycles.
-
I usually have about 7 running but at the moment only 3. I had too do a "cleanup" a couple of weeks ago. lol.
Like Paul, the RPi used for development and testing get a lot of backups and reboots.
Can't say I had a power failure at the exact time I was doing a backup.
-
Aha, okay - I think the power issue is 100% the cause. As I say, I've tested this condition and it for sure causes the backup to be corrupt.
I then guess it's simply a ratio factor - Currently I have just under 2000 pi's running 24/7.
So hopefully my md5sum check can help stop the few failures I do get.
Big Thanks again for your thoughts on this - Here's to what is a superb OS - Hopefully soon I will attempt upgrading to 9.X or more - But sadly the modules / tcz's I require are not yet on 9.x - So for now 90% of my pi's are on 8.1.5 and lower.
If anyone is ever interested in getting involved commercially with my project I would only be to happy to welcome some needed guru advice.
Keep safe - Kind Regards,
-
We've moved onto piCore-11.x now - which extensions you believe are missing?
-
Aha, I last tried beta 9.X and at that time many were missing. My onboot.lst as example is :
mc.tcz
mpg123.tcz
rsync.tcz
bash.tcz
coreutils.tcz
vtun.tcz
lzo.tcz
visudo.tcz
cpulimit.tcz
dermixd.tcz
java.tcz
libmagic.tcz
libsndfile.tcz
studio.tcz
ntp.tcz
libvorbis.tcz
flac.tcz
firmware-ralinkwifi.tcz
libusb.tcz
usbutils.tcz
wifi.tcz
usb-modeswitch-data.tcz
usb-modswitch.tcz
firmware-brcmfmac43430.tcz
wireless-4.4.13-piCore+.tcz
rtc-4.4.13-piCore+.tcz
usb-serial-4.4.13-piCore+.tcz
alsa.tcz
alsa-plugins.tcz
alsa-utils.tcz
wireless-4.4.13-piCore_v7+.tcz
rtc-4.4.13-piCore_v7+.tcz
usb-serial-4.4.13-piCore_v7+.tcz
libacl.tcz
openssh.tcz
alsa-modules-4.4.19-piCore_v7+.tcz
alsa-modules-4.4.19-piCore+.tcz
alsa-modules-4.4.39-piCore_v7+.tcz
usb-serial-4.4.39-piCore_v7+.tcz
wireless-4.4.39-piCore_v7+.tcz
firmware.tcz
alsa-modules-4.4.39-piCore+.tcz
libevdev.tcz
libevent.tcz
libopus.tcz
beamer.tcz
netcat.tcz
md5mydata.tcz
ffmpeg.tcz
There are a few custom tcz's such as :
dermixd
java
studio
beamer
md5mydata
But the critical missing parts were example, alsa modules, usb serial and ffmpeg.
I'm not sure if the repo for 11.X is now "open" that we can see the available tcz's now?
Maybe just maybe it's now further along and I can attempt an 11.x build :)
Thank you!
[EDIT]: Added code tags. Rich
-
alsa-modules, usb-serial and ffmpeg are present in both the 9.x and 11.x repos - you can see what's in the 11.x repos at:
http://tinycorelinux.net/11.x/armv6/tcz/
http://tinycorelinux.net/11.x/armv7/tcz/
Note that deps are recursive, so there are a lot of lines that can be eliminated from your onboot.list
-
Hi jazzbiker
... Maybe my 2 cents are misplaced, but for slow flash drives sync doesn't guarantee that data have found their place, and sometimes those sleeps are the only way to success. ...
Taking a look at this:
https://man7.org/linux/man-pages/man2/sync.2.html
It states:
NOTES top
Since glibc 2.2.2, the Linux prototype for sync() is as listed above,
following the various standards. In glibc 2.2.1 and earlier, it was
"int sync(void)", and sync() always returned 0.
According to the standard specification (e.g., POSIX.1-2001), sync()
schedules the writes, but may return before the actual writing is
done. However Linux waits for I/O completions, and thus sync() or
syncfs() provide the same guarantees as fsync called on every file in
the system or filesystem respectively.
To me that says that even if sync returns right away, Linux will wait for any current I/O to complete before starting any new
commands. Typically sync is used to flush cached writes to disk so it an be unmounted or to shutdown the system.
Take for example this:
else
sudo mv -f /mnt/mmcblk0p2/tce/mydata.tgz /mnt/mmcblk0p2/tce/mydata.corrupt
sleep 1
sudo mv -f /mnt/mmcblk0p2/tce/mydatabk.tgz /mnt/mmcblk0p2/tce/mydata.tgz
sleep 1
sync
sleep 2
sudo reboot
fi
That first sleep is unnecessary. Linux will not execute the second mv command until the first one has completed.
If you did this:
sudo mv -f /mnt/mmcblk0p2/tce/mydata.tgz /mnt/mmcblk0p2/tce/mydata.corrupt &
sudo mv -f /mnt/mmcblk0p2/tce/mydatabk.tgz /mnt/mmcblk0p2/tce/mydata.tgz &
you could have a race condition and I don't know what will happen.
As for the other 2 sleep commands and the sync command, I'm pretty sure the reboot command calls sync too.
As an experiment, I copied a large file to a thumb drive (/mnt/sdh). Then in another terminal I monitored the busy file for the device:
cat /sys/block/sdh/device/device_busy
This always returned 1 while the cp command was active. It also returned 1 after cp command completed. I then issued
a sync command. It did not return for about 10 seconds. The device_busy file did not get set to zero until the sync command
completed.
-
look at the visual diagrams
https://schinagl.priv.at/nt/hardlinkshellext/linkshellextension.html
https://en.wikipedia.org/wiki/Symbolic_link
https://en.wikipedia.org/wiki/Hard_link
Hi there Forum,
I seem to be going in circles with making a custom TCZ that will place a boot script in the /usr/local/tce.installed/ directory that runs md5sum check on the mydata.tgz backup file.
The issue is not that the tcz isn't loading, but for some reason I am getting a 2nd copy in a duplicate foler called /usr_1/ - seems very odd.
Example :
snb@snb:~$ md5sum /usr_1/local/tce.installed/md5mydata
baffab92405c6f437de2e7653dc523b4 /usr_1/local/tce.installed/md5mydata
snb@snb:~$ md5sum /usr/local/tce.installed/md5mydata
baffab92405c6f437de2e7653dc523b4 /usr/local/tce.installed/md5mydata
Both folders have my exact script and md5 shows the files are identical - I don't know how maybe I have told it to load twice?
By squashfs folder path is all in order from what I know as :
/squash-root/usr/local/tce.installed/md5mydata
Any suggestions on how I fix this issue?
BIG Thanks.
-
Hi xor
look at the visual diagrams
https://schinagl.priv.at/nt/hardlinkshellext/linkshellextension.html
https://en.wikipedia.org/wiki/Symbolic_link
https://en.wikipedia.org/wiki/Hard_link
The problem that onelife was experiencing has nothing to do with hard or symbolic links.
If you read reply #1 by Juanito, he identified the cause of the problem and a solution.
If you read reply #2, I posted an alternate solution.
If you read reply #3 by onelife, he confirmed that his problem had been solved.
-
Hi, Rich!
Thnks for testing my suspicious statement :)
for slow flash drives sync doesn't guarantee that data have found their place, and sometimes those sleeps are the only way to success.
This sentence consists of the two statements, in fact.
1. "for slow flash drives sync doesn't guarantee that data have found their place" - it is false statement. I've found very slow 4 class SD card and made some tough tests upon it and now can fully agree with You, that sync does its job very well. And more, I feel much more comfortable dealing with flash devices :) Of course I was not modelling power faults and other horror movies, but probably excessive sleeps will not heal the hardware and physical issues.
2. "sometimes those sleeps are the only way to success" - this is true statement at least for one scenario, which I cited from tc-install.sh and which I faced myself. Repartitioning of connected device needs some time for udev to recognize newcomers, and there is nothing to do with this but wait. But this is very special case and sync is out of suspicions, really. I don't know are the other such scenarios existing, it will be good to know about.
Thanks again for shooting my suspicions! Life becomes better when they're gone :)