Tiny Core Linux
Tiny Core Base => Raspberry Pi => Topic started by: drtebi on November 10, 2018, 03:46:15 AM
-
Hello,
I have successfully built a raspberry pi based audio streamer with piCore, using cifs (to mount my network share with its music), mpd, alsa, and busybox-httpd (to serve album images).
Everything is working just perfectly, but I realized now that all my extensions are mounted from /mnt/mmcblk0p2/. As far as I understand, this is not good if I now want to do a "hard" shutdown. My device is configured with only a power knob, which turns on and off the power supply inside the case... this was intentional, since the case mimics a vintage Sony line, so a soft-shutdown is no option.
So my question now is, how do I change all the extensions to be loaded into RAM, so that I don't have to worry when doing a hard shutdown?
I read about copy2fs, but all documentation I can find about it relates to the graphical installer. I don't have the GUI installed, and don't really want to, either.
I see there is a bunch of tce-* scripts, but I am a bit lost as to which one to use in order to do what I want...
-
$ touch /mnt/mmcblk0p2/tce/copy2fs.flg
-
Thank you!
Damn, it's that simple? To be honest, I was surprised how simple setting up piCore was in general... I waited too long to do this.
One more question: In order to compile mpd, I had to install a few things like gcc etc. I won't need these anymore, so it would be best if I could not have these extensions loaded. How do I get rid of them via CLI? Is there an overview somewhere for all the tce-* commands? That would really help...
-
The cli tool you need is tce-audit - builddb and then delete
Somebody started to explain the tce-* commands in the wiki, but it looks to be a wip.
You could take a look at the book:
http://tinycorelinux.net/book.html
-
OK, I will try that.
I looked through the book already, I have only found a chapter on tce-load, everything else seems to refer to the graphical app interface.
-
Take a look at
/etc/sysconfig/tcedir/onboot.lst
and remove extensions you think are not needed.
Installing extensions via tce-load -wi ... automatically creates entries in that file for those extensions to loaded on boot.
-
Take a look at
/etc/sysconfig/tcedir/onboot.lst
and remove extensions you think are not needed.
Installing extensions via tce-load -wi ... automatically creates entries in that file for those extensions to loaded on boot.
OK, I will check that out, although I think I like using tce-audit for this better... it could be considered a CLI front-end for extensions I suppose.
I can for example use tce-audit notrequired to list extensions that are not strictly required, then pick the ones I think I don't need for my setup and verify this with e.g. tce-audit requiredby boost... and then use tce-audit delete boost for example.
One question I still have is, should the second partition (/dev/mmcblk0p2) be unmounted when using the copy2fs flag? In the book there is an example for a "Web Kiosk", where they mention the copy2fs flag, and then further suggest that one adds something like
umount /dev/mmcblk0p2
to /opt/bootlocal.sh
-
Yes, if you've used the copy2fs.flg you can unmount the partition.
-
Perfect!
Thanks for all the quick responses.
-
hi drtebi,
piCorePlayer is a music client for the LMS/Squeezebox environment based on piCore. In the 4 or 5 years of existence, we have hundreds, maybe thousands of pCP's installations and we haven't seen the corruption issue you are trying to avoid. Lots of users just pull the plug.
We implemented a copy2fs option years ago and it never when from "development" to "production" because the need never surfaced.
regards
Greg
-
hi drtebi,
piCorePlayer is a music client for the LMS/Squeezebox environment based on piCore. In the 4 or 5 years of existence, we have hundreds, maybe thousands of pCP's installations and we haven't seen the corruption issue you are trying to avoid. Lots of users just pull the plug.
We implemented a copy2fs option years ago and it never when from "development" to "production" because the need never surfaced.
regards
Greg
Hi,
yes I know about piCorePlayer. I was tempted to use it, but I prefer MPD, which I have already installed on a few other computers. I liked the challenge to start from scratch with piCore... it was surprisingly easy, following mostly a great tutorial by a French guy (http://bz31.tuxfamily.org/dokuwiki/doku.php?id=core:picore_rpi3).
It's interesting to hear that piCorePlayer does not use copy2fs. I feel safer using it... although I have noticed quite a longer boot time now (which makes sense... since a bunch of stuff is copied into RAM at boot time now).
Anyway, I am quite happy with the outcome, only my second project where I solder electronics together (linear power supply etc), my first experience milling aluminum, etching etc! :D
Here a picture, where it is sitting underneath the vintage Sonys, almost done with the case:
(http://drtebi.com/dump/ternowski-np1/np1-with-companions.jpg)
-
When data is written there is usually cache involved. In case of power interruption this data might not be completely written, i.e. still in the cache and therefore lost. If the system just reads I can imagine this might be negligible. I wouldn't do this with other OSes where I don't know whats running in the background.
-
Hi drtebi
That's one fine looking enclosure you built. Counter-boring the mounting hole for the power switch was a nice touch. What did
use to relieve the ears for rack mount, a fly cutter with a profiled bit? Did you make the slots by drilling first and then cutting
the slots, or is that just an artifact of the picture? I am also curious about what technique you used to apply the markings and
whether it was done before or after applying the brushed finish. At any rate, it looks very clean and professional.
-
Good job drtebi,
It's interesting to hear that piCorePlayer does not use copy2fs. I feel safer using it... although I have noticed quite a longer boot time now (which makes sense... since a bunch of stuff is copied into RAM at boot time now).
Personally, I really like the idea of using copy2fs and do from time to time. But working with a team means you have a decision process to follow. We haven't had many people complaining about SD card corruptions but do get a lot of "How do I make pCP boot faster?".
regards
Greg
-
Hi drtebi
That's one fine looking enclosure you built. Counter-boring the mounting hole for the power switch was a nice touch. What did
use to relieve the ears for rack mount, a fly cutter with a profiled bit? Did you make the slots by drilling first and then cutting
the slots, or is that just an artifact of the picture? I am also curious about what technique you used to apply the markings and
whether it was done before or after applying the brushed finish. At any rate, it looks very clean and professional.
Thanks ;)
You pretty much nailed it... however, the big challenge was, that I don't have a mill! I used my old Atlas drill press (50s or so) to mill the "ears", first with an end mill, then with a router bit that had more or less the right profile to create the edge that leads back to the flat plane.
The slots were made just as you said, drilling a hole first, then sawing and filing.
The markings ("Ternowski...") were done with electro etching. It took a few practice pieces to figure out how long to etch, which solution to use etc.. The metal (aluminum in this case) must be very clean for this, so it was sanded and cleaned thoroughly before etching.
The counter-bored hole for the power knob was a challenge. Since I have a woodworking shop, not a machinist shop, I ended up doing it with a router and a small circle jig that I build. It worked so well, if I had to do it all over again, I would use a router to make the "ears" as well.
I will post a couple of pictures later to give you some visuals...
-
Good job drtebi,
It's interesting to hear that piCorePlayer does not use copy2fs. I feel safer using it... although I have noticed quite a longer boot time now (which makes sense... since a bunch of stuff is copied into RAM at boot time now).
Personally, I really like the idea of using copy2fs and do from time to time. But working with a team means you have a decision process to follow. We haven't had many people complaining about SD card corruptions but do get a lot of "How do I make pCP boot faster?".
regards
Greg
That makes a lot of sense... I figure that, as mentioned by Misalf, there is very little writing going on, so piCorePlayer does not really face the problem of data corruption.
-
Here are a few pictures of the process... I hope this is not unorthodox to post here, as this is not directly related to piCore...
First the "milling" of the ears of the faceplate. It worked, but I don't recommend doing it this way. Either use a proper mill for this, or, if only woodworking tools at hand, a router with a carbide bit can be used, as long as the aluminum is very well secured.
(http://drtebi.com/dump/ternowski-np1/038-.JPG)
(http://drtebi.com/dump/ternowski-np1/041-.JPG)
(http://drtebi.com/dump/ternowski-np1/054-.JPG)
Here a picture of the circle jig I made to cut the counter-bore for the large hole (on a test piece).
(http://drtebi.com/dump/ternowski-np1/064-.JPG)
The electro etching is a bit more elaborate. I use "Press-n-Peel Blue Transfer Film" to transfer the printed image to the aluminum. Any spots are touched up with nail polish, or simply taped. Then I build a sort of cage around the print, glued down with hot glue. For the actual etching process, I added a vinegar and salt solution, then connected the test piece to the positive of battery charger, and the negative to a copper piece that is suspended in the solution above the print. I etched for four minutes.
(http://drtebi.com/dump/ternowski-np1/078-.JPG)
(http://drtebi.com/dump/ternowski-np1/079-.JPG)
(http://drtebi.com/dump/ternowski-np1/079-b.jpg)
The result will look somewhat like this. Note that it needs to be sanded again, usually the etching bites at least a little into some areas that aren't supposed to be etched.
(http://drtebi.com/dump/ternowski-np1/080-.JPG)
It's a lot of work, but a hell lot of fun :)
Eventually I will post one big blog post on my website with more images. At this time though, I have to figure out how to get the green LED in there, and then find some matching paint for the case (not the front, of course). I am also thinking about nickel-plating the frontplate, since aluminum is a bit sensitive to fingerprints etc.
-
Hi drtebi
Some very creative solutions, especially how you accomplished the counter-bore. Vinegar and salt, very low tech, I like it.
... First the "milling" of the ears of the faceplate. It worked, but I don't recommend doing it this way. ...
As long as you keep the cuts light you should be OK.
... if only woodworking tools at hand, a router with a carbide bit can be used, as long as the aluminum is very well secured. ...
You can also use HSS bits if you control your RPMs to keep the SFM in the 100 to 200 range.
-
Hi drtebi
Some very creative solutions, especially how you accomplished the counter-bore. Vinegar and salt, very low tech, I like it.
... First the "milling" of the ears of the faceplate. It worked, but I don't recommend doing it this way. ...
As long as you keep the cuts light you should be OK.
... if only woodworking tools at hand, a router with a carbide bit can be used, as long as the aluminum is very well secured. ...
You can also use HSS bits if you control your RPMs to keep the SFM in the 100 to 200 range.
Vinegar, salt and electricity that is :) Nickel plating is done in a similar way... I thought I will try that rather than anodizing, which requires a bunch of "ugly" chemicals, temperature checks etc.
Yes, you are totally right, routing aluminum with a woodworking router should be done in very small steps, light cuts. It works very nicely though.
I am actually not sure what the slowest speed is on my drill press. I believe the cutters I used for the ears were both HSS. But I didn't particularly like the setup... it wasn't dangerous, but just not stiff enough to get smooth results. If I had made a jig to hold the aluminum, and then routed it, I am sure I could have gotten a bit more perfection...
-
And by the way... I should give this guy some credit; for my etching, I followed for the most part exactly what he did in his video:
https://www.youtube.com/watch?v=bOMRcqxZe4I
He explains every detail. Only that he does it on some kind of pistol grip or something... oh well ;)
-
Awesome work drtebi 8)
-
@drtebi
Please do not hijack topics! What you posted has nothing to do with piCore!
It will be moved out. Next time will be deleted without notice.
Thank you for understanding.
Béla
-
@drtebi
Please do not hijack topics! What you posted has nothing to do with piCore!
It will be moved out. Next time will be deleted without notice.
Thank you for understanding.
Béla
I did hijack my own topic... as I am the OP. But I understand nevertheless, and I apologize for getting into non-related piCore stuff.
Someone asked questions about the case work... so I posted some pictures. It would have been a bit rude not to answer.
-
Hi drtebi
Someone asked questions about the case work... so I posted some pictures. It would have been a bit rude not to answer.
That was my fault. Sorry about that.
-
Also, if you want to make it even less likely to corrupt itself when pulling the power (and make the SDCard last for several years longer (since they generally support far far more read cycles than write cycles)) you may want to do everything in: http://forum.tinycorelinux.net/index.php/topic,21821.0.html
Cheers.
-
Also, if you want to make it even less likely to corrupt itself when pulling the power (and make the SDCard last for several years longer (since they generally support far far more read cycles than write cycles)) you may want to do everything in: http://forum.tinycorelinux.net/index.php/topic,21821.0.html
Cheers.
Thanks, that's helpful.
Can you explain what is being done there, or rather, why that is the better way? I thought once the SD card is unmounted, it would be safe.
-
I thought once the SD card is unmounted, it would be safe.
That is true. I guess I wasn't that clear, but above when I said "even less likely" I meant to imply that the initial thing was already pretty good.
... why that is the better way?
Booting, having the OS mount in rw mode, and then unmounting has two (minor) problems:
- You can still loose power while booting (before it's unmounted) and corrupt the card (like when the power flicks on after a power outage or when someone plugs it in and then tries to quickly pull it out again, etc.).
- The OS might write to the filesystem when it mounts it in rw mode (access times, fsck dirty checks, etc.). While this won't corrupt the card it will use up a write cycle, which will make the card fail sooner. (Since most SD cards support far far far more reads than writes before they fail.)
So, if you make Linux only ever mount filesystems on the card in readonly mode, both the above problems are avoided. That is the remastering thing (changing the mount flags in the compressed image) in that link.
Now that is pretty good and projects like piCorePlayer say they have thousands of users and never had a corruption, so it is likely good enough. But even in that case the SD card can still corrupt itself because the SD card itself (that is, the hardware) sometimes rearranges it's internal sectors (maybe because some have gone bad and it is using some unused sectors or something) and if the power is cut while it is doing that it might (if you are really really unlucky) corrupt the data.
The other thing that xpector does in that link is use sdtool to change a bit in the hardware of the SD card to locked (read only mode). When that is done the OS cannot mount anything on it as rw as writes will be blocked by the hardware of the SD card itself (as a note this is totally different than the little plastic switch on the side of regular-sized SD cards which don't change anything in the card itself, they just tell the reader to not write, which it can ignore). Additionally some SD cards might use this internal setting to avoid rearranging sectors, or they might not, there isn't really any way to know. I guess it just makes me feel better.
-
Hi san.gh
... Now that is pretty good and projects like piCorePlayer say they have thousands of users and never had a corruption, so it is likely good enough. But even in that case the SD card can still corrupt itself because the SD card itself (that is, the hardware) sometimes rearranges it's internal sectors (maybe because some have gone bad and it is using some unused sectors or something) and if the power is cut while it is doing that it might (if you are really really unlucky) corrupt the data. ...
The other thing that xpector does in that link is use sdtool to change a bit in the hardware of the SD card to locked (read only mode). When that is done the OS cannot mount anything on it as rw as writes will be blocked by the hardware of the SD card itself (as a note this is totally different than the little plastic switch on the side of regular-sized SD cards which don't change anything in the card itself, they just tell the reader to not write, which it can ignore). Additionally some SD cards might use this internal setting to avoid rearranging sectors, or they might not, there isn't really any way to know. I guess it just makes me feel better.
I'm afraid I'm going to have to question some of the content of those paragraphs. Correct me if I'm wrong, but this is the scenario
I see you describing:
1. Your SD card is always running in Read Only mode.
2. The cards internal logic detects one of its sectors reliability may have become questionable and wants to remap it.
3. To prevent the possibility that a power glitch occurs at exactly the same time the card remaps that sector and
corrupts the data, you want to prevent the card from remapping that sector.
4. You wait for the sector to fail and corrupt your data.
-
Okay, though I don't understand what you are asking, exactly. I also feels like you are conflating a few different cases (three kinds of "read only mode" and two failure scenarios), so I'm going to list them and then refer to them by names.
Read only modes:
- RO Mode 1: Only the copy2fs flag is set, so the card (p2) is mounted readwrite on boot, the extensions are copied to ram, then at the end of boot (in bootlocal.sh) p2 is unmounted.
- RO Mode 2: The kernel images are remastered to never mount p2 readwrite on boot and mount it readonly the first time. Then the extensions are copied out, and at this point it doesn't matter much if p2 is unmounted at the end of boot.
- RO Mode 3: The SD card itself is locked in hardware, so the kernel cannot mount it readwrite even if it wants to (it will mount it readonly), and neither can the user (accidentally or otherwise). (And of course it doesn't matter if it is unmounted in this mode.)
SD Card Failure modes:
- End of Life: The SD card (or a sector on the card) has exceeded the number of reads or writes that the hardware is capable of and the hardware itself has failed.
- Corruption: The data on the SD card is in an inconsistent state so the OS can't read or write to the filesystem properly, but the hardware itself is fine.
Okay, so corruption can happen either because the OS is in the middle of a write when the power is cut or because the SD itself was rearranging things when the power was cut. End of Life is basically just based on the number of writes (barring physical damage).
There is no way to know or prevent the SD card from internally doing anything, different manufacturers do different things, etc., it's a black box. Which is why above I said it basically just makes me feel better because I suspect (but do not know and cannot prove or even show that it is likely) that some manufactures will refrain from rearranging as much as they can if the hardware bit is set (since internally they can see that, whereas the card hardware can't know if the OS has it mounted readwrite or readonly).
The other issue is that in RO Mode 1 the card might or might not be written to on every boot. If you don't care about power being cut while booting changing to RO Mode 2 or 3 (instead of RO Mode 1) is only about extending the time before End of Life (NOT about avoiding corruption).
Hopefully that helps clear up what I meant. Does it?
-
Guys, you are missing an important element of the puzzle, file system and journaling.
-
...also MLC flash drive controllers may or may not run various implementations of error correction, garbage collection and Wear-leveling operations whenever the flash device is powered. No data connection is required for these maintenance operations.
Sent from my iPhone using Tapatalk
-
Well, one reason why I don't think I will take it any further than the "copy2fs" mode (referred to here as RO Mode 1), is because I do sometimes want to run filetool.sh. For example, when I do put together a playlist, I like to login and just have to type a couple of commands to ensure this new playlist is saved to the SD, and then I reboot.
I am also not worried about a power failure during boot... I don't know, I just feel like that is one very unlucky situation that one would have to encounter.
So I think I will stick with what I have setup for now... and do a backup of the entire SD card once in a while. By the way, this can simply be done with "dd" on another machine, correct?
-
Guys, you are missing an important element of the puzzle, file system and journaling.
Well, that's kind of a different thing. Journalling makes OS corruption way way less likely, but not impossible, (and of course to use it the card has to be writable (shortening its life)).
-
Well, one reason why I don't think I will take it any further than the "copy2fs" mode (referred to here as RO Mode 1), is because I do sometimes want to run filetool.sh. For example, when I do put together a playlist, I like to login and just have to type a couple of commands to ensure this new playlist is saved to the SD, and then I reboot.
I am also not worried about a power failure during boot... I don't know, I just feel like that is one very unlucky situation that one would have to encounter.
So I think I will stick with what I have setup for now... and do a backup of the entire SD card once in a while. By the way, this can simply be done with "dd" on another machine, correct?
Cool. Sticking with RO Mode 1 is fine, and likely good enough mostly corruption-wise. (Even though it shortens the life of the card.)
But, I want to say that even with RO Mode 3 you can log in (to a Raspberry Pi which has the MMC connected to the CPU bus (it has /dev/mm* instead of /dev/sd* devices)), run some commands and save files to the card. In RO Mode 3 you have _more_ commands to run, though, and have to copy the sdtool onto the pi; so that may be too much work for too little gain.
The general process (from the original link) is something like:
#making changes after you locked it read-only
sudo ./arm-sdtool /dev/mmcblk0 unlock
mount /mnt/mmcblk0p2
sudo mount -o remount,rw /mnt/mmcblk0p2
#now make the change
filetool.sh -bv
sudo umount /mnt/mmcblk0p2
sudo ./arm-sdtool /dev/mmcblk0 lock
instead of just running filetool.
And yes, if you want to back up you can just take an image of the card with dd from another computer.
-
Thanks for the info.
One question, where it says
#Now make the changeDoes that imply that one cannot do it before those steps? In other words, I would have to run the commands before
filetool.sh -bvbefore I make changes like adding a new playlist?
Anyway, I suppose one could just put all that into a script, and life would be simpler.
A question about "dd":
Is it possible to write the image over the network? I am thinking I could do the backup through the terminal, logged in to the piCore, and copy the image (over rsync?) directly to my NAS.
Again... would make life easier. It's a bit of work to open up my case :)
-
Hi drtebi
... One question, where it says
#Now make the changeDoes that imply that one cannot do it before those steps? In other words, I would have to run the commands before
filetool.sh -bvbefore I make changes like adding a new playlist? ...
Yes.
-
Does that imply that one cannot do it before those steps?
This confuses me. What is "it"?
You can make changes to the file system in RAM at any moment. Like saving a playlist at /home/tc .
In order for those changes to be able to be backed up (filetool.sh) or to write anything to the SD card the SD card has to be mounted.