Formatting confusions(2) for USB

[goossbears]

Have already perused the requisite posting-guideline webpages at

• Intro, http://distro.ibiblio.org/tinycorelinux/intro.html
• Overview, http://distro.ibiblio.org/tinycorelinux/overview.html
• Concepts, http://distro.ibiblio.org/tinycorelinux/concepts.html
• FAQ, http://distro.ibiblio.org/tinycorelinux/faq.html

Microcore 4.0.2

According to one thread, at http://forum.tinycorelinux.net/index.php/topic,9213 , one can  optimally format a single USB drive with both fat32 (using the mkdosfs command) +plus+ extX (using the mke2fs command).  The advantage to this is that the fat32-formatted /persistent /home and /opt for example can be recognized by Windows and also be used to boot microcore + extensions. At the same time, one can have a persistent /home and /opt in that second extX-formatted /dev/sda2 partition.

The first source of confusion I have is; What is really the "best" extX for formatting /dev/sda2 in this case, where X is 2, 3, or 4 (ext2. ext3, ext4) ??
According to Guy in http://forum.tinycorelinux.net/index.php/topic,9213.msg50391.html#msg50391,

Use ext2 filesystem. This is not a journalling filesystem, and the usb drive will last longer.

The reason for this is essentially that any journalling extX filesystem (3 or 4) will wear down a USB drive because of the extra overhead from journalling read-writes.
Guy even writes:

If you use persistent home and opt, every time you save something, it is written to the usb drive

So therefore, having ext3 or ext4 on /dev/sda2 with its /home and /opt would seem really inefficient!

On the other hand, maybe ext3 and especially ext4 may be good enough here if a) if you don't save files on a super-frequent basis and b) if you are much more concerned about the data integrity of a possibly-faulty USB drive than you are about any performance degradation and current read-write weardown.
a) For using microcore, chances are that you're not really using memory-hungry and high reading-writing X-apps such as Firefox, Chromium, Gimp, ...etc.
b) Apparently, from webpage http://kernelnewbies.org/Ext4 , ext4 offers some improvements in files-storage data integrity ("significant-enough" improvements??)


The second source of confusion is; If you're not using Windows at all with your microcore (or tinycore) USB drive, then Why have two separately-formatted partitions (e.g., as above or with one as ext2 and another as ext4) instead of having just a single extX partition??

Take the common example of using a low-end 1 GB USB drive.
Maybe it is best to partition and format this drive as
/dev/sda1 ; extX formatted (X probably=2?) ; the full 1 GB capacity for  /boot, all other microcore startup folders&files including the PPR tce, the PPI folders /home and /opt, and any other possible data-files.

On the other hand, maybe you can multi-partition and multi-format this same 1 GB USB drive as
/dev/sda1 ; ext2 formatted ; 25 MB or less just for  /boot and other microcore startup files
/dev/sda2 ; extX formatted ; 450 MB or greater for the standard PPR and PPI folders as well as for any other possible data files
/dev/sda3 ; extX formatted ; 450 MB or greater as a duplicate copy of the previous-mentioned PPR and PPI contents of /dev/sda2 (is there a type of minimal fault-tolerant RAID 1 going on here??)

Any further good feedback or pointers on these?

[gerald_clark]

You cannot have /home or /opt on a fat drive.

Do not attempt to backup one partition on a flash to another on the same drive.
When a USB flash drive fails, the whole drive becomes inaccessible.

[goossbears]

You cannot have /home or /opt on a fat drive.

Where does it say the requirement that /home or /opt cannot be on a fat32 USB partition??
Does the webpage http://wiki.tinycorelinux.net/wiki:backup_usb make any reference to /opt, /home/ and even /tce specifically needing to be on an extX primary partition for the USB device, such as on /dev/sda1??
As in the bootloader configuration file entry

Code: [Select]

opt=UUID="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" home=UUID="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" tce=UUID="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" solely for ext2

Do not attempt to backup one partition on a flash to another on the same drive.
When a USB flash drive fails, the whole drive becomes inaccessible.

That really doesn't resolve my confusion over which filesystem -- ext2, ext3, or ext4 -- offers "significant enough" advantages here for one or more USB partitions, given that Microcore doesn't require anywhere near as many read/writes as X-apps in standard tinycore.
And yes, I've already viewed http://wiki.tinycorelinux.net/wiki:usb_life which recommends against (for some reason) using a journalling filesystem.
 

[bmarkus]

Where does it say the requirement that /home or /opt cannot be on a fat32 USB partition??

In general LINUX requires an appropriate file system supporting for examle symlinks and few other stuff. It can be ext2, ext3, ext4, jfs, xfs, reiser, ...  FAT filesystems do not have such features, not LINUX compatible.

It is not a TC specific. Usage of LINUX systems requires some basic LINUX knowledge. Of course a WEB page can be always enhanced, but concentrates on TC specific stuff and not a LINUX 101 curse.

[curaga]

For the partition layout and linux fs, use what you need. You can have zero partitions (floppy style) or 128 (GPT) and TC will run.

[goossbears]

Hmm...
Back to my second source of confusion if its just microcore (or tinycore) with no need for any Windows-access.
I previously wrote:

If you're not using Windows at all with your microcore (or tinycore) USB drive, then Why have two separately-formatted partitions (e.g., as above or with one as ext2 and another as ext4) instead of having just a single extX partition??

Maybe I should have better asked Why NOT instead of Why for this.


In other words, now that gerald_clark revealed the info of one-part-affects-all USB drive-failure,
Does it make the best sense to....

(a) As I first wrote,

partition and format this drive as
/dev/sda1 ; extX formatted (X probably=2?) ; the full 1 GB capacity for  /boot, all other microcore startup folders&files including the PPR tce, the PPI folders /home and /opt, and any other possible data-files.

All the boot and the persistence folders & files are in a single partition here, and I suppose that I could use extra identically-configured 1 GB USB drives, with one ext2 partition only each, for purposes of fault-tolerance.
 

-or-

(b) Somewhat similar to what I proposed second, but with a significant change,
/dev/sda1 ; ext3 formatted ; 10 MB or less just for the 7.8 MB contents of microcore's /boot folder + a max of 5% overhead for the ordered journal.
/dev/sda2 ; ext2 formatted ; 990 MB or greater for the standard PPR and PPI folders (/tce, /opt, /home) as well as for any and all other possible data files.
The key difference here is that there are TWO partitions here; UNidentically formatted.
The first ext3-formatted /dev/sda1 partition here will infrequently be read-from and written-to (just once each the USB drive is booted from), whereas the second partition -- containing solely persistence and data folders & files -- will  almost always be read-from and written-to each time tc extensions and data are accessed .
Again, for this option too, I suppose that I could use extra identically-configured 1 GB USB drives, with precisely two extX partitions-only each, for purposes of fault-tolerance.
 

It should be noted for both these two scenarios here, that if any single 1 GB USB drive becomes inaccessible though failure, the identically-configured extras can be plugged right in as bootable backup(s).
And I'm not even certain how the journaled ext3 partition in scenario (b) offers any significant advantage(s) here, given this availability of extra, identically-configured 1 GB USB drives and given the extX-related issues noted within a previous thread http://forum.tinycorelinux.net/index.php/topic,10430.0.html (?)


Any clearer insight on the advantages and disadvantages of each of these two scenarios??

[gerald_clark]

You have it backwards. /boot is rarely written.
Persistence folders, data and files are frequently written.

[goossbears]

bmarkus wrote:

In general LINUX requires an appropriate file system supporting for examle symlinks and few other stuff. It can be ext2, ext3, ext4, jfs, xfs, reiser, ...  FAT filesystems do not have such features, not LINUX compatible.

Yes, and the appropriate file-system partitioning step in microcore "base" to carry out before actually formatting with any of these file systems is

Code: [Select]

fdisk /dev/sda, as per bmarkus's Reply #12 in http://forum.tinycorelinux.net/index.php/topic,10430.0.html

Questions still remain in my two above scenarios whether two partitions are better than one here, and which Linux formats any of these partitions "should" use(?)
I'm wondering in the second scenario (b) whether the first partition could optimally use ext3 as a journaling-format alternative instead of ext2 (but probably not using ext4, jfs, xfs, reiser, ... )

I am aware that I would format the desired Linux ext partitions in microcore "base" with

Code: [Select]

mke2fs [ switches ] /dev/sda# .

[Rich]

Hi goossbears
I've read claims of running a journaling file system with no problems and claims of a USB stick
wearing out in a week because the sectors with the journal wore out. I've also read claims that
due to wear leveling it's OK to use a journaled files system. Unfortunately, those claims are based
on some very optimistic math, and don't take into account how the leveling is implemented or
how full the drive is. Since these devices support a finite number of write cycles, I personally feel
that EXT2 is the right way to go. I see no reason not to format the whole drive as one partition.
As far as "fault-tolerance" is concerned, you could use the  dd  command once a day or week or
whatever suits you and duplicate the contents onto a second device. Since the /opt directory does
not get changed very often you could make it persistent. If you want to make /home persistent you
should beware of any apps that may do frequent writes to a subdirectory in /home. Files that you
edit are fine. Since many devices support 100,000 write cycles, saving the same file 50 times a
day will take over 5 years to wear it out. A browsers cache on the other hand, would probably be
too hard on the device and should probably be pointed to a RAM based file instead.

[Guy]

Firstly, it is a free world. You can do it however you like. Take the information into consideration, and decide which method suits you.

But be aware:

Expect malfunctions if you don't use a Linux filesystem.

Grub4dos had not been updated to boot Ext4. But I have not checked recently, it may have changed. Not an issue if you are using a different bootloader.


I will explain about non-journalling filesystems.

Windows 3.1 came out with a non-journalling filesystem. When you saved a file, it would save it, then write a record of where it saved it. If you were saving a file and there was a power failure or malfunction, there would be this partly saved file, but no record of what it was. So you finished up with a number of lost files on the hard drive. You could run scandisk (I think) and it would find them, and name them file00001, file00002, file00003, etc. You could then open these files and see if there was anything valuable there. Then save them with a useful name, or delete them. Some of them were just parts of files, not entire files.

To overcome this journalling filesystems were introduced. When you save a file, it writes what it will save, it then saves it, then writes a record of what it saved. If there is a power failure or malfunction, if enough has been saved, the computer can recover it. If not enough has been saved, it reverts to the previously saved version. So you don't finish up with lost files.

Journalling filesystems should be used on hard drives.

For those using solid state memory, such as usb drives:

Non journalling filesystems can be fine if you backup important files.

A journalling filesystem may write approximately twice as much as a non-journalling filesystem when you save a file. So it may shorten the life of a usb drive. But it should still last many thousands of writes.


Persistent opt is probably ideal for most people, as you don't change settings often.

I have a usb drive I use for installation and transferring files. I choose to have persistent home. Rewriting all of your home files every time you turn the computer off, is a lot of writing. But it depends on how you use the usb drive. You need to decide for yourself. I use hard drives for serious operating systems.


There are no set rules. Others choose to do it differently. You can do it how you like.


A bit of history

Windows continued to use a non-journalling filesystem until it brought out Windows NT. Windows 95 and Windows 98 used non-journalling filesystems. Windows XP could be installed on FAT32 - non-journalling, or NTFS journalling filesystems.

[Guy]

Most people are best using a single partition. Particularly if you don't want to access it with Windows.

But again, some people choose to do it differently.

[floppy]

Hello,
I am using since few months CF and SD cards 8GB with 2 partitions: one big with ext2 (boot, extlinux on it, several distro.. and TCL on it) and one small with FAT (as exchange partition: I store there files I want to exchange with other PCs.. like windows).
I dont have /home /opt persistent on it as per advice in this forum. So, its not wearing out.
I am happy; no issue till now.
I should perhaps have made a bigger FAT partition for exchanging more files.. but no acute need till now.

[nick65go]

Hi, this is how I use my USB pen; 16GB USB stick is partitioned in 4 parts,
- first is a 4GB FAT32 for use with windows (just in case I need transfer with other PC); Windows OS is dumb, it does see only first partition on a USB drive.
- the second partition is 4GB linux ext2 formated, used for Microcore 4; It is also bootable (active flag). All grub and MC/TC files are here;
- and third partitions is 4GB ext4 (formated WITHOUT journaling, yes this can be done by choosing mkext4 parameters); Here all TC testing files live, for other TC4rc testing version; This avoid messing with TC from second partition; Also here could live other linux system in parallel with TC4;
- forth partition is for my personal data, 4GB ext4 also; it helps to keep files here but format/reinstall TC on second or third partition;

The first 4GB FAT32 is a good idea on a big 16 GB USB pen (of $13 price); And all TC versions DO NOT use backup, neither persistence folders, only custom made extension, so no write back on USB, except when I chose so for my files. Every time TC starts virgin and dies virgin; All temporary files are in RAM only. TC is nomad OS, for fun and rescue HDD/ or hack not bootable Win XP.)

Oh, USB is not used often, I use my Laptop for daily fun. TC customizations are tested on laptop, then transfered to USB for up to date.

[goossbears]

Thanks for the advice and bits of clarification; it is up to us all in the end.
Seems best for me to stick with solely one or two ext2 (non-journaling) partitions on small USB drives.
Especially with no need for Windows'-specific  files at the present time and for the short-term future.