dCore-jessie f2fs remaster script, bugfixes

[AdM_dNess]

Hi everyone!

Bit of background:
   I was until very recently a dedicated PuppyLinux fan, running Quirky April as a hobbyst-level user, having it running most of the time on my CX2720 Gateway laptop, off a half-split USB stick with F2FS second partition.  Very pleased with the experience, especially the initial setup, locale keboard and all, aside from touchpad tap not on by default and out-of-the-box Wifi utils not compatible with WEP 128bit/restricted ("frisbee" is the way to go on this platform). Even working langpacks are available; had to say it, it's so great.

    I first used DSL some years back, then I used TC for a while and followed its development from afar, got on the PuppyLinux and then Quirky bandwagons, then got the urge to try dCore. I tried dCore-jessie over the past few days, made it work so-so with F2FS and along the way found couple annoyances/bugs. It was a great re-learning experience

Before migrating to ub-dCore-Vivid, I thought I'd share what i accomplished so far, feel free to rate, comment and correct my thinking where I'm mistaken.


• Missing f2fs.ko.gz kernel module in the 3.16.6 kernel config, had to compile it separately, thanks Internet for how to compile modules only and tinycore for providing sources. (Should be backported from latest drivers...?)
• Bug in i915 drm driver (warningwhen external monitor plugged in (EDID checksum failing) , it's fixed since then, but not in 3.16.6.]
• util-linux.sce doesn't include dependencies libuuid1, libsmartcols1 and libblkid1 needed for fdisk and blkid, I used the -dev ones when available; blkid -k doesn't recognize F2FS, so had to recompile fdisk and blkid static from scratch
• Xorg has a broken 40-generic.conf that generates warnings, that also lie for no reason in /etc. Perhaps the .deb2sce script can arrange that: http://cromwell-intl.com/linux/font-config-warnings.html
• tc-config still thinks loadsce exists, twice Included a modprobe f2fs and adjusted checkfs for f2fs check, dosfsck could be included from .sce, didn't change since 2014; (DONE just before posting, not tested )
• Better default behavior: deb2sce didn't delete the leftover orphan $TARGET.sce.dep after an aborted interactive "sce-import -d" run (fixed)
• Long time for swap partitions finding, really long load time before loading extensions, like previously mentionned by jls-legalize, after that a "${NORMAL} /" on a line by istelf then extensions load messages in yellow. Probable cause autoscan script with the outdated f2fs driver, maybe linked to next item:
• During boot, buffer i/o errors on sr0 sector 0 block 0, from a blank DVD-R

Here's an .sce.dep orphan file cleanup snippet,
for a sce-clean maybe, or a cleanup() addition:

Code: [Select]

cd /etc/sysconfig/tcedir/sce;
if ls *.sce.dep > /dev/null 2>&1; then
for D in $(find -name "*.sce.dep"); do
[ -f "${D%.dep}" ] || rm -v "$D"
done
fi

Remaster-f2fs-dCore-jessie
    During the last few days, wrote a simple script that unpacks in /tmp the original dCore-jessie.gz initrd, copies it into /tmp/bootgz and the to-be-repacked /tmp/bootgzz, then pauses for manual edits, then proceeds to git clone in /tmp util-linux and f2fs-tools, (/opt/src could be better), static-build fdisk, blkid, mkfs.f2fs and fsck.f2fs, copy them along with dosfsck from dosfsck.sce and the modified tc-config and deb2sce scripts to /tmp/bootgzz, then pauses, then repacks. It does work, albeit couple shortcomings.

Attachement
The archive, meant to be extracted in "/home/tc", (different home dir: modifiy script accordingly) contains my first take at a remastering shell script, which contains lots of TODOs and sudos, you've been warned, and the minimum needed files taken from my dCore-jessie-dev directory: the modified deb2sce and tc-config files and the needed kernel module f2fs.ko.gz, residing in "/home/tc/dCore-jessie-dev/modules/3.16.6-tinycore/kernel/fs/f2fs". I also included the modified 40-generic.conf file to patch a running system. did a fc-cache -f afterward, don't know if truly necessary, but the warnings disappeared. Edit the script once, in order to at least set your paths right and it should work. Only one problem remaining, before and after directory size reporting, gives a "numfmt: invalid number: ‘\033[1;31m0’" error.

Took me *days* to write this script. I'm posting this after an all-nighter... Hence the username
At least I'm having fun doing it! =) Taking my leave to try ub-dCore-Vivid...  and the T2 SDE running on it =D

[curaga]

f2fs was not stable in the kernel we use, its use is not recommended. Likely will change with newer kernels.

[Jason W]

AdM_dNess, thanks for your investigating and reporting of issues you have found, today I will look into the things you have mentioned.   And maybe f2fs will be in our next kernel upgrade.

[Jason W]

Added to the dependency list of util-linux:  libuuid1, libsmartcols1 and libblkid1.

The broken 40-generic.conf has been removed from /etc (forget how it got there) and it was in Xtc that the file exists in it's normal directory.  Deprecated and does not exist in Debian/Ubuntu so now removed.  JLS had mentioned the errors and that file, I just did not find the solution at the time.

Changed the remaining references to loadsce to sce-load in tc.config.

The I/O errors when a blank or audio cd is in the tray during boot is common across dCore and Core6.  It seems to be udev related.

If a script is aborted before completion, it obviously does not do all that was intended.  And keeping dep and list files is an option of sce-remove.  But I could make a sce-clean type of utility that would remove any orphaned md5.txt, .lst, and .dep files in the case one wants to just clean his sce directory.

Xtc has been updated, sce-update will fetch the update.  And x86 dCore base images have been updated in release and release_candidates.

[Jason W]

Oh, I just saw your 40-generic.conf file.  I will add that to Xtc as that is the fix mentioned on the site referenced.

[AdM_dNess]

Hi Jason, Curaga,

   thanks for the prompt responses.

Forgot to mention earlier that since there's no /home directory in Quirky, its especially compatible with dCore/Tinycore's persistent home & opt approach.

I just played with ub-dCore-vivid, some questions emerged, perhaps they're answered elsewhere on the forum already, sittc:

• Thought at first that the ub- prefix was about using the original Ubuntu kernel, got me excited, but ub-dCore-vivid doesn't use the official release 3.19.05 kernel... Most probable answer: it wouldn't be Tinycore anymore. Tried using it with the provided ub-dCore-vivid.gz, it failed miserably, as expected. Perhaps dCore could migrate to sce-* tools + .deb2sce glue code repository for every debian repository, with a solid remastering script included 
• With the current one kernel approach, why not unify all dCore variants inside a single initrd.gz with a choice of repo to be made when fresh starting , with the established possibility of embedding multi-kernel compatibilty. Clearly, I don't get some of the complexity involved.

For the sake of completeness,  here's a small fix to the checkfs() function in the "/etc/init.d/tc-config" I provided earlier, reformating just before posting  (*sigh*) led me to forget three closing square braces.

Code: [Select]

checkfs() {
FS=/dev/"$1"
if ! grep "^$FS$" /tmp/checkfs > /dev/null 2>&1; then
      if grep "$FS" /etc/mtab > /dev/null 2>&1; then
:
      else
if [ grep "$FS" /etc/fstab | grep ext[2,3,4] ]> /dev/null 2>&1 ]; then
  e2fsck -n "$FS" > /tmp/checkfs."$1" 2>&1
  checkfs_prompt "$?" "$2"
elif [ grep "$FS" /etc/fstab | grep f2fs ] > /dev/null 2>&1 ]; then
  fsck.f2fs -f "$FS" > /tmp/checkfs."$1" 2>&1
  checkfs_prompt "$?" "$2"
elif [ grep "$FS" /etc/fstab | grep vfat ] > /dev/null 2>&1 ]; then
  dosfsck -V "$FS" > /tmp/checkfs."$1" 2>&1
  checkfs_prompt "$?" "$2"
else
  echo "Filesystem is not ext[2,3,4] nor f2fs nor vfat and cannot be checked." > /tmp/checkfs."$1"
fi
    fi
  fi
}


[AdM_dNess]

Hi everyone,

   quick status update: kept reading on the forum and on the wiki about dCore, found http://git.tinycorelinux.net, most notably the dCore scripts, but also tinyx revived, the C tools source code, and the kernel module sorter script.

Most notably I've also found http://shellcheck.net, found information about trapping signals such as err and ctrl-c signals, also found out about the rationale behind the symlinks of fsck "$(ls --color -ls /sbin/fsck.*)" and thus stumbled upon a non-negligible amount of shame about posting what I did earlier, main script and also code in earlier post

Thus I'm heading toward v0.2 from included road map, to at least obtain a better working version of this script, thinking about working with pretce=RAM, let say as first example: dosfstools.sce inside /tmp/tce, then packed as a cat-concatenated cpio "addon" (eager to speed test), also learning about 0install technology, as specified in the roadmap. Should provide at the end a grub4dos compatible minimum menu entry

I'm still concerned about long boot times (delays after setting home and opt on f2fs, long delay between /opt set line and swap partition status line, even with noswap, and longer delay before loading extensions.
Can i investigate on a live system, another virtual terminal, using the multivt boot code...?

Thanks again for your time, feel free to comment or propose a better course of action.