mksquasfs 4.7 increased speed [20% up to 10x]

[nick65go]

For TC16_x64, with better CPU (multi-core) and better disks (SSD), I think it worths to use latest mksquasfs 4.7 (increased speed) when you build big/huge TCZ (libre office, etc).
https://github.com/plougher/squashfs-tools/tree/master/Documentation/4.7

"Modern computers can have 16 cores/32 threads or more, and systems with 8-cores/16 threads are becoming standard.
What this increase in computational power means is Mksquashfs is increasingly I/O bound rather than CPU bound.

Mksquashfs by default uses four small file reader threads and four block file reader threads.
A small file reader thread only reads files which are smaller than a block in size, and a block file reader thread only reads files which are a block or larger in size.."

[CentralWare]

Small files are very slow to read and reading in parallel can significantly speed up Mksquashfs.  The greatest speed-up seen is reading 1Kbyte files with 256 small reader threads, which is 12.22 times faster than a single reader thread (33.43 seconds vs 6 minutes 48.55 seconds).

256 threads for 1kb files

Obviously the larger the files are, the less performance increase is seen reading files in parallel.  But even reading 128Kbyte files in parallel can achieve a 1.4 times speed increase over a single reader thread. On this SSD drive (and ext4) there doesn't seem to be any performace gain reading files in parallel where the files are 512Kbytes or larger.

"...when you build big/huge TC (libre office, etc)."
See quote above - it doesn't sound like large files have any performance gain, regardless of their misspelling.
I have, however, added 4.7 to our build queue and it tested successfully; we'll see if the admins here take you up on your suggestion.

[Rich]

Hi CentralWare
This sounds like it's more of a case of someone promoting the
hype they read and less about someone that would benefit by
any speed improvement.

I decided to take a closer look at how bad the current version
of mksquashfs is:

Code: [Select]

tc@E310:~$ mksquashfs -version
mksquashfs version 4.3-git (2017/07/18)
 ----- Snip -----
tc@E310:~$ mksquashfs --help
SYNTAX:mksquashfs source1 source2 ...  dest [options] [-e list of exclude
dirs/files]
 ----- Snip -----
-processors <number>    Use <number> processors.  By default will use number of
                        processors available
-mem <size>             Use <size> physical memory.  Currently set to 640M
                        Optionally a suffix of K, M or G can be given to specify
                        Kbytes, Mbytes or Gbytes respectively
 ----- Snip -----Well it does allow tuning processor count and memory use.

I decided to squash a large directory to see just how long a
wait I would be forced to endure.

This was run under TC10 X86 (32 bit) with the following hardware:

Code: [Select]

tc@E310:~$ sudo inxi -CMm
Machine:
  Type: Desktop System: Dell product: Dell DV051 v: N/A serial: 9Q5X2B1
  Mobo: Dell model: 0JC474 serial: ..CN481115B2077P. BIOS: Dell v: A04
  date: 04/04/2006
Memory:
  RAM: total: 2.96 GiB used: 1.16 GiB (39.2%)
  Array-1: capacity: 4 GiB note: est. slots: 2 EC: Single-bit ECC
  Device-1: DIMM_1 size: 2 GiB speed: 533 MHz
  Device-2: DIMM_2 size: 2 GiB speed: 533 MHz
CPU:
  Topology: Single Core model: Intel Pentium 4 bits: 64 type: MT
  L2 cache: 1024 KiB
  Speed: 2793 MHz min/max: N/A Core speeds (MHz): 1: 2793 2: 2793
This was the target:

Code: [Select]

tc@E310:/mnt/sdb1/data/www$ du -ca 8.x/x86/tcz/src/
 ----- Snip -----
632M    8.x/x86/tcz/src/
632M    total
tc@E310:/mnt/sdb1/data/www$ tree 8.x/x86/tcz/src/
 ----- Snip -----
108 directories, 260 files
Here's the result:

Code: [Select]

tc@E310:/mnt/sdb1/data/www$ time mksquashfs  8.x/x86/tcz/src/ TC8.tcz
Parallel mksquashfs: Using 2 processors
Creating 4.0 filesystem on TC8.tcz, block size 4096.
[=========================================================|] 161469/161469 100%

Exportable Squashfs 4.0 filesystem, gzip compressed, data block size 4096
        compressed data, compressed metadata, compressed fragments, no xattrs
        duplicates are removed
Filesystem size 642663.68 Kbytes (627.60 Mbytes)
        99.51% of uncompressed filesystem size (645848.42 Kbytes)
Inode table size 29302 bytes (28.62 Kbytes)
        4.46% of uncompressed inode table size (657350 bytes)
Directory table size 5163 bytes (5.04 Kbytes)
        46.86% of uncompressed directory table size (11018 bytes)
Number of duplicate files found 1
Number of inodes 369
Number of files 260
Number of fragments 40
Number of symbolic links  0
Number of device nodes 0
Number of fifo nodes 0
Number of socket nodes 0
Number of directories 109
Number of ids (unique uids + gids) 2
Number of uids 1
        tc (1001)
Number of gids 1
        staff (50)
real    0m 54.09s
user    1m 25.28s
sys     0m 17.80s
632M comprised of 108 directories containing 260 files
in 1 minute 25 seconds doesn't sound so bad.

Of course I'm no expert.

Maybe Juanito will weigh in on this. He builds more
extensions than anyone.

[nick65go]

Hi, I have no personal gain to promote this or that software.

But I personally consider that kernel and squashfs (+its tools) are the most important pieces of resistance in TinyCore. And I took the info from the horse mouth (squash inventor), as a innovation in the last 2 years.[ If I remember correctly the author paid from his pocket to promote squash into kernel tree]. Similar to the pain endured by busybox (and kernel tempfs) author Rob Landley...

The main idea was that you loose (next to) nothing (in therms of TCZ size) for new version, but you MAY gain something for "nothing". TC is stable (mature enough) and I can not see many improvements soon. So for me, on the envelope, this idea sounds good to share. That's all.

[Paul_123]

This is only applicable to making the extension….  Making the squashfs is peanuts in the amount of time it takes to compile and test.

In use the squashfs extension is handled by the kernel.

[CentralWare]

Good evening, @Everyone!

mksquashfs v4.3 is in @Rich's example using a dual core (assuming two thread) processor...
x86 assumes 3GB~4GB ceiling, but there's no specs for storage which may also factor into the mix.

v4.6.1 has been in play since TCL 14.x and continues onto 16.x
v4.7 (including git download, repackaging as tar.gz, extracting to ./src and compiling) takes a whole 2.95 seconds on the machine I'm compiling with at the moment once all the z-compression dependencies were located.

I repeated the test using 8.x/x86/tcz/src as our source

Code: [Select]

10.51s using 2 cores, <3s using 8 cores, <2s open throttle w/ 40 threads ~ 12 in use during trial
...compiling 4.6.1...
10.64s using 2 cores, <3s using 8 cores...  yeah, I'm not seeing any real differences so far...
...compiling 4.5.1...
0.24s slower using 2 cores...  virtually same using 8 cores...

Now, when you adjust memory (-mem 3072M) things get a little more interesting
v4.7    2 cores: 10.43s
v4.6.1 2 cores: 10.41s
v4.5.1 2 cores: 13.94s
But that's still 630MB being managed, many of the files found in tcz/src are already archived so results may vary.

As @Paul_123 notes, this only applies to creating an archive/extension... it does not affect efficiency when loading the TCZ

[Rich]

Hi CentralWare

... but there's no specs for storage which may also factor into the mix. ...

I take it you are referring to my hard drives.

Both are WD5003ABYX  500GB  7200RPM  64MB cache  SATA 3.0Gb/sec

/mnt/sdb1/  which was the target had 80 Gbytes out of 192GBytes free.
/mnt/sda1/home/tc  had 3.8 Gbytes out of 46GBytes free.

[Juanito]

Maybe Juanito will weigh in on this. He builds more
extensions than anyone.

Not much to add except that this is probably like zstd in that the version in the kernel is behind that used separately?