Tiny Core Extensions > TCE Talk
mksquasfs 4.7 increased speed [20% up to 10x]
nick65go:
For TC16_x64, with better CPU (multi-core) and better disks (SSD), I think it worths to use latest mksquasfs 4.7 (increased speed) when you build big/huge TCZ (libre office, etc).
https://github.com/plougher/squashfs-tools/tree/master/Documentation/4.7
"Modern computers can have 16 cores/32 threads or more, and systems with 8-cores/16 threads are becoming standard.
What this increase in computational power means is Mksquashfs is increasingly I/O bound rather than CPU bound.
Mksquashfs by default uses four small file reader threads and four block file reader threads.
A small file reader thread only reads files which are smaller than a block in size, and a block file reader thread only reads files which are a block or larger in size.."
CentralWare:
--- Quote ---Small files are very slow to read and reading in parallel can significantly speed up Mksquashfs. The greatest speed-up seen is reading 1Kbyte files with 256 small reader threads, which is 12.22 times faster than a single reader thread (33.43 seconds vs 6 minutes 48.55 seconds).
--- End quote ---
256 threads for 1kb files
--- Quote ---Obviously the larger the files are, the less performance increase is seen reading files in parallel. But even reading 128Kbyte files in parallel can achieve a 1.4 times speed increase over a single reader thread. On this SSD drive (and ext4) there doesn't seem to be any performace gain reading files in parallel where the files are 512Kbytes or larger.
--- End quote ---
"...when you build big/huge TC (libre office, etc)."
See quote above - it doesn't sound like large files have any performance gain, regardless of their misspelling.
I have, however, added 4.7 to our build queue and it tested successfully; we'll see if the admins here take you up on your suggestion.
Rich:
Hi CentralWare
This sounds like it's more of a case of someone promoting the
hype they read and less about someone that would benefit by
any speed improvement.
I decided to take a closer look at how bad the current version
of mksquashfs is:
--- Code: ---tc@E310:~$ mksquashfs -version
mksquashfs version 4.3-git (2017/07/18)
----- Snip -----
tc@E310:~$ mksquashfs --help
SYNTAX:mksquashfs source1 source2 ... dest [options] [-e list of exclude
dirs/files]
----- Snip -----
-processors <number> Use <number> processors. By default will use number of
processors available
-mem <size> Use <size> physical memory. Currently set to 640M
Optionally a suffix of K, M or G can be given to specify
Kbytes, Mbytes or Gbytes respectively
----- Snip -----
--- End code ---
Well it does allow tuning processor count and memory use.
I decided to squash a large directory to see just how long a
wait I would be forced to endure.
This was run under TC10 X86 (32 bit) with the following hardware:
--- Code: ---tc@E310:~$ sudo inxi -CMm
Machine:
Type: Desktop System: Dell product: Dell DV051 v: N/A serial: 9Q5X2B1
Mobo: Dell model: 0JC474 serial: ..CN481115B2077P. BIOS: Dell v: A04
date: 04/04/2006
Memory:
RAM: total: 2.96 GiB used: 1.16 GiB (39.2%)
Array-1: capacity: 4 GiB note: est. slots: 2 EC: Single-bit ECC
Device-1: DIMM_1 size: 2 GiB speed: 533 MHz
Device-2: DIMM_2 size: 2 GiB speed: 533 MHz
CPU:
Topology: Single Core model: Intel Pentium 4 bits: 64 type: MT
L2 cache: 1024 KiB
Speed: 2793 MHz min/max: N/A Core speeds (MHz): 1: 2793 2: 2793
--- End code ---
This was the target:
--- Code: ---tc@E310:/mnt/sdb1/data/www$ du -ca 8.x/x86/tcz/src/
----- Snip -----
632M 8.x/x86/tcz/src/
632M total
tc@E310:/mnt/sdb1/data/www$ tree 8.x/x86/tcz/src/
----- Snip -----
108 directories, 260 files
--- End code ---
Here's the result:
--- Code: ---tc@E310:/mnt/sdb1/data/www$ time mksquashfs 8.x/x86/tcz/src/ TC8.tcz
Parallel mksquashfs: Using 2 processors
Creating 4.0 filesystem on TC8.tcz, block size 4096.
[=========================================================|] 161469/161469 100%
Exportable Squashfs 4.0 filesystem, gzip compressed, data block size 4096
compressed data, compressed metadata, compressed fragments, no xattrs
duplicates are removed
Filesystem size 642663.68 Kbytes (627.60 Mbytes)
99.51% of uncompressed filesystem size (645848.42 Kbytes)
Inode table size 29302 bytes (28.62 Kbytes)
4.46% of uncompressed inode table size (657350 bytes)
Directory table size 5163 bytes (5.04 Kbytes)
46.86% of uncompressed directory table size (11018 bytes)
Number of duplicate files found 1
Number of inodes 369
Number of files 260
Number of fragments 40
Number of symbolic links 0
Number of device nodes 0
Number of fifo nodes 0
Number of socket nodes 0
Number of directories 109
Number of ids (unique uids + gids) 2
Number of uids 1
tc (1001)
Number of gids 1
staff (50)
real 0m 54.09s
user 1m 25.28s
sys 0m 17.80s
--- End code ---
632M comprised of 108 directories containing 260 files
in 1 minute 25 seconds doesn't sound so bad.
Of course I'm no expert.
Maybe Juanito will weigh in on this. He builds more
extensions than anyone.
nick65go:
Hi, I have no personal gain to promote this or that software.
But I personally consider that kernel and squashfs (+its tools) are the most important pieces of resistance in TinyCore. And I took the info from the horse mouth (squash inventor), as a innovation in the last 2 years.[ If I remember correctly the author paid from his pocket to promote squash into kernel tree]. Similar to the pain endured by busybox (and kernel tempfs) author Rob Landley...
The main idea was that you loose (next to) nothing (in therms of TCZ size) for new version, but you MAY gain something for "nothing". TC is stable (mature enough) and I can not see many improvements soon. So for me, on the envelope, this idea sounds good to share. That's all.
Paul_123:
This is only applicable to making the extension…. Making the squashfs is peanuts in the amount of time it takes to compile and test.
In use the squashfs extension is handled by the kernel.
Navigation
[0] Message Index
[#] Next page
Go to full version