Sugestion: XZ compression in kernel + tcz; squashfs with xz + biger block size

[andyj]

Dictionaries are good for words, which is why text files compress so much better than binaries. Since the majority of extensions are stripped binaries, compression might be working against us except when the binaries have lots of embedded strings.

[andyj]

Since I have 103 extension trees on hand I decided to try a few tests on 64-bit with 2 processors. The input:

Code: [Select]

apache2.4/TCZ
apache2.4/TCZ-dev
apache2.4/TCZ-doc
apr-util/TCZ
apr-util/TCZ-dev
apr/TCZ
apr/TCZ-dev
bind/TCZ
bind/TCZ-dev
bind/TCZ-doc
cyrus-sasl-lite/TCZ
cyrus-sasl-lite/TCZ-dev
cyrus-sasl/TCZ
cyrus-sasl/TCZ-dev
dhcp/TCZ
dhcp/TCZ-dev
google-earth-pro/TCZ
gvim/TCZ-base
gvim/TCZ-doc
gvim/TCZ-gvim
gvim/TCZ-tutor
gvim/TCZ-vim
iftop/TCZ
jemalloc/TCZ
jemalloc/TCZ-dev
libaio/TCZ
libaio/TCZ-dev
libdnet/TCZ
libdnet/TCZ-dev
libestr/TCZ
libestr/TCZ-dev
libevent/TCZ
libevent/TCZ-dev
libfastjson/TCZ
libfastjson/TCZ-dev
libgd/TCZ
libgd/TCZ-dev
liblogging/TCZ
liblogging/TCZ-dev
liblognorm/TCZ
liblognorm/TCZ-dev
libnet/TCZ
libnet/TCZ-dev
libnghttp2/TCZ
libnghttp2/TCZ-dev
libproxy/TCZ
libsodium/TCZ
libsodium/TCZ-dev
libtidy/TCZ
libtidy/TCZ-dev
libwebp-nox/TCZ
libwebp-nox/TCZ-dev
libzip/TCZ
libzip/TCZ-dev
lighttpd/TCZ
net-snmp/TCZ
net-snmp/TCZ-dev
nginx/TCZ
open-vm-tools/TCZ
open-vm-tools/TCZ-desktop
open-vm-tools/TCZ-dev
openldap/TCZ
openldap/TCZ-dev
openldap/TCZ-doc
pgtclng/TCZ
php7/TCZ-cgi
php7/TCZ-cli
php7/TCZ-dev
php7/TCZ-ext
php7/TCZ-fpm
php7/TCZ-lsp
php7/TCZ-mod
postgresql-10/TCZ
postgresql-10/TCZ-client
postgresql-10/TCZ-dev
postgresql-11/TCZ
postgresql-11/TCZ-client
postgresql-11/TCZ-dev
postgresql-9.5/TCZ
postgresql-9.5/TCZ-client
postgresql-9.5/TCZ-dev
postgresql-9.6/TCZ
postgresql-9.6/TCZ-client
postgresql-9.6/TCZ-dev
recode/TCZ
recode/TCZ-dev
rsyslog/TCZ
rxvt/TCZ
rxvt/TCZ-doc
tcl8.6/TCZ
tcl8.6/TCZ-dev
tcl8.6/TCZ-doc
tcllib/TCZ
tcllib/TCZ-doc
tcltls/TCZ
tcludp/TCZ
tk8.6/TCZ
tk8.6/TCZ-dev
tk8.6/TCZ-doc
unixODBC/TCZ
unixODBC/TCZ-dev
xt_geoip_LE_IPv4/TCZ
xtables-addons-4.19.10-tinycore64/TCZ

I compressed them all using defaults, the only thing I set was -comp.
The results, decompressing all of them 10 times:

Code: [Select]

Compression       Total size     Decomp Time
gzip                  296 MB           123 s
lzo                   337 MB            87 s
xz                    285 MB           185 s
zstd                  285 MB            98 s

The winner seems to be zstd. How it would be on 32-bit or different number of processors is left as an exercise for the students.

[andyj]

32-bit, 2 processors (same VM as 64-bit results, but booted into 32-bit kernel):

Code: [Select]

Compression       Total size     Decomp Time
gzip                  197 MB           131 s
lzo                   233 MB           136 s
xz                    192 MB           150 s
zstd                  192 MB           126 s

The winner again is zstd. Interesting that the 32-bit files are about 30% smaller than their 64-bit counterparts.

[Rich]

Hi andyj

Dictionaries are good for words, which is why text files compress so much better than binaries. Since the majority of extensions are stripped binaries, compression might be working against us except when the binaries have lots of embedded strings.

Even a binary with minimal strings and stripped with  sstrip  shows decent compression after being processed with mksquashfs:

Code: [Select]

tc@box:~/guilib/tinyclock$ ls -l
-rwxr-xr-x 1 tc staff  6693 Aug  6  2012 tinyclock
-rw-r--r-- 1 tc staff  4096 May 18 00:06 tinyclock.tcz
tc@box:~/guilib/tinyclock$
The reason is lots of zeros and repeating sequences:

Code: [Select]

tc@box:~/guilib/tinyclock$ od -vt x4 tinyclock
0000000 464c457f 00010101 00000000 00000000
0000020 00030002 00000001 08049238 00000034
0000040 00000000 00000000 00200034 00280007
0000060 00000000 00000006 00000034 08048034
0000100 08048034 000000e0 000000e0 00000005
0000120 00000004 00000003 00000114 08048114
0000140 08048114 00000013 00000013 00000004
0000160 00000001 00000001 00000000 08048000
0000200 08048000 0000185c 0000185c 00000005
0000220 00001000 00000001 0000185c 0804a85c
0000240 0804a85c 000001c9 000001f8 00000006
0000260 00001000 00000002 00001870 0804a870
0000300 0804a870 000000d0 000000d0 00000006
0000320 00000004 00000004 00000128 08048128
0000340 08048128 00000020 00000020 00000004
0000360 00000004 6474e551 00000000 00000000
0000400 00000000 00000000 00000000 00000006
0000420 00000004 62696c2f 2d646c2f 756e696c
0000440 6f732e78 0000322e 00000004 00000010
0000460 00000001 00554e47 00000000 00000002
0000500 00000006 00000016 00000025 00000032
0000520 0000001a 00000008 0000001c 00000028
0000540 0000000a 00000000 00000027 00000014
0000560 00000007 00000000 00000022 00000000
0000600 00000017 00000000 00000020 0000001b
0000620 00000000 00000000 0000002f 00000031
0000640 0000002b 00000021 0000002e 00000026
0000660 00000000 0000000d 0000000e 00000015
0000700 00000000 00000030 00000024 0000002a
0000720 0000002c 00000023 00000000 00000029
0000740 00000000 00000000 00000000 00000000
0000760 00000000 00000000 00000000 00000000
0001000 00000000 00000000 00000000 00000000
0001020 00000000 00000002 00000000 00000005
0001040 00000000 00000000 0000000b 00000000
0001060 00000003 00000004 00000006 00000000
0001100 00000000 00000000 00000000 00000000
0001120 00000018 00000000 00000010 00000016
0001140 00000000 0000001d 0000001e 0000000f
0001160 00000000 00000000 00000011 00000009
0001200 00000001 00000013 00000025 00000019
0001220 0000001f 00000012 00000000 0000002d
0001240 00000000 00000000 0000000c 00000000
0001260 00000000 00000000 00000000 0000010c
0001300 00000000 00000000 00000012 0000014f
0001320 00000000 00000000 00000012 0000015d
0001340 00000000 00000000 00000012 000000ff
0001360 00000000 00000000 00000012 000000d7
0001400 00000000 00000000 00000012 000001f4
0001420 00000000 00000000 00000012 0000000d
0001440 00000000 00000000 00000020 0000001c
0001460 00000000 00000000 00000020 0000016c
0001500 00000000 00000000 00000012 000001cb
0001520 00000000 00000000 00000012 00000056
0001540 00000000 00000000 00000012 000001a2
0001560 00000000 00000000 00000012 00000030
0001600 08049730 00000000 000d0012 0000006f
0001620 00000000 00000000 00000012 0000017a
0001640 00000000 00000000 00000012 000001dd
0001660 00000000 00000000 00000012 0000003c
0001700 00000000 00000000 00000012 0000008f
0001720 00000000 00000000 00000012 00000078
0001740 00000000 00000000 00000012 00000047
0001760 00000000 00000000 00000012 000001b7
0002000 00000000 00000000 00000012 00000134
0002020 08049750 00000004 000e0011 000001be
0002040 00000000 00000000 00000012 00000173
0002060 00000000 00000000 00000012 000001ef
0002100 00000000 00000000 00000012 000001a9
0002120 00000000 00000000 00000012 0000018d
0002140 00000000 00000000 00000012 000000ed
0002160 00000000 00000000 00000012 00000157
0002200 00000000 00000000 00000012 00000143
0002220 00000000 00000000 00000012 00000187
0002240 00000000 00000000 00000012 000000bf
0002260 00000000 00000000 00000012 00000085
0002300 00000000 00000000 00000012 000000ca
0002320 00000000 00000000 00000012 000001d5
0002340 00000000 00000000 00000012 00000180
0002360 00000000 00000000 00000012 00000203
0002400 0804aa28 00000000 fff10010 00000194
0002420 00000000 00000000 00000012 000000e2
0002440 00000000 00000000 00000012 00000060
0002460 00000000 00000000 00000012 0000019b
0002500 00000000 00000000 00000012 0000020f
0002520 0804aa54 00000000 fff10010 000000a3
0002540 00000000 00000000 00000012 000000af
0002560 00000000 00000000 00000012 00000118
0002600 00000000 00000000 00000012 00000163
0002620 00000000 00000000 00000012 000001fc
0002640 0804aa28 00000000 fff10010 0000014a
0002660 00000000 00000000 00000012 00000036
0002700 080489fc 00000000 000a0012 62696c00
0002720 2e313158 362e6f73 675f5f00 5f6e6f6d
0002740 72617473 005f5f74 5f764a5f 69676552
0002760 72657473 73616c43 00736573 6e69665f
0003000 695f0069 0074696e 73694c58 6e6f4674
0003020 58007374 65657246 746e6f46 656d614e
0003040 46580073 46656572 00746e6f 616f4c58
0003060 65755164 6f467972 5800746e 46746553
0003100 00746e6f 65704f58 7369446e 79616c70
0003120 72435800 65746165 58004347 61657243
0003140 69536574 656c706d 646e6957 5800776f
0003160 65746e49 74416e72 58006d6f 6e616843
0003200 72506567 7265706f 58007974 5770614d
0003220 6f646e69 53580077 63656c65 706e4974
0003240 58007475 726f7453 6d614e65 54580065
0003260 57747865 68746469 6f4d5800 65526576
0003300 657a6973 646e6957 5800776f 61656c43
0003320 6e695772 00776f64 61724458 72745377
0003340 00676e69 65684358 69576b63 776f646e
0003360 6e657645 696c0074 732e6362 00362e6f
0003400 5f4f495f 69647473 73755f6e 73006465
0003420 70637274 78650079 73007469 6e697270
0003440 66006674 6e65706f 646b6d00 73007269
0003460 74667274 00656d69 69776572 7300646e
0003500 6f747274 6766006c 00737465 73727473
0003520 63007274 72696468 6c636600 0065736f
0003540 6c6c616d 7300636f 61637274 65670074
0003560 766e6574 635f5f00 65707974 6c5f625f
0003600 7500636f 65656c73 65670070 6d697474
0003620 64666f65 6c007961 6c61636f 656d6974
0003640 72706600 66746e69 6c5f5f00 5f636269
0003660 72617473 616d5f74 66006e69 00656572
0003700 73785f5f 00746174 6164655f 5f006174
0003720 7373625f 6174735f 5f007472 00646e65
0003740 42494c47 2e325f43 4c470031 5f434249
0003760 00332e32 42494c47 2e325f43 00000030
0004000 00020000 00000002 00020000 00000000
0004020 00020002 00020000 00000001 00020002
0004040 00000000 00000000 00010002 00020002
0004060 00030002 00000004 00020004 00000002
0004100 00000000 00020002 00020001 00000000
0004120 00010002 00000000 00020000 00020001
0004140 00000001 00030001 0000012a 00000010
0004160 00000000 0d696911 00040000 00000214
0004200 00000010 0d696913 00030000 0000021e
0004220 00000010 0d696910 00020000 00000228
0004240 00000000 0804a940 00000706 0804a950
0004260 00000107 0804a954 00000207 0804a958
0004300 00000307 0804a95c 00000407 0804a960
0004320 00000507 0804a964 00000607 0804a968
0004340 00000707 0804a96c 00000907 0804a970
0004360 00000a07 0804a974 00000b07 0804a978
0004400 00000c07 0804a97c 00000e07 0804a980
0004420 00000f07 0804a984 00001007 0804a988
0004440 00001107 0804a98c 00001207 0804a990
0004460 00001307 0804a994 00001407 0804a998
0004500 00001507 0804a99c 00001707 0804a9a0
0004520 00001807 0804a9a4 00001907 0804a9a8
0004540 00001a07 0804a9ac 00001b07 0804a9b0
0004560 00001c07 0804a9b4 00001d07 0804a9b8
0004600 00001e07 0804a9bc 00001f07 0804a9c0
0004620 00002007 0804a9c4 00002107 0804a9c8
0004640 00002207 0804a9cc 00002307 0804a9d0
0004660 00002407 0804a9d4 00002607 0804a9d8
0004700 00002707 0804a9dc 00002807 0804a9e0
0004720 00002907 0804a9e4 00002b07 0804a9e8
0004740 00002c07 0804a9ec 00002d07 0804a9f0
0004760 00002e07 0804a9f4 00003007 53e58955
0005000 0000e851 815b0000 001f3ec3 fcbb8300
0005020 00ffffff 81e80574 e8000000 0000087f
0005040 000cefe8 5d5b5a00 000000c3 a94835ff
0005060 25ff0804 0804a94c 00000000 a95025ff
0005100 00680804 e9000000 ffffffe0 a95425ff
0005120 08680804 e9000000 ffffffd0 a95825ff
0005140 10680804 e9000000 ffffffc0 a95c25ff
0005160 18680804 e9000000 ffffffb0 a96025ff
0005200 20680804 e9000000 ffffffa0 a96425ff
0005220 28680804 e9000000 ffffff90 a96825ff
0005240 30680804 e9000000 ffffff80 a96c25ff
0005260 38680804 e9000000 ffffff70 a97025ff
0005300 40680804 e9000000 ffffff60 a97425ff
0005320 48680804 e9000000 ffffff50 a97825ff
0005340 50680804 e9000000 ffffff40 a97c25ff
0005360 58680804 e9000000 ffffff30 a98025ff
0005400 60680804 e9000000 ffffff20 a98425ff
0005420 68680804 e9000000 ffffff10 a98825ff
0005440 70680804 e9000000 ffffff00 a98c25ff
0005460 78680804 e9000000 fffffef0 a99025ff
0005500 80680804 e9000000 fffffee0 a99425ff
0005520 88680804 e9000000 fffffed0 a99825ff
0005540 90680804 e9000000 fffffec0 a99c25ff
0005560 98680804 e9000000 fffffeb0 a9a025ff
0005600 a0680804 e9000000 fffffea0 a9a425ff
0005620 a8680804 e9000000 fffffe90 a9a825ff
0005640 b0680804 e9000000 fffffe80 a9ac25ff
0005660 b8680804 e9000000 fffffe70 a9b025ff
0005700 c0680804 e9000000 fffffe60 a9b425ff
0005720 c8680804 e9000000 fffffe50 a9b825ff
0005740 d0680804 e9000000 fffffe40 a9bc25ff
0005760 d8680804 e9000000 fffffe30 a9c025ff
0006000 e0680804 e9000000 fffffe20 a9c425ff
0006020 e8680804 e9000000 fffffe10 a9c825ff
0006040 f0680804 e9000000 fffffe00 a9cc25ff
0006060 f8680804 e9000000 fffffdf0 a9d025ff
0006100 00680804 e9000001 fffffde0 a9d425ff
0006120 08680804 e9000001 fffffdd0 a9d825ff
0006140 10680804 e9000001 fffffdc0 a9dc25ff
0006160 18680804 e9000001 fffffdb0 a9e025ff
0006200 20680804 e9000001 fffffda0 a9e425ff
0006220 28680804 e9000001 fffffd90 a9e825ff
0006240 30680804 e9000001 fffffd80 a9ec25ff
0006260 38680804 e9000001 fffffd70 a9f025ff
0006300 40680804 e9000001 fffffd60 a9f425ff
0006320 48680804 e9000001 fffffd50 04244c8d
0006340 fff0e483 8955fc71 be5657e5 080497fe
0006360 0bb95153 81000000 000324ec c47d8d00
0006400 7d8da4f3 9809becf 09b10804 006aa4f3
0006420 fffe27e8 10c483ff c189c085 04aa48a3
0006440 6b7a7408 00008490 90035000 0000008c
0006460 8b0c428b 728b107a 085a8b14 a318528b
0006500 0804aa50 00fec069 3d890000 0804aa40
0006520 00feff69 006a0000 5153006a fcf49589
0006540 148dffff 0af66bb6 8999d001 04aa341d
0006560 6bfef708 fffcf4b5 38a30aff 6b0804aa
0006600 fffcf485 f80105ff a3fef799 0804aa44
0006620 fffe77e8 10c483ff 04aa4ca3 fc9d8d08
0006640 89fffffc 0597e8d8 bd8d0000 fffffdfc
0006660 310cec83 0005b9c0 abf30000 ff5cb58d
0006700 de68ffff 6a00b0c4 6a006a00 6a056a05
0006720 ff056a05 04aa3435 4835ff08 e80804aa
0006740 fffffd48 6a2cc483 97dc6800 35ff0804
0006760 0804aa48 fdfc85c7 0003ffff 3ca30000
0007000 c70804aa fffe0085 000000ff 0485c700
0007020 02fffffe e8000000 fffffe70 fdfc958d
0007040 056affff 6a006a52 ff505020 04aa3c35
0007060 4835ff08 e80804aa fffffe60 ff28c483
0007100 04aa3c35 4835ff08 e80804aa fffffdac
0007120 680cc483 00020030 aa3c35ff 35ff0804
0007140 0804aa48 fffdb3e8 0cc483ff 04984168
0007160 3c35ff08 ff0804aa 04aa4835 fbfae808
0007200 5a58ffff db315356 000863e8 10c48300
0007220 fce085c7 0000ffff 85c70000 fffffcec
0007240 00000000 006a5757 50d8458d fffcbbe8
0007260 d8458bff 3910c483 fffcec85 f48589ff
0007300 89fffffc 840fe045 0000032c 858d5656
0007320 ffffff5c fcb58d50 56fffffc 00080fe8
0007340 9c458b00 3910c483 fffce085 f08589ff
0007360 0ffffffc 0000af84 e8f08900 00000441
0007400 83cf458d 04aa1c3d 85891808 fffffce8
0007420 458d0974 e88589c4 83fffffc 458d0cec
0007440 95e850e0 8dfffffb f789b475 e8b5ff50
0007460 6afffffc 81e85610 8bfffffd 04aa3015
0007500 ffc98308 0f1cc483 893e42bf fffce485
0007520 42bf0fff e4850340 89fffffc fffcec85
0007540 f2c031ff 49d1f7ae e8525651 fffffcec
0007560 b5ff595a fffffcec 5006c083 aa2035ff
0007600 35ff0804 0804aa24 aa3c35ff 35ff0804
0007620 0804aa48 fffc23e8 f0858bff 83fffffc
0007640 858920c4 fffffce0 8d0cec83 e850e045
0007660 fffffb08 89b4758d b5ff50f7 fffffce8
0007700 e856106a fffffcf4 ff18c483 04aa3c35
0007720 4835ff08 e80804aa fffffa90 c983c031
0007740 83aef2ff d1f70cc4 ff565149 fffce4b5
0007760 ff006aff 04aa4c35 3c35ff08 ff0804aa
0010000 04aa4835 fa32e808 858bffff fffffcf4
0010020 8920c483 fffcec85 01dae9ff 858b0000
0010040 fffffefc 0f07f883 0001cb8c 08f88300
0010060 f883367e bd850f16 83000001 ffff04bd
0010100 840f00ff 000001b0 ff14858b 01bbffff
0010120 a3000000 0804aa24 ff18858b 20a3ffff
0010140 e90804aa 00000190 840fdb85 00000188
0010160 59685050 8d080497 fffcfc85 49e850ff
0010200 83fffffb c08510c4 2675c389 000167e9
0010220 f085ff00 31fffffc ffc983c0 aef2f789
0010240 8d39d1f7 fffffcf4 8d89227d fffffcf4
0010260 85c71aeb fffffcf4 00000000 fdfcb58d
0010300 85c7ffff fffffcf0 00000000 00685357
0010320 56000001 fffa23e8 10c483ff b175c085
0010340 fcf4858b ec83ffff 85af0f0c fffffcf0
0010360 fb56e850 c483ffff 89c08510 830e75c7
0010400 e8530cec fffffaa4 0000e6e9 0cec8300
0010420 e853fe89 fffff994 5110c483 f4b5ff53
0010440 56fffffc fff9d3e8 10c483ff fcf4b503
0010460 c085ffff ec83e575 6de8530c 58fffffa
0010500 fc858d5a 68fffffc 0804975b fa7ae850
0010520 c483ffff 89c08510 8b1f74c3 fffcf085
0010540 85af0fff fffffcf4 fcf085c7 0000ffff
0010560 85890000 fffffcdc ec835beb 8b6eeb0c
0010600 fffcf0b5 e4558dff f089fe01 000137e8
0010620 02f88300 f8830774 eb1e7503 35ff500e
0010640 0804aa24 0497ec68 5015eb08 aa2035ff
0010660 f5680804 eb080497 68565007 080497c4
0010700 fa66e853 858bffff fffffcf4 0110c483
0010720 fffcf085 dc858bff 39fffffc fffcf085
0010740 839c7cff e8530cec fffff9c0 99e8575e
0010760 83fffff9 db3110c4 fefc858d 6850ffff
0011000 00ffffff aa3c35ff 35ff0804 0804aa48
0011020 fffa97e8 10c483ff 850fc085 fffffdfe
0011040 680cec83 0003d090 fff92fe8 10c483ff
0011060 fffc6fe9 909090ff 895eed31 f0e483e1
0011100 68525450 080496e7 04969868 68565108
0011120 08048cdc fff8b3e8 9090f4ff 53e58955
0011140 283d8052 000804aa 68b83175 2d0804a8
0011160 0804a864 8d02f8c1 0eebff58 2c158942
0011200 ff0804aa a8649514 158b0804 0804aa2c
0011220 e872da39 aa2805c6 58010804 55c35d5b
0011240 ec83e589 6c3d8308 000804a8 00b81674
0011260 85000000 830d74c0 6c680cec ff0804a8
0011300 10c483d0 9090c3c9 ffc98355 5657e589
0011320 83c38953 c0311cec 89e45589 83aef2df
0011340 d1f7ffc8 744aca89 137c8050 08750aff
0011360 c6fe518d 00fe0b44 2eebf631 53575050
0011400 e8e05589 fffff934 8b10c483 d839e055
0011420 c0311775 f2ffc983 e4458bae 3949d1f7
0011440 890889ca eb127ff0 3c8b460d 04aa00b5
0011460 003f8008 c883c675 f4658dff 5d5f5e5b
0011500 e58955c3 c6895657 38ec8153 68000001
0011520 08049754 01e045c7 e8000000 fffff77c
0011540 8510c483 9e840fc0 52000000 e8565052
0011560 fffff868 4c685b59 56080498 fff8fbe8
0011600 243489ff fff863e8 10c483ff 1074c085
0011620 c0685757 56000001 fff6bfe8 10c483ff
0011640 40685353 56080498 fff8cfe8 243489ff
0011660 fff837e8 10c483ff 1074c085 c0685151
0011700 56000001 fff693e8 10c483ff 30685757
0011720 56080498 fff8a3e8 685a58ff 08049759
0011740 f7e6e856 c483ffff 89c08510 20850fc3
0011760 53000001 975b6853 e8560804 fffff7cc
0012000 8510c483 75c389c0 0cec830a b9e8ff6a
0012020 83fffff8 35ff0cec 0804aa1c aa2035ff
0012040 35ff0804 0804aa24 aa4035ff 35ff0804
0012060 0804aa50 aa4435ff 35ff0804 0804aa38
0012100 fee0b58d 5e68ffff 50080497 fff7dbe8
0012120 30c483ff aa4435ff 35ff0804 0804aa38
0012140 04979868 e1e85608 8dfffff5 ff50e045
0012160 ff56e075 04aa4835 f69ee808 c483ffff
0012200 89c08520 eb4775c6 059c887e fffffee0
0012220 3c801e8b 32752d13 07f98341 f9830574
0012240 4017750c e00584c6 2afffffe 01eb1e8b
0012260 137c8042 f8752d01 7c8010eb 752d0113
0012300 0584c609 fffffee1 4042402a c03108eb
0012320 c931d231 1e8bdf89 84131c8a c6aa75db
0012340 fee00584 8d00ffff fffee085 51fb89ff
0012360 97bf6850 e8570804 fffff730 e8243489
0012400 fffff648 8310c483 e8530cec fffff59c
0012420 a110c483 0804aa30 1174c085 ff505252
0012440 04aa4835 f5a2e808 c483ffff 3005c710
0012460 000804aa 8d000000 fffee0b5 00c7e9ff
0012500 558d0000 e8f089e4 fffffd7c 8902f883
0012520 b28c0fc7 83000000 0e7e04f8 0f05f883
0012540 0000a485 0084e900 558b0000 16048de4
0012560 fecc9589 8589ffff fffffed4 fff61be8
0012600 cc958bff 89fffffe fffec895 89008bff
0012620 fffed085 d4858bff ebfffffe be0f4001
0012640 d0958b08 f6fffffe 08014a44 853bef75
0012660 fffffed4 fec8958b 4d74ffff 010000c6
0012700 0a6a50f2 e852006a fffff5b0 8310c483
0012720 077502ff 04aa24a3 832eeb08 077503ff
0012740 04aa20a3 a322eb08 0804aa1c 50501beb
0012760 01e4458b 35ff50f0 0804aa48 fff66be8
0013000 10c483ff 04aa30a3 68535708 00000100
0013020 f4e6e856 c483ffff 0fc08510 ffff2185
0013040 0cec83ff f582e853 c483ffff 303d8310
0013060 000804aa 53531a75 0497c868 4835ff08
0013100 e80804aa fffff624 a310c483 0804aa30
0013120 aa303d83 75000804 6851511a 080497d6
0013140 aa4835ff 01e80804 83fffff6 30a310c4
0013160 a10804aa 0804aa30 0470ff52 aa4c35ff
0013200 35ff0804 0804aa48 fff45fe8 10c483ff
0013220 5bf4658d c35d5f5e 57e58955 5356ff31
0013240 000047e8 9fc38100 83000012 49e80cec
0013260 8dfffff3 ffff18b3 18838dff 29ffffff
0013300 02fec1c6 ff5015eb 75ff1075 0875ff0c
0013320 18bb94ff 47ffffff 3910c483 8de775f7
0013340 5e5bf465 55c35d5f c35de589 c3241c8b
0013360 53e58955 fffff3e8 4bc381ff ff000012
0013400 75ff0c75 e8036a08 fffff380 c9fc5d8b
0013420 909090c3 53e58955 04a85cbb 05eb5208
0013440 eb83d0ff 83038b04 f475fff8 c35d5b58
0013460 53e58955 0000e850 815b0000 00120ac3
0013500 fb16e800 5b58ffff 0000c35d 00000003
0013520 00020001 454d4f48 77007200 4423002b
0013540 3d784950 44206425 3d794950 20206425
0013560 6b736544 3d706f74 25786425 70780a64
0013600 253d736f 70790a64 253d736f 32310a64
0013620 3d34322f 000a6425 2a2d2a2d 2a2d2a2d
0013640 2a2d2a2d 312d2a2d 252d3034 64252d64
0013660 2a2d6d2d 6f73692d 39353838 6600312d
0013700 3d746e6f 000a7325 766c6568 63697465
0013720 31722d61 69660032 00646578 544f4d5f
0013740 575f4649 49485f4d 0053544e 736f7078
0013760 0a64253d 6f707900 64253d73 4925000a
0014000 3a4d253a 70255325 3a482500 253a4d25
0014020 00230053 736f7078 7079003d 003d736f
0014040 322f3231 66003d34 3d746e6f 00000000
0014060 6e69742f 6f6c6379 632e6b63 00666e6f
0014100 6e69742f 6f6c6379 00006b63 6f632e2f
0014120 6769666e 00000000 00000000 ffffffff
0014140 00000000 ffffffff 00000000 00000000
0014160 00000001 00000001 00000001 0000012a
0014200 0000000c 080489fc 0000000d 08049730
0014220 00000004 08048148 00000005 080485cc
0014240 00000006 080482ac 0000000a 00000232
0014260 0000000b 00000010 00000015 00000000
0014300 00000003 0804a944 00000002 00000150
0014320 00000014 00000011 00000017 080488ac
0014340 00000011 080488a4 00000012 00000008
0014360 00000013 00000008 6ffffffe 08048864
0014400 6fffffff 00000001 6ffffff0 080487fe
0014420 00000000 00000000 00000000 00000000
0014440 00000000 00000000 00000000 00000000
0014460 00000000 00000000 00000000 00000000
0014500 00000000 0804a870 00000000 00000000
0014520 08048a42 08048a52 08048a62 08048a72
0014540 08048a82 08048a92 08048aa2 08048ab2
0014560 08048ac2 08048ad2 08048ae2 08048af2
0014600 08048b02 08048b12 08048b22 08048b32
0014620 08048b42 08048b52 08048b62 08048b72
0014640 08048b82 08048b92 08048ba2 08048bb2
0014660 08048bc2 08048bd2 08048be2 08048bf2
0014700 08048c02 08048c12 08048c22 08048c32
0014720 08048c42 08048c52 08048c62 08048c72
0014740 08048c82 08048c92 08048ca2 08048cb2
0014760 08048cc2 08048cd2 00000000 00000000
0015000 08049812 08049796 08049814 0804981a
0015020 08049820 08049827 08049797 0000000c
0015040 00000064 00000064
0015045
tc@box:~/guilib/tinyclock$ I suspect a lot of the zeros are for addresses that get filled in when the program gets loaded.

[Pats]

Though not related to zstd or XZ  ,  but While working on a project an year back , I saw an interesting link about VDO ( not Video ) compression  :

... Interested readers , may google for:  * news  ycombinator A look at VDO, the new Linux compression layer * .

[nick65go]

thanks to andyj, the main thing to take home (for me) is that the actual compression gz (used in tcx64) is not the winner. zstd beating gz (for modest tc-users), both in size and decompresion speed, is a revelation

EDIT: i am talking about (relativly) big sized tcz (libreoffice, firefox), or eventualy personaly combined tcz into only one (remaster) - when HDD/USB size is not constrained, and flash USB/SSD speed helping ever better versus rotational HDD.

[hiro]

Dictionaries are good for words, which is why text files compress so much better than binaries. Since the majority of extensions are stripped binaries, compression might be working against us except when the binaries have lots of embedded strings.

*compression* is easier with low entropy stuff like words.
dictionaries help smoothen the trade-offs with smaller block-sizes, in all relevant cases (including binary data).

[andyj]

... compression might be working against us ...

What I was trying to say here is that the tradeoff between space saved and time spent decompressing depends a lot on how often something has to be decompressed. The kernel modules are already compressed before they are archived into the squashfs file. Compressing an already compressed file more often than not uses more space to support the overhead while not gaining on space saved. But the files are only decompressed once when insmod is run, so the tradeoff makes sense for consistency's sake. Programs which are run frequently would have to be decompressed every time, which will add up unless we put the extension in copy2fs.lst. Bottom line, compression can be beneficial but it's not always the case.

[Rich]

Hi andyj

... the tradeoff between space saved and time spent decompressing depends a lot on how often something has to be decompressed. ...

 ... Programs which are run frequently would have to be decompressed every time, which will add up unless we put the extension in copy2fs.lst. ...

That depends on the size of the programs and the amount of installed RAM. It also depends on what other activities were performed
on the computer since the last time you ran the program (read a large file into RAM or execute cache-clear for example).

If you have lots of RAM your program may still be in the cache from the last time you ran it. As long as it remains in the cache it
doesn't need to be decompressed again.

[andyj]

If you have lots of RAM your program may still be in the cache from the last time you ran it. As long as it remains in the cache it doesn't need to be decompressed again.

Actually, rootfs is basically an ever present ramfs, which is basically cache mapped as a file system (as I understand it, because it would be rather inefficient to have a RAM cache for a RAM based file system), so unless you are using copy2fs (because you have lots of RAM) then the compressed file (the squashfs archive) will be the only file in RAM. Thus, every access to any file in an extension will require decompression.

[Rich]

Hi andyj

so unless you are using copy2fs (because you have lots of RAM) then the compressed file (the squashfs archive) will be the only file in RAM. Thus, every access to any file in an extension will require decompression.

Maybe I'm missing something, but my understanding is if you don't use  copy2fs  then extensions get loop mounted. The mount
point is in the ramfs (/tmp/tcloop/) but the extension resides on the hard drive. The files in the extensions get linked into the ramfs
(/usr/local/bin for example) but the extensions files still reside on the hard drive. When you go to execute the program it follows
the link to read the file into RAM for execution. When you exit the program, it remains in RAM as long as that RAM doesn't need to
be freed up for something else.

[curaga]

That's right, linux has a separate cache for (recently) opened files and executables. The uncompressed squashfs data stays in that until the memory is needed for other things.

[andyj]

On a diskless system (PXE booted), the extensions are in RAM. But this is becoming a circular discussion. In most distros the files are not compressed because the assumption is that they are not resource constrained, thus the cost of decompression is seen as a liability because saving space isn't an asset. Compression is what helps resource constrained systems to function, so the space saved is an asset if it offsets the cost of decompressing on a low power CPU.

[jazzbiker]

HI,all!

Maybe this new compression algorythm will be usefull for zswap?

[hiro]

likely. if lz4 isn't even better...