upx for large executables?

[GNUser]

neonix mentioned compressing with upx -9 in a different thread. I decided to investigate.

The largest extension on my system is brave-browser.tcz at 180 MiB.

If I unsquash brave-browser.tcz, compress just the main brave executable using upx -9, and recreate brave-browser.tcz, size of the extension drops to 152 MiB. I'm testing the shrunken extension now and it's working fine. Importantly, upx.tcz does not need to be loaded for the shrunken brave-browser.tcz to work.

Should I be using the upx (or similar) compression tool to reduce the size of the large extensions that I maintain (e.g., brave-browser.tcz)? Are there any downsides? I'm very interested in hearing the opinion of the TCL developers on this.

[GNUser]

In the real world there's no such thing as a free lunch, so there must be a downside somewhere...

[hiro]

generally, most higher compression methods also take much longer to decompress, especially when the implementation isn't multithreaded...
personally i don't like to go harder than lz4. this algorithm is efficient enough that i don't waste too many cpu cycles, but it compresses most stuff well enough for my needs.

[CardealRusso]

Perhaps it might be allowed to change the block size, maybe even the compression to zstd, for specific extensions that would benefit from this.

Brave-broser with 64kb (default is 4kb) gets 155MB.

[GNUser]

Hi hiro. Decompression overhead is a good point. I saw your post in the other thread. You're right--having upx inside a gzip-compressed squashfs sounds overcomplicated.

It seems best compression algorithm for tcz has been discussed at length over the years, including in the "Into the Core" book (chapter 18). I'm sorry for beating on this dead horse some more

Hi CardealRusso. Tweaking block size has been discussed before but not pursued due to added complexity (keeping track of which block size to use on which architecture).

I'm going to continue to keep it simple with the extensions that I create and avoid being too clever.

[Rich]

Hi GNUser
I think an executable in a squash file system can load
only the code it needs as it's executing because the
the driver decompresses and fetches the data for the
operating system transparently.

I suspect UPX basically creates a self extracting compressed
file with a loader. This suggests the entire file gets loaded
immediately into RAM even if all sections of code are not
yet (or ever) needed.

[GNUser]

I suspect UPX basically creates a self extracting compressed
file with a loader.

Hi Rich. Based on my cursory reasearch that sounds about right.

This suggests the entire file gets loaded
immediately into RAM even if all sections of code are not
yet (or ever) needed.

Yikes. We definitely don't want that.

It's so easy to just focus on the size of the extension and get carried away into all kinds of trouble!

[CardealRusso]

...Tweaking block size has been discussed before...

I may be mistaken but the discussion (which I took part in) was about changing the default block size, not specific cases.
I hope to hear from an administrator, as I'm also interested.

[hiro]

increasing the block size will lead to more stuff being loaded into ram unnecessarily. there's a good reason it's at 4k right now.
more useful would be to figure out how to minimize data getting cached twice (once in compressed form and once uncompressed).
but there's diminishing returns in all of this. our current system is plenty good already, maybe find something else to hack on

[curaga]

I may be mistaken but the discussion (which I took part in) was about changing the default block size, not specific cases.
I hope to hear from an administrator, as I'm also interested.

The block size was fixed at 4k for the 32bit platforms, but larger sizes are acceptable on the 64-bit ones. The RAM hit is quite different on a 64mb system vs a 4gb one. That is, if you're maintaining some big extension for x86-64 or arm64, and it significantly benefits, it's fine to send it with a larger block size.

[neonix]

upx -9 is gzip compression. All x86_64 cpus are so powerfull that you don't see a difference with gzip.

Try upx --ultra-brute and tell us what is the result when you use browser for many hours. How much RAM you have in your PC? How much RAM typical 64-bit TC user has?

On old wiki there was recomendation to use upx when you create new tczs. But it was in upx-9 era, now we have lzma era.

If you want for sure, how upx works, you need to read documentation or upx source code.
As far as I remember, compressed (upxed) program is decompressed in RAM on the fly (I don' sure about lzma) and it don't impact RAM in negative way. More pros than cons.

There's no way to change anything in tcz block compression, there's more cons than pros.

[hiro]

can you back it up with real numbers?
i still didn't see any meaningful benchmarks.
i assume it's because you didn't try it out at all, you're just running your ideas by us. but next time, if you don't intend to run the experiment please at least put this into a disclaimer and don't pretend you know already the outcome of the experiment.

[GNUser]

Hi CardealRusso. I can reproduce your experiment: If I squash brave-browser without specifying block size (i.e., using the 4K default), extension size is 180 MiB. If I squash using 64K block size, extension size is 155 MiB.

Hi curaga. I will follow your advice. For large extensions being submitted for x86_64 repo, I will check if there is significant benefit to using a block size larger than the 4K default. Is there a block size you would recommend as a sweet spot for x86_64 architecture? 64K? 128K?

[curaga]

It depends on the data being compressed, can't really give a number.

[GNUser]

Fair enough. Is there at least a range of sizes that you would consider to be reasonable?

EDIT: I just need a range to guide experimentation. I don't want to go above a block size that would be considered ridiculous