WelcomeWelcome | FAQFAQ | DownloadsDownloads | WikiWiki

Author Topic: squashfs zstd vs lz4  (Read 5542 times)

Offline CardealRusso

  • Full Member
  • ***
  • Posts: 178
Re: squashfs zstd vs lz4
« Reply #15 on: June 23, 2023, 05:38:04 PM »
CompSizeRamDecomp (Real)
zstd(128k)119MB391MB0.37s
lz4(128k)181MB381MB0.35s
gzip(128k)131MB379MB0.32s
zstd(4k)154MB385MB1.12s
gzip(4k)160MB374MB1.15s
« Last Edit: June 23, 2023, 05:49:28 PM by CardealRusso »

Offline Juanito

  • Administrator
  • Hero Member
  • *****
  • Posts: 14770
Re: squashfs zstd vs lz4
« Reply #16 on: June 27, 2023, 06:06:27 AM »
squashfs-tools and deps updated in x86/x86_64 repos in case anybody wishes to test further.

Offline GNUser

  • Wiki Author
  • Hero Member
  • *****
  • Posts: 1490
Re: squashfs zstd vs lz4
« Reply #17 on: June 27, 2023, 12:11:55 PM »
Hi, Juanito. I updated and ran into this:
Code: [Select]
$ mksquashfs --help
mksquashfs: error while loading shared libraries: liblz4.so.1: cannot open shared object file: No such file or directory
It seems liblz4.tcz needs to be added to squashfs-tools.tcz.dep

P.S. I also noticed that liblz4.tcz.list, liblzma.tcz.list, etc. are empty.
« Last Edit: June 27, 2023, 12:22:29 PM by GNUser »

Offline Juanito

  • Administrator
  • Hero Member
  • *****
  • Posts: 14770
Re: squashfs zstd vs lz4
« Reply #18 on: June 28, 2023, 04:28:11 AM »
dep file adjusted - I'm not sure what happened there.

It seems:
Code: [Select]
for file in *.tcz; do unsquashfs -ll -d '' "$file" | grep -v '^d' | sed -e 's#.* /#/#' -e 's# -> .*##' -e 1,3d > "$file".list ; done..no longer works - *list files reposted.

Offline GNUser

  • Wiki Author
  • Hero Member
  • *****
  • Posts: 1490
Re: squashfs zstd vs lz4
« Reply #19 on: June 28, 2023, 08:26:03 AM »
Hi, Juanito.

It seems:
Code: [Select]
for file in *.tcz; do unsquashfs -ll -d '' "$file" | grep -v '^d' | sed -e 's#.* /#/#' -e 's# -> .*##' -e 1,3d > "$file".list ; done..no longer works

I decided to investigate this.

With old version of squashfs-tools:
Code: [Select]
$ unsquashfs -ll -d '' liblz4.tcz
Parallel unsquashfs: Using 4 processors
3 inodes (29 blocks) to write

drwxr-xr-x tc/staff                 26 2023-06-26 13:29
drwxr-xr-x root/root                28 2023-06-26 13:28 /usr
drwxr-xr-x root/root                26 2023-06-26 13:30 /usr/local
drwxr-xr-x root/root                74 2023-06-26 13:30 /usr/local/lib
lrwxrwxrwx root/root                15 2023-06-26 11:10 /usr/local/lib/liblz4.so -> liblz4.so.1.9.4
lrwxrwxrwx root/root                15 2023-06-26 11:10 /usr/local/lib/liblz4.so.1 -> liblz4.so.1.9.4
-rwxr-xr-x root/root            108856 2023-06-26 13:30 /usr/local/lib/liblz4.so.1.9.4

With new version of squashfs-tools, notice that there is no 3-line header:
Code: [Select]
$ unsquashfs -ll -d '' liblz4.tcz
drwxr-xr-x tc/staff                 26 2023-06-26 13:29
drwxr-xr-x root/root                28 2023-06-26 13:28 /usr
drwxr-xr-x root/root                26 2023-06-26 13:30 /usr/local
drwxr-xr-x root/root                74 2023-06-26 13:30 /usr/local/lib
lrwxrwxrwx root/root                15 2023-06-26 11:10 /usr/local/lib/liblz4.so -> liblz4.so.1.9.4
lrwxrwxrwx root/root                15 2023-06-26 11:10 /usr/local/lib/liblz4.so.1 -> liblz4.so.1.9.4
-rwxr-xr-x root/root            108856 2023-06-26 13:30 /usr/local/lib/liblz4.so.1.9.4

So with new version of squashfs-tools we need to change the command that generates .list files from this:
Code: [Select]
for file in *.tcz; do unsquashfs -ll -d '' "$file" | grep -v '^d' | sed -e 's#.* /#/#' -e 's# -> .*##' -e 1,3d > "$file".list ; doneTo this:
Code: [Select]
for file in *.tcz; do unsquashfs -ll -d '' "$file" | grep -v '^d' | sed -e 's#.* /#/#' -e 's# -> .*##' > "$file".list ; done
P.S. I just took a peek at submitqc to see whether it is affected by this issue. It is not affected because its strategy to create the .list file does not use unsquashfs -ll:
Code: [Select]
find -not -type d | sort | sed 's/^\.//' > ${NEWLIST}

« Last Edit: June 28, 2023, 08:40:48 AM by GNUser »

Offline Juanito

  • Administrator
  • Hero Member
  • *****
  • Posts: 14770
Re: squashfs zstd vs lz4
« Reply #20 on: June 28, 2023, 10:43:44 AM »
Thanks for that  :)

This also works:
Code: [Select]
for file in *.tcz; do unsquashfs -lc "$file" > "$file".list ; done && sed -i 's/squashfs-root//g' *tcz.list

Offline GNUser

  • Wiki Author
  • Hero Member
  • *****
  • Posts: 1490
Re: squashfs zstd vs lz4
« Reply #21 on: June 28, 2023, 11:45:01 AM »
This also works:
Code: [Select]
for file in *.tcz; do unsquashfs -lc "$file" > "$file".list ; done && sed -i 's/squashfs-root//g' *tcz.list
It sure does. And is a lot easier to understand.

Thanks for that  :)
You're welcome :)


Offline Rich

  • Administrator
  • Hero Member
  • *****
  • Posts: 11560
Re: squashfs zstd vs lz4
« Reply #22 on: June 28, 2023, 12:24:27 PM »
Hi Juanito
This also works:
Code: [Select]
for file in *.tcz; do unsquashfs -lc "$file" > "$file".list ; done && sed -i 's/squashfs-root//g' *tcz.list
That will also include empty directories.

This will work with both versions of  unsquashfs:
Code: [Select]
tc@E310:~$ unsquashfs -ll -d '' /mnt/sda1/tce/optional/sbc.tcz | grep -v '^d' | grep '/' | sed -e 's#.* /#/#' -e 's# -> .*##'
/usr/local/bin/sbcdec
/usr/local/bin/sbcenc
/usr/local/bin/sbcinfo
/usr/local/lib/libsbc.so
/usr/local/lib/libsbc.so.1
/usr/local/lib/libsbc.so.1.1.0
tc@E310:~$
This was run with the old version of  unsquashfs.