Link Time Optimization (LTO) + distcc

[polikuo]

Hi, all.

I'm documenting my results here in case anyone is interested.

Some related forum posts:
Setting up distcc
Forefox-ESR
Assist requested for distcc to-be-extension

Raspberry Pi cluster setup: (piCore 15.0, armhf)

• host name: pi4-1
     
  • Position - Master
  • Model - 4B, 8G
  • Kernel - 6.6.47-piCore-v8
  • IP - 192.168.18.11
   
• host name: pi4-2
     
  • Position - Slave
  • Model - 4B, 4G
  • Kernel - 6.6.47-piCore-v8
  • IP - 192.168.18.12
   
• host name: pi3-1
     
  • Position - Slave
  • Model - 3B, 1G
  • Kernel - 6.6.47-piCore-v7
  • IP - 192.168.18.13
   
• host name: pi3-2
     
  • Position - Slave
  • Model - 3B, 1G
  • Kernel - 6.6.47-piCore-v7
  • IP - 192.168.18.14
   

- Master's distcc configuration -
(.distcc/tc.sh is empty)

Code: [Select]

tc@pi4-1:~$ cat .distcc/hosts
localhost/4
192.168.18.12,lzo,cpp/4
192.168.18.13,lzo,cpp/4
192.168.18.14,lzo,cpp/4

- Slaves' distcc configuration -
(.distcc/tc.sh, adjusted flags for consistency)

Code: [Select]

tc@pi4-2:~$ cat .distcc/tc.sh
#!/bin/sh

CFLAGS='-march=armv6zk -mtune=arm1176jzf-s -mfpu=vfp -Os -pipe'
CXXFLAGS='-march=armv6zk -mtune=arm1176jzf-s -mfpu=vfp -Os -pipe'
EXTRA_C=' -L/usr/local/lib -flto=full -fuse-ld=lld -fno-exceptions'
EXTRA_CXX=' -L/usr/local/lib -flto=full -fuse-ld=lld -fno-exceptions -fno-rtti'
CFLAGS="${CFLAGS}${EXTRA_C} --gcc-install-dir=/usr/local/lib/gcc/armv7l-unknown-linux-gnueabihf/13.2.0"
CXXFLAGS="${CXXFLAGS}${EXTRA_CXX} --gcc-install-dir=/usr/local/lib/gcc/armv7l-unknown-linux-gnueabihf/13.2.0"
LDFLAGS='-L/usr/local/lib -Wl,-O1 -Wl,-rpath=/usr/local/lib'
export CFLAGS CXXFLAGS LDFLAGS

if [ $(id -u) = 0 ]; then
  TCUSER=$(cat /etc/sysconfig/tcuser)
  [ "$TCUSER" ] || TCUSER='tc'
  CMD="--user $TCUSER"
fi

distccd --daemon $CMD \
  --allow 192.168.0.0/16 \
  --allow 172.16.0.0/12 \
  --allow 10.0.0.0/8 \
  --allow 127.0.0.0/8 \
  --verbose --log-file=/var/log/distccd.log
(onboot.lst)

Code: [Select]

tc@pi4-2:/etc/sysconfig/tcedir$ cat onboot.lst
mylocale.tcz
openssh.tcz
clang.tcz
distcc.tcz


LTO comparison: ( GCC + bfd ) V.S. ( Clang + lld )

- CMAKE configuration -

Code: [Select]

# GCC
cmake -DCMAKE_C_COMPILER='/usr/local/lib/distcc/gcc' -DCMAKE_CXX_COMPILER='/usr/local/lib/distcc/g++' \
 -DCMAKE_C_FLAGS_MINSIZEREL='-march=armv6zk -mtune=arm1176jzf-s -mfpu=vfp -Os -pipe -L/usr/local/lib -flto -fno-exceptions -Wno-unused-value -fuse-ld=bfd -Wl,-O1 -Wl,-v' \
 -DCMAKE_CXX_FLAGS_MINSIZEREL='-march=armv6zk -mtune=arm1176jzf-s -mfpu=vfp -Os -pipe -L/usr/local/lib -flto -fno-exceptions -fno-rtti -Wno-unused-value -fuse-ld=bfd -Wl,-O1 -Wl,-v' \
 -DFLTK_BACKEND_WAYLAND=ON -DFLTK_BACKEND_X11=ON -DFLTK_BUILD_SHARED_LIBS=ON -DFLTK_OPTION_CAIRO_EXT=ON -DFLTK_OPTION_CAIRO_WINDOW=ON \
 -DFLTK_OPTION_OPTIM='-Os' -DFLTK_OPTION_STD=ON -LAH .. | tee /tmp/cmake.info

# Clang
cmake -DCMAKE_C_COMPILER='/usr/local/lib/distcc/clang' -DCMAKE_CXX_COMPILER='/usr/local/lib/distcc/clang++' \
 -DCMAKE_C_FLAGS_MINSIZEREL='-march=armv6zk -mtune=arm1176jzf-s -mfpu=vfp -Os -pipe -L/usr/local/lib -flto=full -fno-exceptions -Wno-unused-value -fuse-ld=lld -Wl,-L/usr/local/lib -Wl,-rpath=/usr/local/lib -Wl,-O1 -Wl,-v' \
 -DCMAKE_CXX_FLAGS_MINSIZEREL='-march=armv6zk -mtune=arm1176jzf-s -mfpu=vfp -Os -pipe -L/usr/local/lib -flto=full -fno-exceptions -fno-rtti -Wno-unused-value -fuse-ld=lld -Wl,-L/usr/local/lib -Wl,-rpath=/usr/local/lib -Wl,-O1 -Wl,-v' \
 -DCMAKE_AR='/usr/local/bin/llvm-ar' -DCMAKE_RANLIB='/usr/local/bin/llvm-ranlib' -DCMAKE_NM='/usr/local/bin/llvm-nm' \
 -DCMAKE_STRIP='/usr/local/bin/llvm-strip' -DCMAKE_BUILD_TYPE=MINSIZEREL -DCMAKE_INSTALL_PREFIX=/usr/local \
 -DFLTK_BACKEND_WAYLAND=ON -DFLTK_BACKEND_X11=ON -DFLTK_BUILD_SHARED_LIBS=ON -DFLTK_OPTION_CAIRO_EXT=ON -DFLTK_OPTION_CAIRO_WINDOW=ON \
 -DFLTK_OPTION_OPTIM='-Os' -DFLTK_OPTION_STD=ON -LAH .. | tee /tmp/cmake.info

- Build time -

Code: [Select]

time pump make -j16
-------------------------------------------------------------------------
# GCC
-------------------------------------------------------------------------
__________Warning: 11 pump-mode compilation(s) failed on server, but succeeded locally.
real    3m 4.19s
user    9m 56.64s
sys     1m 26.71s
-------------------------------------------------------------------------
# Clang
-------------------------------------------------------------------------
__________Warning: 13 pump-mode compilation(s) failed on server, but succeeded locally.
real    33m 34.26s
user    2h 6m 26s
sys     1m 53.38s

- Size comparison -

- Overall size -

Code: [Select]

tc@pi4-1:/mnt/sda1/unstripped-copies$ du -hd0 fltk-full-clang fltk-full-gcc
11.5M   fltk-full-clang
9.6M    fltk-full-gcc
tc@pi4-1:/tmp/stripped$ du -hd0 fltk-full-clang fltk-full-gcc
10.6M   fltk-full-clang
8.4M    fltk-full-gcc

- Individual ELF Binaries -

Code: [Select]

-------------------------------------------------------------------------
# Unstripped
-------------------------------------------------------------------------
tc@pi4-1:/mnt/sda1/unstripped-copies$ find fltk-full-clang fltk-full-gcc -type f -exec head -c4 {} ';' -print | awk '/^\x7fELF/{print(substr($0,5))}' | xargs du -h | awk 'BEGIN{print "clang\tgcc\tfile"}NR<8{A[NR]=$1}NR>7{sub(/[^/]*/, "", $2); print A[NR-7] "\t" $1 "\t" $2}'
clang   gcc     file
8.0K    8.0K    /usr/local/lib/libfltk_cairo.so.1.4.0
1.5M    1.5M    /usr/local/lib/libfltk.so.1.4.0
28.0K   32.0K   /usr/local/lib/libfltk_forms.so.1.4.0
192.0K  200.0K  /usr/local/lib/libfltk_images.so.1.4.0
196.0K  204.0K  /usr/local/lib/libfltk_gl.so.1.4.0
1.8M    2.1M    /usr/local/bin/fluid
592.0K  904.0K  /usr/local/bin/fltk-options
-------------------------------------------------------------------------
# Stripped
-------------------------------------------------------------------------
tc@pi4-1:/tmp/stripped$ find fltk-full-clang fltk-full-gcc -type f -exec head -c4 {} ';' -print | awk '/^\x7fELF/{print(substr($0,5))}' | xargs du -h | awk 'BEGIN{print "clang\tgcc\tfile"}NR<8{A[NR]=$1}NR>7{sub(/[^/]*/, "", $2); print A[NR-7] "\t" $1 "\t" $2}'
clang   gcc     file
4.0K    8.0K    /usr/local/lib/libfltk_cairo.so.1.4.0
1.1M    1.2M    /usr/local/lib/libfltk.so.1.4.0
16.0K   24.0K   /usr/local/lib/libfltk_forms.so.1.4.0
144.0K  160.0K  /usr/local/lib/libfltk_images.so.1.4.0
124.0K  144.0K  /usr/local/lib/libfltk_gl.so.1.4.0
1.4M    1.5M    /usr/local/bin/fluid
432.0K  612.0K  /usr/local/bin/fltk-options

- Individual Static Libraries -

Code: [Select]

tc@pi4-1:/mnt/sda1/unstripped-copies$ find fltk-full-clang fltk-full-gcc -name '*.a' | xargs du -h | awk 'BEGIN{print "clang\tgcc\tfile"}NR<6{A[NR]=$1}NR>5{sub(/[^/]*/, "", $2); print A[NR-5] "\t" $1 "\t" $2}'
clang   gcc     file
3.0K    1.0K    /usr/local/lib/libfltk_cairo.a
4.8M    2.6M    /usr/local/lib/libfltk.a
72.0K   48.0K   /usr/local/lib/libfltk_forms.a
556.0K  252.0K  /usr/local/lib/libfltk_images.a
420.0K  276.0K  /usr/local/lib/libfltk_gl.a