Tiny Core Linux
Tiny Core Extensions => TCE Q&A Forum => Topic started by: polikuo on January 24, 2022, 11:29:45 AM
-
Has anyone managed to get it working on TC ?
I compiled distcc on Raspberry Pi 4 4G (aarch64) a few days ago, but couldn't find a working configuration, so I never mail it.
Here are some notes
Compile Flags:
CFLAGS='-march=armv8-a+crc -mtune=cortex-a72 -Os -pipe -flto -fuse-linker-plugin -fno-exceptions'
CXXFLAGS='-march=armv8-a+crc -mtune=cortex-a72 -Os -pipe -flto -fuse-linker-plugin -fno-exceptions -fno-rtti'
LDFLAGS='-Wl,-O1'
Dependencies:
tce-load -ci compiletc coreutils procps-ng python3.8-dev gtk3-dev
Note that I use copy mode because I feel like doing so. :)
LTO plug-in fix, see Topic: Wrong path gcc:bfd-plugins (http://forum.tinycorelinux.net/index.php/topic,25488.0.html)
P=/usr/local/libexec/gcc/aarch64-unknown-linux-gnu/10.2.0/liblto_plugin.so.0.0.0
sudo mkdir -p /usr/local/lib/bfd-plugins
sudo ln -sfv $P /usr/local/lib/bfd-plugins/liblto_plugin.so
Commands for building and checking:
$ tar xf distcc-3.4.tar.gz
$ cd distcc-3.4
$ ./configure --prefix=/usr/local --without-libiberty
$ time make -j4
real 0m 17.39s
user 0m 47.03s
sys 0m 8.63s
$ time make check
real 3m 36.39s
user 0m 36.88s
sys 0m 19.73s
## add . to PATH
$ export PATH=".:$PATH"
## add user temporarily for installcheck
$ sudo adduser -DH distcc
$ time make installcheck
real 3m 21.65s
user 0m 23.87s
sys 0m 16.88s
I have 2 RPIs, a Pi3B and a Pi4B.
Both running PiCore-13.1 with latest tczs
I've tried the instructions by dentonlt here (http://tinycorelinux.net/5.x/x86_64/tcz/distcc.tcz.info).
His thread (http://forum.tinycorelinux.net/index.php/topic,16533.0.html) isn't helping, and it's been more than a year since his last active.
The instructions on distcc official page (https://distcc.github.io/index.html) sounds easy but doesn't work, most likely did something wrong somewhere.
I check the dependencies, the daemon wouldn't run without procps-ng and coreutils.
I edit the hosts file, both /usr/local/etc/distcc/hosts ~/.distcc/hosts
I export the environment variables DISTCC_HOSTS, DISTCC_POTENTIAL_HOSTS, PATH
I create symbolic links for the masquerade directory.
I try the pump command when calling make.
All above only leads me to a local build on pi4.
Pi3 sometimes helps, but I can't replicate when I retry.
I'm running out of ideas~ :P
-
Hi polikuo
... All above only leads me to a local build on pi4.
Pi3 sometimes helps, but I can't replicate when I retry.
I'm running out of ideas~ :P
Maybe try turning on the debugging options to see if they provide anything useful:
5. If you run into problems it is highly recommended to use DISTCC_VERBOSE=1
on the client and "--log-level debug" on the server.
Found here:
https://raw.githubusercontent.com/distcc/distcc/master/INSTALL
-
Hi polikuo
Just a couple more notes:
You should create a "distcc" account on server machines so that distcc
can run with minimal privilege. It is not necessary for this account
to own any files or have a home directory. If this account doesn't
exist, distccd uses the "nobody" account.
Tinycore has "nobody" account, I presume piCore does too, so it should still work without a "distcc" account.
By default distccd writes messages to the "daemon" syslog, which
typically ends up in /var/log/messages or /var/log/daemon.
You may want to start syslog to capture messages.
Distcc will only improve performance if your build is parallelized.
So you need to use the "-j" option to make, or its equivalent with
your build tools. If your build contains too many sequential steps,
e.g. if your Makefile contains
all:
for subdir in `SUBDIRS`; ... Removed due to forum error
then you may need to rewrite your Makefile to get better parallelism.
This is especially important if you're using pump mode.
If you forget the -j , the pi3 won't help.
-
Maybe try turning on the debugging options to see if they provide anything useful:
5. If you run into problems it is highly recommended to use DISTCC_VERBOSE=1
on the client and "--log-level debug" on the server.
Found here:
https://raw.githubusercontent.com/distcc/distcc/master/INSTALL
Hi Rich
Thanks for the input.
I had turned on the debug log before I started this thread.
They were somewhat useful, I managed to fix some issues by adding some additional links and directories.
I'm starting this thread because they stop reporting anything that looks helpful.
Tinycore has "nobody" account, I presume piCore does too, so it should still work without a "distcc" account.
From my observation, I've tried with all three users, tc distcc nobody, it's OK to run as any of these.
You may want to start syslog to capture messages.
There's a log-file option for distccd, so you don't need syslog.
If you forget the -j , the pi3 won't help.
Of course I had that, I use -j10 all the time.
The thing is, I can't replicate what I had done.
There were times that, distcc stop working without any changes to the setting.
make -j10 # I can see pi3 CPU running with htop
make clean
make -j10 # pi3 is idling, it sees nothing
-
Hi polikuo
If you're using TCP connections, it should look like
this:
localhost red green blue
For SSH connections
localhost @red @green @blue
The hosts should be listed in descending order of speed. localhost
should normally be first, unless it is significantly slower than
another machine.
If the job is small enough and localhost is fast enough, the local machine may start compiling locally and sometimes
finish before it has a chance to give the next part to a remote machine. Since the local machine is once again free, it
may process that part itself. Try removing localhost from your DISTCC_HOSTS list. It should send all jobs to remote
machines.
-
Hi Rich.
I think I found something (https://wiki.voidlinux.org/Distcc#Caveats)
Compile Flags:
CFLAGS='-march=armv8-a+crc -mtune=cortex-a72 -Os -pipe -flto -fuse-linker-plugin -fno-exceptions'
CXXFLAGS='-march=armv8-a+crc -mtune=cortex-a72 -Os -pipe -flto -fuse-linker-plugin -fno-exceptions -fno-rtti'
LDFLAGS='-Wl,-O1'
I usually compile with LTO enabled.
When I remove that from the flags, pi3 receives the packages.
Then I take the chance and try that out with fltk-1.3.8
# with gcc on pi4
time make -j5
real 0m 42.95s
user 2m 30.94s
sys 0m 17.59s
# with distcc
time make -j10
real 2m 57.41s
user 0m 11.21s
sys 0m 10.55s
-
OK, I think I have things sort out.
Before I pack the extensions,
I'd like to hear out your opinions and leave my notes here.
Distcc has various modes: pump TCP, plain TCP, ssh ... etc
I'm using pump mode TCP, I believe it's more favorable in most use case.
I could setup the tce.installed script for that.
However, there are Client/Worker and Server(s)/Volunteer(s), different machine require different setting.
I'm planning to start the TCP daemon as server by default.
This way, folks can build clusters much easier, just load up the compiler and headers, no extra configuration needed.
This is what I use for my pies (Volunteer, pi3).
distccd --daemon --allow 192.168.0.40 --verbose --log-file=/home/tc/distccd.log
To make it more general purpose, I could go:
# touch /var/log/distccd.log
# chmod 666 /var/log/distccd.log
# distccd --daemon --allow 192.168.0.0/24 --allow 192.168.1.0/24 --allow 10.0.0.0/24 --allow 10.1.1.0/24 --verbose --log-file=/var/log/distccd.logNote that "/24" is a CIDR mask.
For the Worker (pi4)
$ vi ~/.distcc/hosts
localhost
192.168.0.30,lzo,cpp
The ",lzo,cpp" suffix tells distcc to pump for this server.
You don't need that for localhost, it will only slow down the compilation.
For the record, you can limit the job counts (threads) here.
localhost/2
192.168.0.30/2,lzo,cpp
Here, "/2" states job limit, not a CIDR mask
To compile (on pi4):
Export PATH with /usr/lib/distcc (can't post the code because forum error)
Configure like you normally would
pump make -j10
P.S. The internal IP of my pies
Pi3: 192.168.0.30
Pi4: 192.168.0.40
P.P.S. It's faster now that things are sorted
pump make -j10
real 0m 57.49s
user 1m 32.38s
sys 0m 13.43s
-
Here's an update.
I just compiled gfortran with these pies.
The result is quite satisfying. (gcc 10.2.0, piCore64 13.1)
$ time pump make -j10
real 39m 30.38s
user 2h 1m 20s
sys 11m 38.38s
Compare to Juanito's note. (gcc 10.1.0, piCore64 12)
make [1h 49m 48s]
@Juanito, if you're reading this, mind if I post this for you ?
There's some adjustment I'd like to add for gfortran to be compatible with distcc.
-
Please go ahead
-
Hi polikuo
... To make it more general purpose, I could go:
# touch /var/log/distccd.log
# chmod 666 /var/log/distccd.log
# distccd --daemon --allow 192.168.0.0/24 --allow 192.168.1.0/24 --allow 10.0.0.0/24 --allow 10.1.1.0/24 --verbose --log-file=/var/log/distccd.log ...
Just wanted to point out, --allow 192.168.0.0/23 is the same as --allow 192.168.0.0/24 --allow 192.168.1.0/24.
Also, the man page mentions a predefined option for private networks (LAN):
--allow-private
Allow private networks (10.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12, and
127.0.0.0/8).
-
Hi polikuo
Just wanted to point out, --allow 192.168.0.0/23 is the same as --allow 192.168.0.0/24 --allow 192.168.1.0/24.
Also, the man page mentions a predefined option for private networks (LAN):
--allow-private
Allow private networks (10.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12, and
127.0.0.0/8).
Hi Rich
Thanks for the input
However, when I try this out, it reports error.
$ distccd --daemon --allow-private --verbose --log-file=/var/log/distccd.log
distccd[4901] (dcc_parse_mask) ERROR: can't parse internet address "fe80::"I could't find the --allow-private option with distccd --help
Perhaps there are some internal bugs to be fixes by the developers.
I'll add these values to the tce.installed script manually.
-
Hi polikuo
... $ distccd --daemon --allow-private --verbose --log-file=/var/log/distccd.log
distccd[4901] (dcc_parse_mask) ERROR: can't parse internet address "fe80::" ...
That "fe80::" is IPv6 notation. If you comment out the IPv6 lines in /e tc/hosts I think that error will go away.
Or maybe installing ipv6-netfilter would fix that too.
-
That "fe80::" is IPv6 notation. If you comment out the IPv6 lines in /etc/hosts I think that error will go away.
Hi Rich
I don't see "fe80::" anywhere in /etc/hosts
127.0.0.1 pi3 localhost localhost.local
# The following lines are desirable for IPv6 capable hosts
# (added automatically by netbase upgrade)
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
Or maybe installing ipv6-netfilter would fix that too.
Nope, same result.
Guess I'll just stick with manual configuration.
-
Hi polikuo
Yes, comment out those 6 ip6- lines.
-
Hi polikuo
Yes, comment out those 6 ip6- lines.
Nope, not this one.
Tried both your ideas.
If only people invented IPv6 in the first place, we won't have to deal with this.
-
Hi polikuo
In that case, forget what I said and stick with your working version.
-
Hi
Just wanna report that, with distcc, I can see some binaries increase in size.
tc@pi4:/tmp$ du */liblto_plugin.so.0.0.0
132K distcc-build/liblto_plugin.so.0.0.0
112K distcc-stripped/liblto_plugin.so.0.0.0
96K local-build/liblto_plugin.so.0.0.0
76K local-stripped/liblto_plugin.so.0.0.0
This is disappointing :(
-
Indeed :(
-
Hi polikuo
Maybe this is the reason for the size increase:
... I usually compile with LTO enabled.
When I remove that from the flags, pi3 receives the packages. ...
-
The last time I tried, gcc would not compile itself with -flto.
-
No, just the regular flags
CFLAGS='-march=armv8-a+crc -mtune=cortex-a72 -Os -pipe'
CXXFLAGS='-march=armv8-a+crc -mtune=cortex-a72 -Os -pipe'
LDFLAGS='-Wl,-O1'
-
OK I see the problem
You need to export the flags on the Sever machines as well.
Right before you start the daemon:
tc@pi3:~$ export CFLAGS="-march=armv8-a+crc -mtune=cortex-a72 -Os -pipe"
tc@pi3:~$ export CXXFLAGS="-march=armv8-a+crc -mtune=cortex-a72 -Os -pipe"
tc@pi3:~$ export LDFLAGS="-Wl,-O1"
tc@pi3:~$ distccd --daemon --allow 192.168.0.40 --log-file=/var/log/distccd.log
Results:
tc@pi4:10.2.0$ du liblto_plugin.so.0.0.0
96K liblto_plugin.so.0.0.0
tc@pi4:10.2.0$ strip --strip-unneeded liblto_plugin.so.0.0.0
tc@pi4:10.2.0$ du liblto_plugin.so.0.0.0
76K liblto_plugin.so.0.0.0
I'll add the flags to the startup script.
-
Hi Juanito
This is the first time I compile LLVM and I'm following your instructions here (http://tinycorelinux.net/13.x/aarch64/tcz/src/llvm/compile_llvm).
I have a few questions, but first I spot a typo here:
The comma is gone, so nothing is passed to the linker
find . -name link.txt -type f -exec sed -i 's/-Wl,-O3/-Wl -Os/g' {} \;
Anyway~
I can't find all the files listed above so I decide to "grep" my way through.
There are only 3 files containing "-O3" or "-g" flags.
sed -i.orig -e 's:-Wl,-O3:-Wl,-O1:g' -e 's: -O3 : :g' -e 's: -g : :g' build.ninja CMakeCache.txt compile_commands.json
[libexecdir needs setting]
[-DLLVM_ENABLE_LTO=Full]
I didn't touch libexecdir and I removed LTO so jobs can be distributed.
Distcc sure helps a lot with the compile time.
time ninja -j8
real 5h 38m 17s
user 16h 17m 23s
sys 44m 3.42s
BTW, you can specify DESTDIR with ninja
DESTDIR=/tmp/packing-llwm ninja install
I'm having troubles when packing things up.
What should I do with this ?
[links against ncursesw without ncursesw-dev]
What about clang ?
It looks like llvm should be packed into multiple packages and I'm not sure what are they.
P.S. I'm pretty sure -Wl,-O[1-9] is identical to -Wl,-O1 so you can just ignore that.
-
The comma is gone, so nothing is passed to the linker
find . -name link.txt -type f -exec sed -i 's/-Wl,-O3/-Wl -Os/g' {} \;
Well spotted :)
What should I do with this ?
[links against ncursesw without ncursesw-dev]
That was a note to remind me to add ncursesw to the dep file
What about clang ?
It looks like llvm should be packed into multiple packages and I'm not sure what are they.
See llvm-bin, llvm-dev, llvm-lib and clang
-
Hi Juanito
I'm not quite sure which binaries should be in llvm-bin, llvm-dev or clang
It's pretty obvious for those with "llvm-" or "clang-" prefix, but the rest is messy.
Different port also has different layout. (comparing aarch64 and x86_64)
Any suggestions ?
llvm-root/usr/local/bin/analyze-build
llvm-root/usr/local/bin/bugpoint
llvm-root/usr/local/bin/c-index-test
llvm-root/usr/local/bin/diagtool
llvm-root/usr/local/bin/dsymutil
llvm-root/usr/local/bin/hmaptool
llvm-root/usr/local/bin/intercept-build
llvm-root/usr/local/bin/llc
llvm-root/usr/local/bin/lli
llvm-root/usr/local/bin/opt
llvm-root/usr/local/bin/sancov
llvm-root/usr/local/bin/sanstats
llvm-root/usr/local/bin/scan-build
llvm-root/usr/local/bin/scan-build-py
llvm-root/usr/local/bin/scan-view
llvm-root/usr/local/bin/split-file
llvm-root/usr/local/bin/verify-uselistorder
-
I think I originally looked what debian did - since several utilities have been added subsequently I should have checked back before now…
-
Hi Juanito
Just mailed the extensions for you to check.
I use distcc this time, see the instructions for more details.
Not sure what's causing the size difference
$ du *tcz
106M clang.tcz
14M llvm-bin.tcz
61M llvm-dev.tcz
19M llvm-lib.tcz
tc@pi4:/tmp/repo$ grep Size *info
clang.tcz.info:Size: 86MB
llvm-bin.tcz.info:Size: 11.7MB
llvm-dev.tcz.info:Size: 48MB
llvm-lib.tcz.info:Size: 15.4MB
It could be the files cause I could see many new ones not presented in the old packages
-
I'd say the size difference is due to:
* updates are almost always bigger (librsvg with rust 10x bigger)
* lto compiles 5-10% smaller binaries/dynamic libs
-
I use distcc this time, see the instructions for more details.
Not sure what's causing the size difference
I'd say the size difference is due to:
* updates are almost always bigger (librsvg with rust 10x bigger)
* lto compiles 5-10% smaller binaries/dynamic libs
I have verified with a local build, it produces the same result as a distcc build.
Not sure why, but -DLLVM_ENABLE_LTO=Full does not compile for me.