Setting up distcc

[polikuo]

Has anyone managed to get it working on TC ?

I compiled distcc on Raspberry Pi 4 4G (aarch64) a few days ago, but couldn't find a working configuration, so I never mail it.

Here are some notes

Compile Flags:

Code: [Select]

CFLAGS='-march=armv8-a+crc -mtune=cortex-a72 -Os -pipe -flto -fuse-linker-plugin -fno-exceptions'
CXXFLAGS='-march=armv8-a+crc -mtune=cortex-a72 -Os -pipe -flto -fuse-linker-plugin -fno-exceptions -fno-rtti'
LDFLAGS='-Wl,-O1'
Dependencies:

Code: [Select]

tce-load -ci compiletc coreutils procps-ng python3.8-dev gtk3-dev
Note that I use copy mode because I feel like doing so.

LTO plug-in fix, see Topic: Wrong path gcc:bfd-plugins

Code: [Select]

P=/usr/local/libexec/gcc/aarch64-unknown-linux-gnu/10.2.0/liblto_plugin.so.0.0.0
sudo mkdir -p /usr/local/lib/bfd-plugins
sudo ln -sfv $P /usr/local/lib/bfd-plugins/liblto_plugin.so

Commands for building and checking:

Code: [Select]

$ tar xf distcc-3.4.tar.gz
$ cd distcc-3.4
$ ./configure --prefix=/usr/local --without-libiberty
$ time make -j4
real    0m 17.39s
user    0m 47.03s
sys     0m 8.63s
$ time make check
real    3m 36.39s
user    0m 36.88s
sys     0m 19.73s
## add . to PATH
$ export PATH=".:$PATH"
## add user temporarily for installcheck
$ sudo adduser -DH distcc
$ time make installcheck
real    3m 21.65s
user    0m 23.87s
sys     0m 16.88s

I have 2 RPIs, a Pi3B and a Pi4B.
Both running PiCore-13.1 with latest tczs

I've tried the instructions by dentonlt here.
His thread isn't helping, and it's been more than a year since his last active.

The instructions on distcc official page sounds easy but doesn't work, most likely did something wrong somewhere.

I check the dependencies, the daemon wouldn't run without procps-ng and coreutils.
I edit the hosts file, both /usr/local/etc/distcc/hosts ~/.distcc/hosts
I export the environment variables DISTCC_HOSTS, DISTCC_POTENTIAL_HOSTS, PATH
I create symbolic links for the masquerade directory.
I try the pump command when calling make.

All above only leads me to a local build on pi4.
Pi3 sometimes helps, but I can't replicate when I retry.
I'm running out of ideas~ 

[Rich]

Hi polikuo

... All above only leads me to a local build on pi4.
Pi3 sometimes helps, but I can't replicate when I retry.
I'm running out of ideas~ 

Maybe try turning on the debugging options to see if they provide anything useful:

5. If you run into problems it is highly recommended to use DISTCC_VERBOSE=1
   on the client and "--log-level debug" on the server.

Found here:
https://raw.githubusercontent.com/distcc/distcc/master/INSTALL

[Rich]

Hi polikuo
Just a couple more notes:

You should create a "distcc" account on server machines so that distcc
can run with minimal privilege.  It is not necessary for this account
to own any files or have a home directory.  If this account doesn't
exist, distccd uses the "nobody" account.

Tinycore has  "nobody"  account, I presume piCore does too, so it should still work without a  "distcc"  account.

By default distccd writes messages to the "daemon" syslog, which
typically ends up in /var/log/messages or /var/log/daemon.

You may want to start  syslog  to capture messages.

Distcc will only improve performance if your build is parallelized.
So you need to use the "-j" option to make, or its equivalent with
your build tools.  If your build contains too many sequential steps,
e.g. if your Makefile contains

   all:
            for subdir in `SUBDIRS`; ... Removed due to forum error

then you may need to rewrite your Makefile to get better parallelism.
This is especially important if you're using pump mode.

If you forget the  -j , the pi3 won't help.

[polikuo]

Maybe try turning on the debugging options to see if they provide anything useful:

5. If you run into problems it is highly recommended to use DISTCC_VERBOSE=1
   on the client and "--log-level debug" on the server.

Found here:
https://raw.githubusercontent.com/distcc/distcc/master/INSTALL

Hi Rich
Thanks for the input.
I had turned on the debug log before I started this thread.
They were somewhat useful, I managed to fix some issues by adding some additional links and directories.
I'm starting this thread because they stop reporting anything that looks helpful.

Tinycore has  "nobody"  account, I presume piCore does too, so it should still work without a  "distcc"  account.

From my observation, I've tried with all three users, tc distcc nobody, it's OK to run as any of these.

You may want to start  syslog  to capture messages.

There's a log-file option for distccd, so you don't need syslog.

If you forget the  -j , the pi3 won't help.

Of course I had that, I use -j10 all the time.

The thing is, I can't replicate what I had done.
There were times that, distcc stop working without any changes to the setting.

Code: [Select]

make -j10 # I can see pi3 CPU running with htop
make clean
make -j10 # pi3 is idling, it sees nothing

[Rich]

Hi polikuo

If you're using TCP connections, it should look like
this:

  localhost red green blue

For SSH connections

  localhost @red @green @blue

The hosts should be listed in descending order of speed.  localhost
should normally be first, unless it is significantly slower than
another machine.

If the job is small enough and localhost is fast enough, the local machine may start compiling locally and sometimes
finish before it has a chance to give the next part to a remote machine. Since the local machine is once again free, it
may process that part itself. Try removing  localhost  from your  DISTCC_HOSTS  list. It should send all jobs to remote
machines.

[polikuo]

Hi Rich.
I think I found something

Compile Flags:

Code: [Select]

CFLAGS='-march=armv8-a+crc -mtune=cortex-a72 -Os -pipe -flto -fuse-linker-plugin -fno-exceptions'
CXXFLAGS='-march=armv8-a+crc -mtune=cortex-a72 -Os -pipe -flto -fuse-linker-plugin -fno-exceptions -fno-rtti'
LDFLAGS='-Wl,-O1'

I usually compile with LTO enabled.
When I remove that from the flags, pi3 receives the packages.
Then I take the chance and try that out with fltk-1.3.8

Code: [Select]

# with gcc on pi4
time make -j5
real    0m 42.95s
user    2m 30.94s
sys     0m 17.59s
# with distcc
time make -j10
real    2m 57.41s
user    0m 11.21s
sys     0m 10.55s


[polikuo]

OK, I think I have things sort out.
Before I pack the extensions,
 I'd like to hear out your opinions and leave my notes here.

Distcc has various modes: pump TCP, plain TCP, ssh ... etc
I'm using pump mode TCP, I believe it's more favorable in most use case.
I could setup the tce.installed script for that.
However, there are Client/Worker and Server(s)/Volunteer(s), different machine require different setting.

I'm planning to start the TCP daemon as server by default.
This way, folks can build clusters much easier, just load up the compiler and headers, no extra configuration needed.

This is what I use for my pies (Volunteer, pi3).

Code: [Select]

distccd --daemon --allow 192.168.0.40 --verbose --log-file=/home/tc/distccd.log
To make it more general purpose, I could go:

Code: [Select]

# touch /var/log/distccd.log
# chmod 666 /var/log/distccd.log
# distccd --daemon --allow 192.168.0.0/24 --allow 192.168.1.0/24 --allow 10.0.0.0/24 --allow 10.1.1.0/24 --verbose --log-file=/var/log/distccd.logNote that "/24" is a CIDR mask.

For the Worker (pi4)

Code: [Select]

$ vi ~/.distcc/hosts
localhost
192.168.0.30,lzo,cpp
The ",lzo,cpp" suffix tells distcc to pump for this server.
You don't need that for localhost, it will only slow down the compilation.

For the record, you can limit the job counts (threads) here.

Code: [Select]

localhost/2
192.168.0.30/2,lzo,cpp
Here, "/2" states job limit, not a CIDR mask

To compile (on pi4):
Export PATH with /usr/lib/distcc (can't post the code because forum error)
Configure like you normally would

Code: [Select]

pump make -j10
P.S. The internal IP of my pies
Pi3: 192.168.0.30
Pi4: 192.168.0.40

P.P.S. It's faster now that things are sorted

Code: [Select]

pump make -j10
real    0m 57.49s
user    1m 32.38s
sys     0m 13.43s


[polikuo]

Here's an update.
I just compiled gfortran with these pies.
The result is quite satisfying. (gcc 10.2.0, piCore64 13.1)

Code: [Select]

$ time pump make -j10
real    39m 30.38s
user    2h 1m 20s
sys     11m 38.38s

Compare to Juanito's note. (gcc 10.1.0, piCore64 12)

make [1h 49m 48s]

@Juanito, if you're reading this, mind if I post this for you ?
There's some adjustment I'd like to add for gfortran to be compatible with distcc.

[Juanito]

Please go ahead

[Rich]

Hi polikuo

... To make it more general purpose, I could go:

Code: [Select]

# touch /var/log/distccd.log
# chmod 666 /var/log/distccd.log
# distccd --daemon --allow 192.168.0.0/24 --allow 192.168.1.0/24 --allow 10.0.0.0/24 --allow 10.1.1.0/24 --verbose --log-file=/var/log/distccd.log ...

Just wanted to point out,  --allow 192.168.0.0/23  is the same as  --allow 192.168.0.0/24 --allow 192.168.1.0/24.

Also, the man page mentions a predefined option for private networks (LAN):

--allow-private
Allow private networks (10.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12, and
127.0.0.0/8).

[polikuo]

Hi polikuo

Just wanted to point out,  --allow 192.168.0.0/23  is the same as  --allow 192.168.0.0/24 --allow 192.168.1.0/24.

Also, the man page mentions a predefined option for private networks (LAN):

--allow-private
Allow private networks (10.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12, and
127.0.0.0/8).

Hi Rich
Thanks for the input
However, when I try this out, it reports error.

Code: [Select]

$ distccd --daemon --allow-private --verbose --log-file=/var/log/distccd.log
distccd[4901] (dcc_parse_mask) ERROR: can't parse internet address "fe80::"I could't find the --allow-private option with distccd --help
Perhaps there are some internal bugs to be fixes by the developers.
I'll add these values to the tce.installed script manually.

[Rich]

Hi polikuo

...

Code: [Select]

$ distccd --daemon --allow-private --verbose --log-file=/var/log/distccd.log
distccd[4901] (dcc_parse_mask) ERROR: can't parse internet address "fe80::" ...

That  "fe80::"  is IPv6 notation. If you comment out the  IPv6  lines in  /e tc/hosts  I think that error will go away.

Or maybe installing  ipv6-netfilter  would fix that too.

[polikuo]

That  "fe80::"  is IPv6 notation. If you comment out the  IPv6  lines in  /etc/hosts  I think that error will go away.

Hi Rich
I don't see "fe80::" anywhere in /etc/hosts

Code: [Select]

127.0.0.1 pi3 localhost localhost.local

# The following lines are desirable for IPv6 capable hosts
# (added automatically by netbase upgrade)

::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

Or maybe installing  ipv6-netfilter  would fix that too.

Nope, same result.

Guess I'll just stick with manual configuration.

[Rich]

Hi polikuo
Yes, comment out those 6  ip6-  lines.

[polikuo]

Hi polikuo
Yes, comment out those 6  ip6-  lines.

Nope, not this one.
Tried both your ideas.
If only people invented IPv6 in the first place, we won't have to deal with this.