I just tested again.
The only extension I have onboot on this RPi5 running piCore64 17beta2 is openssl.
Boot, then: tce-load -i Xorg-3d flwm aterm wbar
sudo cp /tmp/tcloop/xorg-server/usr/local/share/xorg-server/files/99-vc4.conf /usr/local/share/X11/xorg.conf.d
startx
..and the gui starts with 3d acceleration.
Note that very occasionally the gui doesn't start first time, but starts without problems when "startx" is issued a second time.
Edit: I'm using a monitor connected to the first hdmi port.