Thanks for the reply!
I do indeed use the rpi-vc package, but now I checked them and in 13.x, it contains 23 libraries (in /usr/local/lib), including libvcsm, but in 14.x it only contains libbcm_host, libdebug_sym, libdtovl, libvchiq_arm and libvcos.
I can try to copy it from 13.x to see if it still works. I don't think libvcsm should be deprecated, in contrast to MMAL I don't know what it would be replaced with.
Though seeing how in
this issue, VCSM had to be fixed for MMAL 64bit support, it does seem that VCSM and MMAL support is intrinsically linked. Which would be super unfortunate because I actually require VCSM+DMABUF even while using V4L2 instead of MMAL. So maybe it is "correct" that it's gone from aarch64, but I'm not sure about the 32bit OS yet. Maybe I'll check if the current 32Bit Rasberry Pi OS still has libvcsm.so.
EDIT: Can confirm, it doesn't, libraspberrypi installs libbcm_host.so but not mmal nor vcsm. Very unfortunate. Will try old libvcsm
As for the wireless crash, the device driver crashes and does not show up even when never connecting to any access point, so that's not it.
Boot once, let openssh create the host keys, reboot and dmesg has the crash in RSA key check, and wlan0 is not there. Maybe I am missing sha256sum or something similar as the issue suggests.
Alternatively, do you know of a way to prevent openssh from generating the RSA key on boot (or any key at all - can do manually)? Then it should respect that there's no RSA key and it should just work afaik.
Or maybe I'll just delete the RSA key after openssh starts and hope it's ok even when client wants to use RSA.