If piCore boots to a console prompt I'd guess that the problem is more likely to be with the RPi and/or its firmware.
As said, using an RPi3 with piCore-11.x takes about 20 seconds before ssh connects whereas an RPi4 with piCore-11.x takes 5 minutes - and this is using the same sd card and same network cable.