I think you misunderstand what I mean by pi bootloader. I'm referring to a Raspberry pi bootloader binary that's 52K, that is only to get the USB device to boot. That's all that is on the SD card. Without it, there's no USB booting. It's not the piCore bootloader. That's all on the USB.
I did try pulling the SD card with the raspberry pi bootloader out once the multi colour boot screen came up, but still didn't work.
I've gone and set the OTP bit to boot from USB on my pi 3B. With the usual 5 sec waitusb it now works. There's no SD card in the device. Booting without the Raspberry pi bootloader binary on an SD card has less compatibility for USB devices. One of my sticks doesn't work without it. It's now using what's on the system chip, which is old.