KVM works fine in such a setup. 64-bit kernel and 32-bit userspace is a bit rare combo, but it's supported on Debian, and most stuff works. Notably virtualbox and nvidia/amd binary drivers don't.
One VM would still be restricted to max 4gb ram, but you can then run several in the total 8gb.