The kernel headers extension functions as a kind of short cut to avoid downloading and preparing the full kernel source in order to compile a kernel module.
Without the 64bit kernel headers you will need the full kernel source and to prepare with with the 64bit config and Module.symvers:
$ make mrproper
$ cp config-KERNEL .config
$ make oldconfig
$ make prepare
$ make modules_prepare
$ cp Module.symvers-KERNEL Module.symvers
$ make SUBDIRS=scripts/mod