For me using the evdev mouse driver caused the mouse to not function at all. Also, installing the guest additions from the ISO provided by the host app resulted in the same Xorg failure as the additions built from the OSE source, which make sense because there is no difference in what is installed, only the install method.
I am working on a new streamlined build script that only builds the additions which should cut the extension making time in third. I will post the script here once it is complete if it produces the same results, otherwise it will be in the src dir of the repo.