Well for your average package, that is probably pretty easy. But X seems rather complicated. There's Xorg and XFree86, all the various extensions, etc.
I tried installing xfbdev & ran startx, it bombed in waitforX
I tried installing xorg-server & ran startx, it bombed in waitforX also.