It has a huge amount of libs vs 1 for Xlib. And requires python in the building phase. Also, it needs to be built before other X libs (it's a replacement for Xlib), an extension like that might cause conflicts.
Smaller - not really, as those 340kb come in addition to the 1.1mb of libX11, not in place of.
It was used before, in the very first Xorg extension, but decided not to be worth it back then.