Well, Xfce4 is using functions provided by Xorg extension, but works fine Xvesa. When Xorg loaded, it changes /etc/sysconfig/Xserver to Xorg by default. On my notebook Xorg doesn't start however, as the graphic driver (AGP, ... ) is required as a dependency but there are no erason to add it to Xfce4 dep list.
Of course it can be fixed with some backup/script/... tuning, but not plug and play and not trivial for ordinary user. Agree, Xvesa boot option is not elegant and in general I'm against boot options,but it may help.
Other solution is to split up Xorg, or not to override by default /etc/sysconfig/Xserver just because Xorg is loaded.