I've been following this threead because of a similar issue, but the funny thing is xine-vesa works great while xine-ui did not. when opening xine at the terminal there is a similar error but the apps runs perfectly.
Am not using xorg though so obviously can't find the xorg.conf.vesa file. If xine-ui needs to be run in an xorg environment (I too had great trouble running it), why no try xine-vesa which will run fine in the default vesa environment??