One more update to this topic - thought it might be helpful for others if I post the whole process, through creating the Tiny Core extension (tcz):
I ended up needing to get everything working using HDMI and couldn't figure out how to get OSS to use the HDMI audio channel so I compiled it with ALSA support in addition to VDPAU.
I also ended up just following the extension creation wiki (
http://wiki.tinycorelinux.net/wiki:creating_extensions), minus the export flags step. (I tried it first using the export settings but MPlayer gave me a warning that it likes to set its own flags, and the resulting extension didn't work. Leaving them all out seems to work for me, but maybe someone more knowledgeable could chime in on if some of the flags could still be set for better compatibility.)
- Boot TinyCore 4.7.4 CD and exit to prompt.
- tce-load -wi Xorg-7.5 nvidia-glx alsa compiletc svn git yasm squashfs-tools-4.x Xorg-7.5-dev libvdpau-dev alsa-dev
- sudo Xorg -configure
- sudo mv /root/xorg.conf.new /etc/X11/xorg.conf
- sudo nvidia-xconfig
- startx and verify glxgears works.
- wget http://www.mplayerhq.hu/MPlayer/releases/codecs/all-20110131.tar.bz2
- tar xjvf all-20110131.tar.bz2
- sudo mkdir /usr/local/lib/codecs
- sudo cp -v all-20110131/* /usr/local/lib/codecs
- svn checkout svn://svn.mplayerhq.hu/mplayer/trunk mplayer
- cd mplayer
- git clone git://source.ffmpeg.org/ffmpeg.git ffmpeg (OPTIONAL)
- ./configure --prefix=/usr/local --enable-vdpau --enable-alsa
- make -j3
- make DESTDIR=/tmp/mplayer install
- cd /tmp
- mksquashfs mplayer mplayer-xorg-vdpau-alsa.tcz
Using that extension I can now play smooth, high bitrate (40Mbps), 1080p video on my Zotec's HDMI port by invoking mplayer like this: mplayer -vc ffmpeg12vdpau,ffh264vdpau, -ao alsa:device=hdmi=NVidia.1 bluray_test.mp4
Note #1: I don't think the -ao option would be needed if you set HDMI out as the default in an /etc/asound.conf file.
Note #2: The downloaded codecs and ffmpeg are only compile dependencies - to use the new extension you need Xorg-7.5, nvidia-glx, alsa, and libvdpau installed.
fodder