By default 64 Mbyte is allocated to the GPU which is not needed if you are running console only or accessing it via SSH remotely without graphics. Having a 256 MByte board it is a huge loss. Just add this line to your config.txt in the boot to give 16 MByte to video:
gpu_mem=16
and win 48 MByte RAM.