You can use ps aux to see the memory usage of each process.
I use Fedora with xfce4 in dom0, it also doesn’t use more than 800 MB, you have to running something that uses the extra memory.
You probably have other apps running alongside GNOME, or your using a VM. My Windows setup idles at 1.5 GB RAM.
