Elastic Search using a lot of memory despite low heap size

We have configured Elastic Search JVM options /etc/elasticsearch/jvm.options to use a fixed heap size of -Xms512m -Xmx512m

This can be confirmed as working when checking the Elastic Search process:

However you'll notice that it's still using 835M of memory.

If we check the actual used heap size with ES:

root: curl -sS -XGET "localhost:9200/_cat/nodes?h=heap*&v"
heap.current heap.percent heap.max
     276.1mb           55  494.9mb

It's actually only using 276MB of the heap - so where is the other 559MB being used exactly? We're concerned it's causing a lot of swap/paging issues, particularly as the VIRT column has a size of 4GB (but I'm told not to worry about that column, though it does seem a strangely high number to me)

We've also disabled swapping for Elastic Search with the mlockall option, which can be confirmed with:

root:~# curl -s localhost:9200/_nodes?pretty|grep mlock
     "mlockall" : true

However it seems it's STILL swapping:

for file in /proc/*/status ; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done | sort -k 2 -n -r | less
java 128592 kB

So my questions are:

1. What can we do to reduce the size of memory consumed by Elastic Search to ensure it's fixed at 512MB or as close to that as possible?
2. Ensure swapping is disabled
3. Reduce size of the VIRT column (or is this really not worth worrying about?)