When you run the matrix demo, the ant script (target run.demo2) actually starts 4 applications: a node, a driver, the demo client and the administration tool, each running in its own JVM.
I am afraid 256 MB is not enough memory to run the components.
If you can do without the administration GUI, you can try the following instead:
Edit the build.xml in the demo project using your favorite editor, and add the following target:
<fail status="1" message="Demo has ended normally."/>
then run ant run.demo2.noui
Let me know if that works for you.