JPPF, java, parallel computing, distributed computing, grid computing, parallel, distributed, cluster, grid, cloud, open source, android, .net
JPPF

The open source
grid computing
solution

 Home   About   Features   Download   Documentation   Forums 
October 21, 2019, 02:36:43 AM *
Welcome,
Please login or register.

Login with username, password and session length
Advanced search  
News: New users, please read this message. Thank you!
  Home Help Search Login Register  
Pages: [1]   Go Down

Author Topic: heap size settings  (Read 473 times)

broiyan

  • JPPF Grand Master
  • ****
  • Posts: 54
heap size settings
« on: February 02, 2017, 03:44:26 AM »

There are the following defaults for the driver

jppf.jvm.options = -Xmx256m

and for the node

jppf.jvm.options = -Xmx128m

Do the memory requirements of the application affect what should be chosen for these heap size values?
Logged

lolo

  • Administrator
  • JPPF Council Member
  • *****
  • Posts: 2256
    • JPPF Web site
Re: heap size settings
« Reply #1 on: February 03, 2017, 06:31:31 AM »

Hello,

Quote
Do the memory requirements of the application affect what should be chosen for these heap size values?

Yes, if you consider the tasks as part of the application. As a general rule, the tasks are the greatest memory consumers in a JPPF grid. The tasks being executed in nodes, this is where they have the most impact on heap sizing. Some elements to consider for sizing:

- a task will have a different footprint before, during and after its execution. It is best to consider their peak footprint to avoid out of memory conditions
- multiple tasks can be sent to each node at once, therefore their footrprints will add up. In this regard, load-balancing can be used as an effectiive way to limit the number of tasks sent to the nodes, and consequently to exert some measure of control over the peak memory consumption
- when receiving tasks and sending back results, a node respectively deserializes then serializes them, which creates a significant, albeit temporary, additional pressure on memory usage. A very quick approximation is that while a task and its serialized form coexist in the JVM heap, the task's footprint is doubled

The driver has different constraints. In particular, it doesn't need to deserialize the tasks, it only needs the jobs SLA, metadata and header information. The driver leverages this with a built-in mechanism that enables it to offload the tasks to disk when their memory footprint grows over a certain threshold. Of course, the drawback is that offloading tasks comes with a considerable slowdown of their processing.

Additionally, the driver is designed to work with many clients and many nodes, so it is always recommended to size it with as much memory as possible.

I hope this answersq your question.

Sincerely,
-Laurent
Logged

broiyan

  • JPPF Grand Master
  • ****
  • Posts: 54
Re: heap size settings
« Reply #2 on: February 03, 2017, 01:19:39 PM »

Yes, that's helpful. It's was not obvious to me what needed a high or non-default -Xmx setting.

The node property jppf.jvm.options also needs -enableassertions if the application needs it. Of course the invocation of the JVM for the application also needs it but that's not sufficient. Apparently jppf.jvm.options for the driver does NOT need to have -enableassertions. This was not obvious to me. It's probably valuable to know where the option is not needed just to maintain simplicity.

I don't know if this problem is reported in log files but there was nothing printed to standard output that helped me find this bug. (I forgot about the log files since it has been many months since I used JPPF.)

Perhaps I will suggest a "not obvious" FAQ and Answers page drawn from my previous questions on this forum. I will try to produce it if you would like.

Regards,
Brian
Logged
Pages: [1]   Go Up
 
JPPF Powered by SMF 2.0 RC5 | SMF © 2006–2011, Simple Machines LLC Get JPPF at SourceForge.net. Fast, secure and Free Open Source software downloads