adequate
adequate
adequate
adequate
 

JPPF, java, parallel computing, distributed computing, grid computing, parallel, distributed, cluster, grid, cloud, open source, android, .net
JPPF

The open source
grid computing
solution

 Home   About   Features   Download   Documentation   Forums 
July 19, 2018, 07:24:16 AM *
Welcome,
Please login or register.

Login with username, password and session length
Advanced search  
News: New users, please read this message. Thank you!
  Home Help Search Login Register  
Pages: [1]   Go Down

Author Topic: Node initialization and shared objects  (Read 49 times)

mathieu

  • JPPF Padawan
  • *
  • Posts: 3
Node initialization and shared objects
« on: May 25, 2018, 07:35:45 PM »

Hi all,

I need all my nodes to initialize an object before they can process some tasks. This object is my computation engine: it is a quite big object and its initialization is costly, which is why I can't do it at the task level (it really needs to be initialized prior to the execution of any computation task). And, within each node, this object must be accessible to all incoming tasks.

Last thing (but not least) : unfortunately this initialization step depends on parameters sent by the client itself... So, what is the best way to achieve this scenario ?

Given my constraints, I thought I would be able to achieve it in the following way:
1) The client submits a broadcast job containing a single initialization task with appropriate parameters. When this task is executed on each node, the computation engine object is initialized and stored as a singleton in the node.
2) The client can now submit jobs containing computation tasks. When a node receives such a task, it can use the singleton object initialized at step 1 to achieve its computation goal.

The problem here is that my singleton is not available (null) at step 2, probably because different threads and/or different classloaders are used in steps 1 and 2.

Where am I doing wrong here ? Is there a way to keep a reference to a shared object on each individual  node ?

Thanks a lot for your help.
Logged

lolo

  • Administrator
  • JPPF Council Member
  • *****
  • Posts: 2224
    • JPPF Web site
Re: Node initialization and shared objects
« Reply #1 on: May 26, 2018, 09:47:03 AM »

Hello,

As you mentioned, the main problem here is the class loader from which the engine classes are loaded. For reference, and if you haven't already done so, I invite you to read the doucmentation on class loading, and in particlular the section on the class loader hierarchy in the nodes.

If the engine classes are loaded from a JPPF client, that is, if they are only deployed on the client side and used in a (broadcast) task, then the engine singleton will be tied to the node's client class loader, and will disappear as soon as the class loader is garbage collected. Furthermore, if another task, submitted by a separate JPPF client, attemps to use the same engine, you will face a ClassCastException, because the task class will be loaded by a different class loader in the node.

In terms of class loading mitigation, I believe you have 2 choices, each with its pros and cons:

1) You can deploy the engine classes to the node's local classpath. This means the singleton will be loaded by the node's system classloader, and it will be available no matter the node's state, in particular even when the node is disconnected from the server. The disadvantage of that is that you have the burden of deploying the engine classes onto each node. On the other hand, the engine need be initialized only once (unless the node is restarted, but that's a different thing).

2) You can deploy the engine classes to the server/driver local classpath. In this case, you only have to deploy them to a single additional location. The one drawback is that, since the JPPF class loader in the node depends on the connection to the server, the engine will need to be initialized again whenever the node gets disconnected from and then reconnects to the server.

This leads us to the second problem: how to perform the engine initialization? If we assume that the grid topology never changes (i.e. no additional node ever joins or leaves the grid), then a broadcast job will do the trick. However, if you start having a large number of nodes, then the probability of node failure gets closer and closer to 100%. JPPF nodes have failover and recovery mechanisms that will cause the node to reconnect, but then the engine instance will be lost and will have to be initialized again. The same applies to the situation where nodes join the grid arbitrarily.

Based on this, I would recommend using a solution where the engine code is deployed to the server classpath, and where the engine initialization is done lazily from the computation tasks.

I hope this makes sense.

Sincerely,
-Laurent


Logged

mathieu

  • JPPF Padawan
  • *
  • Posts: 3
Re: Node initialization and shared objects
« Reply #2 on: June 04, 2018, 01:32:35 PM »

Hi Laurent,

I wanted to thank you for your prompt and very detailed response.

I followed your recommendation and it works like a charm !

Cheers,
Mathieu.
Logged
Pages: [1]   Go Up
 
JPPF Powered by SMF 2.0 RC5 | SMF © 2006–2011, Simple Machines LLC Get JPPF at SourceForge.net. Fast, secure and Free Open Source software downloads