Print

[rnell@amphorainc.com]

I have some experience with a commercial compute grid product (call it 'GS').  I am considering converting the code to run on JPPF and I noticed a couple of differences that might be challenging to overcome.  I'm hoping a JPPF expert can address these issues in a way that would give me confidence that a conversion is possible.

a)  JPPF appears to need a collection of tasks to be defined and then submitted to the grid.  In GS, as tasks are created they are executed, there is no need for submit.  This allows me to have a session (Job) open all day, doing work as it arrives. Is there similar functionality in JPPF?

b) In GS the remote object is reused between Sessions.  This would be equivalent to reusing the same remote object between JPPF Jobs.  The advantage of this is that all caching, database connections, etc. that were created from one job are available to the next job.  How can i get the benefit of caching and existing DB connections between one job and the next?

c) When a node is started does it launching two processes?  One to do handle restarts, the other to run the VM that processes jobs?  Is there a way to specify that a Job requires the node's VM be restart before running the job?

[lolo]

Hello,

I am the principal developer at JPPF, thus I should qualify as expert. To answer your questions:

a) The closest thing we have is the JPPF executor service. By default, it will process each task as soon as it is submitted, which allows you to emulate a stream of tasks for as long as you need it. Optionally, you may also batch tasks together, according to size or time criteria.

b) Typically this is done via a JPPF extension/plugin, such as a node lifecycle listener or a node startup class. For more "hands-on" information, you might want to look at our node life cycle sample, which provides an example of how a database connection pool may be cached in a node an reused between jobs.

c) "When a node is started does it launching two processes?  One to do handle restarts, the other to run the VM that processes jobs?"

Yes, exactly! The restart-handling process launches the job-handling process and restarts it whenever it terminates with an exit code equal to 2.
You can remotely trigger a node restart using the JMX-based management APIs. This can be done from your client application, by connecting directly to the node, however we recommend forwarding the management request via the server, as this will allow you to reach the node even if you can't connect to it directly (when node and client are on completely separate subnets for instance).

I hope this answers your questions. Please do not hesitate to get back to us if you need additional clarification.

Sincerely,
-Laurent

[rnell@amphorainc.com]

Thank you for your response.

I'm investigating further how to keep our cache (statics) alive between jobs.  I modified the TemplateJPPFTask.java to have a static AtomicInteger counter and modified one of the methods to increment and print the counter for each call.  It seems that the static is maintained between Tasks in a Job, the counter increases as expected, but for every new Job, the static counter is re-initialized.

What is going on here under the hood?  It appears that each instance of a Job gets a new ClassLoader even though the next Job is identical.  Ideally, I'd like to reuse the same classloader if the current job and the previous job are identical, otherwise use a new classloader (drop the cache).

I see that a NodeListener can be statically referenced from a Node Task as shown in the Node Lifecycle sample.  Adopting this mechanism would require significant effort.

[lolo]

Hello,

In fact, the node creates a new class loader for each new JPPFClient instance that submits a job which gets exected on that node. We have a detailed documentation on how this works and I invite you to read further for a better understanding.

When you create a JPPFClient, you can use either the JPPFClient() noargs constructor, in which case a new client UUID is generated and will trigger the creation of a new class loader in the node(s), or you can use JPPFClient(String uuid), which will cause the node to use an existing class laoder (or create it the 1st time) if the uuid you specify remains the same between restarts of your application.

In the second case, you will lose the advantage of on-demand class loading, i.e. if you changed the code of your tasks between restarts, then the changes will not be taken into account, since the node will use the older class definitions it had already loaded. This is why this mode is mostly recommended in production environments rather than in development environements.

Thus, if you need a more persistent cache, you need to have it loaded by a class loader that is a parent (or ancestor) of the class loader associated with the client. This can be achieved by adding the classes for your tasks and jobs to the driver's classpath. In this case, the node will load these classes from the driver class loader, which is the direct parent of all client class loaders. Here again, you will lose some flexibility in terms of class loading, since you will have to deploy the classes to the driver and restart the driver after each code change.

Sincerely,
-Laurent