Print

[dbwiddis]

I've spent the last week optimizing caches and hunting down various memory leaks, and in the process have stumbled on a desire for a functionality that I'm not sure is available.  Or if it is, I'm not sure how to easily access it.  Specifically, I'd like the ability to run code on the nodes in between tasks.  Essentially, define a "mini-task" that always executes upon completion of a JPPFTask/AbstractTask, in between reporting completion of one task and making itself available for the next one.

Because some of my task code uses singleton objects, they remain in the ClassLoader between tasks, and I'd like the ability to do things to them, such as manually clearing my own cache objects, cleaning up connection pools, or executing garbage collection.

Is this an existing functionality? How hard would it be to add it?

[lolo]

Hello,

First, I'd like to fully understand what you're asking.
Are you talking about something that would execute 1) after each individual task, or 2) after a set of tasks is completed and before a new one is received from the server?

In case 1), this is something you can easily accomplish by writing your own abstract task and making all other tasks inherit from it, for instance following this pattern:

Code: [Select]

public abstract class HousekeepingTask<T> extends AbstractTask<T> {
  @Override
  public final void run() {
    setup();    // before executing the task
    execute();  // actually execute
    teardown(); // cleanup after execution
  }

  // replaces the run() method in subclasses
  public abstract void execute();

  private void setup() { ... }

  private void teardown() { ... }

}
In case 2), you might want to try one of the jobXXX() methods in a NodeLifeCycleListener. I know it's not exactly ideal, since in all of these methods, at least a part of the job has already been received from the server, or the full cleanup hasn't yet occurred because the job still has to be sent back to the server.

What I can propose here, is to add another method to the node listener, say "beforeNextJob()", which would be called after a job is sent back to the server and before a new one is read from the server.
As usual this would be done in multiple stages:
- for v5.0.1, the method would be in a separate interface (e.g. "NodeLifeCycleListenerEx") which your listener classes would have to also implement. NodeLifeCycleListenerAdapter would implement it. The node would automatically recognize listeners that implement this interface and call the new method. This should should allow access to the functionality without breaking any user code.
- for v5.1, we would have NodeLifeCycleListener extend NodeLifeCycleListenerEx. This may break some user implementations unless they use the adapter class, but I estimate it is permissible for a new minor version
- for v6.0 (next major version) the new method would be integrated into NodeLifeCycleListener, and NodeLifeCycleListenerEx would be deprecated or simply removed.

What do you think?

-Laurent

[dbwiddis]

Are you talking about something that would execute 1) after each individual task, or 2) after a set of tasks is completed and before a new one is received from the server?

I apologize for not being clear!  I forget that my choice to only process one task at a time is not the standard.

The Case 1 solution you propose doesn't differ much from simply including steps at the beginning or end of my exisiting tasks.  Let me try to be more clear of the intended use of this. 

There are a few bits of code I can envision being run on an idle node, either cleaning up unneeded objects from singleton instances in the classloader, or synchronizing a cache.  These can obviously be done at the beginning of the next task (which is my current implementation), but my goal is to take advantage of the few seconds that a node sits idle between returning a set of tasks and receiving the next dispatched set of tasks so that the nodes don't have to wait to perform these housecleaning steps before getting to work.

you might want to try one of the jobXXX() methods

Since I tend to have a small number of jobs but a large number of tasks, this probably won't work.

What I can propose here, is to add another method to the node listener, say "beforeNextJob()", which would be called after a job is sent back to the server and before a new one is read from the server.

I'm not sure this is a good idea, as the listeners suffer from network delays, so by the time I listened for this notification on the client and responded to it, the driver would have already beat me to the punch by sending the next job.   And wouldn't it simply duplicate the functionality of the JobListener's jobReturned() method?

The ideal solution for me would be to have a place to tell the node (or all nodes, really) what code I want it to execute immediately after firing the jobReturned event, or wherever in your code you have the node doing the "deferred until idle" actions such as rebooting or shutting down.  This woud just be a "execute this command, deferred until idle" step.

[lolo]

[...] as the listeners suffer from network delays, [...] wouldn't it simply duplicate the functionality of the JobListener's jobReturned() method?

I think there is a slight misunderstanding here. I'm talking about node life cycle events, which are emitted and listened to in the nodes exclusively, without any network or JMX call involved. The listener callback method I'm proposing to add would be invoked right after the node has finished sending the results back to the server, and before it makes a network read() to fetch a new job. The previous method of the life cycle before that is jobEnding(NodeLifeCycleEvent), and between the two the node serializes the tasks and sends them back to the server via the network channel (+ some small cleanup and statistics computations).

In terms of global timeline, this would roughly happen in parallel with the jobReturned events which are emitted by the JPPF driver.

I hope this clarifies,
-Laurent

[dbwiddis]

Aha, I get it now. I was confusing myself by thinking the Listener would be in the client.

Now that I see how it would be implemented, I think making it (and possibly other events on the node) part of the node life cycle is probably the best solution.

It's certainly not an urgent request (we're talking about shaving a few seconds off a process that takes several minutes, compared to my current "execute when the task starts" version) and I'd be content to wait until 6.0 for this feature rather than hack in a quick band-aid.   I was more interested in seeing if the functionality already existed.

[lolo]

Thanks! I registered this feature request for JPPF 5.1: JPPF-378 Ability to perform house keeeping between jobs in the node.