Print

[row_perfect]

Hi Guys,

I have a NodeLifeCycleListener; in jobStarting, I call:
final AbstractJPPFClassLoader clientCL = (AbstractJPPFClassLoader) event.getTasks().get(0).getClass().getClassLoader();

The clientCL that comes back is: JPPFClassLoader[id=1, type=server] if the task is submitted via an Executor Service vs client classloader when submitted via a normal JPPFJob.

This means I can't use that classloader to pull in libraries from the client (using similar pattern to http://www.jppf.org/samples-pack/ExtendedClassLoading/Readme.php).

Is this expected, is there a workaround?

Thanks, Paul

[row_perfect]

I figured out a solution, however this is unbelievably dirty:

final Object taskObject = event.getTasks().get(0).getTaskObject();
final Class<?> taskClass = taskObject.getClass();

final Field field = taskClass.getDeclaredField("runnable");
field.setAccessible(true);
final Object realTask = field.get(taskObject);

final AbstractJPPFClassLoader clientCL = (AbstractJPPFClassLoader) realTask.getClass().getClassLoader();

I'm guessing to avoid this blowing up for any normal job I'd have to add a flag to the job metadata to indicate it had been launched via ExecutorService.

Is there an official way to lookup the client classloader?

Thanks

[lolo]

Hi Paul,

This is definitely a design issue in the JPPFExecutorService API. I have registered a bug report for this: JPPF-48 Impossible to get the class loader of a Runnable submitted via JPPFExecutorService .

I also registered a related issue which, when fixed, should make life much easier for looking up the class loader of a task: JPPF-49 JPPFTask.getTaskObject() should not return null

I already have a code fix which I need to test a bit more, and I will publish an offical patch over the weekend. The fix will also be part of the upcoming v3.1.2 maintenance release, due end of this month.

In the meantime, the only workaround I can see would be to have your Runnable task as a subclass of JPPFTask, or use the reflection code you have already implemented.

I hope this helps.

Sincerely,
-Laurent

[row_perfect]

I'm using my own implementation of the LibraryManager pattern to improve performance. Something that struck me as very awkward is that the task cannot reference any classes that are also contained in the downloaded jars.

It feels to me that this is pretty in invasive in client code (the suggestion is to fully qualify all class names), however I'd like to pass my own parameter objects into a task, but you end up with Class X cannot be cast to X exceptions.

If I am right in thinking, the root cause of this is that the job started hook is called after the tasks have been deserialized.

I was wondering if an enhancement could be made to deserialize the job metadata, create a task classloader, then call NodeLifeCycleListener.jobStarting (or something similar) which would add local jar files to the classpath, then deserialize the tasks.

Apologies if I've mis-understood the problem.

Paul

[lolo]

Hello Paul,

You have very well understood the problem, which is that the tasks are deserialized before calling NodeLifeCycleListener.jobStarting(). A consequence of this is that, when your task has an explicit reference (e.g. via an "import" statement) to another class that is in a downloaded jar, this will cause a ClassNotFoundException at deserialization time. Using the fully qualified class name in the sample's tasks instead causes the class to be loaded when the task is executed. You could also use reflection for the same purpose. I agree this is cumbersome and a little akward.

What I would like to propose is to add a notification method like jobHeaderLoaded(NodeLifeCycleEvent) and add a getTaskClassLoader() method to the event class. The new notification will be sent right after the job header (including the JobMetadata) is deserialized and before the tasks are deserialized. Also, since I'm not very keen on breaking the API with regards to existing applications that use NodeLifeCycleListener, I would prefer to create a new interface that would extend it, e.g.:

Code: [Select]

public interface NodeLifeCycleListenerEx extends NodeLifeCycleListener {
  public void jobHeaderLoaded(NodeLifeCycleEvent event);
}
This would allow us to deprecate NodeLifeCycleListener in a first time, while keeping its functionality intact, then eventually remove it in a later version. I've summarized this in a new enhancement request: JPPF-55 Extensions to NodeLifeCycleListener. Is this something that would work for you?

-Laurent

[row_perfect]

That sounds perfect, thanks!

I may be abusing the NodeLifeCycleListener.jobStarting() method, but what I want to achieve is the concept of a job context.

Basically, every task for a given job id needs to write its results to the same socket (not sending via client for performance reasons) and requires some other common state set up, so I've created a DataProvider  inside the lifecycle listener (created as ExecutorService doesn't allow one in 3.1.1):

final MemoryMapDataProvider dataProvider = new MemoryMapDataProvider();

// creates connection, also adds a few other shared items
initialiseDataProvider(config, dataProvider);

for (final Task task : event.getTasks()) {
   if (task.getDataProvider() == null) {
      task.setDataProvider(dataProvider);
   } else {
      initialiseDataProvider(config, task.getDataProvider());
   }
}

This works, but it's a bit ugly having to iterate through the tasks setting the DataProvider.

The other problem is that my NodeLifeCycleListener implementation does work that would be much better done inside a class provided by the client, not deployed as part of the grid - it breaks the purpose of the clever classloading you've provided.

Is there a better way of achieving this?

Thanks, Paul

BTW once I fixed the client/server load balancing config, performance is phenomenal - I need to make task submission parallel as I can't saturate the grid.

[lolo]

Hello Paul,

Currently, I don't see any better way of initializing the data provider, given the  limitations we've discused for jobs submitted via the executor service.

that would be much better done inside a class provided by the client, not deployed as part of the grid

I apologize, I do not understand this statement. Would you mind elaborating?

I may be abusing the NodeLifeCycleListener.jobStarting() method

I certainly hope you are!  The JPPF APIs are supposed to provide as much freedom as possible, and your contributions so far in that area are invaluable and very much appreciated.

BTW, I'm not sure if you have seen it, we have released JPPF 3.1.2 which includes fixes for the bugs JPPF-48 and JPPF-49 discussed earlier.

-Laurent

[row_perfect]

that would be much better done inside a class provided by the client, not deployed as part of the grid

I apologize, I do not understand this statement. Would you mind elaborating?

My point was simply, that the job 'context' I want to create is related to the job I'm running, not the node. Under my current implementation (in lifecycle listener), if I want to change that, I have to change the jars on the driver, and bounce everything.

[row_perfect]

Basically, for my use case, it would be really handy if:

A job initialiser class could be attached to the job. This should be called on jobStart/jobEnd for a given batch of tasks*. It should be given access to the job metadata and DataProvider.

Does that make sense?

*it might be even better if there was another method, say jobChanged, called if the node takes a batch of tasks from a completely different job.

[lolo]

Ok, I believe I understand what you want.
I believe this will be achievable once you have access to the DataProvider from the NodeLifeCycleListener, without needing to add anything else to the listener interface itself.
You will then be able to add your job initializer/finalizer to the data provider (loaded from the client classpath) and process it from jobStarting() and jobEnding().
Similarly, detecting a job change can be easily done in your NodeLifeCycleListener implementation, for instance by storing the last job uuid and comparing with the new one at jobStarting() time.

We're trying to keep the APIs as simple as possible, our goal being to provide the tools to resolve problems, rather than trying to resolve all possible problems.

Sincerely,
-Laurent

[row_perfect]

You will then be able to add your job initializer/finalizer to the data provider (loaded from the client classpath) and process it from jobStarting() and jobEnding().
Similarly, detecting a job change can be easily done in your NodeLifeCycleListener implementation, for instance by storing the last job uuid and comparing with the new one at jobStarting() time.

That's very cunning.

Have you got a target release for Enhancement JPPF-50?

Thanks.