Print

[sangwook]

I've got an error during the jppf job.
JPPF system's version is 3.3.7, Build number is 1180 and has been working good last 4 months until last 2 days. The system is working on daily base and the last 2 consecutive days job failed.
I've shutdowned and restarted the JPPF system. and the failed job finished normally.
The client side creates 70 threads and each thread creates JPPFClient by call 'JPPFClientPool.get();' and sends jobs simultaneously. When error occurred, looks like jppf clients were not even created all.
JPPFClientPool is a subclass of AbstractObjectPoolImpl

public class JPPFClientPool extends AbstractObjectPoolImpl<JPPFClient> {
   static Log logger = LogFactory.getLog(JPPFClientPool.class);
   
   static JPPFClientPool objectPool;
   final static String UUIDString = UUID.randomUUID().toString();

   public JPPFClient create() {
      return new JPPFClient(UUIDString);
   }
   
   public static JPPFClientPool getInstance() {
      if (objectPool == null) {
         objectPool = new JPPFClientPool();
      }
      
      return objectPool;
   }
}
 
The following is the node's log. Could any one help me understand the cause or solve this problem?
----------------------------------------------------------------------------------------------------------------------------------------------
2015-09-13 11:58:25,413 [INFO ][org.jppf.server.node.NodeExecutionManagerImpl.<init>(132)]: running 1 processing thread
2015-09-13 11:58:25,413 [INFO ][org.jppf.server.node.NodeExecutionManagerImpl.createThreadManager(159)]: Using default thread manager
2015-09-13 11:58:25,413 [INFO ][org.jppf.server.node.NodeExecutionManagerImpl.<init>(132)]: running 1 processing thread
2015-09-13 11:58:25,413 [INFO ][org.jppf.server.node.NodeExecutionManagerImpl.createThreadManager(159)]: Using default thread manager
2015-09-13 11:58:25,413 [INFO ][org.jppf.server.node.NodeExecutionManagerImpl.<init>(132)]: running 1 processing thread
2015-09-13 11:58:25,413 [INFO ][org.jppf.server.node.NodeExecutionManagerImpl.createThreadManager(159)]: Using default thread manager
2015-09-14 11:56:04,203 [ERROR][org.jppf.server.node.JPPFContainer.call(211)]: connection with class server ended, re-initializing, exception is: [object index: 1]
org.jppf.JPPFNodeReconnectionNotification: connection with class server ended, re-initializing, exception is:
   at org.jppf.classloader.AbstractJPPFClassLoaderLifeCycle.loadResource(AbstractJPPFClassLoaderLifeCycle.java:141)
   at org.jppf.classloader.AbstractJPPFClassLoader.findRemoteResources(AbstractJPPFClassLoader.java:319)
   at org.jppf.classloader.AbstractJPPFClassLoader.findResource(AbstractJPPFClassLoader.java:226)
   at java.lang.ClassLoader.getResource(Unknown Source)
   at org.jppf.classloader.AbstractJPPFClassLoader.getResource(AbstractJPPFClassLoader.java:524)
   at org.jppf.classloader.AbstractJPPFClassLoader.getResourceAsStream(AbstractJPPFClassLoader.java:260)
   at org.apache.commons.logging.LogFactory$3.run(LogFactory.java:1359)
   at java.security.AccessController.doPrivileged(Native Method)
   at org.apache.commons.logging.LogFactory.getResourceAsStream(LogFactory.java:1355)
   at org.apache.commons.logging.LogFactory.getFactory(LogFactory.java:544)
   at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:685)
   at com.fistglobal.riskcraft.market.jrml.instrument.function.Function.<clinit>(Function.java:23)
   at java.io.ObjectStreamClass.hasStaticInitializer(Native Method)
   at java.io.ObjectStreamClass.computeDefaultSUID(Unknown Source)
   at java.io.ObjectStreamClass.access$100(Unknown Source)
   at java.io.ObjectStreamClass$1.run(Unknown Source)
   at java.security.AccessController.doPrivileged(Native Method)
   at java.io.ObjectStreamClass.getSerialVersionUID(Unknown Source)
   at java.io.ObjectStreamClass.initNonProxy(Unknown Source)
   at java.io.ObjectInputStream.readNonProxyDesc(Unknown Source)
   at java.io.ObjectInputStream.readClassDesc(Unknown Source)
   at java.io.ObjectInputStream.readNonProxyDesc(Unknown Source)
   at java.io.ObjectInputStream.readClassDesc(Unknown Source)
   at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
   at java.io.ObjectInputStream.readObject0(Unknown Source)
   at java.io.ObjectInputStream.readObject(Unknown Source)
   at java.util.HashMap.readObject(Unknown Source)
   at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
   at java.lang.reflect.Method.invoke(Unknown Source)
   at java.io.ObjectStreamClass.invokeReadObject(Unknown Source)
   at java.io.ObjectInputStream.readSerialData(Unknown Source)
   at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
   at java.io.ObjectInputStream.readObject0(Unknown Source)
   at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
   at java.io.ObjectInputStream.defaultReadObject(Unknown Source)
   at com.fistglobal.riskcraft.market.jrml.instrument.AbstractInstrument.readObject(AbstractInstrument.java:355)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
   at java.lang.reflect.Method.invoke(Unknown Source)
   at java.io.ObjectStreamClass.invokeReadObject(Unknown Source)
   at java.io.ObjectInputStream.readSerialData(Unknown Source)
   at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
   at java.io.ObjectInputStream.readObject0(Unknown Source)
   at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
   at java.io.ObjectInputStream.readSerialData(Unknown Source)
   at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
   at java.io.ObjectInputStream.readObject0(Unknown Source)
   at java.io.ObjectInputStream.readObject(Unknown Source)
   at org.jppf.utils.ObjectSerializerImpl.deserialize(ObjectSerializerImpl.java:168)
   at org.jppf.utils.ObjectSerializerImpl.deserialize(ObjectSerializerImpl.java:152)
   at org.jppf.io.IOHelper.unwrappedData(IOHelper.java:226)
   at org.jppf.server.node.JPPFContainer$ObjectDeserializationTask.call(JPPFContainer.java:202)
   at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
   at java.util.concurrent.FutureTask.run(Unknown Source)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
   at java.lang.Thread.run(Unknown Source)
Caused by: java.io.EOFException
   at java.io.DataInputStream.readInt(Unknown Source)
   at org.jppf.comm.socket.AbstractSocketWrapper.readInt(AbstractSocketWrapper.java:264)
   at org.jppf.io.SocketWrapperInputSource.readInt(SocketWrapperInputSource.java:90)
   at org.jppf.io.IOHelper.readData(IOHelper.java:120)
   at org.jppf.io.IOHelper.unwrappedData(IOHelper.java:186)
   at org.jppf.classloader.RemoteResourceRequest.run(RemoteResourceRequest.java:77)
   at org.jppf.classloader.ClassLoaderRequestHandler$PeriodicTask.run(ClassLoaderRequestHandler.java:157)
   ... 1 more

[lolo]

Hello,

The log you provided shows that the connection between the node and the server was interrupted while the node was making a class loading request. The stack trace shows that this class loading request is triggered by a static initializer in the class Function, while creating a commons logging logger object (probably a statment like static Log logger = LogFactory.getLog(Function.class) ). I'm guessing that Function is used from your tasks. The stack trace also shows that the exception is occurring while deserializing a task received from the server.

Unfortunately, that doesn't tell us why the connection was broken. The EOFException in the "caused by" clause seems to indicate that the connection was closed by the server. Is there any error or exception in the server log file? Also is there anything in the client's log?

Additionally, I would like to emphasize that your use of a pool of JPPFClient instances is definitely not a recommended practice. The recommendation is to use a single JPPFClient instance for your entire application, configuring it with as many connections as needed so you can submit jobs concurrently from multiple threads. Thus, I would rewrite your JPPFClientPool as follows:

Code: [Select]

public class JPPFClientPool {
  private final static JPPFClient jppfClient = initClient();

  private static JPPFClient initClient() {
    // override some configuration properties BEFORE creating the client
    TypedProperties config = JPPFConfiguration.getProperties();
    config.setProperty("jppf.drivers", "driver1");
    config.setProperty("driver1.jppf.server.host", "my.server.address");
    config.setProperty("driver1.jppf.server.port", "11111");
    config.setProperty("driver1.jppf.pool.size", "70"); // 70 connections
    return new JPPFClient();
  }

  public static JPPFClient get() {
    return jppfClient;
  }
}
Thanks,
-Laurent