Print

[atzt]

Hello,

First let me describe my program workflow a little bit. My program listens for a message on a message queue and once it finds one executes a function and goes back to listening to the message queue. It is possible that the program waits for hours or days before it gets another message. So during this function I create a JPPF client object and submit several jobs and work with the results.

So I have been attempting to create a client-side failover configuration with just a primary and secondary driver. Both of my drivers have been added to the client configuration manually where the primary driver has a priority of 10 and the secondary driver has a priority of 5. I have also setup a maximum reconnect time of 10 seconds.

So to test my failover configuration I have started up just the secondary driver and executed a job. The client properly tries for 10 seconds to connect to my primary server and then abandons that connection for my secondary driver. The secondary driver connects immediately and succeeds in executing the job. The jppf client object is now closed but my program will continue to run waiting for the next message on the message queue.

At this point I can see in both eclipse and visualVM that there is a thread that is running named JMX connection PRIMARY_SERVER:11198. And for every time I run my "function" in my program I get one leftover JMX thread for the primary server. Therefore if I receive 10 messages, then I will have 10 JMX leftover threads once the last function completes.

And one more interesting part is that I have turned off JPPF management in my client so I would not think that a JMX connection would ever be needed (unless there are internal ones doing something behind the scenes).

My client log information is the following:
2013-02-25 15:04:24,946 INFO  AbstractGenericClient - connection [secondary] created
[primary - ClassServer - delegate] SocketInitializer.initializeSocket(): Could not reconnect to the remote server2013-02-25 15:04:34,890 INFO  AbstractGenericClient - Connection [primary] failed

2013-02-25 15:04:34,893 ERROR JPPFClientConnectionImpl - [primary - ClassServer] Could not reconnect to the class server
org.jppf.JPPFException: [primary - ClassServer] Could not reconnect to the class server
   at org.jppf.client.ClassServerDelegateImpl.init(ClassServerDelegateImpl.java:84)
   at org.jppf.client.JPPFClientConnectionImpl.connect(JPPFClientConnectionImpl.java:138)
   at org.jppf.client.JPPFClientConnectionImpl.init(JPPFClientConnectionImpl.java:110)
   at org.jppf.client.ConnectionInitializer.run(ConnectionInitializer.java:58)
   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
   at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
   at java.util.concurrent.FutureTask.run(FutureTask.java:138)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:619)
[client: secondary - ClassServer] Attempting connection to the class server at SECONDARY_SERVER:XXXXX
2013-02-25 15:04:34,894 INFO  ClassServerDelegateImpl - [client: secondary - ClassServer] Attempting connection to the class server at SECONDARY_SERVER:XXXXX
[client: secondary - ClassServer] Reconnected to the class server
2013-02-25 15:04:34,908 INFO  ClassServerDelegateImpl - [client: secondary - ClassServer] Reconnected to the class server
[client: secondary - TasksServer] Attempting connection to the JPPF task server at SECONDARY_SERVER:XXXXX
[client: secondary - TasksServer] Reconnected to the JPPF task server

Here is my thread dump from visualVM
"JMX connection PRIMARY_SERVER:11198" daemon prio=6 tid=0x1a0ff400 nid=0x1f9c runnable [0x1a2ff000]
   java.lang.Thread.State: RUNNABLE
   at java.net.PlainSocketImpl.socketConnect(Native Method)
   at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
   - locked <0x049e7008> (a java.net.SocksSocketImpl)
   at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
   at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
   at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
   at java.net.Socket.connect(Socket.java:525)
   at java.net.Socket.connect(Socket.java:475)
   at java.net.Socket.<init>(Socket.java:372)
   at java.net.Socket.<init>(Socket.java:186)
   at com.sun.jmx.remote.socket.SocketConnection.connect(SocketConnection.java:94)
   - locked <0x049e6630> (a [I)
   at com.sun.jmx.remote.generic.ClientSynchroMessageConnectionImpl.connect(ClientSynchroMessageConnectionImpl.java:70)
   - locked <0x049e68b0> (a [I)
   at javax.management.remote.generic.GenericConnector.connect(GenericConnector.java:177)
   - locked <0x049e63c0> (a [I)
   at javax.management.remote.jmxmp.JMXMPConnector.connect(JMXMPConnector.java:119)
   at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:248)
   at org.jppf.management.JMXConnectionWrapper.performConnection(JMXConnectionWrapper.java:226)
   - locked <0x0a2a6200> (a java.lang.Object)
   at org.jppf.management.JMXConnectionThread.run(JMXConnectionThread.java:81)
   at java.lang.Thread.run(Thread.java:619)

   Locked ownable synchronizers:
   - None

I am running JPPF 3.2.1
The client configurations I have set are below:

jppf.drivers = primary secondary

primary.jppf.server.host = PRIMARY_SERVER
primary.jppf.server.port = XXXXX
primary.priority = 10
primary.jppf.pool.size = 1

secondary.jppf.server.host = SECONDARY_SERVER.com
secondary.jppf.server.port = XXXXX
secondary.priority = 5
secondary.jppf.pool.size = 1

jppf.management.enabled = false

jppf.remote.execution.enabled = true
jppf.local.execution.enabled = false

jppf.discovery.enabled = false
reconnect.max.time = 10

Any help would be appreciated. I saw the post about JMX connections leaking in the driver that was fixed in a recent patch so maybe this is related.

[lolo]

Hello,

This is indeed an issue in the JPPF client code. I registered abug report for this: JPPF-131 JPPF client does not release JMX thread upon connection failure.

A fix for this will be included in the next maintenance release v3.2.3. In the meantime, there is an easy workaround that you can use, which is to close the JMX connection (and terminate the associated thread) from a ClientListener as in the following code:

Code: [Select]

JPPFClient client = new JPPFClient();
client.addClientListener(new ClientListener() {
  @Override
  public void newConnection(final ClientEvent event) {
  }
  @Override
  public void connectionFailed(final ClientEvent event) {
    AbstractJPPFClientConnection c = (AbstractJPPFClientConnection) event.getConnection();
    if (c.getJmxConnection() != null) {
      try {
        c.getJmxConnection().close();
      } catch (Exception e) {
        e.printStackTrace();
      }
    }
  }
});
Sincerely,
-Laurent

[atzt]

Excellent, that did the trick!

Thanks a bunch
- Zak