JPPF, java, parallel computing, distributed computing, grid computing, parallel, distributed, cluster, grid, cloud, open source, android, .net
JPPF

The open source
grid computing
solution

 Home   About   Features   Download   Documentation   On Github   Forums 
December 13, 2019, 03:30:35 AM *
Welcome,
Please login or register.

Login with username, password and session length
Advanced search  
News: New users, please read this message. Thank you!
  Home Help Search Login Register  
Pages: [1]   Go Down

Author Topic: setting the queue size in jppfdriver  (Read 3655 times)

kenneththo

  • JPPF Padawan
  • *
  • Posts: 8
setting the queue size in jppfdriver
« on: April 07, 2011, 02:33:42 AM »

hi,

I couldn't find that information from the doc, but I am just wondering if the submitted jobs will be first put into a task queue in the driver before it gets distributed to the nodes. If so, how do we configure the queue size such that it won't grow indefinitely? 

In fact, what i am trying to do is that, if all the nodes are currently busy, I want the job submission to fail at the client side, is that possible to do?
in this case, the queue size would be 1.

thanks,
ken
Logged

lolo

  • Administrator
  • JPPF Council Member
  • *****
  • Posts: 2258
    • JPPF Web site
Re: setting the queue size in jppfdriver
« Reply #1 on: April 07, 2011, 12:52:56 PM »

Hi Ken,

Unfortunately, it is currently not possible to limit the job queue size. The current strategy with regards to the risk of indefinite growth of the job queue, is that the JPPF driver has a disk overflow mechanism, which will store jobs on the local file system when the available memory becomes sparse. Only a minimal header will be kept in memory (enough to handle the job without problem).

It is possible to know how many jobs are currently in the server by using the management APIs. We have actually in the documentation a code example that does exactly that, you will find it there.
This gives you the uuids of all currently submitted jobs, including their number. Incidentally, I just realized there is an error in that code :( the last line should read "String[] jobIds = ..." instead of "String jobIds = ...".

To make it easier in your application code, you can modify it as follows:

Code: [Select]
private DriverJobManagementMBean proxy;
public int getNumberOfJobs(JPPFClient client) {
  JMXDriverConnectionWrapper jmx =
    ((JPPFClientConnectionImpl) client.getClientConnection()).getJmxConnection()
  // create a proxy to the job management MBean
  if (proxy == null)
    proxy = jmx.getProxy(JPPFAdminMBean.DRIVER_JOB_MANAGEMENT_MBEAN_NAME, DriverJobManagementMBean.class);
  // get the ids of all jobs in the server queue
  String[] jobIds = proxy.getAllJobIds();
  return jobIds.length;
}

You may also compare it with the number of nodes dynamically. You can fetch the number of nodes via the management APIs as well:
Code: [Select]
public int getNumberOfNodes(JPPFClient client) {
  JMXDriverConnectionWrapper jmx =
    ((JPPFClientConnectionImpl) client.getClientConnection()).getJmxConnection()
  JPPFStats stats = jmx.statistics();
  return stats.getNbNodes();
}

Then in your application code, you could use the following to determine whether a job should be submitted:
Code: [Select]
JPPFClient = ...;
if (getNumberOfJobs(client) >= getNumberOfNodes(client)) {
  // failure, do not submit the job
}
else {
  // we can submit the job
}

I hope this helps.

Sincerely,
-Laurent
Logged

kenneththo

  • JPPF Padawan
  • *
  • Posts: 8
Re: setting the queue size in jppfdriver
« Reply #2 on: April 08, 2011, 07:09:28 PM »

Thanks Laurent! that works great!

 
Logged

kenneththo

  • JPPF Padawan
  • *
  • Posts: 8
Re: setting the queue size in jppfdriver
« Reply #3 on: April 08, 2011, 11:24:36 PM »

in that sense, if i need to support throttling, whats the best way to do it?

Logged

codemonkey

  • JPPF Council Member
  • *****
  • Posts: 138
Re: setting the queue size in jppfdriver
« Reply #4 on: April 12, 2011, 01:43:01 AM »

Hello Laurent, JPPFClientConnectionImpl automatically connects to the server correct (no need to do a .connect())? In the case here, the proxy here be holding a connection to the server? Should the jmx call be inside the the if statement, or is that call always needed because the connection will close because of the scope?

My current implementation is that I have JMXDriverConnectionWrapper member that connects/closes and releases every time I want to make a call to the MBean Server. Is there a more efficient way without chance of causing connection leaks? Possibly a singleton with the connection open throughout the app life?

Code: [Select]
private DriverJobManagementMBean proxy;
public int getNumberOfJobs(JPPFClient client) {
  // create a proxy to the job management MBean
  if (proxy == null){
//shouldn't this be here?
JMXDriverConnectionWrapper jmx =
    ((JPPFClientConnectionImpl) client.getClientConnection()).getJmxConnection()
    proxy = jmx.getProxy(JPPFAdminMBean.DRIVER_JOB_MANAGEMENT_MBEAN_NAME, DriverJobManagementMBean.class);
}
  // get the ids of all jobs in the server queue
  String[] jobIds = proxy.getAllJobIds();
  return jobIds.length;
}

As well, I find using the getJmxConnection() suggested above then after getting the proxy reference eventually returns an error when you call proxy.getAllJobIds(), stating the client has been closed,

Code: [Select]
ERROR [2011-04-11 20:45:46,272] The client has been closed.
java.io.IOException: The client has been closed.
at com.sun.jmx.remote.internal.ClientCommunicatorAdmin.restart(ClientCommunicatorAdmin.java:74)
at com.sun.jmx.remote.internal.ClientCommunicatorAdmin.gotIOException(ClientCommunicatorAdmin.java:34)
at javax.management.remote.rmi.RMIConnector$RMIClientCommunicatorAdmin.gotIOException(RMIConnector.java:1437)
at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.getAttribute(RMIConnector.java:882)
at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:263)
at $Proxy34.getAllJobIds(Unknown Source)


Thank you very much for your feedback
« Last Edit: April 12, 2011, 02:56:23 AM by codemonkey »
Logged

lolo

  • Administrator
  • JPPF Council Member
  • *****
  • Posts: 2258
    • JPPF Web site
Re: setting the queue size in jppfdriver
« Reply #5 on: April 13, 2011, 06:21:25 AM »

Hi,

Yes you are right, the MBean proxy is created from an existing open connection to the MBean server, so the proxy creation should be dependent on the state of the connection. Thank you for pointing this out.

In fact, when you get the JMXDriverConnectionWrapper instance via a call to ((JPPFClientConnectionImpl) client.getClientConnection()).getJmxConnection(), this instance will be closed as soon as you close the JPPFClient. If you want a JMX connection that lives beyond the JPPFClient, you can either create a new instance:
Code: [Select]
JMXDriverConnectionWrapper jmx = new JMXDriverConnectionWrapper(server_host, jmxPort);
jmx.connect();

or you can check whether it was disconnected:
Code: [Select]
JMXDriverConnectionWrapper jmx =
    ((JPPFClientConnectionImpl) client.getClientConnection()).getJmxConnection();
if (!jmx.isConnected())
{
  jmx.connecAndWait(1000L);
  proxy = jmx.getProxy(JPPFAdminMBean.DRIVER_JOB_MANAGEMENT_MBEAN_NAME, DriverJobManagementMBean.class);
}

Sincerely,
-Laurent
Logged
Pages: [1]   Go Up
 
JPPF Powered by SMF 2.0 RC5 | SMF © 2006–2011, Simple Machines LLC Get JPPF at SourceForge.net. Fast, secure and Free Open Source software downloads