JPPF, java, parallel computing, distributed computing, grid computing, parallel, distributed, cluster, grid, cloud, open source, android, .net

The open source
grid computing

 Home   About   Features   Download   Documentation   Forums 
September 21, 2018, 02:08:20 AM *
Please login or register.

Login with username, password and session length
Advanced search  
News: New users, please read this message. Thank you!
  Home Help Search Login Register  
Pages: [1]   Go Down

Author Topic: Reshuffle JPPF queue  (Read 209 times)


  • JPPF Padawan
  • *
  • Posts: 7
Reshuffle JPPF queue
« on: May 29, 2017, 11:51:32 AM »


I have a requirement where we keep consuming few events from Kafka queue and we group these events and submit this micro-batch as a JPPF Job. These events comes from multiple clients. These jobs will stay in the JPPF queue and gets executed in serial manner. Now, my question is, is there a way to reshuffle this queue?

As in, let's say, initially 100 jobs of one particular customer (Customer A)has been submitted. After sometime, only one job of another customer has come (Customer B). We don't want the CustomerB to wait until all the 100 jobs of CustomerA are completed? Is there a way to achieve that? I know JPPF queues are priority queues but it is not sufficient for us as we need to share resources among the customers. If only one customer related microbatches are executing, we want all the resources to be allotted to that customer itself. But, if some other customers job comes, we want to reshuffle the queue by pushing this new micro-batch UP in the JPPF queue.

Pradeep Jaini.


  • Administrator
  • JPPF Council Member
  • *****
  • Posts: 2240
    • JPPF Web site
Re: Reshuffle JPPF queue
« Reply #1 on: May 31, 2017, 08:55:28 AM »

Hello Pradeep,

It is not clear to me in which queue the job for customer B is waiting: is it in the JPPF client queue or in the JPPF driver queue? Depending on which onei is, you will need a different approach.

1) Job waiting in the client queue
This generally happens when the client has more jobs than it can send concurrently. In a JPPF client, the maximum number of concurrent jobs is equal to the number of connections the client has with the server. Therefore, the strategy here is to adjust the size of the client's connection pool(s). This can be done either statically in the configuration, or dynamically via the JPPFConnectionPool.setSize() api. You can even automatically adjust the pool size based on the client queue size, as shown in the example for the client job queue notifications.

2) Job in the driver queue
In this situation, the only available way to push the job up the queue is by increasing its priority. Note that the job priority can be changed dynamically, even after the job was submitted to the server, via the JMX APIs. For instance:
Code: [Select]
JPPFJob job = ...;
JPPFClient client = ...;
// obtain a working JMX connection to the driver
JMXDriverConnectionWrapper jmx = client.awaitWorkingConnectionPool().awaitWorkingJMXConnection();
// get a proxy to the job management MBean
DriverJobManagementMBean jobManager = jmx.getJobManager();
// set new priority to 10 on the server side
jobManager.updatePriority(job.getUuid(), 10);

Another approach could be to adjust the number of nodes, using the node provisioning facility, to decrease the time the jobs remain in the server queue.



  • JPPF Padawan
  • *
  • Posts: 7
Re: Reshuffle JPPF queue
« Reply #2 on: June 06, 2017, 08:34:22 AM »

Hi Laurent,

Thanks a lot for the info. I was looking for the second option. The solution you provided should work for us.

Pradeep Jaini.
Pages: [1]   Go Up
JPPF Powered by SMF 2.0 RC5 | SMF © 2006–2011, Simple Machines LLC Get JPPF at Fast, secure and Free Open Source software downloads