JPPF, java, parallel computing, distributed computing, grid computing, parallel, distributed, cluster, grid, cloud, open source, android, .net
JPPF

The open source
grid computing
solution

 Home   About   Features   Download   Documentation   On Github   Forums 
August 11, 2020, 08:43:22 PM *
Welcome,
Please login or register.

Login with username, password and session length
Advanced search  
News: New users, please read this message. Thank you!
  Home Help Search Login Register  
Pages: [1]   Go Down

Author Topic: Proportional allocation of nodes based on priority  (Read 1965 times)

row_perfect

  • Guest
Proportional allocation of nodes based on priority
« on: March 28, 2013, 04:26:36 PM »

Hi Guys,

As far as I can tell priorities are such that n has absolute priority over n-1, i.e.

I've got a scenario where it would be useful to specify that n is assigned a relatively higher proportion of nodes than n-1.

In my case I have a homogeneous grid so I don't have to be concerned that nodes have differing throughput.

Has anything like this been attempted on JPPF?

It looks to me like it's achievable with a few modifications to TaskQueueChecker and JPPFPriorityQueue, but they don't seem to be pluggable.

Thanks, Paul
Logged

lolo

  • Administrator
  • JPPF Council Member
  • *****
  • Posts: 2262
    • JPPF Web site
Re: Proportional allocation of nodes based on priority
« Reply #1 on: March 28, 2013, 10:18:13 PM »

Hi Paul,

Couldn't this be achieved with a combination of job priority and maximum number of nodes?

Otherwise, do you mind clarifying what you mean by "specify that n is assigned a relatively higher proportion of nodes than n-1", especially the "relative" part ?
Were you thinking of something around providing a probability that a node will be assigned to a job, based on the priority or any arbitrary value configured in the SLA?

Thanks,
-Laurent
Logged

row_perfect

  • Guest
Re: Proportional allocation of nodes based on priority
« Reply #2 on: March 28, 2013, 10:44:47 PM »

Hi Laurent,

The challenge I am facing is that I have a number of user jobs which I would like to run at the same priority. In the absence of anything else, I want these jobs to utilise the entire grid. I also have a number of batch jobs that I need to run; these need to be a lower priority - however I still want them to get some grid time, and in the absence of user jobs, they should too use the entire grid.

The idea of relative weights came from past experience with DataSynapse GridServer - priorities 1-9 could be assigned a weight factor. That weight determined the proportion of available nodes that would be assigned to the set of pending jobs at each priority.

What do you suggest?

Thanks, Paul
Logged

lolo

  • Administrator
  • JPPF Council Member
  • *****
  • Posts: 2262
    • JPPF Web site
Re: Proportional allocation of nodes based on priority
« Reply #3 on: April 17, 2013, 10:23:38 AM »

Hello Paul,

I apologize for this very late answer.
I've had a hard time with your question :). I registered a feature request to track it in the future: JPPF-137 Enabling a more flexible way of matching jobs to nodes.

In the meantime, I figured a way to achieve this, using a custom execution policy.
The main problems to solve were that the execution policy by default doesn't know anything about the job (e.g. priority), nor about the global state of the grid (e.g number of nodes).
To circumvent these issues, I designed the custom policy class as follows:
- the job uuid and priority are provided as constructor arguments
- we use a static local connection to the driver's jmx server, so we can have global information on the grid

This gives us an implementation like this:
Code: [Select]
public class MyCustomPolicy extends CustomPolicy {
  // Local connection to the JMX server.
  private static JMXDriverConnectionWrapper jmx = null;
  // Used to ensure jmx is initialized only once and that all threads wait until the initialization is complete
  // It must be initialized only when running in the driver, so we do it in the accepts() method
  private static Lock lock = new ReentrantLock();
  // The job priority
  private final int priority;
  // The job uuid
  private final String uuid;

  // Initialize this policy with a list of arguments:
  // args[0] is interpreted as the job's uuid
  // args[1] is interpreted as the job's priority
  public MyCustomPolicy(final String... args) {
    super(args);
    this.uuid = args[0];
    int p = 0;
    try {
      p = Integer.valueOf(args[1]);
    }
    catch (NumberFormatException ignore) {
    }
    this.priority = p;
  }

  @Override
  public boolean accepts(final PropertiesCollection info) {
    lock.lock();
    try {
    if (jmx == null) initJmx();
    } finally {
      lock.unlock();
    }
    try {
      // get how many nodes the job is currently dispatched to
      NodeJobInformation[] nodeJobInfo = jmx.getNodeInformation(uuid);
      int nbDispatchedNodes = (nodeJobInfo != null) ? nodeJobInfo.length : 0;
      // current number of nodes in the grid
      int nbGridNodes = jmx.nbNodes();
      return (float) nbDispatchedNodes / (float) nbGridNodes < getPctNodesFromPriority();
    } catch(Exception e) {
      // do something ?
    }
    return true;
  }

  // Compute the % of total nodes a job can be dispatched to.
  private float getPctNodesFromPriority() {
    // this is just an example of calculating the %, feel free to implemnt your own!
    if (this.priority <= 1) return 0.1f;
    else if (this.priority >= 9) return 0.9f;
    return (float) this.priority / 10f;
  }

  // Initialize a local (in-JVM) JMX connection to the driver.
  private static JMXDriverConnectionWrapper initJmx() {
    JMXDriverConnectionWrapper jmx = new JMXDriverConnectionWrapper();
    jmx.connect();
    return jmx;
  }
}

Then, you submit the job with an instance of this policy:
Code: [Select]
JPPFJob job = ....;
MyCustomPolicy policy = new MyCustomPolicy(job.getUuid(), Integer.toString(job.getSLA().getPriority()));
job.getSLA().setExecutionPolicy(policy);

For your convenience, I'm also attaching the source file to this post. For this to work, the custom policy class must be both in the driver's and the client's classpath.

I hope this helps,
-Laurent
« Last Edit: April 17, 2013, 10:26:59 AM by lolo »
Logged
Pages: [1]   Go Up
 
JPPF Powered by SMF 2.0 RC5 | SMF © 2006–2011, Simple Machines LLC Get JPPF at SourceForge.net. Fast, secure and Free Open Source software downloads