Print

[whddnslasla]

Hello,

I am looking for a cluster program which I can compute Java job in parallel. I looked at Rockscluster and Hadoop. The problem of using Rockscluster was that it required scripts in the Unix to run computation in parallel. However, what I wanted to do is sending jobs to workers in Java itself so that workers compute them and return the values. It is because my jobs are decided by many different users and you cannot write script before running the jobs. Also, the problem of using Hadoop was that it uses Map-reduce tool but I think my Java job is not benefited from Map-reduce scheme.

Since I already wasted a lot of time to search proper softwares by myself, I asked other gird softwares and people recommended me to try JPPF. Before getting into JPPF deeply, I just want your advice whether JPPF supports what I am trying to do in a big picture wise. 

What I want to do is simple. Once I set clusters, I want to send jobs to workers(other computer nodes) and receive results. All my jobs sent to workers will be independent (so I don't have to worry about dependency btw jobs. simple jobs). Also, I want to implement those parallelization in Java itself. When I send multiple jobs to the scheduler, I hope scheduler sets queue and automatically send some jobs to available nodes and return the results back to users. (I do not need fancy function like selecting nodes by myself to send jobs...)

For a better explanation, let me give an example below. Let's say there is a user1 who is working on Java. He is doing three computations in main() in his computer. Below is his code.

public class Multiplecal {

public static void main(String[] args){
    Multiplecal calobj= new Multiplecal();
    int result1, result2, result3;
    result1=calobj.addtwo(5);
    result2=calobj.addthree(6);
    result3=calobj.addfour(7);
}
public int addtwo(int n){
    return (n+(n-1));
}
public int addthree(int n){
    return (n+(n-1)+(n-2));
}
public int addfour(int n){
    return (n+(n-1)+(n-2)+(n-3));
}
}

However, user1 wants to get result1, result2, result3 by using some cluster program. If there exists an API called service then his main() code might look like below.

import service.*;

public class Multiplecal {

public static void main(String[] args){
    Multiplecal calobj= new Multiplecal();
    int result1, result2, result3;
    result1=service.send("Multiplecal", "addtwo", 5);
    result2=service.send("Multiplecal", "addthree", 6);
    result3=service.send("Multiplecal", "addfour", 7);
}
.... }

Service API will send each (classname, methodname and input parameters) to parallel program manager. Then parallel program manager distribute these jobs to nodes (workers). Since workers already have Multiplecal class, they can obtain results by matching classes and methods sent from service API. When workers finish their work, they return results back to user1.

What I mentioned above is just big picture of what I am trying to do. Parameter format for the parallelization do not have to be like above in JPPF. If JPPF supports above jobs, please let me know. Also, please let me know if there are great simple examples like above which I can refer to.

Thanks

[lolo]

Hello,

Yes, this is what JPPF is designed to do.
In JPPF, the work can be parallelized by creating a job and adding tasks to it. The tasks may then be distributed to one or more nodes, and each node can execute multiple tasks in parallel. The traditional way to create a task is to instantiate a subclass of JPPFTask. To take an example related to your example, we could have the following:

Code: [Select]

public static class AddTwo extends JPPFTask {
  private int n;

  public AddTwo(final int n) {
    this.n = n;
  }

  @Override
  public void run() {
    setResult(n + (n - 1));
  }
}
This is the representation of your addtwo methods as a JPPF task. Then, all you need to do is create a job, add tasks to it, and submit it to the grid:

Code: [Select]

// create the job and add the tasks
JPPFJob job = new JPPFJob();
job.addTask(new AddTwo(5));
job.addTask(new AddThree(6));
job.addTask(new AddFour(7));
// connect to the grid
JPPFClient client = new JPPFClient();
List<JPPFTask> results =  client.submit(job);
// print the results
for (JPPFTask task: results) System.out.println("task " + task.getClass().getSimpleName() + " result = " + task.getResult);
However, JPPF has a more flexible way to add tasks to a job, which will allow you to reuse existing code almost without change: the only requirement is that any object sent to the grid must be serializable. So we just redefine your class as "public class Multiplecal implements Serializable". Then we can create the job as follows:

Code: [Select]

JPPFJob job = new JPPFJob();
Multiplecal calobj = new Multiplecal();
// method addtwo() will be invoked on calobj with parameter (5)
job.addTask("addtwo", calobj, 5);
job.addTask("addthree", calobj, 6);
job.addTask("addfour", calobj, 7);
// ... submit to the grid and print the results ...
For your convenience, I'm attaching a full working code sample, which uses both methods and shows that the results are identical - 9, 15 and 22 in your example.

We have a number of samples demonstrating various features of JPPF. Please take a look at our samples page. To get started with JPPF, you might want to read the JPPF Overview and try hands-on with the tutorial.

I hope this helps.

Sincerely,
-Laurent