A first taste of JPPF

From JPPF Documentation version 3.x

Jump to: navigation, search

Contents

Main Page > A first taste of JPPF

Required software

In this tutorial, we will be writing a sample JPPF application, and we will run it on a small grid. To this effect, we will need to download and install the following JPPF components:

  • JPPF application template: this is the JPPF-x.y.z-application-template.zip file
  • JPPF driver: this is the JPPF-x.y.z-driver.zip file
  • JPPF node: this is the JPPF-x.y.z-node.zip file
  • JPPF administration console: this is the JPPF-x.y.z-admin-ui.zip file



Note: “x.y.z” designates the latest version of JPPF (major.minor.update). Generally, “x.y.0” is abbreviated into “x.y”.



These files are all available from the JPPF installer and/or from the JPPF download page.

In addition to this, Java 1.7 or later and Apache Ant 1.8.0 or later should already be installed on your machine.

We will assume the creation of a new folder called "JPPF-Tutorial", in which all these components are unzipped. Thus, we should have the following folder structure:

» JPPF-Tutorial
  » JPPF-x.y.z-admin-ui
  » JPPF-x.y.z-application-template
  » JPPF-x.y.z-driver
  » JPPF-x.y.z-node

Overview

Tutorial organization

We will base this tutorial on a pre-existing application template, which is one of the components of the JPPF distribution. The advantage is that most of the low-level wiring is already written for us, and we can thus focus on the steps to put together a JPPF application. The template is a very simple, but fully working, JPPF application, and contains fully commented source code, configuration files and scripts to build and run it.

It is organized with the following directory structure:

  • root directory: contains the scripts to build and run the application
  • src: this is where the sources of the application are located
  • classes: the location where the Java compiler will place the built sources
  • config: contains the JPPF and logging configuration files
  • lib: contains the required libraries to build and run the application

Expectations

We will learn how to:

  • write a JPPF task
  • create a job and execute it
  • process the execution results
  • manage JPPF jobs
  • run a JPPF application


The features of JPPF that we will use:

  • JPPF task and job APIs
  • local code changes automatically accounted for
  • JPPF client APIs
  • management and monitoring console
  • configuring JPPF


By the end of this tutorial, we will have a full-fledged JPPF application that we can build, run, monitor and manage in a JPPF grid. We will also have gained knowledge of the workings of a typical JPPF application and we will be ready to write real-life, grid-enabled applications.

Writing a JPPF task

A JPPF task is the smallest unit of code that can be executed on a JPPF grid. From a JPPF perspective, it is thus defined as an atomic code unit. A task is always defined as an implemntation of the inteface Task. Task extends the Runnable interface. The part of a task that will be executed on the grid is whatever is written in its run() method. From a design point of view, writing a JPPF task will comprise 2 major steps:

  • create an implementation of Task.
  • implement the run() method.

From the template application root folder, navigate to the folder src/org/jppf/application/template. You will see 2 Java files in this folder: "TemplateApplicationRunner.java" and "TemplateJPPFTask.java". Open the file "TemplateJPPFTask.java" in your favorite text editor.

In the editor you will see a full-fledged JPPF task declared as follows:

public class TemplateJPPFTask extends AbstractTask<String>

Here we use the more convenient class AbstractTask, which implements all methods in Task, except for run(). Below this, you will find a run() method declared as:

public void run() {
  // write your task code here.
  System.out.println("Hello, this is the node executing a template JPPF task");

  // ...

  // eventually set the execution results
  setResult("the execution was performed successfully");
} 

We can guess that this task will first print a "Hello …" message to the console, then set the execution result by calling the setResult() method with a string message. The setResult() method is provided as a convenience to store the results of the task execution, for later retrieval.

In this method, to show that we have customized the template, let's replace the line "// ..." with a statement printing a second message, for instance "In fact, this is more than the standard template". The run() method becomes:

public void run() {
  // write your task code here.
  System.out.println("Hello, this is the node executing a template JPPF task");
  System.out.println("In fact, this is more than the standard template");

  // eventually set the execution results
  setResult("the execution was performed successfully");
} 

Do not forget to save the file for this change to be taken into account.

The next step is to create a JPPF job from one or multiple tasks, and execute this job on the grid.

Creating and executing a job

A job is a grouping of tasks with a common set of characteristics and a common SLA. These characteristics include:

  • common data shared between tasks
  • a priority
  • a maximum number of nodes a job can be executed on
  • an execution policy describing which nodes it can run on
  • a suspended indicator, that enables submitting a job in suspended state, waiting for an external command to resume or start its execution
  • a blocking/non-blocking indicator, specifying whether the job execution is synchronous or asynchronous from the application's point of view

Creating and populating a job

In the JPPF APIs, a job is represented as an instance of the class JPPFJob. To see how a job is created, let's open the source file "TemplateApplicationRunner.java" in the folder JPPF-x.y.z-application-template/src/org/jppf/application/template. In this file, navigate to the method createJob().

This method is written as follows:

public JPPFJob createJob(String jobName) throws Exception {
  // create a JPPF job
  JPPFJob job = new JPPFJob();

  // give this job a readable name that we can use to monitor and manage it.
  job.setName(jobName);

  // add a task to the job.
  job.add(new TemplateJPPFTask());

  // add more tasks here ...

  // there is no guarantee on the order of execution of the tasks,
  // however the results are guaranteed to be returned in the same
  // order as the tasks.
  return job;
}

We can see that creating a job is done by calling the default constructor of class JPPFJob. The call to the method job.setName(String) is used to give the job a meaningful and readable name that we can use later to manage it. If this method is not called, an id is automatically generated, as a string of 32 hexadecimal characters.

Adding a task to the job is done by calling the method add(Object task, Object...args). The optional arguments are used when we want to execute other forms of tasks, that are not implementations of Task. We will see their use in the more advanced sections of the JPPF user manual. As we can see, all the work is already done in the template file, so there is no need to modify the createJob() method for now.

Executing a job and processing the results

Now that we have learned how to create a job and populate it with tasks, we still need to execute this job on the grid, and process the results of this execution. Still in the source file "TemplateApplicationRunner.java", let's navigate to the main(String...args) method. we will first take a closer look at the try block, which contains a very important initialization statement:

JPPFClient jppfClient = new JPPFClient();

This single statement initializes the JPPF framework in your application. When it is executed JPPF will do several things:

  • read the configuration file
  • establish a connection with one or multiple servers for job execution
  • establish a monitoring and management connection with each connected server
  • register listeners to monitor the status of each connection

As you can see, the JPPF client has a non-negligible impact on memory and network resources. This is why we recommend to always use the same instance throughout your application. This will also ensure a greater scalability, as it is also designed for concurrent use by multiple threads. To this effect, we have declared it in a try-with-resource block and provide it as a parameter for any method that needs it, in TemplateApplicationRunner.java.

It is always a good practice to release the resources used by the JPPF client when they are no longer used. Since JPPFClient implements AutoCloseable, this can be done conveniently in a try-with-resources statement:

try (JPPFClient jppfClient = new JPPFClient()) {
  // ... use the JPPF client ...
}

Back to the main method, after initializing the JPPF client, the next steps are to initialize our job runner, create a job and execute it:

// create a runner instance.
TemplateApplicationRunner runner = new TemplateApplicationRunner();

// create and execute a blocking job
JPPFJob job = runner.executeBlockingJob(jppfClient);

As we can see, the job creation, its execution and the processing of its rresults are all encapsulated in a call to the method executeBlockingJob(JPPFClient):

/**
 * Execute a job in blocking mode.
 * The application will be blocked until the job execution is complete.
 * @param jppfClient the {@link JPPFClient} instance which submits the job for execution.
 * @throws Exception if an error occurs while executing the job.
 */
public void executeBlockingJob(JPPFClient jppfClient) throws Exception {
  // Create a job
  JPPFJob job = createJob("Template blocking job");

  // set the job in blocking mode.
  job.setBlocking(true);

  // Submit the job and wait until the results are returned.
  // The results are returned as a list of Task<?> instances,
  // in the same order as the one in which the tasks where initially added the job.
  List<Task<?>> results = jppfClient.submitJob(job);

  // process the results
  processExecutionResults(job.getName(), results);
}

The call to createJob(jppfClient) is exactly what we saw in the previous section.

The next statement in this method ensures that the job will be submitted in blocking mode, meaning that the application will block until the job is executed:

job.setBlocking(true);

This is, in fact, optional since submission in blocking mode is the default behavior in JPPF.

The next statement will send the job to the server and wait until it has been executed and the results are returned:

List<Task<?>> results = jppfClient.submitJob(job);

We can see that the results are returned as a list of Task objects. It is guaranteed that each task in this list has the same position as the corresponding task that was added to the job. In other words, the results are always in the same order as the tasks in the the job.

The last step is to interpret and process the results. From the JPPF point of view, there are two possible outcomes of the execution of a task: one that raised a Throwable, and one that did not. When an uncaught Throwable (i.e. generally an instance of a subclass of java.lang.Error or java.lang.Exception) is raised, JPPF will catch it and set it as the outcome of the task. To do so, the method Task.setThrowable(Throwable) is called. JPPF considers that exception processing is part of the life cycle of a task and provides the means to capture this information accordingly.

This explains why, in our template code, we have separated the result processing of each task in 2 blocks:

public void processExecutionResults(List<JPPFTask> results) {
  // process the results
  for (Task<?> task: results) {
    if (task.getThrowable() != null) {
      // process the exception here ...
    } else {
      // process the result here ...
    }
  }
}

The actual results of the computation of a task can be any attribute of the task, or any object accessible from them. The Task<E> API provides two convenience methods to help doing this: setResult(E) and getResult(), however it is not mandatory to use them, and you can implement your own result handling scheme, or it could simply be a part of the task's design.

As an example for this tutorial, let's modify this part of the code to display the exception message if an exception was raised, and to display the result otherwise:

if (task.getThrowable() != null) {
  System.out.println("An exception was raised: " + task.getThrowable ().getMessage());
} else {
  System.out.println("Execution result: " + task.getResult());
}

We can now save the file and close it.

Running the application

We are now ready to test our JPPF application. To this effect, we will need to first start a JPPF grid, as follows:


Step 1: start a server

Go to the JPPF-x.y.z-driver folder and open a command prompt or shell console. Type "startDriver.bat" on Windows or “./startDriver.sh.” on Linux/Unix. You should see the following lines printed to the console:

driver process id: 6112, uuid: 4DC8135C-A22D-2545-E615-C06ABBF04065
management initialized and listening on port 11191
ClientClassServer initialized
NodeClassServer initialized
ClientJobServer initialized
NodeJobServer initialized
Acceptor initialized
-  accepting plain connections on port 11111
- accepting secure connections on port 11443
JPPF Driver initialization complete

The server is now ready to process job requests.


Step 2: start a node

Go to the JPPF-x.y.z-node folder and open a command prompt or shell console. Type "startNode.bat" on Windows or “./startNode.sh.” on Linux/Unix. You will then see the following lines printed to the console:

node process id: 3336, uuid: 4B7E4D22-BDA9-423F-415C-06F98F1C7B6F
Attempting connection to the class server at localhost:11111
Reconnected to the class server
JPPF Node management initialized
Attempting connection to the node server at localhost:11111
Reconnected to the node server
Node successfully initialized

Together, this node and the server constitute the smallest JPPF grid that you can have.


Step 3: run the application

Go to the JPPF-x.y.z-application-template folder and open a command prompt or shell console. Type "ant". This time, the Ant script will first compile our application, then run it. You should see these lines printed to the console:

client process id: 4780, uuid: 011B43B5-AE6B-87A6-C11E-0B2EBCFB9A89
[client: jppf_discovery-1-1 - ClassServer] Attempting connection to the class server ...
[client: jppf_discovery-1-1 - ClassServer] Reconnected to the class server
[client: jppf_discovery-1-1 - TasksServer] Attempting connection to the task server ...
[client: jppf_discovery-1-1 - TasksServer] Reconnected to the JPPF task server
Results for job 'Template blocking job' :
Template blocking job - Template task, execution result: the execution was performed
successfully

You will notice that the last printed line is the same message that we used in our task in the run() method, to set the result of the execution in the statement:

setResult("the execution was performed successfully");

Now, if you switch back to the node console, you should see that 2 new messages have been printed:

Hello, this is the node executing a template JPPF task
In fact, this is more than the standard template

These 2 lines are those that we actually coded at the beginning of the task's run() method:

System.out.println("Hello, this is the node executing a template JPPF task");
System.out.println("In fact, this is more than the standard template");

From these messages, we can conclude that our application was run successfully. Congratulations!

At this point, there is however one aspect that we have not yet addressed: since the node is a separate process from our application, how does it know to execute our task? Remember that we have not even attempted to deploy the application classes to any specific location. We have simply compiled them so that we can execute our application locally. This topic is the object of the next section of this tutorial.

Dynamic deployment

One of the greatest features of JPPF is its ability to dynamically load the code of an application that was deployed only locally. JPPF extends the standard Java class loading mechanism so that, by simply using the JPPF APIs, the classes of an application are loaded to any remote node that needs them. The benefit is that no deployment of the application is required to have it run on a JPPF grid, no matter how many nodes or servers are present in the grid. Furthermore, this mechanism is totally transparent to the application developer.

A second major benefit is that code changes are automatically taken into account, without any need to restart the nodes or the server. This means that, when you change any part of the code executed on a node, all you have to do is recompile the code and run the application again, and the changes will take effect immediately, on all the nodes that execute the application.

We will now demonstrate this by making a small, but visible, code change and running it against the server and node we have already started. If you have stopped them already, just perform again all the steps described in the previous section, before continuing.

Let's open again the source file "TemplateJPPFTask.java" in src/org/jppf/application/template/, and navigate to the run() method. Let's replace the first two lines with the following:

System.out.println("*** We are now running a modified version of the code ***");

The run() method should now look like this:

public void run() {
  // write your task code here.
  System.out.println("*** We are now running a modified version of the code ***");

  // eventually set the execution results
  setResult("the execution was performed successfully");
}

Save the changes to the file, and open or go back to a command prompt or shell console in the JPPF-x.y.z-application-template folder. From there, type "ant" to run the application again. You should now see the same messages as in the initial run displayed in the console. This is what we expected. On the other hand, if you switch back to the node console, you should now see a new message displayed:

*** We are now running a modified version of the code ***

Success! We have successfully executed our new code without any explicit redeployment.

Job Management

Now that we are able to create, submit and execute a job, we can start thinking about monitoring and eventually controlling its life cycle on the grid. To do that, we will use the JPPF administration and monitoring console. The JPPF console is a standalone graphical tool that provides user-friendly interfaces to:

  • obtain statistics on server performance
  • define, customize and visualize server performance charts
  • monitor and control the status and health of servers and nodes
  • monitor and control the execution of the jobs on the grid
  • manage the workload and load-balancing behavior

Preparing the job for management

In our application template, the job that we execute on the grid has a single task. As we have seen, this task is very short-live, since it executes in no more than a few milliseconds. This definitely will not allow us us to monitor or manage it with our bare human reaction time. For the purpose of this tutorial, we will now adapt the template to something more realistic from this perspective.


Step 1: make the tasks last longer

What we will do here is add a delay to each task, before it terminates. It will do nothing during this time, only wait for a specified duration. Let's edit again the source file "TemplateJPPFTask.java" in JPPF-x.y.z-application-template/src/org/jppf/application/template/ and modify the run() method as follows:

public void run() {
  // write your task code here.
  System.out.println("*** We are now running a modified version of the code ***");

  // simply wait for 3 seconds
  try {
    Thread.sleep(3000L);
  } catch(InterruptedException e) {
    setException(e);
    return;
  }

  // eventually set the execution results
  setResult("the execution was performed successfully");
}

Note that here, we make an explicit call to setException(), in case an InterruptedException is raised. Since the exception would be occurring in the node, capturing it will allow us to know what happened from the application side.


Step 2: add more tasks to the job, submit it as suspended

This time, our job will contain more than one task. In order for us to have the time to manipulate it from the administration console, we will also start it in suspended mode. To this effect, we will modify the method createJob() of the application runner "TemplateApplicationRunner.java" as follows:

public JPPFJob createJob() throws Exception {
  // create a JPPF job
  JPPFJob job = new JPPFJob();

  // give this job a readable unique id that we can use to monitor and manage it.
  job.setName("Template Job Id");

  // add 10 tasks to the job.
  for (int i=0; i<10; i++) job.add(new TemplateJPPFTask());

  // start the job in suspended mode
  job.getSLA().setSuspended(true);

  return job;
}


Step 3: start the JPPF components

If you have stopped the server and node, simply start them again as described in the first two step of section 2.5 of this tutorial.

We will also start the administration console:

Go to the JPPF-x.y.z-admin-ui folder and open a command prompt or shell console. Type "ant".

When the console is started, you will see a panel named "Topology" displaying the servers and the nodes attached to them. It should look like this:

CH-2-Job-1.gif

We can see here that a server is started on machine "lolo-quad" and that it has a node attached to it. The color for the server is a health indicator, green meaning that it is running normally and red meaning that it is down.

Let's switch to the "Job Data" panel, which should look like this:

CH-2-Job-2.gif

We also see the color-coded driver health information in this panel. There is currently no other element displayed, because we haven't submitted a job yet.


Step 4: start a job

We will now start a job by running our application: go to the JPPF-x.y.z-application-template folder and open a command prompt or shell console. Type "ant". Switch back to the administration console. We should now see some change in the display:

CH-2-Job-3.gif

We now see that a job is present in the server's queue, in suspended state (yellow highlighting). Here is an explanation of the columns in the table:

  • "Driver / Job / Node" : displays an identifier for a server, for a job submitted to that server, or for a node to which some of the tasks in the job have been dispatched for execution
  • "State" : the current state of a job, either "Suspended" or "Executing"
  • "Initial task count" : the number of tasks in the job at the time it was submitted by the application
  • "Current task count": the number of tasks remaining in the job, that haven't been executed
  • "Priority" : this is the priority, of the job, the default value is 0.
  • "Max nodes" : the maximum number of nodes a job can be executed on. By default, there is no limit, which is represented as the infinity symbol


Step 5: resuming the job execution

Since the job was submitted in suspended state, we will resume its execution manually from the console. Select the line where the job "Template Job Id" is displayed. You should see that some buttons are now activated. Click on the resume button (marked by the icon resume.gif ) to resume the job execution, as shown below:

CH-2-Job-4.gif

As soon as we resume the job, the server starts distributing tasks to the node, and we can see that the current task count starts decreasing accordingly, and the job status has been changed to "Executing":

CH-2-Job-5.gif

You are encouraged to experiment with the tool and the code. For example you can add more tasks to the job, make them last longer, suspend, resume or terminate the job while it is executing, etc...

Conclusion

In this tutorial, we have seen how to write a JPPF-enabled application from end to end. We have also learned the basic APIs that allow us to write an application made of atomic and independent execution units called tasks, and group them into jobs that can be executed on the grid. We have also learned how jobs can be dynamically managed and monitored while executing. Finally, we also learned that, even though an application can be distributed over any number of nodes, there is no need to explicitly deploy the application code, since JPPF implicitly takes care of it.


Main Page > A first taste of JPPF

Support This Project Powered by MediaWiki