Node management

Out of the box in JPPF, each node provides several MBeans that can be accessed remotely using a JMXMP remote connector with the JMX URL “service:jmx:jmxmp://host:port”, where host is the host name or IP address of the machine where the node is running (value of “jppf.management.host” in the node configuration file), and port is the value of the property “jppf.management.port” specified in the node's configuration file.

1 Node-level management and monitoring MBean

MBean name: “org.jppf:name=admin,type=node”

This is also the value of the constant JPPFNodeAdminMBean.MBEAN_NAME.

This MBean's role is to perform management and monitoring at the node level, however we will see that it also has (for historical reasons) some task-level management functions. It exposes the JPPFNodeAdminMBean interface, which provides the functionalities described hereafter.

1.1 Getting a snapshot of the node's state

This is done by invoking the following method on the MBean:

 public interface JPPFNodeAdminMBean extends JPPFAdminMBean {
   // Get the latest state information from the node
   public JPPFNodeState state() throws Exception;
 }

This method returns a JPPFNodeState object, which provides the following information on the node:

 public class JPPFNodeState implements Serializable {
   // the status of the connection with the server
   public String getConnectionStatus()
 
   // the current task execution status
   public String getExecutionStatus()
 
   // the cpu time consumed by the node's execution threads
   // this includes the tasks cpu time and some JPPF processing overhead
   public long getCpuTime()
 
   // the total number of tasks executed
   public int getNbTasksExecuted()
 
   // the current size of the pool of threads used for tasks execution
   public int getThreadPoolSize()
 
   // the current priority assigned to the execution threads
   public int getThreadPriority()
 }

1.2 Updating the execution thread pool properties

 public interface JPPFNodeAdminMBean extends JPPFAdminMBean {
   // Set the size of the node's execution thread pool
   public void updateThreadPoolSize(Integer size) throws Exception;
 
   // Update the priority of all execution threads
   public void updateThreadsPriority(Integer newPriority) throws Exception;
 }

1.3 Shutting down and restarting the node

 public interface JPPFNodeAdminMBean extends JPPFAdminMBean {
   // Restart the node
   public void restart() throws Exception;
 
   // Shutdown the node
   public void shutdown() throws Exception;
 }

These two methods should be used with precautions. Please note that, once shutdown() has been invoked, it is not possible anymore to restart the node remotely.

When any of these methods is invoked, the tasks that were being executed, if any, are automatically resubmitted to the server queue.

1.4 Updating the executed tasks counter

 public interface JPPFNodeAdminMBean extends JPPFAdminMBean {
   // Reset the node's executed tasks counter to zero
   public void resetTaskCounter() throws Exception;
 
   // Reset the node's executed tasks counter to the specified value
   public void setTaskCounter(Integer n) throws Exception;
 }

Please note that resetTaskCounter() is equivalent to setTaskCounter(0).

1.5 Getting information about the node's host

 public interface JPPFAdminMBean extends Serializable {
   // Get detailed information about the node's JVM properties, environment variables,
   // memory usage, available processors andavailable storage space
   JPPFSystemInformation systemInformation() throws Exception;
 }

This method returns an object of type JPPFSystemInformation, which is a snapshot of the environment of the JPPF node, the JVM and the host they run on. The properties defined in this object are also those used by execution policies, as we have seen in section 3.4.1 of this manual.

JPPFSystemInformation provides information about 6 different aspects of the environment:

 public class JPPFSystemInformation implements Serializable {
   // get the system properties
   public TypedProperties getSystem()
   // get runtime information about JVM memory and available processors
   public TypedProperties getRuntime()
   // get the host environment variables
   public TypedProperties getEnv()
   // get IPV4 and IPV6 addresses assigned to the host
   public TypedProperties getNetwork()
   // get the JPPF configuration properties
   public TypedProperties getJppf()
   // get information on available disk storage
   public TypedProperties getStorage()
 }

We encourage the reader to follow the links to the above methods' Javadoc, to obtain details on each set of information, and how the information is formatted and named.

Each of the methods in JPPFSystemInformation returns a TypedProperties object. TypedProperties is a subclass of the standard java.util.Properties that provides convenience methods to read property values as primitive types other than String.

1.6 Canceling a job

 public interface JPPFAdminMBean extends Serializable {
   // Cancel the job with the specified uuid. The requeue parameter determines
   // whether the job should be requeued on the server side or not
   public void cancelJob(String jobUuid, Boolean requeue) throws Exception;
 }

This MBean method is used to cancel a job currently running in the node. The job is identified by its uuid. The requeue parameter is used to notify the server that the canceled job should be requeued on the server and executed again, possibly on an other node. If requeue is false, the job is simply terminated and any remaining task will not be executed.

This method should normally only be used by the JPPF server, in the case where a user requested that the server terminates a job. In effect, a job can contain several tasks, with each task potentially executed concurrently on a separate node. When the server receives a job termination request, it will handle the termination of “sub-jobs” (i.e. subsets of the tasks in the job) by notifying each corresponding node.

1.7 Updating the node's configuration properties

 public interface JPPFAdminMBean extends Serializable {
   // Update the configuration properties of the node. The reconnect parameter
   // specifies whether the node should disconnect then reconnect to the driver
   // after updating the properties
   void updateConfiguration(Map<String, String> config, Boolean reconnect)
      throws Exception;
 }

This method sends a set of configuration properties to the node, that will override those defined in the node's configuration file. The reconnect parameter will allow the node to take the changes into account, especially in the case where the server connection or discovery properties have been changed, for instance to force the node to connect to another server without having to stop it.

2 Task-level monitoring

MBean name : “org.jppf:name=task.monitor,type=node”.

This is also the value of the constant JPPFNodeTaskMonitorMBean.MBEAN_NAME

This MBean monitors the task activity within a node. It exposes the interface JPPFNodeTaskMonitorMBean and also emits JMX notifications of type TaskExecutionNotification.

2.1 Snapshot of the tasks activity

The interface JPPFNodeTaskMonitorMBean provides access to aggregated statistics on the tasks executed within a node:

 public interface JPPFNodeTaskMonitorMBean extends NotificationEmitter {
   // The total number of tasks executed by the node
   Integer getTotalTasksExecuted();
 
   // The total number of tasks that ended in error
   Integer getTotalTasksInError();
 
   // The total number of tasks that executed sucessfully
   Integer getTotalTasksSucessfull();
 
   // The total cpu time used by the tasks in milliseconds
   Long getTotalTaskCpuTime();
 
   // The total elapsed time used by the tasks in milliseconds
   Long getTotalTaskElapsedTime();
 }

2.2 Notification of tasks execution

Each time a task completes its execution in a node, the task monitor MBean will emit a JMX notification of type TaskExecutionNotification defined as follows:

 public class TaskExecutionNotification extends Notification {
   // Get the object encapsulating information about the task
   public TaskInformation getTaskInformation();
 }

This notification essentially encapsulates an object of type TaskInformation, which provides the following information about each executed task:

 public class TaskInformation implements Serializable {
   // Get the task id
   public String getId()
 
   // Get the uuid of the job this task belongs to
   public String getJobId()
 
   // Get the cpu time used by the task
   public long getCpuTime()
 
   // Get the wall clock time used by the task
   public long getElapsedTime()
 
   // Determines whether the task had an exception
   public boolean hasError()
 
   // Get the timestamp for the task completion
   // Caution: this value is related to the node's system time,
   // not to the time of the notification receiver
   public long getTimestamp()
 }

3 Node maintenance

MBean name : “org.jppf:name=node.maintenance,type=node”.
This is also the value of the constant JPPFNodeMaintenanceMBean.MBEAN_NAME

This MBean provides operations for the maintenance of a node. It exposes the interface JPPFNodeMaintenanceMBean defined as follows:

public interface JPPFNodeMaintenanceMBean extends Serializable {
  // object name for this MBean
  String MBEAN_NAME = "org.jppf:name=node.maintenance,type=node";

  // request a reset of the resource caches of all the JPPF class loaders
  // maintained by the node
  void requestResourceCacheReset() throws Exception;
}

Please note that the method requestResourceCacheReset() does not perform the reset immediately. Instead, it sets an internal flag, and the reset will take place when it is safe to do so, as part of the node's life cycle. The outcome of the reset operation is that the temporary files created by the JPPF class loaders wil be deleted, freeing space in the temporary files folder.

4 Accessing and using the node MBeans

JPPF provides an API that simplifies access to the JMX-based management features of a node, by abstracting most of the complexities of JMX programming. This API is represented by the class JMXNodeConnectionWrapper, which provides a simplified way of connecting to the node's MBean server, along with a set of convenience methods to easily access the MBeans' exposed methods and attributes.

4.1 Connecting to an MBean server

Connection to to a node MBean server is done in two steps:

a. Create an instance of JMXNodeConnectionWrapper

To connect to a local (same JVM, no network connection involved) MBean server, use the no-arg constructor:

 JMXNodeConnectionWrapper wrapper = new JMXNodeConnectionWrapper();

To connect to a remote MBean server, use the constructor specifying the management host and port:

 JMXNodeConnectionWrapper wrapper = new JMXNodeConnectionWrapper(host, port, sslEnabled);

Here host and port represent the node's configuration properties “jppf.management.host” and “jppf.management.port”. sslEnabled specifies whether the connection should be established over SSL/TLS.

b. Initiate the connection to the MBean server and wait until it is established

There are two ways to do this:

Synchronously:

 // connect and wait for the connection to be established
 // choose a reasonable value for the timeout, or 0 for no timeout
 wrapper.connectAndWait(timeout);

Asynchronously:

 // initiate the connection; this method returns immediately
 wrapper.connect()
 
 // ... do something else ...
 
 // check if we are connected
 if (wrapper.isConnected()) ...;
 else ...;

4.2 Direct use of the JMX wrapper

JMXNodeConnectionWrapper implements directly the interface JPPFNodeAdminMBean. This means that all the methods of this interface can be used directly from the JMX wrapper. For example:

 JMXNodeConnectionWrapper wrapper = new JMXNodeConnectionWrapper(host, port, sslEnabled);
 wrapper.connectAndWait(timeout);
 
 // get the number of tasks executed since the last reset
 int nbTasks = wrapper.state().getNbTasksExecuted();
 // stop the node
 wrapper.shutdown();

4.3 Use of the JMX wrapper's invoke() method

JMXConnectionWrapper.invoke() is a generic method that allows invoking any exposed method of an MBean.

Here is an example:

 JMXNodeConnectionWrapper wrapper = new JMXNodeConnectionWrapper(host, port, sslEnabled);
 wrapper.connectAndWait(timeout);
 
 // equivalent to JPPFNodeState state = wrapper.state();
 JPPFNodeState state = (JPPFNodeState) wrapper.invoke(
   JPPFNodeAdminMBean.MBEAN_NAME, "state", (Object[]) null, (String[]) null);
 int nbTasks = state.getNbTasksExecuted();
 // get the total CPU time used
 long cpuTime = (Long) wrapper.invoke(JPPFNodeTaskMonitorMBean.MBEAN_NAME,
   "getTotalTaskCpuTime", (Object[]) null, (String[]) null);

4.4 Use of an MBean proxy

A proxy is a dynamically created object that implements an interface specified at runtime.

The standard JMX API provides a way to create a proxy to a remote or local MBeans. This is done as follows:

 JMXNodeConnectionWrapper wrapper = new JMXNodeConnectionWrapper(host, port, sslEnabled);
 wrapper.connectAndWait(timeout);
 
 // create the proxy instance
 JPPFNodeTaskMonitorMBean proxy = wrapper.getProxy(
   JPPFNodeTaskMonitorMBean.MBEAN_NAME, JPPFNodeTaskMonitorMBean.class);
 
 // get the total CPU time used
 long cpuTime = proxy.getTotalTaskCpuTime();

4.5 Subscribing to MBean notifications

We have seen that the task monitoring MBean represented by the JPPFNodeTaskMonitorMBean interface is able to emit notifications of type TaskExecutionNotification. There are 2 ways to subscribe to these notifications:

a. Using a proxy to the MBean

 JMXNodeConnectionWrapper wrapper = new JMXNodeConnectionWrapper(host, port, sslEnabled);
 wrapper.connectAndWait(timeout);
 JPPFNodeTaskMonitorMBean proxy = wrapper.getProxy(
   JPPFNodeTaskMonitorMBean.MBEAN_NAME, JPPFNodeTaskMonitorMBean.class);
 
 // subscribe to all notifications from the MBean
 proxy.addNotificationListener(myNotificationListener, null, null);

b. Using the MBeanServerConnection API

 JMXNodeConnectionWrapper wrapper = new JMXNodeConnectionWrapper(host, port, sslEnabled);
 wrapper.connectAndWait(timeout);
 MBeanServerConnection mbsc = wrapper.getMbeanConnection();
 ObjectName objectName = new ObjectName(JPPFNodeTaskMonitorMBean.MBEAN_NAME);
 
 // subscribe to all notifications from the MBean
 mbsc.addNotificationListener(objectName, myNotificationListener, null, null);

Here is an example notification listener implementing the NotificationListener interface:

 // this class counts the number of tasks executed, along with
 // the total cpu time and wall clock time used by the node
 public class MyNotificationListener implements NotificationListener {
   AtomicInteger taskCount = new AtomicInteger(0);
   AtomicLong cpuTime = new AtomicLong(0L);
   AtomicLong elapsedTime = new AtomicLong(0L);
 
   // Handle an MBean notification
   public void handleNotification(Notification notification, Object handback) {
     TaskExecutionNotification jppfNotif = (TaskExecutionNotification) notification;
     TaskInformation info = jppfNotif.getTaskInformation();
     int n = taskCount.incrementAndGet();
     long cpu = cpuTime.addAndGet(info.getCpuTime());
     long elapsed = elapsedTime.addAndGet(info.getElapsedTime());
     // display the statistics for every 50 tasks executed
     if (n % 50 == 0) {
       System.out.println("nb tasks = " + n + ", cpu time = " + cpu
         + " ms, elapsed time = " + elapsed +" ms");
     }
   }
 };
 
 NotificationListener myNotificationListener = new MyNotificationListener();

5 Remote logging

It is possible to receive logging messages from a node as JMX notifications. Specific implementation are available for Log4j and JDK logging.

To configure Log4j for emitting JMX notifications, edit the log4j configuration files of the node and add the following:

 ### direct messages to the JMX Logger ###
 log4j.appender.JMX=org.jppf.logging.log4j.JmxAppender
 log4j.appender.JMX.layout=org.apache.log4j.PatternLayout
 log4j.appender.JMX.layout.ConversionPattern=%d [%-5p][%c.%M(%L)]: %m\n
 
 ### set log levels - for more verbose logging change 'info' to 'debug' ###
 log4j.rootLogger=INFO, JPPF, JMX

To configure the JDK logging to send JMX notifications, edit the JDK logging configuration file of the node and add the following:

 # list of handlers
 handlers= java.util.logging.FileHandler, org.jppf.logging.jdk.JmxHandler
 
 # Write log messages as JMX notifications
 org.jppf.logging.jdk.JmxHandler.level = FINEST
 org.jppf.logging.jdk.JmxHandler.formatter = org.jppf.logging.jdk.JPPFLogFormatter

To receive the logging notifications from a remote application, you can use the following code:

 // get a JMX connection to the node MBean server
 JMXNodeConnectionWrapper jmxNode = new JMXNodeConnectionWrapper(host, port, sslEnabled);
 jmxNode.connectAndWait(5000L);
 // get a proxy to the MBean
 JmxLogger nodeProxy = jmxNode.getProxy(JmxLogger.DEFAULT_MBEAN_NAME, JmxLogger.class);
 
 // use a handback object so we know where the log messages come from
 String source = "node   " + jmxNode.getHost() + ":" + jmxNode.getPort();
 // subbscribe to all notifications from the MBean
 NotificationListener listener = new MyLoggingHandler();
 nodeProxy.addNotificationListener(listener, null, source);
 
 // Logging notification listener that prints remote log messages
 // to the console
 public class MyLoggingHandler implements NotificationListener {
   // handle the logging notifications
   public void handleNotification(Notification notification, final Object handback) {
     String message = notification.getMessage();
     String toDisplay = handback.toString() + ": " + message;
     System.out.println(toDisplay);
   }
 }