JVM health monitoring

The JPPF management APIs provide some basic abilties to monitor the JVM of remote nodes and drivers in a Grid.

These capabilities include:

• memory usage (heap and non-heap)
• CPU load
• count of live threads and deadlock detection
• triggering remote thread dumps and displaying them locally
• triggering remote garbage collections
• triggering remote heap dumps

These features are available via the built-in MBean interface DiagnosticsMBean, defined as follows:

public interface DiagnosticsMBean {
  // The name of this MBean in a driver
  String MBEAN_NAME_DRIVER = "org.jppf:name=diagnostics,type=driver";

  // The name of this MBean in a node
  String MBEAN_NAME_NODE = "org.jppf:name=diagnostics,type=node";

  // Get the memory usage info for the whole JVM
  MemoryInformation memoryInformation() throws Exception;

  // Perform a garbage collection, equivalent to System.gc()
  void gc() throws Exception;

  // Get a full thread dump, including detection of deadlocks
  ThreadDump threadDump() throws Exception;

  // Determine whether a deadlock is detected in the JVM
  Boolean hasDeadlock() throws Exception;

  // Get a summarized snapshot of the JVM health
  HealthSnapshot healthSnapshot() throws Exception;

  // Trigger a heap dump of the JVM
  String heapDump() throws Exception;

  // Get an approximation of the current CPU load
  Double cpuLoad();
}

Example usage:

// connect to the driver's JMX server
JMXDriverConnectionWrapper jmxDriver =
  new JMXDriverConnectionWrapper(driverHost, driverPort, false);
// obtain a proxy to the diagnostics MBean
DiagnosticsMBean driverProxy =
  jmxDriver.getProxy(DiagnosticsMBean.MBEAN_NAME_DRIVER, DiagnosticsMBean.class);
// get a thread dump of the remote JVM
ThreadDump tdump = proxy.threadDump();
// format the thread dump as easily readable text
String s = TextThreadDumpWriter.printToString(tdump, "driver thread dump");
System.out.println(s);

The MBean interface is exactly the same for nodes and drivers, only the MBean name varies. Also note that, as for other node-related MBeans, it can be be invoked via the node forwarding MBean instead of directly.

Also please keep in mind that, as with all JPPF built-in MBeans, this one is defined as a pluggable MBean, as if it were a custom one.

1 Memory usage

A detailed memory usage can be obtained by calling the method DiagnosticsMBean.memoryInformation(). This method returns an instance of MemoryInformation, defined as follows:

public class MemoryInformation implements Serializable {
  // Get the heap memory usage
  public MemoryUsageInformation getHeapMemoryUsage()

  // Get the non-heap memory usage
  public MemoryUsageInformation getNonHeapMemoryUsage()
}

Both heap and non-heap usage are provided as instances of the class MemoryUsageInformation:

public class MemoryUsageInformation implements Serializable {
  // Get the initial memory size
  public long getInit()

  // Get the current memory size
  public long getCommitted()

  // Get the used memory size
  public long getUsed()

  // Get the maximum memory size
  public long getMax()

  // Return the ratio of used memory over max available memory
  public double getUsedRatio()
}

2 Thread dumps

You can trigger and obtain a full thread dump of the remote JVM by calling DiagnosticsMBean.threadDump(). This method returns an instance of ThreadDump. Rather than detailing this class, we invite you to explore the related Javadoc. Instead, we would like to talk about the provided facilities to translate thread dumps into readable formats.

There are two classes that you can use to print a heap dump to a character stream:

• TextThreadDumpWriter prints a thread dump to a plain text stream
• HTMLThreadDumpWriter prints a thread dump to a styled HTML stream

Both implement the ThreadDumpWriter interface, defined as follows:

public interface ThreadDumpWriter extends Closeable {
  // Print the specified string without line terminator
  void printString(String message);

  // Print the deadlocked threads information
  void printDeadlocks(ThreadDump threadDump);

  // Print information about a thread
  void printThread(ThreadInformation threadInformation);

  // Print the specified thread dump
  void printThreadDump(ThreadDump threadDump);
}

Each of these classes provides a static method to print a thread dump directly into a String:

• static String TextThreadDumpWriter.printToString(ThreadDump tdump, String title)
• static String HTMLThreadDumpWriter.printToString(ThreadDump tdump, String title)

Example usage:

// get a thread dump from a remote node or driver
DiagnosticsMBean proxy = ...;
ThreadDump tdump = proxy.threadDump();
// we will print it to an HTML file
FileWriter fileWriter = new FileWriter("MyThreadDump.html");
HTMLThreadDumpWriter htmlPrinter = new HTMLThreadDumpWriter(fileWriter, "My Title");
htmlPrinter.printTreadDump(tdump);
// close the underlying writer
htmlPrinter.close();

Here is an example of the output, as rendered in the JPPF administration console:

3 Health snapshots

You can obtain a summarized snapshot of the JVM state by calling DiagnosticsMBean.healthSnapshot(), which returns an object of type HealthSnapshot, defined as follows:

public class HealthSnapshot implements Serializable {
  // Get the ratio of used / max for heap memory
  public double getHeapUsedRatio()

  // Get the ratio of used / max for non-heap memory
  public double getNonheapUsedRatio()

  // Determine whether a deadlock was detected
  public boolean isDeadlocked()

  // Get the used heap memory in bytes
  public long getHeapUsed()

  // Get the used non-heap memory in bytes
  public long getNonheapUsed()

  // Get the number of live threads in the JVM
  public int getLiveThreads()

  // Get the cpu load
  public double getCpuLoad()

  // Get this snapshot in an easily readable format, according to the default locale
  public String toFormattedString()

  // Get this snapshot in an easily readable format
  public String toFormattedString(final Locale locale)
}

The toFormattedString() methods return a nicely formatted string which can be used for debugging or testing purposes. Here is an example output with the en_US locale:

HealthSnapshot[heapUsedRatio=  15.7 %; heapUsed=  71.7 MB; nonheapUsedRatio=   8.7 %; 
nonheapUsed=  11.4 MB; deadlocked=false; liveThreads=90; cpuLoad=  23.6 %]

4 CPU load

The CPU load of a remote JVM can be obtained separately by calling DiagnosticsMBean.cpuLoad(). This method returns an approximation of the latest computed CPU load in the JVM. It is important to understand that the CPU load is not computed each time this method is called. Instead, it is computed at regular intervals, and the latest computed value is returned. The purpose of this is to prevent the computation itself from using up too much CPU time, which would throw off the computed value.

The actual computed value is equal to SUM_i{cpuTime(thread_i)} / interval, for all the live threads of the JVM at the time of the computation. Thus, errors may occur, since many threads may have been created then died between two computations. However, in most cases this is a reasonable approximation that does not tax the CPU too heavily.

The interval between computations can be adjusted by setting the following property in a node or driver configuration: jppf.cpu.load.computation.interval = time_in_millis. If unspecified, the default value is 1000 ms.

5 Deadlock indicator

To determine whether a JVM has deadlocked threads, simply call DiagnosticsMBean.hasDeadlock(). This method is useful if you only need that information, without having to process an entire thread dump, which represents a significant overhead, especially from a network perspective.

6 Triggering a heap dump

Remotely triggering a JVM heap dump is done by calling DiagnosticsMBean.heapDump(). This method is JVM implementation-dependent, as it relies on non-standard APIs. Thus, it will only work with the Oracle standard and JRockit JVMs, along with the IBM JVM. The returned value is a description of the outcome: it will contain the name of the genreated heap dump file for Oracle JVMs, and a success indicator only for IBM JVMs.

7 Triggering a garbage collection

A remote garbage collection can be triggered by callling DiagnosticsMBean.gc(). This method simply calls System.gc() and thus has the exact same semantics and constraints.