Print

[gaoisbest]

Hi, I want to build a java Swing application program to deal with large data in PC.
It will time-consuming when there is only one single PC working. So I want to make the
program use other's PC in local area network.

I think JPPF is good enough to meet my demand.
But when I execute a job, I must start a Server and Node first, so my question is 'Can i integrate the  'Server' and 'Node' libs together in one Java project' ? That is to say, when the user want to process their data, they just click the 'Execute' button, and the click event will automatically start the Sever and available Nodes ?

Thanks a lot!

[lolo]

Hello,

If I understand correctly what you want to achieve, you will not need to have a server for each user. Keep in mind that each node is only attached to a single server, so if each user starts their own server and node, they will be able to use only one node at a time, which would defeat the purpose of having a grid.

JPPF has no built-in support for starting remote servers and nodes. Generally, this is done by connecting remotely to remote machines (for instance via SSH on Linux) and running a shell script which starts a server or node.

Another possibility could be to install the JPPF server and nodes as operating system services, as documented here. Then, all you would need would be to start a JPPF client, which would automatically connect to the server, and submit jobs from this client. If no job is submitted, the server and nodes will not be doing any work and will be totally transparent to anyone using the remote machines. Additionally, it is generally easier to deal with remote services (especially on Windows and Linux) than to manage remote processes.

I hope this helps.

Sincerely,
-Laurent

[gaoisbest]

Sincerely thank to your reply.  The following is the scenario:

Suppose i have finished my program with JPPF named "GS" ("GS" has JPPF's server libs and node libs inside). 

There are two users A and B in one local area network. "GS" has been installed on both user A and user B's PC.

One day, user A want to process large data with "GS", when he click the 'Execute' button, "GS" will automatically start JPPF's server and find if there are more available nodes (Now, user A' PC is client, server and node). If user B' PC is power on, then user A' GS will have two nodes (both himself's PC and user B's PC); on the other hand, if user B' PC is power off, then user A' GS will only have one node, that is himself's PC.

So, I want to integrate the 'Server' and 'Node' libs together in GS.

Above is my idea. Is my idea executable ?

Please forgive me if I don't describe my problem clearly.

Again, Sincerely thanks to you.

[lolo]

Hello,

Thank you for clarifying.
I think the best solution in your scenario would to use a local node with each server, and configure the servers to communicate with each other. This will effectively create a P2P network of servers-with-node, exactly as discussed in this article.

To summarize, you will need to have the following in each server configuration file:

Code: [Select]

# enable the broadcast of connection information
jppf.discovery.enabled = true
# enable auto-discovery for peer-to-peer communication between servers
jppf.peer.discovery.enabled = true
# activate the local node
jppf.local.node.enabled = true
In the client configuration, you will probably want to disable server discovery and ensure the client will only connect to the local server you just started:

Code: [Select]

# disable discovery
jppf.discovery.enabled = false
# define a named driver
jppf.drivers = local_driver
# configure the connection to the driver
local_driver.jppf.server.host = localhost
local_driver.jppf.server.port = 11111
local_driver..jppf.management.port = 11198
local_driver.jppf.pool.size = 1
local_driver.priority = 0
Sincerely,
-Laurent

[gaoisbest]

Hi, Laurent. Your answer really give me a great help. Thank you very much.

But I also have four stupid questions about the concrete coding.

[1] How can i  integrate the Server into 'GS' ?
My solution to this question is just add all files in 'JPPF-3.3.5-driver' into my java project in Eclipse.
Are there any other better solutions ? Such as just add few jars into 'Build path' ?

[2] How can i start the Server with code ?
Because I have to integrate the Server into my program 'GS', and I want to start the Server when 'GS' is launched.
Just like the following code ?

Runtime rt = Runtime.getRuntime();   
      Process ps = null;   
      try { 
                  ps = rt.exec("cmd /c start .\\startDriver.bat"); 
      } catch (IOException e1) { 
                  e1.printStackTrace(); 
      }     

[3] How can i stop the Server when 'GS' is stopped ?

[4] I have revised the jppf-driver.properties with the following config
jppf.discovery.enabled = true
jppf.peer.discovery.enabled = true
jppf.local.node.enabled = true

I start the server on two PCs in local area network, but these two servers don't connect to each other ?
I could not see these two servers  in JPPF Monitoring and Administration Tool.

Am i right ? Is there any other better solution to integrate the Server into 'GS' and start it with code ?

Thank you very much. I am sincerely look forward to your reply.

[lolo]

Hello,

I believe the problem is that you need to specify the command line as multiple strings, one for each executable or argument.
For instance, instead of:

Code: [Select]

ps = rt.exec("cmd /c start .\\startDriver.bat"); you should do:

Code: [Select]

ps = rt.exec(new String[] {"cmd", "/c", "start", ".\\startDriver.bat"});
However, the problem with launching a .bat script (especially with "start") is that you will  not have any control over the Java process it will launch. In particular, you will not be able to terminate it properly, so it will remain running and you'll have to kill it with the Windows task manager.

So I think the easiest is to start the driver Java process directly, instead of calling a script. For this, you can use a ProcessBuilder as follows:

Code: [Select]

private static Process process = null;
public static void startDriverAndNode() throws Exception {
  ProcessBuilder builder = new ProcessBuilder();
  // use whatever directory the driver is installed in
  String driverDir = "C:/Workspaces/temp/JPPF-3.3.5-driver";
  builder.directory(new File(driverDir));
  String libDir = driverDir + "/lib";
  String configDir = driverDir + "/config";
  // build the classpath
  String cp = configDir + ";$lib/jppf-common-node.jar;$lib/jppf-common.jar;$lib/jppf-server.jar" +
      ";$lib/jmxremote_optional-1.0_01-ea.jar;$lib/log4j-1.2.15.jar;$lib/slf4j-api-1.6.1.jar;$lib/slf4j-log4j12-1.6.1.jar";
  cp = cp.replace("$lib", libDir);
  builder.command(
      "java", "-cp", cp, "-Xmx512m", "-server",
      "-Dlog4j.configuration=log4j-driver.properties",
      "-Djppf.config=jppf-driver.properties",
      "-Djava.util.logging.config.file=" + configDir + "/logging-driver.properties",
      "org.jppf.server.JPPFDriver",
      "noLauncher");
  System.out.println("command = " + builder.command());
  builder.redirectError(new File("err.log"));
  builder.redirectOutput(new File("out.log"));
  process = builder.start();
}
Then you can use it like this:

Code: [Select]

public static void main(final String... args) {
  JPPFClient client = null;
  try {
    startDriverAndNode();
    client = new JPPFClient();
    // submit a job a get its results ...
  } catch (Throwable t) {
    t.printStackTrace();
  } finally {
    System.out.println("closing the jppf client");
    if (client != null) client.close();
    System.out.println("terminating the driver");
    if (process != null) {
      process.destroy();
      process = null;
    }
    System.out.println("exiting");
    System.exit(0);
  }
}
I'm attaching a full working code sample for your convenience.

Sincerely,
-Laurent

[gaoisbest]

Hi, Laurent. Your sample really give me a great help. Thank you very much.

I copy your sample into my eclipse, but I got a Exception when I execute AllInOne.java.

The exception happened when I delete the note on 'Thread.sleep(3000L);'
that is,
  public static class MyTask extends JPPFTask {
    @Override
    public void run() {
      try {
        Thread.sleep(3000L);
        setResult("execution sucessful");
      } catch (Exception e) {
        setException(e);
      }
    }
  }
The exception is

Exception in thread "Receiver@10.10.115.240:11111" java.lang.OutOfMemoryError: Java heap space
   at org.jppf.comm.discovery.JPPFMulticastReceiver$Receiver.run(JPPFMulticastReceiver.java:255)
Exception in thread "Receiver@2001:0:9d38:90d7:247d:1859:f5f5:8c0f:11111" java.lang.OutOfMemoryError: Java heap space
   at org.jppf.comm.discovery.JPPFMulticastReceiver$Receiver.run(JPPFMulticastReceiver.java:255)

But when i delete 'Thread.sleep(3000L);', I got the results correctly.

I don't konw where is the problem. The thread just sleep 3000L, and can it cause OutOfMemoryError ?

Thank you very much.

[lolo]

Hi,

First off all, can you let us know if the OOME is occurring in the client or in the server/node JVM ? Unfortunately, it's not possible to tell with only the top element of the exception stack trace, and the JPPFMulticastReceiver is actually run in both the client and node: this is the server discovery service. Although it shouldn't be running in the client if you set "jppf.discovery.enabled = false" in the configuration.

If this happens in the client, could you post the Eclipse launch configuration you are using to run the sample? you can export it to a .launch file (XML content) and post it here.
In particular, are your memory settings (-Xms and -Xmx) adequate?

Otherwise, what you can do is add the JVM argument "-XX:+HeapDumpOnOutOfMemoryError" : this will cause the JVM to generate a heap dump (a .hprof file) which can then be analyzed with a tool such as Eclipse MAT to determine which which objects are filling the heap.

Additionally, on my side I do not reproduce this behavior with the uncommented Thread.sleep(), even when using -Xmx32m for the client

Sincerely,
-Laurent

[gaoisbest]

Hi, Laurent. Thank you very much for you reply.

Based on your advise, I decide to build a P2P topology.

So I have revised the jppf-driver.properties with the following config:
jppf.discovery.enabled = true
jppf.peer.discovery.enabled = true
jppf.local.node.enabled = true

It's very strange that the Exception is not happened any more. So we can think this problem is solved.

But there is a new question, may be the last question, because i have caused many problems to you.

The question is about P2P topology, I have started two Servers with revised above jppf-driver.properties in two PCs in local area network, but these two Servers do not find each other, there is no any information about these two Servers in JPPF Monitoring and Administration Tool.

So, how can i say these two P2P Servers has connected to each other ?

Thank you very much.

[lolo]

Hello,

If your are using the  ProcessBuilder API and are doing something like builder.redirectOutput(new File("out.log")), then each driver's console output will be redirected to a file named "out.log", you can look into this file for traces like this:

Code: [Select]

Connecting to  Peer-1@my_host:11111
Reconnected to Peer-1@my_host:11111
You can also use the administration console, for instance in the "Topology > Graph View" tab you should see whether the two servers are connected.

-Laurent

[gaoisbest]

I have revised the jppf-driver.properties with the following config:
jppf.discovery.enabled = true
jppf.peer.discovery.enabled = true
jppf.local.node.enabled = true

Then I run the startConsole.bat, but I cannot find ANY information about "Connecting to  Peer-1@my_host:11111"

The attached pic1 is the jppf driver launch information,

attached pic2 is the "Topology > Graph View" tab information, but there is no connected Server (I have started two Servers with revised above jppf-driver.properties).

Is it enough  only revise the above three properties ?