I am seeking for some help and guidance to better configure JPFF's environment (driver, nodes and tasks) in order to execute the tasks the best way as possible. I have some rough results (I mean, without any fancy network or I/O graphics, just some measurements using milliseconds) that I'd like to share, in order to better provision the hardware needed for each job.
My environment is running on Amazon's EC2, with 1 instance for the server and the client (although the client is not configured to run locally), and up to 5 instances for the nodes. All instances have the same configuration, and there is one type of task that is executed many times, with different sets of data. Each task is independent of each other, there isn't any kind of data sharing and everything runs in memory, so there is no I/O happening.
For this test, I have 412 inputs, which are broken in 412 tasks. The server is configured to use jppf.load.balancing.algorithm = manual, jppf.load.balancing.strategy = manual and strategy.manual.size = 24. Each node is set up to use 24 threads, and the client's classes and all required libraries are in the node's classpath, in order to prevent the remote classloader to take place. All instances are in the same network.
The results I have so far are:
- Command line (without JPPF), with 24 threads using a ThreadPool: completed in ~513 seconds
- Running one node: ~1.017 seconds
- 3 nodes: ~320 seconds
- 5 nodes: ~210 seconds
1) My very first question is: what do you think of such results? Are they expected, or by just looking at them there is something I could do to improve it?
2) Why did it take so much more time to execute the tasks when running on a single node when compared to the command line version, which uses the exact same code? Again, is it an expected result? (please don't get me wrong here, I am asking just to make sure that I will make realistic projections for my needs).
One interesting thing to note is that, although the single node test took twice as much as the simple command line test, the average time to process each step of the task was virtually the same.