Print

[codemonkey]

Hi Laurent,

I have a peer network setup and I've seeing the behavior that only one node ever gets used at time.

There are a lot of jobs sitting in the queue of the other server and never get dispatched to the peer grid.

I have the load balancer set to proportional with the following settings:

strategy.optimized_strategy.performanceCacheSize = 2000
strategy.optimized_strategy.proportionalityFactor = 2
strategy.optimized_strategy.initialMeanTime = 5000000000
strategy.optimized_strategy.initialSize = 4

I highly doubt that these settings have anything to do with it.

I would expect that the peer server's nodes will be utilized should the bandwidth be available.

Can you point me to something that I might have overlooked?

Thanks in advance

CM

[lolo]

Hi CM,

The first thing I would do is set "proportionalityFactor = 1" instead of 2, unless it was a typo on your side?
From all the testing I have done, the value one is always the one that provides the best load distribution.

Also, what configuration are you using on the other driver?

Could you try with an other algorithm, for instance "manual" with size=100 (or some resonable size) and let us know if this makes a difference with regards to what is sent to the peer driver?

Thanks,
-Laurent

[codemonkey]

Hi Laurent and thank you very much for the reply.

I think the 'proportionalityFactor = 2' was a setting of a previous release of JPPF Driver that has been merged into the current jppf-driver.properties file. More importantly is that changing this setting to 1 does not send any more work over to the peer. The peer still only ever gets one job a time while many are sitting queued from the server from which it is attached.

I've changed the load balancer to manual and size of 100 as you suggested, unfortunately the results are the same.

I've attached the jppf-driver.properties. This config is used on both drivers, the non peer and the peer. With the difference being that on the machine I want to make a peer to other has the peer settings enabled. We use a manual peer configuration as well.

All your help is greatly appreciated.

CM

[codemonkey]

Hi Laurent,
I'm at a loss with this. All I can tell at this point is that the driver (non peer) only sees the peer as it exactly sees it, one node, therefore only sends tasks to that one node.

For example, on the peer machine I have a driver and 2 nodes. I see work being swapped between those 2 nodes but never work concurrently, regardless of load balancer settings.

Any other settings that could be the culprit? Managements settings perhaps?

Thanks again very much for all your help. Dying to get this screaming across all available resources

CM

[codemonkey]

Hi,
I've confirmed using every possible load balancer setting that only one job will be submitted to the peer server at a time.

Is this the expected behavior or can a bug be logged for this?

Sincerely,

CM

[lolo]

Hi CM,

This is indeed an issue when using multiple connected drivers. I have reopened JPPF-34 to keep track of this. I am planning on a resolution in the upcoming JPPF 3.3.

In the mean time, if you could provide some information about your jobs and tasks (how many tasks per job, average duration, memory footprint, CPU vs. I/O-bound, etc.), I would like to try and simulate what you observe and find how the task distribution can be improved with the current version.

Sincrely,
-Laurent

[codemonkey]

Hi Laurent, great to hear from you.

I've attached a screen shot of the server stats tab that includes all timings for what is considered an overall run of our calculations done on the grid.

In regards to memory footprint in this particular scenario it averages on the driver around 40% of allocated heap (xmx=2048m)
and on the 8 nodes/2 processing threads average around 60% each (xmx=3096m)

IO Bound vs CPU is roughly an 80/20% split, so pretty IO intensive.

I hope I've provided adequate information here for you but please don't hesitate if you require more information

Thank you again!

Sincerely,

CM
 

[lolo]

Hi CM,

Thank you for the addtional information, this is very helpful.
So I was able to reproduce the behavior that you have described, and discovered an issue in the code (see the comments in JPPF-34).
I would like to ask you if you can try the fix, and let us know if and how it improves the tasks distributions in your environment, is this ok?
To apply the fix, download this jppf-server.jar and replace the existing one in the lib/ folder of your drivers.

Thanks for your help.
-Laurent

[lolo]

Sorrry, after further testing, I realized the fix introduces a memory leak in the driver. I'm working on it and will provide an update version asap.

-Laurent

[lolo]

OK, this is done. I have updated the jar file, you can download it from the same URL http://www.jppf.org/private/3.2.3/jppf-server.jar
-Laurent

[codemonkey]

Hi Laurent, thank you for the quick turnaround on this. I will test as soon as I can

Sincerely,

CM

[codemonkey]

Hi Laurent,

I've tried to test the fix, however the driver fails to even start, no errors or warnings.

I just replaced the existing jppf-server.jar with the one provided from the link.

If I revert this to my previous jppf-server.jar it works as expected, so looks like an issue with this jar.

Any ideas?

Thanks

CM

[lolo]

Can you tell me which version you are using ? I'll try and make a patch for that specific version and we'll see if that works better.

-Laurent

[codemonkey]

Hi Laurent and thanks for the reply. We're using version 3.2.1.

I've tried upgrading to 3.2.3, however in addition to the same result,  I'm getting some other errors around JMXDriverConnectionWrapper 

Given the following code to check for the driver's connection:

Code: [Select]

JMXDriverConnectionWrapper driverCon = new JMXDriverConnectionWrapper(host, port);
driverCon.connectAndWait(2000);

if (driverCon.isConnected()) {
                        _log.info("Driver is started.");
} else {
       _log.info("Driver is not started.");
}
try {
driverCon.close();
                //driverCon.close() never reaches this exception handler.....
} catch (Exception ex) {
_log.warn("Failed to close driver connection", ex);
}
return ret;

Consistently throws exceptions on the closing of the driver's connection:

Code: [Select]

ERROR [2013-03-28 09:59:01,490] sleep interrupted
java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method)
at org.jppf.management.JMXConnectionThread.run(JMXConnectionThread.java:100)
at java.lang.Thread.run(Thread.java:619)


Oddly, is that in my logs I see the following driver output

Code: [Select]

INFO  [2013-03-28 10:00:04,846] driver process id: 5032

However, re-querying the JMXDriverConnectionWrapper states that it's not connected, rightfully so as the process doesn't exist

Hopefully this helps and thanks again very much

CM

[codemonkey]

Hi Laurent, can you confirm that there is a problem with this jar?

Thank you again very much

[lolo]

Hello,

Yes I confirm that this jar is not working with v3.2.1. Unfortunately, I cannot generate a 3.2.1 jar, because I forgot to tag this version in SVN , I apologize for the inconvenience.
So I suggest that you upgrade to 3.2.3 and apply this patch, or just upgrade to 3.3 where this issue is fixed.

Sincerely,
-Laurent

[codemonkey]

Hi Laurent, I've upgraded to 3.3 as you've recommended and i no longer get errors, but unfortunately I'm seeing the same results in regards to the balancing of work over to the peer server  .

I'm going to do some tests using the packaged examples to see if I can reproduce.

Sincerely

CM