JPPF, java, parallel computing, distributed computing, grid computing, parallel, distributed, cluster, grid, cloud, open source, android, .net
JPPF

The open source
grid computing
solution

 Home   About   Features   Download   Documentation   On Github   Forums 
January 19, 2021, 09:47:28 PM *
Welcome,
Please login or register.

Login with username, password and session length
Advanced search  
News: New users, please read this message. Thank you!
  Home Help Search Login Register  
Pages: [1]   Go Down

Author Topic: Peer Grid only ever uses one node  (Read 5706 times)

codemonkey

  • JPPF Council Member
  • *****
  • Posts: 138
Peer Grid only ever uses one node
« on: February 22, 2013, 10:57:37 PM »

Hi Laurent,

I have a peer network setup and I've seeing the behavior that only one node ever gets used at time.

There are a lot of jobs sitting in the queue of the other server and never get dispatched to the peer grid.

I have the load balancer set to proportional with the following settings:

strategy.optimized_strategy.performanceCacheSize = 2000
strategy.optimized_strategy.proportionalityFactor = 2
strategy.optimized_strategy.initialMeanTime = 5000000000
strategy.optimized_strategy.initialSize = 4

I highly doubt that these settings have anything to do with it.

I would expect that the peer server's nodes will be utilized should the bandwidth be available.

Can you point me to something that I might have overlooked?

Thanks in advance

CM







« Last Edit: February 23, 2013, 11:46:58 AM by codemonkey »
Logged

lolo

  • Administrator
  • JPPF Council Member
  • *****
  • Posts: 2262
    • JPPF Web site
Re: Peer Grid only ever uses one node
« Reply #1 on: February 24, 2013, 08:05:40 AM »

Hi CM,

The first thing I would do is set "proportionalityFactor = 1" instead of 2, unless it was a typo on your side?
From all the testing I have done, the value one is always the one that provides the best load distribution.

Also, what configuration are you using on the other driver?

Could you try with an other algorithm, for instance "manual" with size=100 (or some resonable size) and let us know if this makes a difference with regards to what is sent to the peer driver?

Thanks,
-Laurent
Logged

codemonkey

  • JPPF Council Member
  • *****
  • Posts: 138
Re: Peer Grid only ever uses one node
« Reply #2 on: February 25, 2013, 03:53:33 PM »

Hi Laurent and thank you very much for the reply.

I think the 'proportionalityFactor = 2' was a setting of a previous release of JPPF Driver that has been merged into the current jppf-driver.properties file. More importantly is that changing this setting to 1 does not send any more work over to the peer. The peer still only ever gets one job a time while many are sitting queued from the server from which it is attached.

I've changed the load balancer to manual and size of 100 as you suggested, unfortunately the results are the same.

I've attached the jppf-driver.properties. This config is used on both drivers, the non peer and the peer. With the difference being that on the machine I want to make a peer to other has the peer settings enabled. We use a manual peer configuration as well.

All your help is greatly appreciated.

CM

Logged

codemonkey

  • JPPF Council Member
  • *****
  • Posts: 138
Re: Peer Grid only ever uses one node
« Reply #3 on: February 28, 2013, 05:18:58 PM »

Hi Laurent,
I'm at a loss with this. All I can tell at this point is that the driver (non peer) only sees the peer as it exactly sees it, one node, therefore only sends tasks to that one node.

For example, on the peer machine I have a driver and 2 nodes. I see work being swapped between those 2 nodes but never work concurrently, regardless of load balancer settings.

Any other settings that could be the culprit? Managements settings perhaps?

Thanks again very much for all your help. Dying to get this screaming across all available resources

CM



Logged

codemonkey

  • JPPF Council Member
  • *****
  • Posts: 138
Re: Peer Grid only ever uses one node
« Reply #4 on: March 06, 2013, 01:29:16 PM »

Hi,
I've confirmed using every possible load balancer setting that only one job will be submitted to the peer server at a time.

Is this the expected behavior or can a bug be logged for this?

Sincerely,

CM
Logged

lolo

  • Administrator
  • JPPF Council Member
  • *****
  • Posts: 2262
    • JPPF Web site
Re: Peer Grid only ever uses one node
« Reply #5 on: March 14, 2013, 08:30:32 AM »

Hi CM,

This is indeed an issue when using multiple connected drivers. I have reopened JPPF-34 to keep track of this. I am planning on a resolution in the upcoming JPPF 3.3.

In the mean time, if you could provide some information about your jobs and tasks (how many tasks per job, average duration, memory footprint, CPU vs. I/O-bound, etc.), I would like to try and simulate what you observe and find how the task distribution can be improved with the current version.

Sincrely,
-Laurent
Logged

codemonkey

  • JPPF Council Member
  • *****
  • Posts: 138
Re: Peer Grid only ever uses one node
« Reply #6 on: March 14, 2013, 05:56:56 PM »

Hi Laurent, great to hear from you.

I've attached a screen shot of the server stats tab that includes all timings for what is considered an overall run of our calculations done on the grid.

In regards to memory footprint in this particular scenario it averages on the driver around 40% of allocated heap (xmx=2048m)
and on the 8 nodes/2 processing threads average around 60% each (xmx=3096m)

IO Bound vs CPU is roughly an 80/20% split, so pretty IO intensive.

I hope I've provided adequate information here for you but please don't hesitate if you require more information

Thank you again!

Sincerely,

CM
 
Logged

lolo

  • Administrator
  • JPPF Council Member
  • *****
  • Posts: 2262
    • JPPF Web site
Re: Peer Grid only ever uses one node
« Reply #7 on: March 16, 2013, 01:36:40 PM »

Hi CM,

Thank you for the addtional information, this is very helpful.
So I was able to reproduce the behavior that you have described, and discovered an issue in the code (see the comments in JPPF-34).
I would like to ask you if you can try the fix, and let us know if and how it improves the tasks distributions in your environment, is this ok?
To apply the fix, download this jppf-server.jar and replace the existing one in the lib/ folder of your drivers.

Thanks for your help.
-Laurent
Logged

lolo

  • Administrator
  • JPPF Council Member
  • *****
  • Posts: 2262
    • JPPF Web site
Re: Peer Grid only ever uses one node
« Reply #8 on: March 17, 2013, 12:19:27 PM »

Sorrry, after further testing, I realized the fix introduces a memory leak in the driver. I'm working on it and will provide an update version asap.

-Laurent
Logged

lolo

  • Administrator
  • JPPF Council Member
  • *****
  • Posts: 2262
    • JPPF Web site
Re: Peer Grid only ever uses one node
« Reply #9 on: March 17, 2013, 01:03:30 PM »

OK, this is done. I have updated the jar file, you can download it from the same URL http://www.jppf.org/private/3.2.3/jppf-server.jar
-Laurent
Logged

codemonkey

  • JPPF Council Member
  • *****
  • Posts: 138
Re: Peer Grid only ever uses one node
« Reply #10 on: March 17, 2013, 03:12:10 PM »

Hi Laurent, thank you for the quick turnaround on this. I will test as soon as I can

Sincerely,

CM
Logged

codemonkey

  • JPPF Council Member
  • *****
  • Posts: 138
Re: Peer Grid only ever uses one node
« Reply #11 on: March 25, 2013, 08:25:53 PM »

Hi Laurent,

I've tried to test the fix, however the driver fails to even start, no errors or warnings.

I just replaced the existing jppf-server.jar with the one provided from the link.

If I revert this to my previous jppf-server.jar it works as expected, so looks like an issue with this jar.

Any ideas?

Thanks

CM
Logged

lolo

  • Administrator
  • JPPF Council Member
  • *****
  • Posts: 2262
    • JPPF Web site
Re: Peer Grid only ever uses one node
« Reply #12 on: March 28, 2013, 08:48:37 AM »

Can you tell me which version you are using ? I'll try and make a patch for that specific version and we'll see if that works better.

-Laurent
Logged

codemonkey

  • JPPF Council Member
  • *****
  • Posts: 138
Re: Peer Grid only ever uses one node
« Reply #13 on: March 28, 2013, 03:23:08 PM »

Hi Laurent and thanks for the reply. We're using version 3.2.1.

I've tried upgrading to 3.2.3, however in addition to the same result,  I'm getting some other errors around JMXDriverConnectionWrapper 

Given the following code to check for the driver's connection:
Code: [Select]
JMXDriverConnectionWrapper driverCon = new JMXDriverConnectionWrapper(host, port);
driverCon.connectAndWait(2000);

if (driverCon.isConnected()) {
                        _log.info("Driver is started.");
} else {
       _log.info("Driver is not started.");
}
try {
driverCon.close();
                //driverCon.close() never reaches this exception handler.....
} catch (Exception ex) {
_log.warn("Failed to close driver connection", ex);
}
return ret;

Consistently throws exceptions on the closing of the driver's connection:

Code: [Select]
ERROR [2013-03-28 09:59:01,490] sleep interrupted
java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method)
at org.jppf.management.JMXConnectionThread.run(JMXConnectionThread.java:100)
at java.lang.Thread.run(Thread.java:619)


Oddly, is that in my logs I see the following driver output

Code: [Select]
INFO  [2013-03-28 10:00:04,846] driver process id: 5032

However, re-querying the JMXDriverConnectionWrapper states that it's not connected, rightfully so as the process doesn't exist

Hopefully this helps and thanks again very much

CM
Logged

codemonkey

  • JPPF Council Member
  • *****
  • Posts: 138
Re: Peer Grid only ever uses one node
« Reply #14 on: April 18, 2013, 01:21:03 PM »



Hi Laurent, can you confirm that there is a problem with this jar?

Thank you again very much
Logged

lolo

  • Administrator
  • JPPF Council Member
  • *****
  • Posts: 2262
    • JPPF Web site
Re: Peer Grid only ever uses one node
« Reply #15 on: April 19, 2013, 08:12:43 AM »

Hello,

Yes I confirm that this jar is not working with v3.2.1. Unfortunately, I cannot generate a 3.2.1 jar, because I forgot to tag this version in SVN :(, I apologize for the inconvenience.
So I suggest that you upgrade to 3.2.3 and apply this patch, or just upgrade to 3.3 where this issue is fixed.

Sincerely,
-Laurent
Logged

codemonkey

  • JPPF Council Member
  • *****
  • Posts: 138
Re: Peer Grid only ever uses one node
« Reply #16 on: April 30, 2013, 01:13:43 PM »

Hi Laurent, I've upgraded to 3.3 as you've recommended and i no longer get errors, but unfortunately I'm seeing the same results in regards to the balancing of work over to the peer server  :'(.

I'm going to do some tests using the packaged examples to see if I can reproduce.

Sincerely

CM
« Last Edit: April 30, 2013, 01:15:27 PM by codemonkey »
Logged
Pages: [1]   Go Up
 
JPPF Powered by SMF 2.0 RC5 | SMF © 2006–2011, Simple Machines LLC Get JPPF at SourceForge.net. Fast, secure and Free Open Source software downloads