Print

[A.gibault]

Hello,

I'm currently doing an internship in a team of 4 students. Our project use all computers of a company in other to process tasks. The grid we are trying to target is around 10 000 computers.
The computers in the grid won't be always available, we can only use them during their idle time. The server would need to monitor which node is available and dynamically distribute the tasks through the grid. In addition we'll have to manage huge amount of data (hundred of TBytes) and distribute it in a very efficient way.


Therefore I would like to know the following :

• how powerful do(es) the server(s) need to be in order to manage this huge grid? for instance server vs computer ratio?
• We are wondering if JPPF is the right tool to use for this matter? One of our other options is BOINC. Could you give us an advice on which one to use? 

Thank you really much.

Arnaud Gibault

[lolo]

Hello Arnaud,

Given my involvement in JPPF, I don't think I can give a truly objective answer, but I can at least give you some elements.

Given your requirements, I believe JPPF can fulfill all of them:

• dealing with nodes that come in and out of the grid at any time: this is part of the dynamic scalability features in JPPF and shouln't be an issue
• nodes available when the computer is idle: this is a built-in feature in the JPPF nodes, I invite you to take a look at how we implement it. This is not screensaver-based, though, and you may find it works differently from what BOINC offers
• distribution of large amounts of data: this is not a built-in feature in JPPF, however the ease of integration with Java-based distributed cache frameworks (such as Ehcache or Hazelcast or others) makes it almost a no-brainer. I'm not sure how BOINC deals with that, do you have any input on this?
• Regarding how powerful the server(s) should be, this isn't an easy question to answer, and will probably require some trial and error approach, as well as some tuning. With the grid size you are considering, I would recommend a multi-server topology, with no more than a few thousand nodes for each server. The servers themselves do not do much heavy processing, they are mostly dedicated to dispatching work amongst the nodes, so they are essentially network I/O bound. For a server handling 2,000 nodes I would start testing with a 8 to 16 cores machine with at least 1 GB of memory per core. These numbers are not validated by experience, so I would then test various configurations and benchmarks to optimize the hardware utilization. Also, consider the ratio of total nodes vs. active nodes, for instance if they're not all in the same time zone, they won't all be active at the same time.

I hope this helps,
-Laurent

[A.gibault]

That helps me a lot, thanks!