Print

[brian]

Hi,
I am interested in trying out local nodes and "pure P2P topology" mentioned in the docs @ http://www.jppf.org/doc/5.2/index.php?title=Configuring_a_JPPF_server

This is particularly useful if you intend to create a pure P2P topology, where all servers communicate with each other and only one node is attached to each server.

I'd appreciate a pointer on how to get started setting this up. I do not want to use UDP discovery, happy to manually specify driver hostports, however they would not all be known at startup. Thinking client app would on-demand spawn multiple driver+local_node pairs in the cloud then issue tasks to them. Does this design work in theory? When I have a management UI which driver should it connect to, is it the same behavior no matter which driver it connects to? I already have a zookeeper instance running, could JPPF support zk discovery similar to hazelcast? https://github.com/hazelcast/hazelcast-zookeeper

My attraction to the driver+local_node pairs deployment is that it seems masterless like Hazelcast where members can come and go and the grid stays available. Wondering if JPPF can offer a similar deployment with the advantage of still being able to ask a JPPF driver for central view of what is currently running as we can do with single driver.

Appreciate any tips or links to past discussions, I haven't studied JPPF offering in this space.

[lolo]

Hello,

You just put the nail right into the weak point Indeed, the p2p discovery does not offer a dynamic way to discover other servers in environments where UDP multicast is disabled or forbidden, such as cloud envirronments.

One alternative I can see, in the current state of things, would be as follows:

- have all drivers use a JPPF configuration plugin to load the configuration from a source other than a properties file on the file system
- have each driver in the p2p topology register itself at startup time so it becomes part of the configuration available from the configuration source. This could be done from a driver startup class
- have that startup class periodically poll the configuration source for changes in the topology and restart the driver via the local jmx API when a change occurs

Even to me, this approach seems quite heavy and not very efficient performance-wise, but currently that's the only way I see to make discovery dynamic.

What I can propose is a true dynamic mechanism, similar to the node's connection strategy, that would allow you to use any external lookup mechanism or tool to discover the other drivers. Since I have frozen all new developments on the upcoming v5.2, I propose to implement this as an  enhancement for the first maintenance release (v5.2.1). Will ths be OK?

Thanks,
-Laurent

[brian]

That's helpful, thank you.

Assume I use one of these two schemes to have dynamic driver topology, how is the behavior then different from single driver from client / UI point of view? For example
- when one driver dies, what happens to its jobs, are they "taken over" by another driver?
- when a management UI is attached to one of the drivers, does it see complete status for all drivers,nodes,jobs? if not, how would you suggest a user get the full grid big picture view of grid status?
- is there any notion of a leader driver and leader election?
- when a driver dies what is effect on clients who were connected to it and submitted jobs, what about clients connected to other drivers?
- would every driver be connected to every other? i.e. a star topology with bidirectional arrows between each driver pair.
- do you foresee performance suffering with this type of topology? how many drivers should it scale to?
- do you foresee buggy behavior in this type of topology? i.e. are there many users deploying this way.

Making this easier to do in a 5.2.1 update would be great, though I want to be sure this sort of deployment would work for me before asking you to add a feature to make it easier.

[lolo]

Hello,

To try and answer your questions:

- when one driver dies, what happens to its jobs, are they "taken over" by another driver?

The jobs are taken over by whoever submitted them, that is, either a driver or a client. To clarify: when a driver A connects to a driver B, it is seen by driver B as a node. Each driver has a node failure detection and failover mechanism that causes it to re-send the tasks that were sent to this node to another node. Thus, the same mechanism applies to server-to-server connections. When the submitter is a client, the job remains in the client queue until either the driver connection is reestablished or another driver connection is available, in which case it will resubmit the job's tasks for which no result has been received.

- when a management UI is attached to one of the drivers, does it see complete status for all drivers,nodes,jobs? if not, how would you suggest a user get the full grid big picture view of grid status?

A driver only sees its immediate neighbors: nodes and other drivers it is directly connected to. Therefore, to see the full grid topology, the managment UI needs to connect to each and every driver. This makes me realize that whatever discovery mechanism I proposed for driver-to-driver connections should also be used for clients.

- is there any notion of a leader driver and leader election?

No, there is no such thing in JPPF.

- when a driver dies what is effect on clients who were connected to it and submitted jobs, what about clients connected to other drivers?

I believe I already addressed that in my first answer above. A client can have multiple connections to each driver, and can be connected to multiple drivers concurrently. When a driver to which a job was submitted dies, that job (actually the tasks for which no result was received) is resubmitted to another driver connection, according to the connection's priority, the job client-side SLA and the load-balancer settings

- would every driver be connected to every other? i.e. a star topology with bidirectional arrows between each driver pair.

I assume you're refering to the discovery mechanism proposal. It would give each driver the choice to accept or deny connection to any other driver. Therefore you can chose the shape of the entire grid topology by defining its local properties at each driver endpoint. In fact, I'm also thinking it would be a great opportunity to introduce the notion of connection pools (as for the clients) for driver-to-driver connectivity.

- do you foresee performance suffering with this type of topology? how many drivers should it scale to?

Yes, there will be a performance impact. In a p2p topology with N drivers, a job may be routed through up to N drivers to reach a node on which to execute, whereas on a master/worker topology, it's always 1 driver. In short this means the transport time is in O(N) vs. O(1). Similarly, network traffic for the whole grid will grow in O(N²) vs O(N). You will gain resilience, but not performance. From the feedback I've had, multi-server topologies typically remain small, with less than 10 servers involved.

- do you foresee buggy behavior in this type of topology? i.e. are there many users deploying this way.

From the little but precious feedback I've had in the past, load-balancing could be very tricky. Maybe less so if you stick to pure p2p topologies where each driver has a single local node.
In any case, any problem that may occur will be more complex to investigate and resolve.

I hope this clarifies,
-Laurent

[brian]

Hi Laurent,

I'd first like to offer my condolences on the recent tragedy in your country. It is such a shame and I grieve with you.

The jobs are taken over by whoever submitted them

Thank you for explaining jppf multi-driver architecture. I think I'll avoid the pure p2p deployment given your response, opting for the master/worker architecture with backup master.

Being able to have clients and nodes discover drivers (in my case via zookeeper) sounds useful, so they can all re-attach to backup driver if primary fails. But unless I'm missing something it sounds like having drivers know about each other does not offer much benefit in a cloud deployment so I'd probably keep them unaware of each other.

[lolo]

Hello,

Thank you very much for your kind words, we have been through a lot in the past 2 years, and they are wholeheartedly appreciated.

The original intent of driver-to-driver communication was to solve two problems:
- how to avoid the single point of failure induced by a master/worker topology
- how to extend the possible number of nodes beyond the limit of 64k TCP ports on a single machine

Since then, many features were implemented and that there are now other setups that achieve these goals, each with its own tradeoffs:
- a client can connect to multiple drivers in either or both failover and load-balancing configuration
- you can always have a backup driver sitting idle, coupled with a node connection strategy, where nodes attach themselves to the backup when the primary dies, and vice-versa
- offline nodes will allow a much greater number of nodes, since they only connect intermittently to fetch tasks and return the results. Coupled with a connection strategy and one or more backup drivers, this addresses the goals above. For instance, this is what we do with the Android nodes.

Another possible use, which I have not seen implemented, of multiple connected drivers is as gateways to physically, logically or administratively separated grid infrastructures that may chose to join or leave a global grid.

-Laurent

[brian]

Like the nodes, it would be great to be able to use DriverConnectionStrategy in the client as well in a 5.2.1 release. I don't need it in the drivers though.
Edit: also the management GUI

[brian]

Hi Laurent,
What did you think of the idea of extending the DriverConnectionStrategy function to clients and admin UI as well as nodes?

[lolo]

Hi Brian,

What did you think of the idea of extending the DriverConnectionStrategy function to clients and admin UI as well as nodes?

As promised, it was delivered as part of v5.2.1:
- documentation for drivers
- documentation for clients. Since the admin GUI is implemented on top of the client APIs, this extension implicitely applies as well.

-Laurent

[brian]

Aha, I will give it a spin straight away then, thanks!

[brian]

Note, the Tip here references the wrong property for disabling discovery: http://www.jppf.org/doc/5.2/index.php?title=Custom_discovery_of_remote_drivers#Tip

[lolo]

What do you mean exactly? The described property "jppf.remote.execution.enabled" seems to be the proper one for a client configuration. What would you replace it with?

Thanks,
-Laurent

[brian]

My understanding is driver discovery in client is controlled by jppf.discovery.enabled = false as per http://www.jppf.org/doc/5.2/index.php?title=Client_and_administration_console_configuration whereas this remote execution flag is about whether tasks are to be executed locally or remotely.

[lolo]

Thank for the clarifcation! I suppose some part of the documentation is not so clear though: "jppf.discovery.enabled" toggles beween automatic discovery via UDP multicast (when true) and manually defined connections in the configuration (when false), whereas "jppf.remote.execution.enabled = false" disables both. I created a doc bug JPPF-477 Clarify usage of 'jppf.discovery.enabled" and "jppf.remote.execution enabled" in client config for this.

Sincerely,
-Laurent