Print

[cb4jppf]

Dear All,

Thanks and kudos for JPPF, its usability, and all of its features.  Please excuse my lack of familiarity, but I've searched these fora and the examples to no avail so far.

As such, the compute nodes in the the subject cluster can send requests to external platforms (outside the cluster) via TCP, but they are restricted from receiving TCP messages initiated from outside the cluster.

Is this a show stopper (does this prevent my use of JPPF on such a linux cluster)?

To rephrase, I'd like to run my JPPF-server process on my local server platform, which would in turn submit PBS-job requests via ssh to the PBS front end platform, where each such PBS-job consists of starting a JPPF-node-process (on a compute node determined by PBS), requesting from the external JPPF-server and running JPPF-tasks, and then terminating the JPPF-node-process.

1) Can the JPPF-server be configured to wait for the PBS jobs to flow through the PBS job queue until the JPPF-node process gets started on the compute node?
2)  Can the JPPF-node be configured to request tasks from the JPPF-server - rather than the JPPF-server (initiating the communication and) sending jobs to the JPPF-node - as well as terminate the JPPF-node process when there are no more JPPF tasks?

If so to all the above, is there such an example?

Best,
CB

[lolo]

Hello CB,

As such, the compute nodes in the the subject cluster can send requests to external platforms (outside the cluster) via TCP, but they are restricted from receiving TCP messages initiated from outside the cluster.
Is this a show stopper (does this prevent my use of JPPF on such a linux cluster)?
To rephrase, I'd like to run my JPPF-server process on my local server platform, which would in turn submit PBS-job requests via ssh to the PBS front end platform, where each such PBS-job consists of starting a JPPF-node-process (on a compute node determined by PBS), requesting from the external JPPF-server and running JPPF-tasks, and then terminating the JPPF-node-process.

Unfortunately, this is not compatible with the current design of the JPPF grid, which uses its own scheduler, load-balancer, workload manager, etc...
One possiblity could be to have the entire JPPF grid running within the cluster (especially the client and server), and have the PBS front-end submitting jobs via an integrated JPPF client.
I'm not familiar with PBS/Torque, so I'm not sure how difficult it is to integrate a JPPF client into it. However, if that part is feasible, then I believe the erst will be relatively easy:
- JPPF jobs can be assigned to specific nodes (via job SLA and execution policies)
- starting and stopping JPPF nodes on demand may require a little bit of customization, however we already do most of that in the JPPF code. I invite you to look at the code for the package org.jppf.process, and especially for the ProcessLauncher class.

I hope this answers your questions.

Sincerely,
-Laurent

[cb4jppf]

Hello Laurent,

Thanks for your response.  I checked the links that you've provided, and sorry that I did not recognize anything in conflict with what I'm suggesting.  Granted that JPPF must use its own scheduler, etc., I did not intend to suggest otherwise.  Perhaps I could rephrase my question, and please excuse my repetition.

In my case, each of the 6 (for now) compute nodes on the cluster has 32 cores.  For example, as you may know, one could compose a script to start, say, 4 PBS-jobs, where each job does nothing more than start a JPPF-node process and requires, say, 8 threads.  Thus, PBS would be able to run all 4 such jobs on a single PBS compute node, assuming that no other jobs were running on that node, according to the PBS-job directives, etc., included in the script.

In my hypothetical scenario, I would first start the (JPPF-client and) JPPF-server process running on my local server platform, where the JPPF-server's custom initialization at start up could involve starting such PBS-jobs by ssh/running such a script on the PBS front end.  Then the JPPF-server process would wait for the JPPF-node processes to start up while the corresponding PBS-jobs are waiting in the PBS-job queue.  As each such PBS-job starts, it starts a corresponding JPPF-node process.  Each such JPPF-node then contacts the JPPF-server process, requests one JPPF-task from the JPPF-server, completes that task, and sends the results back to the JPPF-server before requesting another task.  Once the JPPF-node receives an empty response for its next task request from the JPPF-server, it terminates - along with the corresponding PBS-job.  Meanwhile, the JPPF-server can send the results of the tasks back to the JPPF-client.

Again, please let me know if any of this is in conflict with the currently configurable JPPF policy.

Best,
CB

[lolo]

Hi CB,

Thank you for the clarification.
If you don't mind, I would like to summarize my understanding of what you want to achieve, and I would appreciate if you could correct me whenever I'm wrong.
- you start a JPPF client and server on your local server
- upon initialization, the JPPF server will submit one or more PBS jobs via ssh + some remote script
- here is the point where I'm not fully clear: what is the exact role of the PBS jobs? My understanding is they would be used to start one or more JPPF nodes on the cluster, acting as a provisioning service. Is this correct? Is there anything else to it?
- once a node is started, it will request tasks, one at a time, from the JPPF server, execute each task, and send the results back to the server.
- when a node receives an empty response for a task request, it terminates, and so does the corresponding PBS job

Assuming the above points are correct, I believe the main diffculty will come from the communication model between JPPF server and nodes. By design, it is not the nodes that request tasks, but the server that sends them to the nodes. So the problem is to figure out how to tell the nodes to terminate. I can see two ways to do that:

1) You can add a custom initialization to the server, which can subscribe to JPFF jobs status events. Upon receiving a job completion event, it would use the JPPF management capabilities to terminate the nodes. Note that you could use this same server initialization to submit the remote PBS jobs, upon receiving notification that a new job was sent by the JPPF client.
The drawback of this solution is that it requires that the JMX connection between server and nodes be allowed in your cluster. This may not be the case, since each node will need to listen to a distinct port for the JMX service. Typically this will not work in a secure or firewalled environment.

2) Another solution is, once the JPPF job is completed, to submit a broadcast job, to trigger the shutdown/termination of each node. The broadcast job would contain a single task, this task would be broadcast to each node and terminate the node upon execution. Any execution policy can be applied to a broadcast job, as for any normal job, so only the designated nodes will be terminated.

I hope this answers your questions, and please feel free to comment.

Sincerely,
-Laurent

[cb4jppf]

Hi Laurent,

Thanks for the info, and please excuse my delay in responding.  I'm waiting for confirmation on the following from the cluster admin, but I'm assuming for now that the cluster's compute nodes are unable to listen for JMX events.  Given that, is it possible for my JPPF server to interact with the cluster front node via SSH (qsub, qstat) in order to both submit PBS job scripts as well as monitor PBS job status?

Best Regards,
CB

[lolo]

Hello CB,

All you need is a hook in the JPPF server to execute the code for sending the commands via SSH. I think the best would be to use a server startup class. From this custom code, you could create a shell process (for instance using java.lang.ProcessBuilder) that sends the command, and then parse the output. You could wrap the different PBS commands into separate java methods and, as I mentioned earlier, tie them to the JPPF jobs life cycle events, so you could for instance send a command that kills a node when notified that a JPPF job is complete.

Does this answer your questions?

Sincerely,
-Laurent

[cb4jppf]

Hello Laurent,

Yes, that answers my question quite well, thanks.

Re command that kills a node:
I'm not sure which type of node you mean, JPPF-server or JPPF-node.  I'm assuming that I would not need a JPPF-node and the that JPPF-server's ProcessBuilder would submit qsub and qstat commands via ssh directly to the cluster's front node, true?  Thus, I would not need to kill the JPPF-server until the user confirms that there are no more jobs needed, also true?  Please explain.

Best Regards,
CB

[lolo]

Hi CB,

'm not sure which type of node you mean, JPPF-server or JPPF-node

I meant JPPF-node.

I'm assuming that I would not need a JPPF-node

Now I'm confused.  If you want to execute a JPPF job, you need at least one JPPF node to execute it, that's how JPPF works. Is there something I missed in your previous explanations? It was my understanding that the PBS commands were intended to start, monitor and kill a JPPF-node, but it looks like I misunderstood, could you confirm?

Thanks,
-Laurent

[cb4jppf]

Hi Laurent,

Ok, I think you've cleared up my confusion, which perhaps I had unintentionally propagated back to you.

All you need is a hook in the JPPF server to execute the code for sending the commands via SSH.

By this, I thought you had meant that the JPPF-server's ProcessBuilder would send jobs directly to the OS on the cluster front-end without a JPPF-node, but now...

If you want to execute a JPPF job, you need at least one JPPF node to execute it, that's how JPPF works.

...I understand that, since there must be a JPPF-node running somewhere, the following would be consequences of the JPPF-client's job request, ...

1) the listening ProcessBuilder sends an ssh-command to the cluster front-end in order to start a JPPF-node on the cluster front-end itself,
2) the JPPF-server (soon discovers the JPPF-node now running on the cluster front-end and) sends the job (tasks) to the JPPF-node, and
3) after the job completes, the still listening ProcessBuilder sends an ssh-command to the cluster front-end in order to kill the remote JPPF-node...

..., true?

(I'm still just thinking out loud here, and thanks for your patience.)

If I may suggest an alternative with a slight modification in order to somewhat reduce resource consumption on the cluster's front-end node, the JPPF-server's ProcessBuilder could instead start the JPPF-node on the local machine rather than on the cluster front-end node, and kill the now local JPPF-node after the job completes.  Meanwhile, the local JPPF-node would submit PBS jobs (via ssh <cluster>; qsub...) directly to the OS on the cluster front-end as well as monitor job progress (via ssh <cluster>; qstat...).

Your thoughts?

Best,
CB

[lolo]

Hi CB,

1) the listening ProcessBuilder sends an ssh-command to the cluster front-end in order to start a JPPF-node on the cluster front-end itself,
2) the JPPF-server (soon discovers the JPPF-node now running on the cluster front-end and) sends the job (tasks) to the JPPF-node, and
3) after the job completes, the still listening ProcessBuilder sends an ssh-command to the cluster front-end in order to kill the remote JPPF-node...
..., true?

That's my understanding and that's definitely in the range of possibilities for JPPF.

If I may suggest an alternative with a slight modification in order to somewhat reduce resource consumption on the cluster's front-end node, the JPPF-server's ProcessBuilder could instead start the JPPF-node on the local machine rather than on the cluster front-end node, and kill the now local JPPF-node after the job completes.  Meanwhile, the local JPPF-node would submit PBS jobs (via ssh <cluster>; qsub...) directly to the OS on the cluster front-end as well as monitor job progress (via ssh <cluster>; qstat...).

Sorry, I'm lost on this one    What you suggest is certainly possible, but I fail to see the benefit of using JPPF in this scenario. You might be better off with a simple client (or shell script) that sends the PBS commands.

There's clearly something I'm missing here. Could you please clarify what it is the PBS-nodes are supposed to be doing and what, in your mind, the JPPF-nodes are supposed to do? What benefit do you expect from JPPF witth this kind of topology?

Thanks for your time,
-Laurent

[cb4jppf]

Hi Laurent,

My colleague already has a working version of his code using JPPF and running on 3 local Macs.  Admittedly one of the Macs has the JPPF-client and JPPF-server  running as well as one JPPF-node.  His JPPF-server code takes a large file (100+ GB) as input and breaks it up into pieces to be operated upon by the JPPF-nodes running on the Macs.

I agree (as does my colleague) that perhaps a simple client would be better.  I was just thinking that he could reuse entirely...

1) his existing GUI code,
2) his existing JPPF-client code,
3) and his existing JPPF-server code

and compose a (ProcessBuilder-listener and a) new JPPF-node in order to submit jobs to the cluster.

Presumably he would need two instances of his GUI in order to submit jobs to 1) the remote cluster and separately 2) to the local Macs.

Hope this helps.

Best,
CB

[lolo]

Hi CB,

If the goal is to submit the actual workload to the PBS cluster, then I can see several  architectures that would allow you to reuse existing code.

One way would be to use the JPPF grid as a front-end to the PBS cluster, by submitting JPPF-jobs made of a number of tasks, each task creating a shell process that would submit and monitor a PBS job via SSH. So basically each task would be a surrogate to a PBS job, you could have many running in parallel, with the actual heavy lifting being done by the PBS cluster. This would involve moving the JPPF-server code to the nodes, but this can be done easily if you implemented it as a JPPF-server startup class: just make it a JPPF-node startup (just a matter of adding JPPFNodeStartupSPI to the implemented interfaces), and it will be downloaded by the JPPF-nodes from the JPPF-server.


If you don't need multiple JPPF-nodes to submit multiple PBS-jobs, one option is to use a single JPPF-client with local execution enabled, remote execution disabled, and no JPPF-server or JPPF-node. You can use the same JPPF-client APIs (including JPPF jobs and tasks) as for remote execution, except that the JPPF jobs will be executed in the same JVM as the JPPF client. Again, you would have to move the JPPF-server code to the JPPF-client, but I'm hoping it would be relatively easy.

If you still want the choice to have the actual work done by the PBS cluster or by the JPPF-grid, I would suggest adding an option in the GUI to select either one, and have the JPPF-client use this choice to determine whether to submit a JPPF-job made of tasks that submit work to the PBS cluster, or that do the actual work in your local cluster.

Is any of this making sense?

Thanks,
-Laurent

[cb4jppf]

Hi Laurent,

Thanks again for all your efforts and, in advance, for your continued patience.  Hopefully, there may be others that share in my confusion and may benefit as well from reading your responses.

At first reading, most of what you've said makes sense to me, but I'm not quite sure how to reconcile some of the consequences of what you've said with the documentation as I understand it.  (Perhaps the online documentation needs some updates as I see references, for example, to missing contents or missing sections (2.4.1) and chapter (6).  Also, when I use the online search field, I'm unable to find, for example, any of the known, existing occurrences of "SLA".)

I'm also assuming for now that my colleague, though he is currently the primary developer, would have a limited stake in this deliverable to the extent that this deliverable is a major part of his degree program and that he would have reduced interest in making any code changes after he graduates.   So, I'm speaking on behalf of the remaining stake-holders, including both the end-users as well as future maintenance developers, presumably including myself to some degree.   As such, and though my colleague has indicated that he plans to implement a separate GUI for cluster jobs, I would nonetheless prefer a single implementation and hence a single GUI (or set of GUI's) that would hypothetically provide a common point of access for the following purposes:

1) users submitting and monitoring both local and remote jobs,
2) developers maintaining a single copy of the resulting shared code library.

Hence, I'm concerned for example that, as I recently observed in the illustration of the "Job Data" panel of the administration console in http://www.jppf.org/doc/v2/index.php?title=A_first_taste_of_JPPF, the resulting display is organized by JPPF-server and thus apparently relies on the existence of one or more instances of the JPPF-server in order to display the job progress.  Thus, in your suggested scenario...

If you don't need multiple JPPF-nodes to submit multiple PBS-jobs, one option is to use a single JPPF-client with local execution enabled, remote execution disabled, and no JPPF-server or JPPF-node...

..., I'm wondering whether the absence of a running JPPF-server (and JPPF-node for that matter) would in effect render that display unusable and leave the user without any such tool in order to monitor job progress.

Also, for remote jobs I would prefer to avoid using resources on the remote PBS-cluster's front end node for the only purpose of running remote JPPF-nodes that could otherwise be running on the user's local Mac node.  Admittedly, such a decision need not be made at design time and could, by design, be deferred to the user at time of job submittal by providing the user with the choice of starting the JPPF-node either on the local Mac node or the remote PBS-cluster front end node, though not without the additional (though perhaps marginal) coding complexity of implementing the corresponding two additional types of JPPF-nodes.

Furthermore, according to http://www.jppf.org/doc/v2/index.php?title=JPPF_Overview

nodes can only process one job at a time; however they can execute multiple tasks in parallel

Hence, as I believe you've already suggested, for each remote job submitted by the JPPF-client, the proposed job-event listener would start and subsequently kill a single JPPF-node - running on either a local node or the remote PBS cluster front end node.  That JPPF-node would survive for the life of the given job and take responsibility for running and monitoring the included JPPF-tasks on the PBS cluster.  I'd appreciate your suggestion as I'm still not quite sure whether to implement a RemoteTaskNode and/or a RemoteTask.

Your thoughts?

Best,
CB

[lolo]

Hi CB,

(Perhaps the online documentation needs some updates as I see references, for example, to missing contents or missing sections (2.4.1) and chapter (6).  Also, when I use the online search field, I'm unable to find, for example, any of the known, existing occurrences of "SLA".)

Could you let us know where you found these dead references? We should be able to fix those quickly.
For the search on "SLA", I don't know why it doesn't work, it's probably a limitation of the wiki. You can look for "service level" instead, this'll provide more useful results.
In the meantime, you can find the documentation dedicated to the SLA here

Regarding the local execution configuration, you are right, there is no monitoring occurring in this context. I was thinking it might be convenient, since you use the exact same code to execute remotely or locally. However, if monitoring via the JPPF gui tool is a requirement, then local execution is not a suitable approach.

... though not without the additional (though perhaps marginal) coding complexity of implementing the corresponding two additional types of JPPF-nodes.

Actually, there is still something that is not clear to me: what would be running on the PBS cluster? would it be
1) a PBS-specific kind of job (not using the JPPF APIs)?
2) or the same kind of JPPF-job as for local jobs, in a JPPF-node that would be started in the PBS cluster for that purpose?
I'm still unclear on whether you want to run JPPF-nodes (for the life span of a job) within your PBS cluster or not.

If the answer is 1), then you can use JPPF-jobs with 2 different kinds of JPPF-tasks:
- either JPPF tasks that perform the actual job execution on the user's local environment,
- or JPPF tasks used as a wrapper for submitting and monitoring a PBS-job. Here I suppose a single JPPF-task would correspond to a PBS-job, so the JPPF-job would contain only this one JPPF-task.
The option provided in the GUI would determine which kind of JPPF-job to compose and submit.

If the answer is 2) (JPPF-node started within the PBS cluster), then it is just a matter of deciding where to start the node, either locally on the user's machine, or remotely in the PBS-cluster.
My take on this is that, since this information would be known at the time the JPPF-job is submitted, if would be easier to start the JPPF-node, or the script that starts the node for remote nodes, from within the client itself. This is definitely less complex than listening to JPPF-job events in the JPPF-server: start the JPPF-node before submitting the JPPF-job, kill it when the job is complete.

Is this making sense?

-Laurent

[cb4jppf]

Hi Laurent,

(Yes, most of what you've said makes sense, thank you.)

Could you let us know where you found these dead references? We should be able to fix those quickly.

1) Please see "The call to runner.createJob() is exactly what we saw in the previous section 2.4.1 ." in http://www.jppf.org/doc/v2/index.php?title=A_first_taste_of_JPPF.
2) Please see "We will see how to define a custom alogrithm in Chapter 6 Extending JPPF of this manual. JPPF now has 4 predefined load-balancing algorithms to compute the distribution of tasks to the nodes, each with its own configuration parameters." in http://www.jppf.org/doc/v2/index.php?title=Configuring_a_JPPF_server.

Regarding the local execution configuration, you are right, there is no monitoring occurring in this context.

Admittedly, at this typing I'm not sure whether the primary developer (as you may recall, other than me) is either aware of or concerned with such lack of monitoring.  He appears to be leaning towards the apparently more expedient route of a separate GUI perhaps using only JPPF-client without JPPF-server or JPPF-node.  In the interest of disclosure, I've recently notified the user representative that such lack of monitoring my come as an unwelcome surprise, though perhaps currently unavoidable under the circumstances.

Actually, there is still something that is not clear to me: what would be running on the PBS cluster? would it be
1) a PBS-specific kind of job (not using the JPPF APIs)?
2) or the same kind of JPPF-job as for local jobs, in a JPPF-node that would be started in the PBS cluster for that purpose?

For now, I'm assuming that the users would overwhelmingly prefer to retain the ability to monitor their jobs via JPPF in the same, existing window as already provided by JPPF - regardless of whether those jobs are running on the local Mac cluster or the remote PBS cluster.   Thus, I would need to (re)use JPPF-nodes and JPPF-server in order to retain the users' ability to monitor jobs via JPPF.  So, in general, I would say the case would be your option 2) - though with the following qualifications.

As you may be aware, I'm proposing that the user make the choice at configuration- and/or run- time in order to decide whether the JPPF-node process runs on the local Mac node or the PBS cluster's front end node (but, recall, not so on any PBS cluster compute node).  In either allowable case, I understand that the JPPF-tasks must run among the JVM's pool of threads for the given JPPF-node.  As such, each task thread submits its corresponding PBS-job directly to the PBS-cluster.

If the user chooses to run the JPPF-node on the local Mac, then the JPPF-task could submit the PBS-job via a sequence of commands such as "scp <user@cluster> <script_name>; ssh <user@cluster>; qsub <script_name>".

If the user chooses to run the JPPF-node on the PBS cluster's front end (perhaps less reliable, but uses less local Mac resource), then the JPPF-tasks could submit its PBS-job to the PBS-cluster via a single command, say, "qsub <script_name>".  Also, presumably the script would be sent over as part of the data for the given task.

My take on this is that, since this information would be known at the time the JPPF-job is submitted, if would be easier to start the JPPF-node, or the script that starts the node for remote nodes, from within the client itself. This is definitely less complex than listening to JPPF-job events in the JPPF-server: start the JPPF-node before submitting the JPPF-job, kill it when the job is complete.

In order to manage the node from within the client, I'm not sure how to avoid blocking the client without using a listener-in-the-client, such as in the following, yes?

<begin client code>
1. Start the JPPF-node.
2. Submit the job to the server.
3. Wait for the job to complete. <client blocks here unless he delegates to some listener, true?>
4. Terminate the JPPF-node.
<end client code>

Your thoughts?

Best,
CB

[lolo]

Hi CB,

Thanks for tellings us about the dead references in the doc. Those have now been fixed.

In order to manage the node from within the client, I'm not sure how to avoid blocking the client without using a listener-in-the-client, such as in the following, yes?

There is already a built-in mechanism which associates a listener with each job, and whose goal is essentially to collect the results of the tasks in the job. You could easily override the listener so that it performs the node cleanup steps.
One reminder, though: in JPPF 2.5, if the JPPF-client terminates, the JPPF-server will automatically cancel the jobs submitted from that client. The intent is to free up the JPPF-nodes when a client can no longer receive the results, so other jobs can be executed. This behavior will be configurable in the upcoming JPPF 3.0, where it can be disabled or enabled.

Sincerely,
-Laurent