adequate
adequate
adequate
adequate
 

JPPF, java, parallel computing, distributed computing, grid computing, parallel, distributed, cluster, grid, cloud, open source, android, .net
JPPF

The open source
grid computing
solution

 Home   About   Features   Download   Documentation   Forums 
June 25, 2018, 12:07:22 PM *
Welcome,
Please login or register.

Login with username, password and session length
Advanced search  
News: New users, please read this message. Thank you!
  Home Help Search Login Register  
Pages: [1]   Go Down

Author Topic: Client Local Node  (Read 742 times)

tamas

  • JPPF Master
  • ***
  • Posts: 30
Client Local Node
« on: August 31, 2015, 05:24:05 PM »

Hello all,

I created a application which run a JPPFClient and distribute some tasks between the connected nodes.

My application/client is running on my PC. Server and nodes are running on amazon EC2 instances.
The client is connecting to the driver via public IP.

The question is. How to execute a job between all remote nodes + to my local PC where the application is running?
Is there a way how to run a client which run a node in the same JVM as a client is running?

I know that it is possible for the Driver with jppf.local.node.enabled = true. It creates a node inside a drivers's JVM.

In a config of the client is the same setting jppf.local.execution.enabled. When I set it to true then all tasks are executed ONLY in my local PC.

I hope there is a solution for that without running a new node or another driver(peer to peer connection) in my local PC.
I want to use also the power of my local pc to compute tasks not only remote nodes.

Thanks for any response.
Tomas
Logged

tamas

  • JPPF Master
  • ***
  • Posts: 30
Re: Client Local Node
« Reply #1 on: August 31, 2015, 05:32:15 PM »

Maybe better example for what I mean is the case with a VPS.

I have my local pc + VPS = 2 computers.

I want to distribute a job with 20 tasks between my local pc and VPS.
How to do it when the driver+node(one JVM) is running on the VPS.

One possibility is run a new node in my local PC which will connect to the VPS and execute the tasks but there is a big time consumption due to internet connection.

Thanks,
Tomas
Logged

lolo

  • Administrator
  • JPPF Council Member
  • *****
  • Posts: 2223
    • JPPF Web site
Re: Client Local Node
« Reply #2 on: September 01, 2015, 08:10:20 AM »

Hello Tomas,

To achieve this, you will just need to configure the client-side load balancing appropriately, and setup the job SLA so it can be submitted over multiple channels (local and remote) on the client-side.

By default, the client is configured so that all the tasks in a job are sent at once over a single channel (local executor or driver connection). The default config in the client is like this:
Code: [Select]
jppf.load.balancing.algorithm = manual
jppf.load.balancing.profile = manual
jppf.load.balancing.profile.manual.size = 1000000
We can see that the client will attempt to distribute up to 1,000,000 tasks at once, which means that in the vast mority of jobs, all the tasks will be sent at once. To alter this behavior, we could change the "size" parameter to a reasonable value, for example 10, or use a different algorithm altogether.

Now, we need to setup the job SLA so it can be sent over multiple channels. This is done by setting the MaxChannels attribute of the job's client-side SLA:
Code: [Select]
JPPFJob job = new JPPFJob();
job.getClientSLA().setMaxChannels(2);
Here we configured the job to be sent over up to 2 channels in parallel (the default is 1). Thus, if you have a local executor and a remote driver connection, tasks from the job will be sent to both, according to the load-balancer.

Sincrely,
-Laurent
« Last Edit: September 01, 2015, 08:12:17 AM by lolo »
Logged

tamas

  • JPPF Master
  • ***
  • Posts: 30
Re: Client Local Node
« Reply #3 on: September 01, 2015, 11:32:16 AM »

Hello lolo,

thanks for your response.
It works as you wrote for simple tasks but it does not work with a broadcast job.

11:15:05.334 [TaskQueueChecker] DEBUG o.j.c.b.queue.JPPFPriorityQueue - 1 connection(s) for broadcast job 'BatchHandler job 1' : [ChannelWrapperRemote[channel=driver1-1[localhost:11111] : ACTIVE]]
11:15:05.334 [TaskQueueChecker] DEBUG o.j.c.balancer.AbstractClientJob - creating ClientJob #613
11:15:05.334 [TaskQueueChecker] ERROR o.j.c.b.queue.TaskQueueChecker - An error occurred while preparing for bundle creation and dispatching.
java.lang.IllegalStateException: Job 8E152120-66AB-D3B8-883A-516407450D26 already enqueued
   at org.jppf.client.balancer.queue.JPPFPriorityQueue.processBroadcastJob(JPPFPriorityQueue.java:247) ~[jppf-client.jar:na]
   at org.jppf.client.balancer.queue.JPPFPriorityQueue.processPendingBroadcasts(JPPFPriorityQueue.java:356) ~[jppf-client.jar:na]
   at org.jppf.client.balancer.queue.TaskQueueChecker.dispatch(TaskQueueChecker.java:211) [jppf-client.jar:na]
   at org.jppf.client.balancer.queue.TaskQueueChecker.run(TaskQueueChecker.java:199) [jppf-client.jar:na]
   at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]

Please see the attachment with the code or a link with the image where I run it in Eclipse http://s27.postimg.org/uzpi5rzeb/broadcast_job.jpg.

What am I doing wrong? First I need to create a broadcast job to distribute the data for each node which will be persistent also for other jobs
NodeRunner.setPersistentData("Data", data);

Sincerely,
Tomas

UPDATE:
http://www.jppf.org/doc/v5/index.php?title=Job_Service_Level_Agreement#Broadcast_jobs

I read in the documentation that broadcast job is not executed in client's local node.
I have a common task which read a data from NodeRunner.getPersistentData("Data") so I have to broadcast my data to all nodes.

I really stuck with this. Please help.
« Last Edit: September 01, 2015, 12:31:08 PM by tamas »
Logged

lolo

  • Administrator
  • JPPF Council Member
  • *****
  • Posts: 2223
    • JPPF Web site
Re: Client Local Node
« Reply #4 on: September 03, 2015, 07:50:52 AM »

Hi Tomas,

I have registered a bug for this: JPPF-409 IllegalStateException in the client when submitting a broadcast job.
The problem for me is that I can't reproduce it, even with the code you posted, and I'd like to understand why we have such different results. You mentionned, and I can see in your screenshot, that you have a driver with a locol node. Do you have any other driver or node in your topology? Also, is there a way you could post your full client configuration file, if you use any?
In the meantime, I will keep investigating and provide updates whenever I find something.

Quote
I read in the documentation that broadcast job is not executed in client's local node
Yes, that is correct. To handle this situation, you may use the Task.isInNode() method, which tells you whether your task is executing within a real node or a client local executor. For instance:
Code: [Select]
public class MyTask extends AbstractTask<Object> {
  @Override
  public void run() {
    String param = null;
    if (isInNode()) param = (String) NodeRunner.getPersistentData("myData");
    else param = (String) MyCache.get("myData");
    ...
  }
}

Sincerely,
-Laurent
Logged

tamas

  • JPPF Master
  • ***
  • Posts: 30
Re: Client Local Node
« Reply #5 on: September 03, 2015, 09:45:57 AM »

Hello Laurent,

I downloaded new zip files node/driver/application from your homepage.
I started a driver + new node with default config files.

Then I wrote a basic broadcast job/task
Code: [Select]
JPPFClient jppfClient = new JPPFClient();

JPPFJob job = new JPPFJob();
job.add(new JPPFTestTask(1));
job.getSLA().setBroadcastJob(true);

jppfClient.submitJob(job);

I got the same exception as in my last example but the task was distributed
Please look at the picture from the link http://s2.postimg.org/wplh5kq2h/broadcast.png

I tried to run a application with and also without the config file. Always the same result.
When I enable jppf.local.execution.enabled = true in client's config file I get the same exception but task was not distributed as before to remote node.

Please give me some advice why it is not working.

To broadcast the job is enough to set job.getSLA().setBroadcastJob(true);???
What am I doing wrong? I started all client/node/server with default configurations so it should work.

Best regards,
Tomas

UPDATE:
I attached a log file from the example above. (with default settings)
« Last Edit: September 03, 2015, 09:52:05 AM by tamas »
Logged

lolo

  • Administrator
  • JPPF Council Member
  • *****
  • Posts: 2223
    • JPPF Web site
Re: Client Local Node
« Reply #6 on: September 04, 2015, 07:08:58 AM »

Hi Tomas,

The good news is I finally managed to reproduce the issue. This should make finding the cause and fixing it much easier from now on.
I've also found a workaround that you could use in the meantime, which consists in waiting until the client has at least one established connection with a remote driver, like this:

Code: [Select]
JPPFClient client = new JPPFClient();
// wait until there is at least one connenction pool with at least one established connection
client.awaitWorkingConnectionPool();

I will provide an update when I have a fix.

-aurent
Logged

tamas

  • JPPF Master
  • ***
  • Posts: 30
Re: Client Local Node
« Reply #7 on: September 04, 2015, 06:02:09 PM »

Thanks Laurent, it works like a charm :)

Best regards,
Tomas
Logged

lolo

  • Administrator
  • JPPF Council Member
  • *****
  • Posts: 2223
    • JPPF Web site
Re: Client Local Node
« Reply #8 on: September 05, 2015, 05:04:11 PM »

Hello,

I just waned to say that I fixed JPPF-409 and delvired the corresponding patch 01 for JPPF 5.0.4

Sincerely,
-Laurent
Logged

tamas

  • JPPF Master
  • ***
  • Posts: 30
Re: Client Local Node
« Reply #9 on: September 07, 2015, 11:55:36 AM »

Thanks for the info Laurent.

Best regards,
Tomas
Logged

outlandishphoton424

  • JPPF Padawan
  • *
  • Posts: 1
Re: Client Local Node
« Reply #10 on: October 17, 2016, 11:39:27 AM »

This is what I was searching for. Thanks for the help lolo
Logged
Pages: [1]   Go Up
 
JPPF Powered by SMF 2.0 RC5 | SMF © 2006–2011, Simple Machines LLC Get JPPF at SourceForge.net. Fast, secure and Free Open Source software downloads