adequate
adequate
adequate
adequate
 

JPPF, java, parallel computing, distributed computing, grid computing, parallel, distributed, cluster, grid, cloud, open source, android, .net
JPPF

The open source
grid computing
solution

 Home   About   Features   Download   Documentation   Forums 
July 22, 2017, 08:37:04 PM *
Welcome,
Please login or register.

Login with username, password and session length
Advanced search  
News: New users, please read this message. Thank you!
  Home Help Search Login Register  
Pages: [1]   Go Down

Author Topic: Interface gridcopmuting project  (Read 71 times)

steve4j

  • JPPF Padawan
  • *
  • Posts: 3
Interface gridcopmuting project
« on: April 27, 2017, 12:22:37 PM »

Hallo Laurent,

thanks for your help in the issue tracker so far http://www.jppf.org/tracker/tbg/jppf/issues/JPPF-500.
Now since i got the DBCP & Hibernate workin my project "gridcomputing for interfaces" i came along this Thread http://www.jppf.org/forums/index.php/topic,1961.0.html
Where you said (4y ago) that one node can only handle one job at a time.

Since my project starts like min. 15 to max. 80 interfaces per min. i have to ask if i realy need like at least 100 nodes with each a single JVM.
This would produce so much overhead from each jvm that i could not use JPPF as ligthweight parallel processing framework.

Is there a way to start lets say like 20 nodes with 5 open "Jobpool" each so i need only 20 JVM not 100?
The nodes are needed for fail-security, but i cannot start that much jvms.

Can there be a feature request for a new Release or something that could help to fix this?

Thank you for your efforts
Logged

lolo

  • Administrator
  • JPPF Council Member
  • *****
  • Posts: 2187
    • JPPF Web site
Re: Interface gridcopmuting project
« Reply #1 on: April 28, 2017, 08:08:05 AM »

Hello,

You do not have to have as many nodes as there are jobs at any given time. When all nodes are busy, the server's queuing mechanism ensures that unexecuted jobs are preserved until a node becomes available. Similar queuing occurs in the client as well, since the maximum number of jobs that can be submitted to a server is determined by the number of connections in a connection pool.

The one-job-per-node model (or one job per connection in the client) is a design choice which significantly simplifies the communication model, makes the grid more robust and less prone to crashes, allows better control of the resources usage , enables load-balancing which adapts to the workload, etc. The benefits are too important to abandon them now.

I'll be happy to provide suggestions on how the performance or topology can be optimized, however I'd need little more information. In particular, it is not clear to me what you mean by "interface". What does it represent in terms of JPPF jobs and tasks? How many tasks do your jobs have? Remember that the granularity of jobs and tasks is a critical factor for performance, and it should be given some consideration at design time when building a distributed application.

Thanks for your time,
-Laurent
Logged

steve4j

  • JPPF Padawan
  • *
  • Posts: 3
Re: Interface gridcopmuting project
« Reply #2 on: April 28, 2017, 08:50:07 AM »

Hallo Laurent,

thanks for your super-fast response.
Interfaces in my project are "connectors" between two systems.
Lets say for a example there is a system wich provides businessdata in CSV formated file on a ftp server. This file needs to be picked up every 15min  and be iterated, mapped and written to a database.
So my project has 75 different interfaces with a total of 150 versions.
My historical data tells me that 15-80 interfaces are started at the same time due the day.
A interface version can only run at the same time.

The current model of this project is a stand alone rich client (swing).
But since we dont want to be bound to hardware / OS / other problems we want to commit the jobs into a cluster and use a scheduler www.quartz-scheduler.org.

Thanks for your help,
Steve
Logged
Pages: [1]   Go Up
 
JPPF Powered by SMF 2.0 RC5 | SMF © 2006–2011, Simple Machines LLC Get JPPF at SourceForge.net. Fast, secure and Free Open Source software downloads