JPPF, java, parallel computing, distributed computing, grid computing, parallel, distributed, cluster, grid, cloud, open source, android, .net
JPPF

The open source
grid computing
solution

 Home   About   Features   Download   Documentation   Forums 
December 15, 2018, 10:36:48 AM *
Welcome,
Please login or register.

Login with username, password and session length
Advanced search  
News: New users, please read this message. Thank you!
  Home Help Search Login Register  
Pages: [1]   Go Down

Author Topic: List of tasks: who needs work to do?  (Read 6580 times)

lolo

  • Administrator
  • JPPF Council Member
  • *****
  • Posts: 2242
    • JPPF Web site
List of tasks: who needs work to do?
« on: June 29, 2006, 02:05:44 AM »

Here is a (long) list of things, tasks, features, ideas, that I think we may want to work on.

1) Expanding the framework to a multi-server architecture
I'm very much in favor of  a model where the nodes are still attached to a single server (i.e. they don't communicate with each other or another server), and where the servers are federated into a kind of P2P network.
As an example: let's say server A sends tasks to server B
My vision of it is to have a A see B as a node, a B see A as a client. The situation is reversed if B sends tasks to A.
I see to huge advantages to this:

1 it allows us to reuse the existing implementations of  nodes and clients, without reiventing the wheel
2 the auto-tuning alogrithm can be reused without any change, which extends the scheduling capabilities to a vastly larger framework

So, here are the things I think we'll have to do:
- define and implement the server-to-server communication
- define and implement the server discovery process, including the bootstraping part
- extend the dynamic class loading mechanism, so that class loading can be propagated accross drivers on the network
- implement the required security mechanisms - see 2)

2) Security
- define a security model to control the interactions between users, servers and nodes.
- implement and plug it into the communication model for the framework components
- build the corresponding administrative tools to define users, permissions, and associated logic.

I think Domingos currently has the clearest ideas on the subject, so here I'm citing him, from a recent email:

About the security, I think that it must the based on users, and making the tasks impersonate it.
About the decision of sending tasks to be processed at other networks, I think it could be done by the admin of driver, using some criteria like,
max size of the queue, security requirements of each task, cost of processing time (is it better to run it here or outsource?), or other stuff, and not based on each applications - which will be many  :-)
As the used processing power will be a commodity, the issue will be making the tasks be processed with a smallest cost.


One major advantage I can see, is that implementing the logic on the administrative side will also much reduce the overhead on the server.

It also means we'll have to extend the admin tools a lot to handle the users management, authentication, and permissions.
By the way, I finally found a Java-based LDAP server that I can install and run on any platform. It's Apache Directory Server (http://directory.apache.org), and it could be useful when we need to do some real-life testing of our security stuff.

3) Administration and monitoring tool
What other functionalities do we need to add here?
I'm sure there's a lot of stuff to do here, and here are some of the features I'm envisioning:
- Usability: maybe adding toolbars, menus, etc.... Also, given that most of the UI is dynamically built from XML documents, it makes it possible to discard or add some panels as the users see fit
- Extensions for users and permissions management, authentication, etc...
- I was thinking of giving the admins the possibility to define their own performance indicators, alerts, and so on. Since we have an API for embedding scripts, users should be able to define and manage alert conditions, as well as responses to those (for instance preventing the server from taking requests with priority below X when the moving average of the time spent in the queue is more than Y)
- There is also the XML-based UI building framework that can be very much expanded (the goal is to use it for the admin tool extensions) and eventually made into it's own subproject.

4) Documentation
I guess we all agree that we have a serious lack of user/developer documentation. I think the Wiki is a great tool for this, and it will probably be the best place to develop the documentation.
What to document? Here's a starting point:
- installation on various environments
- configuration of clients, servers, nodes
- How to use the APIs to distribute the execution of the tasks
- documenting the way things work behind the scene

Well, now we just have to pickup (or add!) things and get to work.
Logged

g_korland

  • JPPF Padawan
  • *
  • Posts: 2
    • http://www.cs.technion.ac.il/~korland
Expanding the framework to a multi-server architecture
« Reply #1 on: July 20, 2006, 05:29:55 PM »

"the nodes are still attached to a single server"
1. What about failover in case this sever failed.
2. What about load balancing, the way you sugested will not share the load between the server.

I think we should put some logic on the client side that will partition the work between the servers, and use the method you sugested in order to provided a bakcup server for failover.

By the way a "discovery process" seems like an extension that can add a lot of use simplicity to the client.
Logged
Regards,
Guy Korland

lolo

  • Administrator
  • JPPF Council Member
  • *****
  • Posts: 2242
    • JPPF Web site
List of tasks: who needs work to do?
« Reply #2 on: July 21, 2006, 01:47:06 PM »

Hi Guy,

Quote from: "g_korland"
1. What about failover in case this sever failed.

Good point. I believe your suggestion to enable access to multiple servers from the client is a very workable one.

Quote from: "g_korland"
2. What about load balancing, the way you sugested will not share the load between the server.

Can you elaborate on this?
My current idea is  based on the following:
- each batch of tasks submitted by the client is partioned on the server side, based on each node's performance profile (this part's already implemented)
- if the "node" is actually a server, it should be able to take larger partitions, thus effectively distributing the load across servers
Am I making wrong assumptions?

Quote from: "g_korland"
I think we should put some logic on the client side that will partition the work between the servers, and use the method you sugested in order to provided a bakcup server for failover.

Agreed. I think we could even extend that mechanism to the nodes, so they can still contribute work even if their primary server is down.

-Laurent
Logged

g_korland

  • JPPF Padawan
  • *
  • Posts: 2
    • http://www.cs.technion.ac.il/~korland
List of tasks: who needs work to do?
« Reply #3 on: July 22, 2006, 01:46:25 PM »

Quote from: "lolocohen"
Can you elaborate on this?
My current idea is based on the following:
- each batch of tasks submitted by the client is partioned on the server side, based on each node's performance profile (this part's already implemented)
- if the "node" is actually a server, it should be a ble to take larger partitions, thus effectively distributing the load across servers
Am I making wrong assumptions?

Most of the times the servers can be overloaded and their queues can be full.
As I wrote:
Quote from: "g_korland"
I think we should put some logic on the client side that will partition the work between the servers, and use the method you sugested in order to provided a bakcup server for failover.
Logged
Regards,
Guy Korland

lolo

  • Administrator
  • JPPF Council Member
  • *****
  • Posts: 2242
    • JPPF Web site
List of tasks: who needs work to do?
« Reply #4 on: July 22, 2006, 11:35:29 PM »

I think you're making 2 good points here. Let's go with it.
Do you want to work on extending the client to communicate with multiple servers?
How about extending the nodes as well? :D

-Laurent
Logged
Pages: [1]   Go Up
 
JPPF Powered by SMF 2.0 RC5 | SMF © 2006–2011, Simple Machines LLC Get JPPF at SourceForge.net. Fast, secure and Free Open Source software downloads