JPPF, java, parallel computing, distributed computing, grid computing, parallel, distributed, cluster, grid, cloud, open source, android, .net
JPPF

The open source
grid computing
solution

 Home   About   Features   Download   Documentation   On Github   Forums 
December 13, 2019, 04:40:10 AM *
Welcome,
Please login or register.

Login with username, password and session length
Advanced search  
News: New users, please read this message. Thank you!
  Home Help Search Login Register  
Pages: [1]   Go Down

Author Topic: What do I consider when planning to divide my app. Into tasks.  (Read 2390 times)

willis

  • Guest

What are the considerations when planning to divide a program so that it can be done in parallel. My application get values from a data base, does some processing on them then updates the same row in the data base. Each row is independent so each – get/process/update could be a task, but I am not sure that is the right way to do it.

Any advice or pointers to docs would be appreciated. 
Logged

lolo

  • Administrator
  • JPPF Council Member
  • *****
  • Posts: 2258
    • JPPF Web site
Re: What do I consider when planning to divide my app. Into tasks.
« Reply #1 on: April 23, 2011, 08:26:54 AM »

Hello willis,

Some of the considerations you might want to take into account:

- how should the database accesses be performed? this will determine the design of your tasks and how your application processes them. For instance, you can design your tasks as you mentioned, where each task will fetch one database row, process it and update it back in the database. However, maybe this isn't the most efficient use of the database. So, you may use a different design where each task processes multiple rows obtained via a SQL query. For instance, if you have millions of rows, you could have each task process 1000 rows. This means the queries must provide a very precise partitioning of your database, to avoid overlaps and gaps between tasks.

- another consideration is about transaction management, failover and crash recovery. First, I believe it is important to understand how jobs and tasks are dispatched to the nodes for execution. You will find very useful information in the JPPF overview chapter of the documentation.
In practical terms, let's say you submit a job with N tasks. The JPPF server (also called "driver") will partition it into M subsets Pi such that SUM(P1, ..., PM) = N. Each Pi will be sent to a node for execution. Now let's imagine that, for any reason whatsoever, the node crashes or gets disconnected from the server. JPPF has a recovery mechanism that will cause it to requeue Pi so that it can be executed on another node. However, some database rows may already have been updated at that time and you need to deal with that. One strategy would be to check in your tasks if each row has already been processed, maybe using a timestamp in the row or some other marker. A probably safer and more efficient way is to use a transaction encompassing all Pi tasks. JPPF provides a sample that demonstrates how this can be done using a node lifecycle listener and an XA-compliant JTA transaction manager: see Control of database transactions via node life cycle events.

- in relation to what I mentioned above, using XA transactions also means that all database operations must be performed in the same thread. This can create a bottleneck if the nodes are configured to use multiple threads to execute tasks concurrently. You may consider simplifying this problem by using more nodes with one thread each, which may result in a different partioning of your jobs into subsets (you can have as many nodes as you wish on any physical or virtual machine, the only limit being the available system resources)

- are your tasks CPU-bound or I/O-bound or somewhere in between? Given the description you provided, I'm guessing yours would be more I/O bound due to the database access. In this case, it would be very benefitial to have as many as possible executed in parallel, in particular this is a scenario where the number of available cores/CPUs is not the main limiting factor.

I hope this helps.

Sincerely,
-Laurent
-
Logged

willis

  • Guest
Re: What do I consider when planning to divide my app. Into tasks.
« Reply #2 on: April 25, 2011, 08:07:14 PM »

We will have millions of rows in the database to deal with so I guess having a task per row would be somewhat inefficient. Is this because of accumulated network latency?

Thinking about it programmatic (I don’t know much about data bases), since each task could hold the “Job” start time, we could check the time the current row being processed was last updated and if it is past the start time of the job we know that task (containing current row) has been re-queued. We could roll back the changes for that 1000 rows to a time before the start of the job and then process normally OR mark that section of rows as corrupt so that it can be handled another way.
Please let me know if this sounds reasonable.

I looked at the example you sited in your post. I was trying to avoid using the listener (for simplicity) and I wanted to keep as much parallelism as possible. That is why I proposed the idea above.
Logged

lolo

  • Administrator
  • JPPF Council Member
  • *****
  • Posts: 2258
    • JPPF Web site
Re: What do I consider when planning to divide my app. Into tasks.
« Reply #3 on: April 26, 2011, 09:30:11 AM »

Hi willis,

Quote
We will have millions of rows in the database to deal with so I guess having a task per row would be somewhat inefficient. Is this because of accumulated network latency?
That is one of the reasons, network latency will accumulate for both the tasks and the database accesses. Also, I'm no DB expert either, but I can guess that one query that handles 1000 rows will be more efficient performance-wise than 1000 queries that handle one row each.

Regarding the solution you are envisioning, the questions I would ask myself are how easy is it to roll-back the database data, and do I know which values to roll back to? If there is a positive answer to these questions (unless to choose to mark the rows as corrupts), then your solution will certainly be easier to implement than the one I described.

Sincerely,
-Laurent

« Last Edit: April 26, 2011, 09:52:59 AM by lolo »
Logged
Pages: [1]   Go Up
 
JPPF Powered by SMF 2.0 RC5 | SMF © 2006–2011, Simple Machines LLC Get JPPF at SourceForge.net. Fast, secure and Free Open Source software downloads