adequate
adequate
adequate
adequate
 

JPPF, java, parallel computing, distributed computing, grid computing, parallel, distributed, cluster, grid, cloud, open source, android, .net
JPPF

The open source
grid computing
solution

 Home   About   Features   Download   Documentation   Forums 
November 21, 2017, 06:24:58 PM *
Welcome,
Please login or register.

Login with username, password and session length
Advanced search  
News: New users, please read this message. Thank you!
  Home Help Search Login Register  
Pages: [1]   Go Down

Author Topic: JPPF task not rendering big files execution stops  (Read 3059 times)

Avin

  • JPPF Padawan
  • *
  • Posts: 13
JPPF task not rendering big files execution stops
« on: July 25, 2013, 10:29:14 AM »

Hi,
I m new to JPPF and doing POC over the same. I am reading 2 different files of 32 MB each. What i saw is execution of JPPF stops in between after it reads 30200 lines of code.
File attached has the code for reading and writing files

If in the while loop( where i am reading file line by line) introduce a condition && hj++ < 30000 Then it reads and writes the files but if this condition is not there execution just stops no error nothing.

Any help will be highly appreciated. Since if this dosent works then we will have to drop idea for using JPPF
Logged

lolo

  • Administrator
  • JPPF Council Member
  • *****
  • Posts: 2207
    • JPPF Web site
Re: JPPF task not rendering big files execution stops
« Reply #1 on: July 25, 2013, 07:23:44 PM »

Hello,

It seems to me that you are facing an out of memory error. Can you tell us how much memory is allocated to each node? This is normally set in the node's configuration file, where you have by default the follwoing property "jppf.jvm.options = -Xmx128m -Djava.util.logging.config.file=config/logging-node.properties".
Also, you should see a related error in the node's log file, can you check it out? Normaly this is the file "jppf-node.log" in the root installation folder of the node.

You might also consider the following, which would help optimize the memory footprint of your tasks:

- from what I've seen in your code, you are reading each file into a string in its entirety, then converting this string into a byte array, then writing the byte array to another file. This is not an approach that scales. Instead what you might want to do is read each line from the input file and print it immediately to the output file. The code should look like this:
Code: [Select]
try {
  FileReader fr = new FileReader(readfileName);
  BufferedReader br = new BufferedReader(fr);

  File file = new File(writeFileName);
  FileOutputStream fop = new FileOutputStream(file);
  BufferedOutputStream bos = new BufferedOutputStream(fop);

  String sCurrentLine;
  int hj = 0;
  while ((sCurrentLine = br.readLine()) != null ) {
    hj++;
    byte[] contentInBytes = sCurrentLine.getBytes();
    fop.write(contentInBytes);
  }

  System.out.println("Counter value  " + hj );
  br.close();

  bos.flush();
  bos.close();

} catch ( Exception e) {
  e.printStackTrace();
}
As you can see, the code is shorter and will work even if your files are much larger than 32MB.

- another point I wanted to outline is that when you do "byte[] contentInBytes = content.getBytes();" the getByte() method will allocate a new byte array. So you end up with twice the footprint of the input file content. Furthermore, if you have more than one task executing in parallel in the node, the actual memory footprint will be multiplied by that many. The code above avoids this problem as well.

Sincerely,
-Laurent
Logged

Avin

  • JPPF Padawan
  • *
  • Posts: 13
Re: JPPF task not rendering big files execution stops
« Reply #2 on: July 26, 2013, 06:25:03 AM »

Hi,
I made changes i have allocated 512m memory to it yet same response. below is the content found in jppf_node.log
Code: [Select]
2013-07-26 09:46:11,308 [INFO ][org.jppf.utils.VersionUtils.logVersionInformation(88)]: --------------------------------------------------------------------------------
2013-07-26 09:46:11,308 [INFO ][org.jppf.utils.VersionUtils.logVersionInformation(89)]: JPPF version information: Version: 3.3.4, Build number: 1146, Build date: 2013-07-08 05:41 PDT
2013-07-26 09:46:11,308 [INFO ][org.jppf.utils.VersionUtils.logVersionInformation(90)]: starting node with PID=14300, UUID=CBEBA46C-945B-9F97-B6B1-B5696250B8B7
2013-07-26 09:46:11,308 [INFO ][org.jppf.utils.VersionUtils.logVersionInformation(91)]: --------------------------------------------------------------------------------
2013-07-26 09:46:12,213 [INFO ][org.jppf.classloader.ClassLoaderRequestHandler.run(152)]: maxBatchSize = 1
2013-07-26 09:46:12,244 [INFO ][org.jppf.server.node.NodeExecutionManagerImpl.<init>(131)]: running 8 processing threads
2013-07-26 09:46:12,244 [INFO ][org.jppf.server.node.NodeExecutionManagerImpl.createThreadManager(158)]: Using default thread manager
It dosent shows any error.

The thing that i am trying to do is to check for how Jppf reacts over heavy data usage. I know that code is not optimal usage but as you rightly pointed out it consumes huge memory

Also both reading , writing of a single file is defined in one single task and one task is executed by 1 job. It means for two different files I have 2 different tasks associated to 2 different jobs for your reference.

I also tried putting two tasks in single job where every task reads and writes a file

Thanks and Regards
Patil Abhijeet
« Last Edit: July 26, 2013, 08:54:41 AM by Avin »
Logged

lolo

  • Administrator
  • JPPF Council Member
  • *****
  • Posts: 2207
    • JPPF Web site
Re: JPPF task not rendering big files execution stops
« Reply #3 on: July 27, 2013, 08:55:40 AM »

Hello Patil,

I tried running the task code you posted initially with the following code:
Code: [Select]
public class Runner {
  public static void main(String[] args) {
    JPPFClient client = null;
    try {
      client = new JPPFClient();
      JPPFJob job = new JPPFJob();
      job.setName("large file");
      job.addTask(new MYJPPFTask("C:/Workspaces/JPPF-b3.3/x-largefiles/data/output.txt", "C:/Workspaces/JPPF-b3.3/x-largefiles/data/input.txt"));
      List<JPPFTask> results = client.submit(job);
      MYJPPFTask task = (MYJPPFTask) results.get(0);
      if (task.getException() != null) System.out.println("task got exception: " + ExceptionUtils.getStackTrace(task.getException()));
      else System.out.println("task result: " + task.getResult());
    } catch (Exception e) {
      e.printStackTrace();
    } finally {
      if (client != null) client.close();
    }
  }
}

I used an input file with size = 32765 KB. When running it on a node with the default heap size (128 mb), the resulting task.getException() showed that an OutOfMemoryError had occured in the node while reading the input file. When I ran it in a node with 512 mb, there was no error, and the output file was properly created.

Could you post the node configuration file(s) that you are using (config/jppf-node.properties), so we can check if the settings are ok? Please don't hesitate to obfuscate any ip address that may be in the files.

Thanks,
-Laurent
Logged

Avin

  • JPPF Padawan
  • *
  • Posts: 13
Re: JPPF task not rendering big files execution stops
« Reply #4 on: July 31, 2013, 04:10:55 PM »

hi,
My Node 1 has 1024m memory and Node has 1024m memory. Memory given to driver is 1024m
Still Node 1 reads data and gets stuck Similiarly Node 2 reads data completely and gets stucked i.e no further processing takes place.
Same code when executed in eclipse / as simple java program it runs without any issue.

Zip file attached contains screenshot of the same
I cannot increase memory ofr driver or node since it dosent starts i tried giving 2048m to them

Any help that you can provide me ?
Thanks and Regards
Patil Abhijeet
Logged

lolo

  • Administrator
  • JPPF Council Member
  • *****
  • Posts: 2207
    • JPPF Web site
Re: JPPF task not rendering big files execution stops
« Reply #5 on: August 01, 2013, 08:28:31 AM »

Hi Patil,

Ok, we need to determine what is going on.
Would it be possible for you to wrap your task's run() method code in a try { } catch() { } block similar to this:
Code: [Select]
@Override
public void run() {
  try {
    ... your code here ...
  } catch(Throwable t) {
    t.printStackTrace();
    setException(t instanceof Exception ? (Exception) t : new Exception(t));
  }
}

With a little luck this should allow us to get more information on what's happening.
Another thing you could try is to add the "-XX:+HeapDumpOnOutOfMemoryError" flag to the node's "jppf.jvm.options" property, so if there is an OOM we will have a proof of it, along with a heap dump to anlayze.

Also, given the problem you have setting 2048m to your JVMs, it seems you are using a 32 bits JVM, could you confirm this? Could you also let us know how much physical memory you have on your machine?

Thanks for your time,
-Laurent
Logged

Avin

  • JPPF Padawan
  • *
  • Posts: 13
Re: JPPF task not rendering big files execution stops
« Reply #6 on: August 01, 2013, 10:53:19 AM »

Hi,
I tried by using heapDump option. You are right it gives heap space error java.lang.outofMemoryError : java heap space
I cannot give you .prof file since it is huge 61 MB and more moreover its not human readable it has dumped its error there.

I dont know why this error is getting generated here. It runs fine in eclipse but here it gives this error.
I have give xmx i.e max memory usage as 1024m to each node and its driver.Then why this error crops ?
My system has 8 GB memory. so say with other applications and OS running it consumes 2 GB max
the point here i am trying to make is it can consume all 3 gb which is available to it
My System is 64 bits windows 7 and jvm version is 32 bits
java version 1.6.16

If this has to create problem then i think even java program should create same issue but it dosent do so
Thanks and Regards
Patil Abhijeet
« Last Edit: August 01, 2013, 10:58:00 AM by Avin »
Logged

lolo

  • Administrator
  • JPPF Council Member
  • *****
  • Posts: 2207
    • JPPF Web site
Re: JPPF task not rendering big files execution stops
« Reply #7 on: August 01, 2013, 08:54:16 PM »

Hi,

Indeed, hprof files are not human-readable in general. You can analyze it with a specialized tool. A very good tool that I use is Eclipse MAT. However here we can already already make a guess as to what is filling the memory.

If this is working as a standalone application in Eclipse, but not in the node, then there is a difference in the configuration or environment of the node's JVM. Also, if the other node doesn't have that problem, maybe the 2 nodes do not have the same configuration, can you check this out? Also, are both jobs reading files of similar sizes?

Maybe you could add a trace in your task to display the maximum memory available to the JVM:
Code: [Select]
long maxMemoryMB = Runtime.getRuntime().maxMemory() / (1024L * 1024L);
System.out.println("JVM max memory = " + maxMemoryMB + " MB");

Sincerely,
-Laurent
Logged

Avin

  • JPPF Padawan
  • *
  • Posts: 13
Re: JPPF task not rendering big files execution stops
« Reply #8 on: August 01, 2013, 10:03:15 PM »

Hi,
No separate configuration in eclipe. And both noeds stop at a moment only thing is one node stops little latter  ;D i can pot you nodes cconfiguration filee if you want to no separate changes done only in jvm options

Did you try wwith jppf try reading and writing 400 mb of file . I dont know how to proceed or i it JPPF I snot ssuitable for this kind of tasks ?
Logged

lolo

  • Administrator
  • JPPF Council Member
  • *****
  • Posts: 2207
    • JPPF Web site
Re: JPPF task not rendering big files execution stops
« Reply #9 on: August 02, 2013, 08:59:19 AM »

Hello,

Maybe there is something I misunderstood. Could you please clarify what you meant when you said: "It runs fine in eclipse but here it gives this error."
Are you running a different code in Eclipse? can you elaborate on this, maybe post the settings you use in the eclipse launch configuration, especially the VM arguments in the "Arguments" tab? Could you  actually post the .launch file?

I also tried your task code with a file of 407 MB and a node with -Xmx1024m, and I did get the OOME while reading the input file.
Why is this happening? This is because the task reads the file into a StringBuilder using a character stream (i.e. a Reader). StringBuilder and String internally use an array of char. A char is a 16 bits (2 bytes!) unicode representation of a character. What this means is that you will end up with as many chars as you have bytes in the input file. So, if you have 400 MB in the input file, the StringBuilder will end up with a 2*400 MB footprint. So we might say that 800 MB is still less than 1024 MB. True. However, the StringBuilder has a dynamic capacity. This means that, possibly many times, it will resize its internal char buffer, by creating a new, larger one, then copy the smaller buffer content into the new one and then the old one will become garbage-collectible. Thus, at some points in time, there will be 2 large char buffers for a very short time. However, when the buffer becomes large, the JVM will be unable to allocate a new one, because there isn't enough heap space.

Furthermore, your code then does a String.getBytes() to write the resulting byte[] to a byte stream (which seems inconsistent with the reading side using a character stream). I checked the code in String.getBytes() : to do this, it allocates a byte array whose size is calculated as number_of_chars * max_number_of_bytes_to represent_a_char_in_the_current_encoding. The current encoding (your platform default, for instance ISO-8859-1 if you're in the US) will generally use at least 2 bytes to represent a single char. Thus yet another array of 800 MB is needed.

Thus, my conclusion is there is no way this code can work with a 400mb file and a 1024mb JVM, whether the JVM is a JPPF node or not. Actually it may not work even with a 2gb JVM, because of heap fragmentation preventing the allocation of large contiguous blocks of memory (contiguous memory is required for arrays). So we need to understand why it is working in the "eclipse version". I'm suspecting it is using a different JVM (64 bits) with a much larger allocated heap, whether implicitely or explicitely. Implicitely means the hotspot JVM identifies the class of machine it's running on and allocates heap size accordingly. Your machine would be considered a "server class" machine, leading to a max heap size of 1/4th the available physical memory (i.e. 2GB on your machine).

Additionally, I believe you should change the design of your task since, as I mentioned earlier, it will not scale if you need to handle even larger files, for instance files much larger than your JVM heap size.
In JPPF we have a simple API that handles file copy (and much more) in a way which scales to any file size: the location API. Using this API, I rewrte your task's run method as follows:
Code: [Select]
package jppf.task;

import org.jppf.server.protocol.*;

public class MYJPPFTask2 extends JPPFTask {
  String readfileName;
  String writeFileName;

  public MYJPPFTask2(String writeFileName, String readfileName) {
    this.readfileName = readfileName;
    this.writeFileName = writeFileName;
  }

  public void run() {
    long start = System.currentTimeMillis();
    Location input = new FileLocation(readfileName);
    Location output = new FileLocation(writeFileName);
    try {
      input.copyTo(output);
      long elapsed = System.currentTimeMillis() - start;
      System.out.println("copied file with length " + input.size() + " in " + elapsed + " ms");
    } catch (Exception e) {
      e.printStackTrace();
    }
  }
}

With this code, there is no problem with a 1024mb node. It also runs much faster: 500 ms versus 3700 ms in the previous version.

Sincerely,
-Laurent
« Last Edit: August 02, 2013, 09:12:14 AM by lolo »
Logged

Avin

  • JPPF Padawan
  • *
  • Posts: 13
Re: JPPF task not rendering big files execution stops
« Reply #10 on: August 02, 2013, 09:31:07 AM »

Hi,
Sorry to keep u bugging.
yes you are right eclipse uses IBM java while my path points to oracle java.

Can we set jdk path in Nodes so that it is valid only for that nodal enviroment ?

Thanks and Regards
Patil Abhijeet
Logged

lolo

  • Administrator
  • JPPF Council Member
  • *****
  • Posts: 2207
    • JPPF Web site
Re: JPPF task not rendering big files execution stops
« Reply #11 on: August 02, 2013, 10:00:10 AM »

Hi,

I'm not sure if you're asking how to change the JVM for launching the node in Eclipse, or for launching the node standalone, so I'll answer both  :)

To change in Eclipse (see attached screenshot):
- open your launch configuration
- select the "JRE" tab
- select "Alternate JRE"
- choose the desired JRE from the combo box
- if your JRE is not in the list, you need to add it via the Eclipse preferences: in "Java > Installed JRE", click the "Add" button and browse to the root installation folder of the JRE/JDK you need.

For a standalone node, you will need to modify the provided startNode.bat script.
The script provided in the node distribution is as follows:
call java -cp config;lib/* -Xmx32m -Dlog4j.configuration=log4j-node.properties -Djppf.config=jppf-node.properties -Djava.util.logging.config.file=config/logging-node.properties org.jppf.node.NodeLauncher

So if your JVM is located in C:\java\jdk1.6.0 then you can just change the script as:
call C:\java\jdk1.6.0\bin\java -cp config;lib/* -Xmx32m -Dlog4j.configuration=log4j-node.properties -Djppf.config=jppf-node.properties -Djava.util.logging.config.file=config/logging-node.properties org.jppf.node.NodeLauncher

Sincerely,
-Laurent
Logged

Avin

  • JPPF Padawan
  • *
  • Posts: 13
Re: JPPF task not rendering big files execution stops
« Reply #12 on: August 02, 2013, 10:30:30 AM »

Hi,
I was asking for standalone node and i guess you have helped me with that.
Since you are from jppf team can you take this as enhancement where java path should be set in properties file which will be searched by batch file and if it dosent gets it, it will use default java path ?

Thanks and Regards
Patil Abhijeet
Logged

lolo

  • Administrator
  • JPPF Council Member
  • *****
  • Posts: 2207
    • JPPF Web site
Re: JPPF task not rendering big files execution stops
« Reply #13 on: August 03, 2013, 05:25:36 AM »

Hello,

Not a problem. I registered the feature request JPPF-176 Enable specifying the JVM location for the driver and node launcher

Sincerely,
-Laurent
Logged
Pages: [1]   Go Up
 
JPPF Powered by SMF 2.0 RC5 | SMF © 2006–2011, Simple Machines LLC Get JPPF at SourceForge.net. Fast, secure and Free Open Source software downloads