adequate
adequate
adequate
adequate
 

JPPF, java, parallel computing, distributed computing, grid computing, parallel, distributed, cluster, grid, cloud, open source, android, .net
JPPF

The open source
grid computing
solution

 Home   About   Features   Download   Documentation   Forums 
September 19, 2018, 03:45:43 AM *
Welcome,
Please login or register.

Login with username, password and session length
Advanced search  
News: New users, please read this message. Thank you!
Home Help Search Login Register
Pages: [1]   Go Down

Author Topic: Probable out of memory exception?  (Read 293 times)

aishwarya93

  • JPPF Padawan
  • *
  • Posts: 7
Probable out of memory exception?
« on: June 13, 2017, 04:42:47 PM »

Hi,

I have the following - single JPPF client, single JPPF driver, 3 JPPF nodes on 3 different AWS instances. The client submits a single job that has 1 million tasks. My program hangs at the line of code which says

My thread dump is as follows -

Code: [Select]
Full thread dump OpenJDK 64-Bit Server VM (25.131-b11 mixed mode):

"RemoteChannelWrapper-driver1-1-0001" #24 daemon prio=5 os_prio=0 tid=0x00007fb9c8008000 nid=0x7bc9 runnable [0x00007fb9fabc1000]
   java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
- locked <0x00000006cf83ff48> (a java.io.BufferedInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at org.jppf.serialization.SerializationUtils.readToBuf(SerializationUtils.java:448)
at org.jppf.serialization.SerializationUtils.readInt(SerializationUtils.java:259)
at org.jppf.comm.socket.AbstractSocketWrapper.readInt(AbstractSocketWrapper.java:241)
at org.jppf.io.SocketWrapperInputSource.readInt(SocketWrapperInputSource.java:85)
at org.jppf.io.IOHelper.readData(IOHelper.java:110)
at org.jppf.io.IOHelper.unwrappedData(IOHelper.java:188)
at org.jppf.client.BaseJPPFClientConnection.receiveBundleAndResults(BaseJPPFClientConnection.java:245)
at org.jppf.client.BaseJPPFClientConnection.receiveResults(BaseJPPFClientConnection.java:290)
at org.jppf.client.JPPFClientConnectionImpl.receiveResults(JPPFClientConnectionImpl.java:34)
at org.jppf.client.balancer.ChannelWrapperRemote$RemoteRunnable.run(ChannelWrapperRemote.java:220)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)

"oneagentperiodicrequests" #23 daemon prio=5 os_prio=0 tid=0x00007fba28001000 nid=0x7b98 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"oneagentperiodiceventsmanaged" #22 daemon prio=5 os_prio=0 tid=0x00007fba24001000 nid=0x7b9b runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"JobExecutor_0001" #19 daemon prio=5 os_prio=0 tid=0x00007fb9ac02f000 nid=0x7bb4 runnable [0x00007fb9facc2000]
   java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
- locked <0x00000006cf854210> (a java.io.BufferedInputStream)
at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2598)
at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2905)
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2915)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1502)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
at com.sun.jmx.remote.socket.SocketConnection.readMessage(SocketConnection.java:300)
at com.sun.jmx.remote.generic.ClientSynchroMessageConnectionImpl$MessageReader.run(ClientSynchroMessageConnectionImpl.java:372)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)

"ChannelsExecutor-0001" #17 daemon prio=5 os_prio=0 tid=0x00007fb9c0021800 nid=0x7bb1 waiting on condition [0x00007fb9fb0c4000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000006cf861290> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)

"driver1-1 - ClassServer" #16 daemon prio=5 os_prio=0 tid=0x00007fb9bc01f000 nid=0x7bb0 runnable [0x00007fb9fb1c5000]
   java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
- locked <0x00000006cf8445a0> (a java.io.BufferedInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at org.jppf.serialization.SerializationUtils.readToBuf(SerializationUtils.java:448)
at org.jppf.serialization.SerializationUtils.readInt(SerializationUtils.java:259)
at org.jppf.comm.socket.AbstractSocketWrapper.readInt(AbstractSocketWrapper.java:241)
at org.jppf.io.SocketWrapperInputSource.readInt(SocketWrapperInputSource.java:85)
at org.jppf.io.IOHelper.readData(IOHelper.java:110)
at org.jppf.io.IOHelper.unwrappedData(IOHelper.java:188)
at org.jppf.client.AbstractClassServerDelegate.readResource(AbstractClassServerDelegate.java:115)
at org.jppf.client.AbstractClassServerDelegate.processNextRequest(AbstractClassServerDelegate.java:155)
at org.jppf.client.ClassServerDelegateImpl.run(ClassServerDelegateImpl.java:103)
at java.lang.Thread.run(Thread.java:748)

"TaskQueueChecker" #12 prio=5 os_prio=0 tid=0x00007fba3de82800 nid=0x7bac in Object.wait() [0x00007fb9fb4c8000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:460)
at org.jppf.utils.ThreadSynchronization.goToSleep(ThreadSynchronization.java:61)
- locked <0x00000006cf83d8a8> (a org.jppf.client.balancer.queue.TaskQueueChecker)
at org.jppf.client.balancer.queue.TaskQueueChecker.run(TaskQueueChecker.java:201)
at java.lang.Thread.run(Thread.java:748)

"oneagentsubpathsender" #10 daemon prio=5 os_prio=0 tid=0x00007fba18001000 nid=0x7b9c runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"oneagentautosensor" #9 daemon prio=5 os_prio=0 tid=0x00007fb9c4001000 nid=0x7b9a runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007fba3caaf000 nid=0x7ba8 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007fba3caa1800 nid=0x7ba7 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007fba3ca37000 nid=0x7ba6 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007fba3ca8d000 nid=0x7ba5 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007fba3ca2c000 nid=0x7ba4 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fba3ca03000 nid=0x7ba3 in Object.wait() [0x00007fba00a9e000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x00000006cf7e92b8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fba3ca13000 nid=0x7ba2 in Object.wait() [0x00007fba30168000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x00000006cf7e72f0> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

"main" #1 prio=5 os_prio=0 tid=0x00007fba3c482000 nid=0x7b95 in Object.wait() [0x00007fba4071b000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.jppf.utils.ThreadSynchronization.goToSleep(ThreadSynchronization.java:49)
- locked <0x00000006cf959c50> (a org.jppf.client.JobResults)
at org.jppf.utils.ConcurrentUtils.awaitCondition(ConcurrentUtils.java:53)
at org.jppf.client.JPPFJob.await(JPPFJob.java:427)
at org.jppf.client.JPPFJob.awaitResults(JPPFJob.java:330)
at org.jppf.client.JPPFJob.awaitResults(JPPFJob.java:318)
at org.jppf.client.JPPFClient.submitJob(JPPFClient.java:153)
at com.company.claims.mpp.jppf.src.ClaimRunner.executeBlockingJob(ClaimRunner.java:96)
at com.company.claims.mpp.jppf.src.ClaimRunner.main(ClaimRunner.java:26)

"VM Thread" os_prio=0 tid=0x00007fba3c48d800 nid=0x7ba1 runnable

"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007fba3c483000 nid=0x7b9d runnable

"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007fba3c460800 nid=0x7b9e runnable

"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007fba3c492800 nid=0x7b9f runnable

"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007fba3c48e800 nid=0x7ba0 runnable

"VM Periodic Task Thread" os_prio=0 tid=0x00007fba3caa5800 nid=0x7ba9 waiting on condition

JNI global references: 288

Heap
 PSYoungGen      total 529408K, used 98431K [0x000000076c700000, 0x0000000795d80000, 0x00000007c0000000)
  eden space 515072K, 16% used [0x000000076c700000,0x0000000771977408,0x000000078be00000)
  from space 14336K, 97% used [0x000000078be00000,0x000000078cba8820,0x000000078cc00000)
  to   space 80384K, 0% used [0x0000000790f00000,0x0000000790f00000,0x0000000795d80000)
 ParOldGen       total 2026496K, used 1474326K [0x00000006c5400000, 0x0000000740f00000, 0x000000076c700000)
  object space 2026496K, 72% used [0x00000006c5400000,0x000000071f3c5b68,0x0000000740f00000)
 Metaspace       used 13313K, capacity 13560K, committed 13568K, reserved 1060864K
  class space    used 1695K, capacity 1787K, committed 1792K, reserved 1048576K

2017-06-08 14:26:24
Full thread dump OpenJDK 64-Bit Server VM (25.131-b11 mixed mode):

"RemoteChannelWrapper-driver1-1-0001" #24 daemon prio=5 os_prio=0 tid=0x00007fb9c8008000 nid=0x7bc9 runnable [0x00007fb9fabc1000]
   java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
- locked <0x00000006cf83ff48> (a java.io.BufferedInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at org.jppf.serialization.SerializationUtils.readToBuf(SerializationUtils.java:448)
at org.jppf.serialization.SerializationUtils.readInt(SerializationUtils.java:259)
at org.jppf.comm.socket.AbstractSocketWrapper.readInt(AbstractSocketWrapper.java:241)
at org.jppf.io.SocketWrapperInputSource.readInt(SocketWrapperInputSource.java:85)
at org.jppf.io.IOHelper.readData(IOHelper.java:110)
at org.jppf.io.IOHelper.unwrappedData(IOHelper.java:188)
at org.jppf.client.BaseJPPFClientConnection.receiveBundleAndResults(BaseJPPFClientConnection.java:245)
at org.jppf.client.BaseJPPFClientConnection.receiveResults(BaseJPPFClientConnection.java:290)
at org.jppf.client.JPPFClientConnectionImpl.receiveResults(JPPFClientConnectionImpl.java:34)
at org.jppf.client.balancer.ChannelWrapperRemote$RemoteRunnable.run(ChannelWrapperRemote.java:220)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)

"oneagentperiodicrequests" #23 daemon prio=5 os_prio=0 tid=0x00007fba28001000 nid=0x7b98 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"oneagentperiodiceventsmanaged" #22 daemon prio=5 os_prio=0 tid=0x00007fba24001000 nid=0x7b9b runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"JobExecutor_0001" #19 daemon prio=5 os_prio=0 tid=0x00007fb9ac02f000 nid=0x7bb4 runnable [0x00007fb9facc2000]
   java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
- locked <0x00000006cf854210> (a java.io.BufferedInputStream)
at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2598)
at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2905)
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2915)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1502)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
at com.sun.jmx.remote.socket.SocketConnection.readMessage(SocketConnection.java:300)
at com.sun.jmx.remote.generic.ClientSynchroMessageConnectionImpl$MessageReader.run(ClientSynchroMessageConnectionImpl.java:372)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)

"ChannelsExecutor-0001" #17 daemon prio=5 os_prio=0 tid=0x00007fb9c0021800 nid=0x7bb1 waiting on condition [0x00007fb9fb0c4000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000006cf861290> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)

"driver1-1 - ClassServer" #16 daemon prio=5 os_prio=0 tid=0x00007fb9bc01f000 nid=0x7bb0 runnable [0x00007fb9fb1c5000]
   java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
- locked <0x00000006cf8445a0> (a java.io.BufferedInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at org.jppf.serialization.SerializationUtils.readToBuf(SerializationUtils.java:448)
at org.jppf.serialization.SerializationUtils.readInt(SerializationUtils.java:259)
at org.jppf.comm.socket.AbstractSocketWrapper.readInt(AbstractSocketWrapper.java:241)
at org.jppf.io.SocketWrapperInputSource.readInt(SocketWrapperInputSource.java:85)
at org.jppf.io.IOHelper.readData(IOHelper.java:110)
at org.jppf.io.IOHelper.unwrappedData(IOHelper.java:188)
at org.jppf.client.AbstractClassServerDelegate.readResource(AbstractClassServerDelegate.java:115)
at org.jppf.client.AbstractClassServerDelegate.processNextRequest(AbstractClassServerDelegate.java:155)
at org.jppf.client.ClassServerDelegateImpl.run(ClassServerDelegateImpl.java:103)
at java.lang.Thread.run(Thread.java:748)

"TaskQueueChecker" #12 prio=5 os_prio=0 tid=0x00007fba3de82800 nid=0x7bac in Object.wait() [0x00007fb9fb4c8000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:460)
at org.jppf.utils.ThreadSynchronization.goToSleep(ThreadSynchronization.java:61)
- locked <0x00000006cf83d8a8> (a org.jppf.client.balancer.queue.TaskQueueChecker)
at org.jppf.client.balancer.queue.TaskQueueChecker.run(TaskQueueChecker.java:201)
at java.lang.Thread.run(Thread.java:748)

"oneagentsubpathsender" #10 daemon prio=5 os_prio=0 tid=0x00007fba18001000 nid=0x7b9c runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"oneagentautosensor" #9 daemon prio=5 os_prio=0 tid=0x00007fb9c4001000 nid=0x7b9a runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007fba3caaf000 nid=0x7ba8 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007fba3caa1800 nid=0x7ba7 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007fba3ca37000 nid=0x7ba6 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007fba3ca8d000 nid=0x7ba5 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007fba3ca2c000 nid=0x7ba4 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fba3ca03000 nid=0x7ba3 in Object.wait() [0x00007fba00a9e000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x00000006cf7e92b8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fba3ca13000 nid=0x7ba2 in Object.wait() [0x00007fba30168000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x00000006cf7e72f0> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

"main" #1 prio=5 os_prio=0 tid=0x00007fba3c482000 nid=0x7b95 in Object.wait() [0x00007fba4071b000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.jppf.utils.ThreadSynchronization.goToSleep(ThreadSynchronization.java:49)
- locked <0x00000006cf959c50> (a org.jppf.client.JobResults)
at org.jppf.utils.ConcurrentUtils.awaitCondition(ConcurrentUtils.java:53)
at org.jppf.client.JPPFJob.await(JPPFJob.java:427)
at org.jppf.client.JPPFJob.awaitResults(JPPFJob.java:330)
at org.jppf.client.JPPFJob.awaitResults(JPPFJob.java:318)
at org.jppf.client.JPPFClient.submitJob(JPPFClient.java:153)
at com.company.claims.mpp.jppf.src.ClaimRunner.executeBlockingJob(ClaimRunner.java:96)
at com.company.claims.mpp.jppf.src.ClaimRunner.main(ClaimRunner.java:26)

"VM Thread" os_prio=0 tid=0x00007fba3c48d800 nid=0x7ba1 runnable

"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007fba3c483000 nid=0x7b9d runnable

"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007fba3c460800 nid=0x7b9e runnable

"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007fba3c492800 nid=0x7b9f runnable

"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007fba3c48e800 nid=0x7ba0 runnable

"VM Periodic Task Thread" os_prio=0 tid=0x00007fba3caa5800 nid=0x7ba9 waiting on condition

JNI global references: 288

Heap
 PSYoungGen      total 529408K, used 98431K [0x000000076c700000, 0x0000000795d80000, 0x00000007c0000000)
  eden space 515072K, 16% used [0x000000076c700000,0x0000000771977408,0x000000078be00000)
  from space 14336K, 97% used [0x000000078be00000,0x000000078cba8820,0x000000078cc00000)
  to   space 80384K, 0% used [0x0000000790f00000,0x0000000790f00000,0x0000000795d80000)
 ParOldGen       total 2026496K, used 1474326K [0x00000006c5400000, 0x0000000740f00000, 0x000000076c700000)
  object space 2026496K, 72% used [0x00000006c5400000,0x000000071f3c5b68,0x0000000740f00000)
 Metaspace       used 13313K, capacity 13560K, committed 13568K, reserved 1060864K
  class space    used 1695K, capacity 1787K, committed 1792K, reserved 1048576K


IntelliJ provides a method to read the dump and I used that. What I understood was that when we submit the job and wait for the results, the thread is killed as it is waiting for “too long”. I looked into the config files of the JPPF client, server and node but couldn’t find a way to increase the time a thread can stay alive and wait without being killed. Is my line of thinking right or am I missing something?
Logged

lolo

  • Administrator
  • JPPF Council Member
  • *****
  • Posts: 2240
    • JPPF Web site
Re: Probable out of memory exception?
« Reply #1 on: June 15, 2017, 07:33:59 AM »

Hello,

The thread dump clearly shows that the JPPF client is still waiting for the job results. This is visible in the call stack of the "main" thread:

Code: [Select]
"main" #1 prio=5 os_prio=0 tid=0x00007fba3c482000 nid=0x7b95 in Object.wait() [0x00007fba4071b000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
  at java.lang.Object.wait(Native Method)
  at org.jppf.utils.ThreadSynchronization.goToSleep(ThreadSynchronization.java:49)
  - locked <0x00000006cf959c50> (a org.jppf.client.JobResults)
  at org.jppf.utils.ConcurrentUtils.awaitCondition(ConcurrentUtils.java:53)
  at org.jppf.client.JPPFJob.await(JPPFJob.java:427)
  at org.jppf.client.JPPFJob.awaitResults(JPPFJob.java:330)
  at org.jppf.client.JPPFJob.awaitResults(JPPFJob.java:318)
  at org.jppf.client.JPPFClient.submitJob(JPPFClient.java:153)
  at com.company.claims.mpp.jppf.src.ClaimRunner.executeBlockingJob(ClaimRunner.java:96)
  at com.company.claims.mpp.jppf.src.ClaimRunner.main(ClaimRunner.java:26)

We can see the JPPFClient.submitJob() call. Thus, we can exclude a possible job submission/execution timeout, which is not implemented in the JPPFClient API in any case.

You mentioned that you submitted 1 million tasks. This is a very large number of tasks and this makes the possibility of an OutOfMemoryError very plausible. Normally, the OOME should show in the log file or in the console output, but it's not always easy to catch. One sure way to determine whether an OOME occurred is to start the JVM with the -XX:+HeapDumpOnOutOfMemoryError option, so that a heap dump will be generated when an OOME happens. I would recommend to use this option in your cliient application, but also for the drivers and nodes (in the "jppf.jvm.options" configuration property).

Once a heap dump is generated, you can analyze it with a tool such as Eclipse MAT to determine exactly what is using the Java heap. If it turns out that this is caused by the sheer number of tasks you submit, then you might need to increase the heap allocated to your client, drivers and nodes.

Also keep in mind that the driver's load-balancing configuration can provide a measure of control over the memory pressure in the nodes, by limiting the number of tasks that can be sent to the nodes a once.

Sincerely,
-Laurent
Logged
Pages: [1]   Go Up
JPPF Powered by SMF 2.0 RC5 | SMF © 2006–2011, Simple Machines LLC Get JPPF at SourceForge.net. Fast, secure and Free Open Source software downloads