You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@tez.apache.org by Chris K Wensel <ch...@wensel.net> on 2014/06/19 01:53:04 UTC

OutOfMemoryError on DefaultSorter

Hey all

I've got a DAG with a Vertex that outputs to two Vertices, with each edge SCATTER_GATHER, PERSISTED, SEQUENTIAL. See attached pdf.

The input file has 5 text lines. and i'm passing 2048 via the resource to the Vertex.

I'm on commit TEZ-1172. 

And am getting the below stacktrace. 

is there another memory setting I need to be adjusting? Or is this just not supported.

Note i'm calling #start on the OnFileSortedOutput class manually. I know I cannot call #stop. 

2014-06-18 16:28:46,322 INFO  dflt.DefaultSorter (DefaultSorter.java:setEquator(327)) - (EQUATOR) 0 kvi 21757948(87031792)
2014-06-18 16:28:46,322 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 83
2014-06-18 16:28:46,323 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 69625448
2014-06-18 16:28:46,323 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 87031808
2014-06-18 16:28:46,323 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 21757948; length = 5439488
2014-06-18 16:28:46,335 INFO  impl.ExternalSorter (ExternalSorter.java:<init>(192)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2014-06-18 16:28:46,335 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(107)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2014-06-18 16:28:46,335 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(63)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2014-06-18 16:28:46,459 ERROR task.TezTaskRunner (TezTaskRunner.java:run(218)) - Exception of type Error. Exiting now
java.lang.OutOfMemoryError: Java heap space
	at org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter.<init>(DefaultSorter.java:137)
	at org.apache.tez.runtime.library.output.OnFileSortedOutput.start(OnFileSortedOutput.java:98)
	at cascading.flow.tez.stream.TezGroupGate.prepare(TezGroupGate.java:85)
	at cascading.flow.stream.StreamGraph.prepare(StreamGraph.java:169)
	at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:120)
	at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:307)
	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:181)
	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:173)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:173)
	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:722)
2014-06-18 16:28:46,462 INFO  util.ExitUtil (ExitUtil.java:terminate(124)) - Exiting with status -1


--
Chris K Wensel
chris@concurrentinc.com
http://concurrentinc.com

Re: OutOfMemoryError on DefaultSorter

Posted by Chris K Wensel <ch...@wensel.net>.
that was it, thanks!

ckw

On Jun 18, 2014, at 6:37 PM, Bikas Saha <bi...@hortonworks.com> wrote:

> Chris,
>  
> If you can confirm that the JVM opts is the issue behind this then we can speed up https://issues.apache.org/jira/browse/TEZ-699 so that you can use that instead of working around in your code.
>  
> Bikas
>  
> From: Siddharth Seth [mailto:sseth@apache.org] 
> Sent: Wednesday, June 18, 2014 5:12 PM
> To: user@tez.incubator.apache.org
> Subject: Re: OutOfMemoryError on DefaultSorter
>  
> Chris
> Are you configuring the JVM options via Vertex.setJavaOpts ?
> Setting the container size to 2048 doesn't automatically set the java options at the moment. There's a jira under TEZ-690, which would start setting a default based on the specified container size (if users don't set it) - but that isn't in yet.
> What's likely happening here is that the heap size on the JVM is far too low, and allocating an 83MB buffer for sort causes the OOM.
>  
> Thanks
> - Sid
>  
> On Wed, Jun 18, 2014 at 4:53 PM, Chris K Wensel <ch...@wensel.net> wrote:
> Hey all
>  
> I've got a DAG with a Vertex that outputs to two Vertices, with each edge SCATTER_GATHER, PERSISTED, SEQUENTIAL. See attached pdf.
>  
> The input file has 5 text lines. and i'm passing 2048 via the resource to the Vertex.
>  
> I'm on commit TEZ-1172. 
>  
> And am getting the below stacktrace. 
>  
> is there another memory setting I need to be adjusting? Or is this just not supported.
>  
> Note i'm calling #start on the OnFileSortedOutput class manually. I know I cannot call #stop. 
>  
> 2014-06-18 16:28:46,322 INFO  dflt.DefaultSorter (DefaultSorter.java:setEquator(327)) - (EQUATOR) 0 kvi 21757948(87031792)
> 2014-06-18 16:28:46,322 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 83
> 2014-06-18 16:28:46,323 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 69625448
> 2014-06-18 16:28:46,323 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 87031808
> 2014-06-18 16:28:46,323 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 21757948; length = 5439488
> 2014-06-18 16:28:46,335 INFO  impl.ExternalSorter (ExternalSorter.java:<init>(192)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
> 2014-06-18 16:28:46,335 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(107)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
> 2014-06-18 16:28:46,335 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(63)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
> 2014-06-18 16:28:46,459 ERROR task.TezTaskRunner (TezTaskRunner.java:run(218)) - Exception of type Error. Exiting now
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter.<init>(DefaultSorter.java:137)
> at org.apache.tez.runtime.library.output.OnFileSortedOutput.start(OnFileSortedOutput.java:98)
> at cascading.flow.tez.stream.TezGroupGate.prepare(TezGroupGate.java:85)
> at cascading.flow.stream.StreamGraph.prepare(StreamGraph.java:169)
> at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:120)
> at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:307)
> at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:181)
> at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:173)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:173)
> at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> 2014-06-18 16:28:46,462 INFO  util.ExitUtil (ExitUtil.java:terminate(124)) - Exiting with status -1
>  
>  
> --
> Chris K Wensel
> chris@concurrentinc.com
> http://concurrentinc.com
>  
>  
>  
> 
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.

--
Chris K Wensel
chris@concurrentinc.com
http://concurrentinc.com

RE: OutOfMemoryError on DefaultSorter

Posted by Bikas Saha <bi...@hortonworks.com>.
Chris,



If you can confirm that the JVM opts is the issue behind this then we can
speed up https://issues.apache.org/jira/browse/TEZ-699 so that you can use
that instead of working around in your code.



Bikas



*From:* Siddharth Seth [mailto:sseth@apache.org]
*Sent:* Wednesday, June 18, 2014 5:12 PM
*To:* user@tez.incubator.apache.org
*Subject:* Re: OutOfMemoryError on DefaultSorter



Chris

Are you configuring the JVM options via Vertex.setJavaOpts ?

Setting the container size to 2048 doesn't automatically set the java
options at the moment. There's a jira under TEZ-690, which would start
setting a default based on the specified container size (if users don't set
it) - but that isn't in yet.

What's likely happening here is that the heap size on the JVM is far too
low, and allocating an 83MB buffer for sort causes the OOM.



Thanks

- Sid



On Wed, Jun 18, 2014 at 4:53 PM, Chris K Wensel <ch...@wensel.net> wrote:

Hey all



I've got a DAG with a Vertex that outputs to two Vertices, with each
edge SCATTER_GATHER, PERSISTED, SEQUENTIAL. See attached pdf.



The input file has 5 text lines. and i'm passing 2048 via the resource to
the Vertex.



I'm on commit TEZ-1172.



And am getting the below stacktrace.



is there another memory setting I need to be adjusting? Or is this just not
supported.



Note i'm calling #start on the OnFileSortedOutput class manually. I know I
cannot call #stop.



2014-06-18 16:28:46,322 INFO  dflt.DefaultSorter
(DefaultSorter.java:setEquator(327)) - (EQUATOR) 0 kvi 21757948(87031792)

2014-06-18 16:28:46,322 INFO  dflt.DefaultSorter
(DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 83

2014-06-18 16:28:46,323 INFO  dflt.DefaultSorter
(DefaultSorter.java:<init>(151)) - soft limit at 69625448

2014-06-18 16:28:46,323 INFO  dflt.DefaultSorter
(DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 87031808

2014-06-18 16:28:46,323 INFO  dflt.DefaultSorter
(DefaultSorter.java:<init>(153)) - kvstart = 21757948; length = 5439488

2014-06-18 16:28:46,335 INFO  impl.ExternalSorter
(ExternalSorter.java:<init>(192)) - Instantiating Partitioner:
[cascading.tuple.tez.util.TuplePartitioner]

2014-06-18 16:28:46,335 INFO  common.TezRuntimeUtils
(TezRuntimeUtils.java:instantiatePartitioner(107)) - Using partitioner
class: cascading.tuple.tez.util.TuplePartitioner

2014-06-18 16:28:46,335 INFO  common.TezRuntimeUtils
(TezRuntimeUtils.java:instantiateCombiner(63)) - No combiner specified via
tez.runtime.combiner.class. Combiner will not be used

2014-06-18 16:28:46,459 ERROR task.TezTaskRunner
(TezTaskRunner.java:run(218)) - Exception of type Error. Exiting now

java.lang.OutOfMemoryError: Java heap space

at
org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter.<init>(DefaultSorter.java:137)

*at
org.apache.tez.runtime.library.output.OnFileSortedOutput.start(OnFileSortedOutput.java:98)*

at cascading.flow.tez.stream.TezGroupGate.prepare(TezGroupGate.java:85)

at cascading.flow.stream.StreamGraph.prepare(StreamGraph.java:169)

at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:120)

at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:307)

at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:181)

at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:173)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:415)

at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)

at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:173)

at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168)

at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)

at java.util.concurrent.FutureTask.run(FutureTask.java:166)

at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:722)

2014-06-18 16:28:46,462 INFO  util.ExitUtil (ExitUtil.java:terminate(124))
- Exiting with status -1





--

Chris K Wensel

chris@concurrentinc.com

http://concurrentinc.com

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: OutOfMemoryError on DefaultSorter

Posted by Siddharth Seth <ss...@apache.org>.
Chris
Are you configuring the JVM options via Vertex.setJavaOpts ?
Setting the container size to 2048 doesn't automatically set the java
options at the moment. There's a jira under TEZ-690, which would start
setting a default based on the specified container size (if users don't set
it) - but that isn't in yet.
What's likely happening here is that the heap size on the JVM is far too
low, and allocating an 83MB buffer for sort causes the OOM.

Thanks
- Sid


On Wed, Jun 18, 2014 at 4:53 PM, Chris K Wensel <ch...@wensel.net> wrote:

> Hey all
>
> I've got a DAG with a Vertex that outputs to two Vertices, with each
> edge SCATTER_GATHER, PERSISTED, SEQUENTIAL. See attached pdf.
>
> The input file has 5 text lines. and i'm passing 2048 via the resource to
> the Vertex.
>
> I'm on commit TEZ-1172.
>
> And am getting the below stacktrace.
>
> is there another memory setting I need to be adjusting? Or is this just
> not supported.
>
> Note i'm calling #start on the OnFileSortedOutput class manually. I know I
> cannot call #stop.
>
> 2014-06-18 16:28:46,322 INFO  dflt.DefaultSorter
> (DefaultSorter.java:setEquator(327)) - (EQUATOR) 0 kvi 21757948(87031792)
> 2014-06-18 16:28:46,322 INFO  dflt.DefaultSorter
> (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 83
> 2014-06-18 16:28:46,323 INFO  dflt.DefaultSorter
> (DefaultSorter.java:<init>(151)) - soft limit at 69625448
> 2014-06-18 16:28:46,323 INFO  dflt.DefaultSorter
> (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 87031808
> 2014-06-18 16:28:46,323 INFO  dflt.DefaultSorter
> (DefaultSorter.java:<init>(153)) - kvstart = 21757948; length = 5439488
> 2014-06-18 16:28:46,335 INFO  impl.ExternalSorter
> (ExternalSorter.java:<init>(192)) - Instantiating Partitioner:
> [cascading.tuple.tez.util.TuplePartitioner]
> 2014-06-18 16:28:46,335 INFO  common.TezRuntimeUtils
> (TezRuntimeUtils.java:instantiatePartitioner(107)) - Using partitioner
> class: cascading.tuple.tez.util.TuplePartitioner
> 2014-06-18 16:28:46,335 INFO  common.TezRuntimeUtils
> (TezRuntimeUtils.java:instantiateCombiner(63)) - No combiner specified via
> tez.runtime.combiner.class. Combiner will not be used
> 2014-06-18 16:28:46,459 ERROR task.TezTaskRunner
> (TezTaskRunner.java:run(218)) - Exception of type Error. Exiting now
> java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter.<init>(DefaultSorter.java:137)
> *at
> org.apache.tez.runtime.library.output.OnFileSortedOutput.start(OnFileSortedOutput.java:98)*
> at cascading.flow.tez.stream.TezGroupGate.prepare(TezGroupGate.java:85)
> at cascading.flow.stream.StreamGraph.prepare(StreamGraph.java:169)
> at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:120)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:307)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:181)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:173)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:173)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> 2014-06-18 16:28:46,462 INFO  util.ExitUtil (ExitUtil.java:terminate(124))
> - Exiting with status -1
>
>
> --
> Chris K Wensel
> chris@concurrentinc.com
> http://concurrentinc.com
>
>
>