You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Gautam <ga...@gmail.com> on 2016/02/11 02:28:42 UTC
Record too large for Tez in-memory buffer...
Hello ,
Trying to benchmark with Hive on Tez causes the following error. Admittedly
these are some very large looking records .. the same job runs fine on MR2.
I'v attached the query explain tree. It fails in the very last reducer
phase ..
*Execution:*
--------------------------------------------------------------------------------
VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED
KILLED
--------------------------------------------------------------------------------
Map 1 .......... SUCCEEDED 477 477 0 0 0
0
Reducer 10 ..... SUCCEEDED 250 250 0 0 0
0
Reducer 11 ..... SUCCEEDED 250 250 0 0 1
0
Reducer 12 ..... SUCCEEDED 250 250 0 0 1
0
Reducer 13 ..... SUCCEEDED 250 250 0 0 1
0
Reducer 14 ..... SUCCEEDED 250 250 0 0 0
0
Reducer 15 ..... SUCCEEDED 250 250 0 0 0
0
Reducer 16 ... KILLED 250 187 0 63 0
63
Reducer 17 FAILED 250 0 0 250 62
249
Reducer 2 ...... SUCCEEDED 250 250 0 0 1
0
Reducer 3 ...... SUCCEEDED 250 250 0 0 0
0
Reducer 4 ...... SUCCEEDED 250 250 0 0 0
0
Reducer 5 ...... SUCCEEDED 250 250 0 0 0
0
Reducer 6 ...... SUCCEEDED 250 250 0 0 1
0
Reducer 7 ...... SUCCEEDED 250 250 0 0 0
0
Reducer 8 ...... SUCCEEDED 250 250 0 0 0
0
Reducer 9 ...... SUCCEEDED 250 250 0 0 0
0
--------------------------------------------------------------------------------
VERTICES: 15/17 [========================>>--] 93% ELAPSED TIME:
17600.11 s
--------------------------------------------------------------------------------
*Error: *
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.tez.runtime.library.common.sort.impl.ExternalSorter$MapBufferTooSmallException:
Record too large for in-memory buffer. Exceeded buffer overflow limit,
bufferOverflowRecursion=2, bufferList.size=1, blockSize=268435456
at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:397)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
at org.apache.hadoop.hive.ql.exec.FilterOperator.processOp(FilterOperator.java:120)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:157)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:497)
... 18 more
Caused by: org.apache.tez.runtime.library.common.sort.impl.ExternalSorter$MapBufferTooSmallException:
Record too large for in-memory buffer. Exceeded buffer overflow limit,
bufferOverflowRecursion=2, bufferList.size=1, blockSize=268435456
at org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.collect(PipelinedSorter.java:315)
at org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.collect(PipelinedSorter.java:320)
at org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.write(PipelinedSorter.java:272)
at org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput$1.write(OrderedPartitionedKVOutput.java:164)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor$TezKVOutputCollector.collect(TezProcessor.java:211)
at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.collect(ReduceSinkOperator.java:534)
at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:380)
... 24 more
-Gautam.
Re: Record too large for Tez in-memory buffer...
Posted by Gautam <ga...@gmail.com>.
setting tez.task.scale.memory.enabled=false; failed the job sooner.
But the fix from TEZ-2575 worked!
Upgraded from 0.7.0 to 0.7.1 and applied
https://github.com/apache/tez/commit/0e155e7185d1350f64dead488103777295ac76d1
Goes through without any fatal issues. Will continue testing / benchmarking
further.
thanks!
-Gautam.
On Wed, Feb 10, 2016 at 8:12 PM, Gopal Vijayaraghavan <go...@apache.org>
wrote:
>
> > Good to know there's a fix .. Is there a jira that talks about this
> >issue? Coz I couldn't find one.
>
> https://github.com/apache/tez/commit/714461f47e6408ec331acd0ddd640335e6a7a0
> 6c
>
>
> Also, it looks like Reducer 16 is the one failing - not Reducer 17.
>
> You can draw out the explain using https://github.com/t3rmin4t0r/lipwig
>
> PTF doesn't actually tell the UDAF name in the explain, so I'm guessing it
> a ROW_NUMBER() <= 50 - because that's the only one which didn't get
> optimized.
>
> I see absolutely no broadcast edges in this, so it's possible to disable
> the weighted memory scaler in Tez to sort of dumb it down to MRv2 mode.
>
> set tez.task.scale.memory.enabled=false;
>
> *or* do extensive tuning for it (see
> tez.task.scale.memory.additionalreservation.fraction.max).
>
> Cheers,
> Gopal
>
>
--
"If you really want something in this life, you have to work for it. Now,
quiet! They're about to announce the lottery numbers..."
Re: Record too large for Tez in-memory buffer...
Posted by Gopal Vijayaraghavan <go...@apache.org>.
> Good to know there's a fix .. Is there a jira that talks about this
>issue? Coz I couldn't find one.
https://github.com/apache/tez/commit/714461f47e6408ec331acd0ddd640335e6a7a0
6c
Also, it looks like Reducer 16 is the one failing - not Reducer 17.
You can draw out the explain using https://github.com/t3rmin4t0r/lipwig
PTF doesn't actually tell the UDAF name in the explain, so I'm guessing it
a ROW_NUMBER() <= 50 - because that's the only one which didn't get
optimized.
I see absolutely no broadcast edges in this, so it's possible to disable
the weighted memory scaler in Tez to sort of dumb it down to MRv2 mode.
set tez.task.scale.memory.enabled=false;
*or* do extensive tuning for it (see
tez.task.scale.memory.additionalreservation.fraction.max).
Cheers,
Gopal
Re: Record too large for Tez in-memory buffer...
Posted by Gautam <ga...@gmail.com>.
Thanks Gopal! I'l look at options provided.
On Wed, Feb 10, 2016 at 7:46 PM, Gautam <ga...@gmail.com> wrote:
> Here's the json version.
>
> On Wed, Feb 10, 2016 at 7:44 PM, Gautam <ga...@gmail.com> wrote:
>
>> Whoops.. meant to send the tez explain earlier. Here's the Tez query
>> plan. Good to know there's a fix .. Is there a jira that talks about this
>> issue? Coz I couldn't find one. Maybe I can alter the query a bit to
>> filter these out.
>>
>> Cheers,
>> -Gautam.
>>
>>
>> On Wed, Feb 10, 2016 at 7:32 PM, Gopal Vijayaraghavan <go...@apache.org>
>> wrote:
>>
>>> Hey,
>>>
>>> > Trying to benchmark with Hive on Tez causes the following error.
>>> >Admittedly these are some very large looking records .. the same job
>>> runs
>>> >fine on MR2.
>>> ...
>>> > I'v attached the query explain tree. It fails in the very last reducer
>>> >phase ..
>>>
>>> Can you attach the explain plan with hive.execution.engine=tez (even
>>> better would be the JSON output of "explain formatted").
>>>
>>> > Error: Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
>>>
>>> >org.apache.tez.runtime.library.common.sort.impl.ExternalSorter$MapBufferTo
>>> >oSmallException: Record too large for in-memory buffer. Exceeded buffer
>>> >overflow limit, bufferOverflowRecursion=2, bufferList.size=1,
>>> >blockSize=268435456
>>>
>>> You have a single row which is > 200Mb?
>>>
>>> The tez 0.8.2 out there should already have this issue fixed, but single
>>> spill record mode is likely to spin your disks all wrong (unless you have
>>> SSDs).
>>>
>>> I'm guessing there's a UDTF + PTF producing humongous rows & that it's
>>> being planned wrong because that's not a standard pattern.
>>>
>>> Cheers,
>>> Gopal
>>>
>>>
>>>
>>
>>
>> --
>> "If you really want something in this life, you have to work for it. Now,
>> quiet! They're about to announce the lottery numbers..."
>>
>
>
>
> --
> "If you really want something in this life, you have to work for it. Now,
> quiet! They're about to announce the lottery numbers..."
>
--
"If you really want something in this life, you have to work for it. Now,
quiet! They're about to announce the lottery numbers..."
Re: Record too large for Tez in-memory buffer...
Posted by Gautam <ga...@gmail.com>.
Here's the json version.
On Wed, Feb 10, 2016 at 7:44 PM, Gautam <ga...@gmail.com> wrote:
> Whoops.. meant to send the tez explain earlier. Here's the Tez query plan.
> Good to know there's a fix .. Is there a jira that talks about this issue? Coz
> I couldn't find one. Maybe I can alter the query a bit to filter these
> out.
>
> Cheers,
> -Gautam.
>
>
> On Wed, Feb 10, 2016 at 7:32 PM, Gopal Vijayaraghavan <go...@apache.org>
> wrote:
>
>> Hey,
>>
>> > Trying to benchmark with Hive on Tez causes the following error.
>> >Admittedly these are some very large looking records .. the same job runs
>> >fine on MR2.
>> ...
>> > I'v attached the query explain tree. It fails in the very last reducer
>> >phase ..
>>
>> Can you attach the explain plan with hive.execution.engine=tez (even
>> better would be the JSON output of "explain formatted").
>>
>> > Error: Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
>>
>> >org.apache.tez.runtime.library.common.sort.impl.ExternalSorter$MapBufferTo
>> >oSmallException: Record too large for in-memory buffer. Exceeded buffer
>> >overflow limit, bufferOverflowRecursion=2, bufferList.size=1,
>> >blockSize=268435456
>>
>> You have a single row which is > 200Mb?
>>
>> The tez 0.8.2 out there should already have this issue fixed, but single
>> spill record mode is likely to spin your disks all wrong (unless you have
>> SSDs).
>>
>> I'm guessing there's a UDTF + PTF producing humongous rows & that it's
>> being planned wrong because that's not a standard pattern.
>>
>> Cheers,
>> Gopal
>>
>>
>>
>
>
> --
> "If you really want something in this life, you have to work for it. Now,
> quiet! They're about to announce the lottery numbers..."
>
--
"If you really want something in this life, you have to work for it. Now,
quiet! They're about to announce the lottery numbers..."
Re: Record too large for Tez in-memory buffer...
Posted by Gautam <ga...@gmail.com>.
Whoops.. meant to send the tez explain earlier. Here's the Tez query plan.
Good to know there's a fix .. Is there a jira that talks about this issue? Coz
I couldn't find one. Maybe I can alter the query a bit to filter these out.
Cheers,
-Gautam.
On Wed, Feb 10, 2016 at 7:32 PM, Gopal Vijayaraghavan <go...@apache.org>
wrote:
> Hey,
>
> > Trying to benchmark with Hive on Tez causes the following error.
> >Admittedly these are some very large looking records .. the same job runs
> >fine on MR2.
> ...
> > I'v attached the query explain tree. It fails in the very last reducer
> >phase ..
>
> Can you attach the explain plan with hive.execution.engine=tez (even
> better would be the JSON output of "explain formatted").
>
> > Error: Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> >org.apache.tez.runtime.library.common.sort.impl.ExternalSorter$MapBufferTo
> >oSmallException: Record too large for in-memory buffer. Exceeded buffer
> >overflow limit, bufferOverflowRecursion=2, bufferList.size=1,
> >blockSize=268435456
>
> You have a single row which is > 200Mb?
>
> The tez 0.8.2 out there should already have this issue fixed, but single
> spill record mode is likely to spin your disks all wrong (unless you have
> SSDs).
>
> I'm guessing there's a UDTF + PTF producing humongous rows & that it's
> being planned wrong because that's not a standard pattern.
>
> Cheers,
> Gopal
>
>
>
--
"If you really want something in this life, you have to work for it. Now,
quiet! They're about to announce the lottery numbers..."
Re: Record too large for Tez in-memory buffer...
Posted by Gopal Vijayaraghavan <go...@apache.org>.
Hey,
> Trying to benchmark with Hive on Tez causes the following error.
>Admittedly these are some very large looking records .. the same job runs
>fine on MR2.
...
> I'v attached the query explain tree. It fails in the very last reducer
>phase ..
Can you attach the explain plan with hive.execution.engine=tez (even
better would be the JSON output of "explain formatted").
> Error: Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
>org.apache.tez.runtime.library.common.sort.impl.ExternalSorter$MapBufferTo
>oSmallException: Record too large for in-memory buffer. Exceeded buffer
>overflow limit, bufferOverflowRecursion=2, bufferList.size=1,
>blockSize=268435456
You have a single row which is > 200Mb?
The tez 0.8.2 out there should already have this issue fixed, but single
spill record mode is likely to spin your disks all wrong (unless you have
SSDs).
I'm guessing there's a UDTF + PTF producing humongous rows & that it's
being planned wrong because that's not a standard pattern.
Cheers,
Gopal