You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Srinivas Pogiri <po...@gmail.com> on 2017/02/16 22:49:48 UTC

Hive UDAF function throwing "java.lang.OutOfMemoryError: GC overhead limit exceeded" error

Hi,

We have a custom Hive UDAF function which aggregates a lot of data for
grouping. The reduce task fails with the below stack trace. Any suggestion
would be very help.

MR job was having 5 Maps which completed fine. There were 6 reduces out of
which 5 only completed. Here is a sample MR job_1476197655848_2037226
Error:'
INFO communication thread org.apache.hadoop.mapred.Task: Communication
exception: java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.io.BufferedReader.<init>(BufferedReader.java:105)
at java.io.BufferedReader.<init>(BufferedReader.java:116)
at
org.apache.hadoop.yarn.util.ProcfsBasedProcessTree.constructProcessInfo(ProcfsBasedProcessTree.java:525)
at
org.apache.hadoop.yarn.util.ProcfsBasedProcessTree.updateProcessTree(ProcfsBasedProcessTree.java:223)
at org.apache.hadoop.mapred.Task.updateResourceCounters(Task.java:847)
at org.apache.hadoop.mapred.Task.updateCounters(Task.java:986)
at org.apache.hadoop.mapred.Task.access$500(Task.java:79)
at org.apache.hadoop.mapred.Task$TaskReporter.run(Task.java:735)
at java.lang.Thread.run(Thread.java:745)

Thank you,
Srinivas Pogiri

Re: Hive UDAF function throwing "java.lang.OutOfMemoryError: GC overhead limit exceeded" error

Posted by Deepak Khandelwal <dk...@gmail.com>.
How much data is there? Is your table partitioned ?

On Thursday, February 16, 2017, Srinivas Pogiri <po...@gmail.com>
wrote:

> Hi,
>
> We have a custom Hive UDAF function which aggregates a lot of data for
> grouping. The reduce task fails with the below stack trace. Any suggestion
> would be very help.
>
> MR job was having 5 Maps which completed fine. There were 6 reduces out of
> which 5 only completed. Here is a sample MR job_1476197655848_2037226
> Error:'
> INFO communication thread org.apache.hadoop.mapred.Task: Communication
> exception: java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.io.BufferedReader.<init>(BufferedReader.java:105)
> at java.io.BufferedReader.<init>(BufferedReader.java:116)
> at org.apache.hadoop.yarn.util.ProcfsBasedProcessTree.
> constructProcessInfo(ProcfsBasedProcessTree.java:525)
> at org.apache.hadoop.yarn.util.ProcfsBasedProcessTree.updateProcessTree(
> ProcfsBasedProcessTree.java:223)
> at org.apache.hadoop.mapred.Task.updateResourceCounters(Task.java:847)
> at org.apache.hadoop.mapred.Task.updateCounters(Task.java:986)
> at org.apache.hadoop.mapred.Task.access$500(Task.java:79)
> at org.apache.hadoop.mapred.Task$TaskReporter.run(Task.java:735)
> at java.lang.Thread.run(Thread.java:745)
>
> Thank you,
> Srinivas Pogiri
>
>