You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by "Martin, Nick" <Ni...@pssd.com> on 2014/02/06 01:08:19 UTC
RE: Execution failed with exit status: 3
Hi all,
Still having trouble with OOM/GC Overhead limit on queries. Happens very frequently. The tables being joined aren't terribly large (~6m rows, ~12 columns) and we've upped our HADOOP_HEAPSIZE and several of the other heap-related settings to 12GB. Sample of one of the more recent errors is below. The error seems to occur anytime I attempt to join two or more tables with several million rows of data. Error occurs both in the Hue Hive shell and via Hive CLI. Is there anything else I should be checking? Any other info I can provide to help isolate what the issue might be?
Running Hive 11
2014-02-05 17:29:44,603 FATAL org.apache.hadoop.mapred.Child: Error running child : java.lang.OutOfMemoryError: GC overhead limit exceeded
at org.apache.hadoop.hive.serde2.io.HiveDecimalWritable.getHiveDecimal(HiveDecimalWritable.java:84)
at org.apache.hadoop.hive.serde2.io.HiveDecimalWritable.<init>(HiveDecimalWritable.java:51)
at org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableHiveDecimalObjectInspector.copyObject(WritableHiveDecimalObjectInspector.java:44)
at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject(ObjectInspectorUtils.java:260)
at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject(ObjectInspectorUtils.java:296)
at org.apache.hadoop.hive.ql.exec.persistence.MapJoinObjectValue.readExternal(MapJoinObjectValue.java:105)
at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1791)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1750)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
at java.util.HashMap.readObject(HashMap.java:1030)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:969)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1848)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
at org.apache.hadoop.hive.ql.exec.persistence.HashMapWrapper.initilizePersistentHash(HashMapWrapper.java:128)
at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:194)
at org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:212)
at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1377)
at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1381)
at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1381)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:611)
at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:144)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:365)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
From: Martin, Nick [mailto:NiMartin@pssd.com]
Sent: Tuesday, October 08, 2013 3:21 PM
To: user@hive.apache.org
Subject: RE: Execution failed with exit status: 3
Thanks for the suggestion. We have the user perms configured appropriately, as we're able to run other Hive queries that kick off MR jobs and complete them successfully.
MR1
HWX suggested it might have something to do with hive.auto.convert.join = true and hive.mapjoin.smalltable.filesize=25MB. The smaller table in the join is 250mb so we're going to play with that and see if it fixes anything. I'll update this thread after we test in case anyone else hits this issue down the road.
From: Sanjay Subramanian [mailto:Sanjay.Subramanian@wizecommerce.com]
Sent: Tuesday, October 08, 2013 2:53 PM
To: user@hive.apache.org<ma...@hive.apache.org>
Subject: Re: Execution failed with exit status: 3
Hi
Are u running this thru Beeswax in Hue ? If I recall right then u might need to provide access to "hue" user to submit and run MR jobs on the cluster
Also r u using YARN or MR1 ?
Thanks
Regards
sanjay
From: <Martin>, Nick <Ni...@pssd.com>>
Reply-To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Date: Tuesday, October 8, 2013 10:27 AM
To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Subject: RE: Execution failed with exit status: 3
Update on this...
When I run this in Hive CLI it works perfectly. Only has a bug in Hue. I'll send this thread over to hue_user@ and see what they say.
From: Martin, Nick [mailto:NiMartin@pssd.com]
Sent: Tuesday, October 08, 2013 12:17 PM
To: user@hive.apache.org<ma...@hive.apache.org>
Subject: RE: Execution failed with exit status: 3
Hi Sanjay, thanks for the suggestion.
There are no partitions on either table.
From: Sanjay Subramanian [mailto:Sanjay.Subramanian@wizecommerce.com]
Sent: Monday, October 07, 2013 8:19 PM
To: user@hive.apache.org<ma...@hive.apache.org>
Subject: Re: Execution failed with exit status: 3
Hi Nick
How many partitions are there in table t1 and table t2
If there are many partitions in either t1 or t2 or both can u mod your query as follows and see if the error comes up
SELECT
T1.somecolumn,
T2.someothercolumn
FROM
(SELECT * FROM t1 WHERE partition_column1='<some_val>') T1
JOIN
(SELECT * FROM t2 WHERE partition_column2='<some_val>') T2
ON (T1.idfield=T2.idfield)
Thanks
Regards
sanjay
email : sanjay.subramanian@wizecommerce.com<ma...@wizecommerce.com>
Irc : sanjaysub (channel #noc)
skype : sanjaysubramanian
mobile : (925) 399 2692
From: <Martin>, Nick <Ni...@pssd.com>>
Reply-To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Date: Monday, October 7, 2013 5:13 PM
To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Subject: Execution failed with exit status: 3
Hi all,
I'm doing a very basic join in Hive and getting the error below. The HiveQL join syntax I'm using is:
SELECT
T1.somecolumn,
T2.someothercolumn
FROM t1
JOIN t2
ON (t1.idfield=t2.idfield)
Driver returned: 3. Errors: OK
Total MapReduce jobs = 1
setting HADOOP_USER_NAME someuser
Execution failed with exit status: 3
Obtaining error information
Task failed!
Task ID:
Stage-4
Logs:
FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.MapredLocalTask
I searched through the logs and couldn't find anything terribly useful, although perhaps I'm missing something. Is this a common error I'm just now coming across?
On Hive 0.11
Thanks!
Nick
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.