You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Ryan LeCompte <le...@gmail.com> on 2009/10/12 01:01:55 UTC
Failures during map/reduce when running a hive query
Hello all,
My hive queries are returning back successfully, however if I did in the
map/reduce job that's running as a result of running the query, I see the
following errors in a lot of map tasks that get killed:
java.lang.RuntimeException: Map operator initialization failed
at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:110)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:328)
at org.apache.hadoop.mapred.Child.main(Child.java:155)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hive.ql.exec.GroupByOperator.initializeOp(GroupByOperator.java:207)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:316)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:353)
at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:338)
at org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:59)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:316)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:353)
at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:338)
at org.apache.hadoop.hive.ql.exec.Operator.initializeOp(Operator.java:324)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:316)
at org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:301)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:316)
at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:82)
... 7 more
The reducers also fail at times with similar exceptions/errors.
Is this expected, or do I have something misconfigured? I don't have the
hive distribution on all of my hadoop nodes, just the single node where I
run the queries. I am using 0.4-rc2.
Thanks,
Ryan
Re: Failures during map/reduce when running a hive query
Posted by Ryan LeCompte <le...@gmail.com>.
Hi Ashish,
I recently upgraded to Hadoop 0.20.1 and also am now feeding log data in
compressed gzip form to a hive sequence table, and I haven't seen the errors
any more.
Thanks!
Ryan
On Mon, Oct 12, 2009 at 4:44 PM, Ashish Thusoo <at...@facebook.com> wrote:
> This is not expected. Can you send the query and the explain plan for the
> query.
>
> Ashish
>
> ------------------------------
> *From:* Ryan LeCompte [mailto:lecompte@gmail.com]
> *Sent:* Sunday, October 11, 2009 4:02 PM
> *To:* hive-user@hadoop.apache.org
> *Subject:* Failures during map/reduce when running a hive query
>
> Hello all,
>
> My hive queries are returning back successfully, however if I did in the
> map/reduce job that's running as a result of running the query, I see the
> following errors in a lot of map tasks that get killed:
>
> java.lang.RuntimeException: Map operator initialization failed
> at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:110)
> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
>
> at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83)
> at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
>
> at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:328)
> at org.apache.hadoop.mapred.Child.main(Child.java:155)
> Caused by: java.lang.NullPointerException
>
> at org.apache.hadoop.hive.ql.exec.GroupByOperator.initializeOp(GroupByOperator.java:207)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:316)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:353)
>
> at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:338)
> at org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:59)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:316)
>
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:353)
> at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:338)
> at org.apache.hadoop.hive.ql.exec.Operator.initializeOp(Operator.java:324)
>
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:316)
> at org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:301)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:316)
>
> at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:82)
>
> ... 7 more
>
>
> The reducers also fail at times with similar exceptions/errors.
>
> Is this expected, or do I have something misconfigured? I don't have the
> hive distribution on all of my hadoop nodes, just the single node where I
> run the queries. I am using 0.4-rc2.
>
> Thanks,
> Ryan
>
>
RE: Failures during map/reduce when running a hive query
Posted by Ashish Thusoo <at...@facebook.com>.
This is not expected. Can you send the query and the explain plan for the query.
Ashish
________________________________
From: Ryan LeCompte [mailto:lecompte@gmail.com]
Sent: Sunday, October 11, 2009 4:02 PM
To: hive-user@hadoop.apache.org
Subject: Failures during map/reduce when running a hive query
Hello all,
My hive queries are returning back successfully, however if I did in the map/reduce job that's running as a result of running the query, I see the following errors in a lot of map tasks that get killed:
java.lang.RuntimeException: Map operator initialization failed
at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:110)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:328)
at org.apache.hadoop.mapred.Child.main(Child.java:155)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hive.ql.exec.GroupByOperator.initializeOp(GroupByOperator.java:207)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:316)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:353)
at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:338)
at org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:59)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:316)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:353)
at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:338)
at org.apache.hadoop.hive.ql.exec.Operator.initializeOp(Operator.java:324)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:316)
at org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:301)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:316)
at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:82)
... 7 more
The reducers also fail at times with similar exceptions/errors.
Is this expected, or do I have something misconfigured? I don't have the hive distribution on all of my hadoop nodes, just the single node where I run the queries. I am using 0.4-rc2.
Thanks,
Ryan