You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by xinyan Yang <mo...@gmail.com> on 2013/10/11 14:44:32 UTC
NPE org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable
Hi,
when i run this sql,it fails,can anyone give me a advise
--------------------------------------------------------------------------------------------------------------------------------
select e.udid as udid,e.app_id as app_id
from acorn_3g.ClientChannelDefine cc
join (
select udid,app_id,from_id
from (
select u.device_id as udid,u.app_id as app_id,g.device_id as
3gdid,u.from_id as from_id from acorn_3g.user_device_info u
left outer join (select device_id from acorn_3g.3g_device_id where
log_date<'2013-09-15') g
on u.device_id=g.device_id where u.log_date='2013-09-15' and u.from_id<>0
and u.type=1) f1
where 3gdid is null ) e
on(e.from_id=cc.from_id)
--------------------------------------------------------------------------------------------------------------------------------
error info:
Task with the most failures(4):
-----
Task ID:
task_201305281414_236693_m_000001
URL:
http://YZSJHL18-22.opi.com:50030/taskdetails.jsp?jobid=job_201305281414_236693&tipid=task_201305281414_236693_m_000001<http://yzsjhl18-22.opi.com:50030/taskdetails.jsp?jobid=job_201305281414_236693&tipid=task_201305281414_236693_m_000001>
-----
Diagnostic Messages for this Task:
java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:162)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:198)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:212)
at
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1377)
at
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1381)
at
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1381)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:611)
at
org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:144)
... 8 more
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:186)
... 14 more
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.MapRedTask
MapReduce Jobs Launched:
Job 0: Map: 343 Reduce: 2 Cumulative CPU: 3478.61 sec HDFS Read:
1862106687 HDFS Write: 3838425 SUCCESS
Job 1: Map: 2 HDFS Read: 0 HDFS Write: 0 FAIL
Re: NPE org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable
Posted by xinyan Yang <mo...@gmail.com>.
thank you,I explain sql success,try hive trunk also success,but I do not
know what patch fixed this problem
2013/10/11 Yin Huai <hu...@gmail.com>
> Hello Xinyang,
>
> Can you attach the query plan (the output of "EXPLAIN")? I think a bad
> plan caused the error.
>
> Also, can you try hive trunk? Looks like it is a bug fixed after the
> release of 0.11.
>
> Thanks,
>
> Yin
>
>
> On Fri, Oct 11, 2013 at 9:21 AM, xinyan Yang <mo...@gmail.com>wrote:
>
>> Development environment，hive 0.11、hadoop 1.0.3
>>
>>
>> 2013/10/11 xinyan Yang <mo...@gmail.com>
>>
>>> Hi,
>>> when i run this sql,it fails,can anyone give me a advise
>>>
>>> --------------------------------------------------------------------------------------------------------------------------------
>>> select e.udid as udid,e.app_id as app_id
>>> from acorn_3g.ClientChannelDefine cc
>>> join (
>>> select udid,app_id,from_id
>>> from (
>>> select u.device_id as udid,u.app_id as app_id,g.device_id as
>>> 3gdid,u.from_id as from_id from acorn_3g.user_device_info u
>>> left outer join (select device_id from acorn_3g.3g_device_id where
>>> log_date<'2013-09-15') g
>>> on u.device_id=g.device_id where u.log_date='2013-09-15' and
>>> u.from_id<>0 and u.type=1) f1
>>> where 3gdid is null ) e
>>> on(e.from_id=cc.from_id)
>>>
>>> --------------------------------------------------------------------------------------------------------------------------------
>>>
>>> error info:
>>> Task with the most failures(4):
>>> -----
>>> Task ID:
>>> task_201305281414_236693_m_000001
>>>
>>> URL:
>>>
>>> http://YZSJHL18-22.opi.com:50030/taskdetails.jsp?jobid=job_201305281414_236693&tipid=task_201305281414_236693_m_000001<http://yzsjhl18-22.opi.com:50030/taskdetails.jsp?jobid=job_201305281414_236693&tipid=task_201305281414_236693_m_000001>
>>> -----
>>> Diagnostic Messages for this Task:
>>> java.lang.RuntimeException:
>>> org.apache.hadoop.hive.ql.metadata.HiveException:
>>> java.lang.NullPointerException
>>> at
>>> org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:162)
>>> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
>>> at
>>> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
>>> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>> at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
>>> at org.apache.hadoop.mapred.Child.main(Child.java:249)
>>> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
>>> java.lang.NullPointerException
>>> at
>>> org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:198)
>>> at
>>> org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:212)
>>> at
>>> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1377)
>>> at
>>> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1381)
>>> at
>>> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1381)
>>> at
>>> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:611)
>>> at
>>> org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:144)
>>> ... 8 more
>>> Caused by: java.lang.NullPointerException
>>> at
>>> org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:186)
>>> ... 14 more
>>>
>>>
>>> FAILED: Execution Error, return code 2 from
>>> org.apache.hadoop.hive.ql.exec.MapRedTask
>>> MapReduce Jobs Launched:
>>> Job 0: Map: 343 Reduce: 2 Cumulative CPU: 3478.61 sec HDFS Read:
>>> 1862106687 HDFS Write: 3838425 SUCCESS
>>> Job 1: Map: 2 HDFS Read: 0 HDFS Write: 0 FAIL
>>>
>>>
>>
>
Re: NPE org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable
Posted by Yin Huai <hu...@gmail.com>.
Hello Xinyang,
Can you attach the query plan (the output of "EXPLAIN")? I think a bad plan
caused the error.
Also, can you try hive trunk? Looks like it is a bug fixed after the
release of 0.11.
Thanks,
Yin
On Fri, Oct 11, 2013 at 9:21 AM, xinyan Yang <mo...@gmail.com> wrote:
> Development environment，hive 0.11、hadoop 1.0.3
>
>
> 2013/10/11 xinyan Yang <mo...@gmail.com>
>
>> Hi,
>> when i run this sql,it fails,can anyone give me a advise
>>
>> --------------------------------------------------------------------------------------------------------------------------------
>> select e.udid as udid,e.app_id as app_id
>> from acorn_3g.ClientChannelDefine cc
>> join (
>> select udid,app_id,from_id
>> from (
>> select u.device_id as udid,u.app_id as app_id,g.device_id as
>> 3gdid,u.from_id as from_id from acorn_3g.user_device_info u
>> left outer join (select device_id from acorn_3g.3g_device_id where
>> log_date<'2013-09-15') g
>> on u.device_id=g.device_id where u.log_date='2013-09-15' and
>> u.from_id<>0 and u.type=1) f1
>> where 3gdid is null ) e
>> on(e.from_id=cc.from_id)
>>
>> --------------------------------------------------------------------------------------------------------------------------------
>>
>> error info:
>> Task with the most failures(4):
>> -----
>> Task ID:
>> task_201305281414_236693_m_000001
>>
>> URL:
>>
>> http://YZSJHL18-22.opi.com:50030/taskdetails.jsp?jobid=job_201305281414_236693&tipid=task_201305281414_236693_m_000001<http://yzsjhl18-22.opi.com:50030/taskdetails.jsp?jobid=job_201305281414_236693&tipid=task_201305281414_236693_m_000001>
>> -----
>> Diagnostic Messages for this Task:
>> java.lang.RuntimeException:
>> org.apache.hadoop.hive.ql.metadata.HiveException:
>> java.lang.NullPointerException
>> at
>> org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:162)
>> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
>> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
>> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
>> at org.apache.hadoop.mapred.Child.main(Child.java:249)
>> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
>> java.lang.NullPointerException
>> at
>> org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:198)
>> at
>> org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:212)
>> at
>> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1377)
>> at
>> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1381)
>> at
>> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1381)
>> at
>> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:611)
>> at
>> org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:144)
>> ... 8 more
>> Caused by: java.lang.NullPointerException
>> at
>> org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:186)
>> ... 14 more
>>
>>
>> FAILED: Execution Error, return code 2 from
>> org.apache.hadoop.hive.ql.exec.MapRedTask
>> MapReduce Jobs Launched:
>> Job 0: Map: 343 Reduce: 2 Cumulative CPU: 3478.61 sec HDFS Read:
>> 1862106687 HDFS Write: 3838425 SUCCESS
>> Job 1: Map: 2 HDFS Read: 0 HDFS Write: 0 FAIL
>>
>>
>
Re: NPE org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable
Posted by xinyan Yang <mo...@gmail.com>.
Development environment，hive 0.11、hadoop 1.0.3
2013/10/11 xinyan Yang <mo...@gmail.com>
> Hi,
> when i run this sql,it fails,can anyone give me a advise
>
> --------------------------------------------------------------------------------------------------------------------------------
> select e.udid as udid,e.app_id as app_id
> from acorn_3g.ClientChannelDefine cc
> join (
> select udid,app_id,from_id
> from (
> select u.device_id as udid,u.app_id as app_id,g.device_id as
> 3gdid,u.from_id as from_id from acorn_3g.user_device_info u
> left outer join (select device_id from acorn_3g.3g_device_id where
> log_date<'2013-09-15') g
> on u.device_id=g.device_id where u.log_date='2013-09-15' and
> u.from_id<>0 and u.type=1) f1
> where 3gdid is null ) e
> on(e.from_id=cc.from_id)
>
> --------------------------------------------------------------------------------------------------------------------------------
>
> error info:
> Task with the most failures(4):
> -----
> Task ID:
> task_201305281414_236693_m_000001
>
> URL:
>
> http://YZSJHL18-22.opi.com:50030/taskdetails.jsp?jobid=job_201305281414_236693&tipid=task_201305281414_236693_m_000001<http://yzsjhl18-22.opi.com:50030/taskdetails.jsp?jobid=job_201305281414_236693&tipid=task_201305281414_236693_m_000001>
> -----
> Diagnostic Messages for this Task:
> java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:162)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:198)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:212)
> at
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1377)
> at
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1381)
> at
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1381)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:611)
> at
> org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:144)
> ... 8 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:186)
> ... 14 more
>
>
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.MapRedTask
> MapReduce Jobs Launched:
> Job 0: Map: 343 Reduce: 2 Cumulative CPU: 3478.61 sec HDFS Read:
> 1862106687 HDFS Write: 3838425 SUCCESS
> Job 1: Map: 2 HDFS Read: 0 HDFS Write: 0 FAIL
>
>