You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Vikram Dixit K (JIRA)" <ji...@apache.org> on 2013/06/09 09:06:20 UTC
[jira] [Updated] (HIVE-4650) Getting Execution Error, return code 2
from org.apache.hadoop.hive.ql.exec.MapRedTask on auto convert to MapJoin
after upgrade to Hive-0.11.0.x from hive-0.10.0.x
[ https://issues.apache.org/jira/browse/HIVE-4650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vikram Dixit K updated HIVE-4650:
---------------------------------
Attachment: join32_lessSize.q.out
Bruce: I meant the patch HIVE-4502.D10695.patch is the one that fixes the issue. HIVE-4502-1.patch is not going to be the patch for it as it does not fix this case. Those differences that you mention is because of the inherent version differences and will need to be resolved. I have attached the results from a test I ran to verify a case similar to yours.
> Getting Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask on auto convert to MapJoin after upgrade to Hive-0.11.0.x from hive-0.10.0.x
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-4650
> URL: https://issues.apache.org/jira/browse/HIVE-4650
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.11.0
> Environment: HortonWorks 1.3 distro on x86_64 Centos 6
> Reporter: Bruce Nelson
> Attachments: join32_lessSize.q.out
>
>
> working from a simple table in Hive
> hive> desc cmnt
> > ;
> OK
> x1 int None
> x2 int None
> x3 int None
> x4 int None
> y double None
> hive> select * from cmnt;
> OK
> 7 26 6 60 78.5
> 1 29 15 52 74.3
> 11 56 8 20 104.3
> 11 31 8 47 87.6
> 7 52 6 33 95.9
> 11 55 9 22 109.2
> 3 71 17 6 102.7
> 1 31 22 44 72.5
> 2 54 18 22 93.1
> 21 47 4 26 115.9
> 1 40 23 34 83.8
> 11 66 9 12 113.3
> 10 68 8 12 109.4
> A query that joins and transforms against this table :
> select * from (select VAL001 x1,VAL002 x2,VAL003 x3,VAL004 x4,VAL005 y from ( select /*+ mapjoin(v2) */ (VAL001- mu1) * 1/(sd1) VAL001,(VAL002- mu2) * 1/(sd2) VAL002,(VAL003- mu3) * 1/(sd3) VAL003,(VAL004- mu4) * 1/(sd4) VAL004,(VAL005- mu5) * 1/(sd5) VAL005 from ( select * from ( select x1 VAL001,x2 VAL002,x3 VAL003,x4 VAL004,y VAL005 from cmnt ) obj1_3 ) v3 join (select count(*) c, avg(VAL001) mu1,avg(VAL002) mu2,avg(VAL003) mu3,avg(VAL004) mu4,avg(VAL005) mu5, stddev_pop(VAL001) sd1,stddev_pop(VAL002) sd2,stddev_pop(VAL003) sd3,stddev_pop(VAL004) sd4,stddev_pop(VAL005) sd5 from ( select * from ( select x1 VAL001,x2 VAL002,x3 VAL003,x4 VAL004,y VAL005 from cmnt ) obj1_3 ) v1) v2 ) obj1_7) obj1_6 ;
> Generates during Stage-3 :
> setting HADOOP_USER_NAME test
> Execution log at: /tmp/test/.log
> 2013-06-03 12:40:55 Starting to launch local task to process map join; maximum memory = 1065484288
> 2013-06-03 12:40:56 Processing rows: 1 Hashtable size: 1 Memory usage: 7175528 rate: 0.007
> 2013-06-03 12:40:56 Dump the hashtable into file: file:/tmp/test/hive_2013-06-03_00-40-21_708_6820064283161196136/-local-10003/HashTable-Stage-3/MapJoin-mapfile00--.hashtable
> 2013-06-03 12:40:56 Upload 1 File to: file:/tmp/test/hive_2013-06-03_00-40-21_708_6820064283161196136/-local-10003/HashTable-Stage-3/MapJoin-mapfile00--.hashtable File size: 334
> 2013-06-03 12:40:56 End of local task; Time Taken: 0.726 sec.
> Execution completed successfully
> Mapred Local Task Succeeded . Convert the Join into MapJoin
> Mapred Local Task Succeeded . Convert the Join into MapJoin
> Launching Job 2 out of 2
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_201306022123_0045, Tracking URL = http://sun1vm3:50030/jobdetails.jsp?jobid=job_201306022123_0045
> Kill Command = /usr/lib/hadoop/libexec/../bin/hadoop job -kill job_201306022123_0045
> Hadoop job information for Stage-3: number of mappers: 1; number of reducers: 0
> 2013-06-03 00:41:05,895 Stage-3 map = 0%, reduce = 0%
> 2013-06-03 00:41:40,687 Stage-3 map = 100%, reduce = 100%
> Ended Job = job_201306022123_0045 with errors
> Error during job, obtaining debugging information...
> Job Tracking URL: http://sun1vm3:50030/jobdetails.jsp?jobid=job_201306022123_0045
> Examining task ID: task_201306022123_0045_m_000002 (and more) from job job_201306022123_0045
> Task with the most failures(4):
> -----
> Task ID:
> task_201306022123_0045_m_000000
> URL:
> http://sun1vm3:50030/taskdetails.jsp?jobid=job_201306022123_0045&tipid=task_201306022123_0045_m_000000
> -----
> Diagnostic Messages for this Task:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:162)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:365)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:198)
> at org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:212)
> at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1377)
> at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1381)
> at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1381)
> at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:611)
> at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:144)
> ... 8 more
> Caused by: java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:186)
> ... 14 more
> FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
> If "hive.auto.convert.join = false" is set then the all the query stages work OK. The same scenario worked OK in Hive-0.10.0.x and Hive-0.9.x with MapJoin working.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira