You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org> on 2010/10/01 07:41:33 UTC
[jira] Created: (HIVE-1678) NPE in MapJoin
NPE in MapJoin
---------------
Key: HIVE-1678
URL: https://issues.apache.org/jira/browse/HIVE-1678
Project: Hadoop Hive
Issue Type: Bug
Components: Query Processor
Reporter: Amareshwari Sriramadasu
The query with two map joins and a group by fails with following NPE:
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:177)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:457)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:697)
at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:457)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:697)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:464)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1678) NPE in MapJoin
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HIVE-1678:
------------------------------------------
Attachment: patch-1678.txt
The bug is in plan generation when MapJoin is followed MapJoin, and is followed by ReduceSink. ReduceSink operator reads the input from oldMapJoin instead of current MapJoin.
Attached patch has one line fix in GenMapRedUtils.initMapJoinPlan to fix the bug. Also includes the testcase.
> NPE in MapJoin
> ---------------
>
> Key: HIVE-1678
> URL: https://issues.apache.org/jira/browse/HIVE-1678
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Amareshwari Sriramadasu
> Assignee: Amareshwari Sriramadasu
> Attachments: patch-1678.txt
>
>
> The query with two map joins and a group by fails with following NPE:
> Caused by: java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:177)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:457)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:697)
> at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:457)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:697)
> at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:464)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1678) NPE in MapJoin
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12916789#action_12916789 ]
Amareshwari Sriramadasu commented on HIVE-1678:
-----------------------------------------------
The following query :
select /*+MAPJOIN(src, myinput1) */ count(srcpart.key) from srcpart join src on (srcpart.value=src.value) join myinput1 on (srcpart.key=myinput1.key);
launches three MapReduce jobs in which third one fails with NPE.
The task logs :
{noformat}
2010-10-01 11:07:02,324 INFO ExecMapper:
<MAP>Id =11
<Children>
<SEL>Id =0
<Children>
<MAPJOIN>Id =1
<Children>
<FS>Id =2
<Parent>Id = 1 null<\Parent>
<\FS>
<\Children>
<Parent>Id = 0 nullId = 3
<TS>Id =3
<Children>null
<\Children>
<\TS><\Parent>
<\MAPJOIN>
<\Children>
<Parent>Id = 11 null<\Parent>
<\SEL>
<\Children>
<\MAP>
2010-10-01 11:07:02,324 INFO org.apache.hadoop.hive.ql.exec.MapOperator: Initializing Self 11 MAP
2010-10-01 11:07:02,324 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: Initializing Self 0 SEL
2010-10-01 11:07:02,327 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: SELECT struct<_col0:int>
2010-10-01 11:07:02,332 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: Operator 0 SEL initialized
2010-10-01 11:07:02,332 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: Initializing children of 0 SEL
2010-10-01 11:07:02,332 INFO org.apache.hadoop.hive.ql.exec.MapJoinOperator: Initializing child 1 MAPJOIN
2010-10-01 11:07:02,332 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: Initialization Done 0 SEL
2010-10-01 11:07:02,332 INFO org.apache.hadoop.hive.ql.exec.MapOperator: Initialization Done 11 MAP
2010-10-01 11:07:02,334 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 11 forwarding 1 rows
2010-10-01 11:07:02,334 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 0 forwarding 1 rows
2010-10-01 11:07:02,338 FATAL ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"_col0":238}
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:483)
at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:180)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:395)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:329)
at org.apache.hadoop.mapred.Child$4.run(Child.java:219)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1021)
at org.apache.hadoop.mapred.Child.main(Child.java:213)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:177)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:459)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:710)
at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:459)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:710)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:464)
... 9 more
{noformat}
>From the logs we can see that MapJoinOperator is not intialized at all.
> NPE in MapJoin
> ---------------
>
> Key: HIVE-1678
> URL: https://issues.apache.org/jira/browse/HIVE-1678
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Amareshwari Sriramadasu
>
> The query with two map joins and a group by fails with following NPE:
> Caused by: java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:177)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:457)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:697)
> at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:457)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:697)
> at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:464)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1678) NPE in MapJoin
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HIVE-1678:
------------------------------------------
Status: Patch Available (was: Open)
> NPE in MapJoin
> ---------------
>
> Key: HIVE-1678
> URL: https://issues.apache.org/jira/browse/HIVE-1678
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Amareshwari Sriramadasu
> Assignee: Amareshwari Sriramadasu
> Attachments: patch-1678.txt
>
>
> The query with two map joins and a group by fails with following NPE:
> Caused by: java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:177)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:457)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:697)
> at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:457)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:697)
> at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:464)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1678) NPE in MapJoin
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-1678:
-----------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Committed. Thanks Amareshwari
> NPE in MapJoin
> ---------------
>
> Key: HIVE-1678
> URL: https://issues.apache.org/jira/browse/HIVE-1678
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Amareshwari Sriramadasu
> Assignee: Amareshwari Sriramadasu
> Attachments: patch-1678.txt
>
>
> The query with two map joins and a group by fails with following NPE:
> Caused by: java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:177)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:457)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:697)
> at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:457)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:697)
> at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:464)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HIVE-1678) NPE in MapJoin
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu reassigned HIVE-1678:
---------------------------------------------
Assignee: Amareshwari Sriramadasu
> NPE in MapJoin
> ---------------
>
> Key: HIVE-1678
> URL: https://issues.apache.org/jira/browse/HIVE-1678
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Amareshwari Sriramadasu
> Assignee: Amareshwari Sriramadasu
>
> The query with two map joins and a group by fails with following NPE:
> Caused by: java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:177)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:457)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:697)
> at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:457)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:697)
> at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:464)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1678) NPE in MapJoin
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12916794#action_12916794 ]
Amareshwari Sriramadasu commented on HIVE-1678:
-----------------------------------------------
The same query succeeds when no MapJoin is used.
Looks like plan generation went wrong in MapJoinProcessor. explain output for the query:
{noformat}
explain
select /*+MAPJOIN(src, myinput1) */ count(srcpart.key) from srcpart join src on (srcpart.value=src.value) join myinput1 on (srcpart.key=myinput1.key);
OK
ABSTRACT SYNTAX TREE:
(TOK_QUERY (TOK_FROM (TOK_JOIN (TOK_JOIN (TOK_TABREF srcpart) (TOK_TABREF src) (= (. (TOK_TABLE_OR_COL srcpart) value) (. (TOK_TABLE_OR_COL src) value))) (TOK_TABREF myinput1) (= (. (TOK_TABLE_OR_COL srcpart) key) (. (TOK_TABLE_OR_COL myinput1) key)))) (TOK_INSERT (TOK_DESTINATION (TOK_DIR TOK_TMP_FILE)) (TOK_SELECT (TOK_HINTLIST (TOK_HINT TOK_MAPJOIN (TOK_HINTARGLIST src myinput1))) (TOK_SELEXPR (TOK_FUNCTION count (. (TOK_TABLE_OR_COL srcpart) key))))))
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-2 depends on stages: Stage-1
Stage-3 depends on stages: Stage-2
Stage-0 is a root stage
STAGE PLANS:
Stage: Stage-1
Map Reduce
Alias -> Map Operator Tree:
srcpart
TableScan
alias: srcpart
Common Join Operator
condition map:
Inner Join 0 to 1
condition expressions:
0 {key}
1
handleSkewJoin: false
keys:
0 [Column[value]]
1 [Column[value]]
outputColumnNames: _col0
Position of Big Table: 0
File Output Operator
compressed: false
GlobalTableId: 0
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
Local Work:
Map Reduce Local Work
Alias -> Map Local Tables:
src
Fetch Operator
limit: -1
Alias -> Map Local Operator Tree:
src
TableScan
alias: src
Common Join Operator
condition map:
Inner Join 0 to 1
condition expressions:
0 {key}
1
handleSkewJoin: false
keys:
0 [Column[value]]
1 [Column[value]]
outputColumnNames: _col0
Position of Big Table: 0
File Output Operator
compressed: false
GlobalTableId: 0
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
Stage: Stage-2
Map Reduce
Alias -> Map Operator Tree:
hdfs://localhost:19000/tmp/hive-amarsri/hive_2010-10-01_11-24-15_198_629889728835692043/-mr-10002
Select Operator
expressions:
expr: _col0
type: int
outputColumnNames: _col0
Common Join Operator
condition map:
Inner Join 0 to 1
condition expressions:
0 {_col0}
1
handleSkewJoin: false
keys:
0 [Column[_col0]]
1 [Column[key]]
outputColumnNames: _col0
Position of Big Table: 0
File Output Operator
compressed: false
GlobalTableId: 0
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
Local Work:
Map Reduce Local Work
Alias -> Map Local Tables:
myinput1
Fetch Operator
limit: -1
Alias -> Map Local Operator Tree:
myinput1
TableScan
alias: myinput1
Common Join Operator
condition map:
Inner Join 0 to 1
condition expressions:
0 {_col0}
1
handleSkewJoin: false
keys:
0 [Column[_col0]]
1 [Column[key]]
outputColumnNames: _col0
Position of Big Table: 0
File Output Operator
compressed: false
GlobalTableId: 0
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
Stage: Stage-3
Map Reduce
Alias -> Map Operator Tree:
hdfs://localhost:19000/tmp/hive-amarsri/hive_2010-10-01_11-24-15_198_629889728835692043/-mr-10002
Select Operator
expressions:
expr: _col0
type: int
outputColumnNames: _col0
Common Join Operator
condition map:
Inner Join 0 to 1
condition expressions:
0 {_col0}
1
handleSkewJoin: false
keys:
0 [Column[_col0]]
1 [Column[key]]
outputColumnNames: _col0
Position of Big Table: 0
File Output Operator
compressed: false
GlobalTableId: 0
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
Reduce Operator Tree:
Group By Operator
aggregations:
expr: count(VALUE._col0)
bucketGroup: false
mode: mergepartial
outputColumnNames: _col0
Select Operator
expressions:
expr: _col0
type: bigint
outputColumnNames: _col0
File Output Operator
compressed: false
GlobalTableId: 0
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
Stage: Stage-0
Fetch Operator
limit: -1
Time taken: 0.202 seconds
{noformat}
If I'm not wrong, Join operation should not be there in 3rd stage.
> NPE in MapJoin
> ---------------
>
> Key: HIVE-1678
> URL: https://issues.apache.org/jira/browse/HIVE-1678
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Amareshwari Sriramadasu
>
> The query with two map joins and a group by fails with following NPE:
> Caused by: java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:177)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:457)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:697)
> at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:457)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:697)
> at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:464)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1678) NPE in MapJoin
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12917684#action_12917684 ]
Namit Jain commented on HIVE-1678:
----------------------------------
Nice catch - Thanks
+1
will commit if the tests pass
> NPE in MapJoin
> ---------------
>
> Key: HIVE-1678
> URL: https://issues.apache.org/jira/browse/HIVE-1678
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Amareshwari Sriramadasu
> Assignee: Amareshwari Sriramadasu
> Attachments: patch-1678.txt
>
>
> The query with two map joins and a group by fails with following NPE:
> Caused by: java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:177)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:457)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:697)
> at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:457)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:697)
> at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:464)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.