You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Phabricator (Updated) (JIRA)" <ji...@apache.org> on 2012/02/24 08:26:51 UTC
[jira] [Updated] (HIVE-2820) Invalid tag is used for
MapJoinProcessor
[ https://issues.apache.org/jira/browse/HIVE-2820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HIVE-2820:
------------------------------
Attachment: HIVE-2820.D1935.1.patch
navis requested code review of "HIVE-2820 [jira] Invalid tag is used for MapJoinProcessor".
Reviewers: JIRA
DPAL-889 Invalid tag is used for MapJoinProcessor
Testing HIVE-2810, I've found tag and alias are used in very confusing manner. For example, query below fails..
hive> set hive.auto.convert.join=true;
hive> select /*+ STREAMTABLE(a) */ * from myinput1 a join myinput1 b on a.key=b.key join myinput1 c on a.key=c.key;
Total MapReduce jobs = 4
Ended Job = 1667415037, job is filtered out (removed at runtime).
Ended Job = 1739566906, job is filtered out (removed at runtime).
Ended Job = 1113337780, job is filtered out (removed at runtime).
12/02/24 10:27:14 WARN conf.HiveConf: DEPRECATED: Ignoring hive-default.xml found on the CLASSPATH at /home/navis/hive/conf/hive-default.xml
Execution log at: /tmp/navis/navis_20120224102727_cafe0d8d-9b21-441d-bd4e-b83303b31cdc.log
2012-02-24 10:27:14 Starting to launch local task to process map join; maximum memory = 932118528
java.lang.NullPointerException
at org.apache.hadoop.hive.ql.exec.HashTableSinkOperator.processOp(HashTableSinkOperator.java:312)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:83)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
at org.apache.hadoop.hive.ql.exec.MapredLocalTask.startForward(MapredLocalTask.java:325)
at org.apache.hadoop.hive.ql.exec.MapredLocalTask.executeFromChildJVM(MapredLocalTask.java:272)
at org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:685)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
Execution failed with exit status: 2
Obtaining error information
Failed task has a plan which doesn't make sense.
Stage: Stage-8
Map Reduce Local Work
Alias -> Map Local Tables:
b
Fetch Operator
limit: -1
c
Fetch Operator
limit: -1
Alias -> Map Local Operator Tree:
b
TableScan
alias: b
HashTable Sink Operator
condition expressions:
0 {key} {value}
1 {key} {value}
2 {key} {value}
handleSkewJoin: false
keys:
0 [Column[key]]
1 [Column[key]]
2 [Column[key]]
Position of Big Table: 0
c
TableScan
alias: c
Map Join Operator
condition map:
Inner Join 0 to 1
Inner Join 0 to 2
condition expressions:
0 {key} {value}
1 {key} {value}
2 {key} {value}
handleSkewJoin: false
keys:
0 [Column[key]]
1 [Column[key]]
2 [Column[key]]
outputColumnNames: _col0, _col1, _col4, _col5, _col8, _col9
Position of Big Table: 0
Select Operator
expressions:
expr: _col0
type: int
expr: _col1
type: int
expr: _col4
type: int
expr: _col5
type: int
expr: _col8
type: int
expr: _col9
type: int
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
File Output Operator
compressed: false
GlobalTableId: 0
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
Stage: Stage-4
Map Reduce
Alias -> Map Operator Tree:
a
TableScan
alias: a
HashTable Sink Operator
condition expressions:
0 {key} {value}
1 {key} {value}
2 {key} {value}
handleSkewJoin: false
keys:
0 [Column[key]]
1 [Column[key]]
2 [Column[key]]
Position of Big Table: 0
Local Work:
Map Reduce Local Work
TEST PLAN
EMPTY
REVISION DETAIL
https://reviews.facebook.net/D1935
AFFECTED FILES
ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractMapJoinOperator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java
ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/CommonJoinResolver.java
ql/src/java/org/apache/hadoop/hive/ql/plan/HashTableSinkDesc.java
ql/src/java/org/apache/hadoop/hive/ql/plan/JoinDesc.java
ql/src/test/queries/clientpositive/join_reorder2.q
ql/src/test/results/clientpositive/join_reorder2.q.out
MANAGE HERALD DIFFERENTIAL RULES
https://reviews.facebook.net/herald/view/differential/
WHY DID I GET THIS EMAIL?
https://reviews.facebook.net/herald/transcript/4101/
Tip: use the X-Herald-Rules header to filter Herald messages in your client.
> Invalid tag is used for MapJoinProcessor
> ----------------------------------------
>
> Key: HIVE-2820
> URL: https://issues.apache.org/jira/browse/HIVE-2820
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.9.0
> Environment: ubuntu
> Reporter: Navis
> Assignee: Navis
> Fix For: 0.9.0
>
> Attachments: HIVE-2820.D1935.1.patch
>
>
> Testing HIVE-2810, I've found tag and alias are used in very confusing manner. For example, query below fails..
> {code}
> hive> set hive.auto.convert.join=true;
> hive> select /*+ STREAMTABLE(a) */ * from myinput1 a join myinput1 b on a.key=b.key join myinput1 c on a.key=c.key;
> Total MapReduce jobs = 4
> Ended Job = 1667415037, job is filtered out (removed at runtime).
> Ended Job = 1739566906, job is filtered out (removed at runtime).
> Ended Job = 1113337780, job is filtered out (removed at runtime).
> 12/02/24 10:27:14 WARN conf.HiveConf: DEPRECATED: Ignoring hive-default.xml found on the CLASSPATH at /home/navis/hive/conf/hive-default.xml
> Execution log at: /tmp/navis/navis_20120224102727_cafe0d8d-9b21-441d-bd4e-b83303b31cdc.log
> 2012-02-24 10:27:14 Starting to launch local task to process map join; maximum memory = 932118528
> java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.exec.HashTableSinkOperator.processOp(HashTableSinkOperator.java:312)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762)
> at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:83)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
> at org.apache.hadoop.hive.ql.exec.MapredLocalTask.startForward(MapredLocalTask.java:325)
> at org.apache.hadoop.hive.ql.exec.MapredLocalTask.executeFromChildJVM(MapredLocalTask.java:272)
> at org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:685)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
> Execution failed with exit status: 2
> Obtaining error information
> {code}
> Failed task has a plan which doesn't make sense.
> {noformat}
> Stage: Stage-8
> Map Reduce Local Work
> Alias -> Map Local Tables:
> b
> Fetch Operator
> limit: -1
> c
> Fetch Operator
> limit: -1
> Alias -> Map Local Operator Tree:
> b
> TableScan
> alias: b
> HashTable Sink Operator
> condition expressions:
> 0 {key} {value}
> 1 {key} {value}
> 2 {key} {value}
> handleSkewJoin: false
> keys:
> 0 [Column[key]]
> 1 [Column[key]]
> 2 [Column[key]]
> Position of Big Table: 0
> c
> TableScan
> alias: c
> Map Join Operator
> condition map:
> Inner Join 0 to 1
> Inner Join 0 to 2
> condition expressions:
> 0 {key} {value}
> 1 {key} {value}
> 2 {key} {value}
> handleSkewJoin: false
> keys:
> 0 [Column[key]]
> 1 [Column[key]]
> 2 [Column[key]]
> outputColumnNames: _col0, _col1, _col4, _col5, _col8, _col9
> Position of Big Table: 0
> Select Operator
> expressions:
> expr: _col0
> type: int
> expr: _col1
> type: int
> expr: _col4
> type: int
> expr: _col5
> type: int
> expr: _col8
> type: int
> expr: _col9
> type: int
> outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
> File Output Operator
> compressed: false
> GlobalTableId: 0
> table:
> input format: org.apache.hadoop.mapred.TextInputFormat
> output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> Stage: Stage-4
> Map Reduce
> Alias -> Map Operator Tree:
> a
> TableScan
> alias: a
> HashTable Sink Operator
> condition expressions:
> 0 {key} {value}
> 1 {key} {value}
> 2 {key} {value}
> handleSkewJoin: false
> keys:
> 0 [Column[key]]
> 1 [Column[key]]
> 2 [Column[key]]
> Position of Big Table: 0
> Local Work:
> Map Reduce Local Work
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira