You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Mafish (JIRA)" <ji...@apache.org> on 2010/02/26 08:46:27 UTC
[jira] Created: (HIVE-1202) "Unknown exception : null" while join
"Unknown exception : null" while join
-------------------------------------
Key: HIVE-1202
URL: https://issues.apache.org/jira/browse/HIVE-1202
Project: Hadoop Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.4.1
Environment: hive-0.4.1
hadoop 0.19.1
Reporter: Mafish
Fix For: 0.4.1
Hive throws "Unknown exception : null" with query:
select * from
(
select name from classes
) a
join classes b
where a.name > b.number
After tracing the code, I found this bug will occur with following
conditions:
1. It is join operation.
2. At least one of the source of join is physical table (right side in
above case).
3. With where condition and condition(s) of where clause must include
columns from both side of join (a.name and b.number in case)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1202) "Unknown exception : null" while join
Posted by "Mafish (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12839606#action_12839606 ]
Mafish commented on HIVE-1202:
------------------------------
@Yongqiang
We have a internal hive originate from hive-0.4.2rc2. So we are not able to using the latest hive release.
This bug afftects 0.4.x only since hive-0.5.x uses a different pruning strategy.
I'm wondering if hive-0.4.x still need bug fixing since hive-0.5.x has been released?
> "Unknown exception : null" while join
> -------------------------------------
>
> Key: HIVE-1202
> URL: https://issues.apache.org/jira/browse/HIVE-1202
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.4.1
> Environment: hive-0.4.1
> hadoop 0.19.1
> Reporter: Mafish
> Fix For: 0.4.1
>
> Attachments: HIVE-1202.branch-0.4.1.patch
>
>
> Hive throws "Unknown exception : null" with query:
> select * from
> (
> select name from classes
> ) a
> join classes b
> where a.name > b.number
> After tracing the code, I found this bug will occur with following
> conditions:
> 1. It is join operation.
> 2. At least one of the source of join is physical table (right side in
> above case).
> 3. With where condition and condition(s) of where clause must include
> columns from both side of join (a.name and b.number in case)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1202) "Unknown exception : null" while join
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach updated HIVE-1202:
---------------------------------
Fix Version/s: (was: 0.4.1)
Description:
Hive throws "Unknown exception : null" with query:
select * from
(
select name from classes
) a
join classes b
where a.name > b.number
After tracing the code, I found this bug will occur with following
conditions:
1. It is join operation.
2. At least one of the source of join is physical table (right side in
above case).
3. With where condition and condition(s) of where clause must include
columns from both side of join (a.name and b.number in case)
was:
Hive throws "Unknown exception : null" with query:
select * from
(
select name from classes
) a
join classes b
where a.name > b.number
After tracing the code, I found this bug will occur with following
conditions:
1. It is join operation.
2. At least one of the source of join is physical table (right side in
above case).
3. With where condition and condition(s) of where clause must include
columns from both side of join (a.name and b.number in case)
> "Unknown exception : null" while join
> -------------------------------------
>
> Key: HIVE-1202
> URL: https://issues.apache.org/jira/browse/HIVE-1202
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.4.1
> Environment: hive-0.4.1
> hadoop 0.19.1
> Reporter: Mafish
> Attachments: HIVE-1202.branch-0.4.1.patch
>
>
> Hive throws "Unknown exception : null" with query:
> select * from
> (
> select name from classes
> ) a
> join classes b
> where a.name > b.number
> After tracing the code, I found this bug will occur with following
> conditions:
> 1. It is join operation.
> 2. At least one of the source of join is physical table (right side in
> above case).
> 3. With where condition and condition(s) of where clause must include
> columns from both side of join (a.name and b.number in case)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1202) "Unknown exception : null" while join
Posted by "He Yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12839555#action_12839555 ]
He Yongqiang commented on HIVE-1202:
------------------------------------
Please try
http://svn.apache.org/repos/asf/hadoop/hive/trunk
or you can download the latest stable 0.5 version from
http://hadoop.apache.org/hive/releases.html#Download
> "Unknown exception : null" while join
> -------------------------------------
>
> Key: HIVE-1202
> URL: https://issues.apache.org/jira/browse/HIVE-1202
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.4.1
> Environment: hive-0.4.1
> hadoop 0.19.1
> Reporter: Mafish
> Fix For: 0.4.1
>
> Attachments: HIVE-1202.branch-0.4.1.patch
>
>
> Hive throws "Unknown exception : null" with query:
> select * from
> (
> select name from classes
> ) a
> join classes b
> where a.name > b.number
> After tracing the code, I found this bug will occur with following
> conditions:
> 1. It is join operation.
> 2. At least one of the source of join is physical table (right side in
> above case).
> 3. With where condition and condition(s) of where clause must include
> columns from both side of join (a.name and b.number in case)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1202) "Unknown exception : null" while join
Posted by "He Yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12839551#action_12839551 ]
He Yongqiang commented on HIVE-1202:
------------------------------------
I tried this query with the trunk code. it works fine.
hive> select a.name, b.* from (select name from classes) a join classes b on a.name = b.number where a.name>b.number;
Total MapReduce jobs = 1
> "Unknown exception : null" while join
> -------------------------------------
>
> Key: HIVE-1202
> URL: https://issues.apache.org/jira/browse/HIVE-1202
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.4.1
> Environment: hive-0.4.1
> hadoop 0.19.1
> Reporter: Mafish
> Fix For: 0.4.1
>
> Attachments: HIVE-1202.branch-0.4.1.patch
>
>
> Hive throws "Unknown exception : null" with query:
> select * from
> (
> select name from classes
> ) a
> join classes b
> where a.name > b.number
> After tracing the code, I found this bug will occur with following
> conditions:
> 1. It is join operation.
> 2. At least one of the source of join is physical table (right side in
> above case).
> 3. With where condition and condition(s) of where clause must include
> columns from both side of join (a.name and b.number in case)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1202) "Unknown exception : null" while join
Posted by "He Yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12838946#action_12838946 ]
He Yongqiang commented on HIVE-1202:
------------------------------------
Actually hive does not support this kind of join. It only support equal join.
please try sth like this:
select a.name, b.* from classes a join classes b on a.name = b.number where a.name > b.number
> "Unknown exception : null" while join
> -------------------------------------
>
> Key: HIVE-1202
> URL: https://issues.apache.org/jira/browse/HIVE-1202
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.4.1
> Environment: hive-0.4.1
> hadoop 0.19.1
> Reporter: Mafish
> Fix For: 0.4.1
>
> Attachments: HIVE-1202.branch-0.4.1.patch
>
>
> Hive throws "Unknown exception : null" with query:
> select * from
> (
> select name from classes
> ) a
> join classes b
> where a.name > b.number
> After tracing the code, I found this bug will occur with following
> conditions:
> 1. It is join operation.
> 2. At least one of the source of join is physical table (right side in
> above case).
> 3. With where condition and condition(s) of where clause must include
> columns from both side of join (a.name and b.number in case)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1202) "Unknown exception : null" while join
Posted by "Mafish (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12839959#action_12839959 ]
Mafish commented on HIVE-1202:
------------------------------
@Zheng
With my patch, hive will omit the outmost where clauses while joining.
So it doesn't do partition pruning for this case, Which may be a problem in this situation.
> "Unknown exception : null" while join
> -------------------------------------
>
> Key: HIVE-1202
> URL: https://issues.apache.org/jira/browse/HIVE-1202
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.4.1
> Environment: hive-0.4.1
> hadoop 0.19.1
> Reporter: Mafish
> Fix For: 0.4.1
>
> Attachments: HIVE-1202.branch-0.4.1.patch
>
>
> Hive throws "Unknown exception : null" with query:
> select * from
> (
> select name from classes
> ) a
> join classes b
> where a.name > b.number
> After tracing the code, I found this bug will occur with following
> conditions:
> 1. It is join operation.
> 2. At least one of the source of join is physical table (right side in
> above case).
> 3. With where condition and condition(s) of where clause must include
> columns from both side of join (a.name and b.number in case)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1202) "Unknown exception : null" while join
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12839940#action_12839940 ]
Zheng Shao commented on HIVE-1202:
----------------------------------
{code}
select * from
(
select name from classes
) a
join classes b
where a.date_partition = '2010-02-01' AND b.date_partition = '2010-03-01';
{code}
It seems with the patch, we won't do partition pruning for this case?
Is that a problem?
> "Unknown exception : null" while join
> -------------------------------------
>
> Key: HIVE-1202
> URL: https://issues.apache.org/jira/browse/HIVE-1202
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.4.1
> Environment: hive-0.4.1
> hadoop 0.19.1
> Reporter: Mafish
> Fix For: 0.4.1
>
> Attachments: HIVE-1202.branch-0.4.1.patch
>
>
> Hive throws "Unknown exception : null" with query:
> select * from
> (
> select name from classes
> ) a
> join classes b
> where a.name > b.number
> After tracing the code, I found this bug will occur with following
> conditions:
> 1. It is join operation.
> 2. At least one of the source of join is physical table (right side in
> above case).
> 3. With where condition and condition(s) of where clause must include
> columns from both side of join (a.name and b.number in case)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1202) "Unknown exception : null" while join
Posted by "Mafish (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mafish updated HIVE-1202:
-------------------------
Attachment: HIVE-1202.branch-0.4.1.patch
Attachment is the patch for this BUG.
It limits hive to perform pruning action only when the current query block contains only on table.
This is fixed according to my understanding and I'm not sure it is the original idea of author. Author of the pruner is Yongqiang, right?
Please comment and evaluate it.
> "Unknown exception : null" while join
> -------------------------------------
>
> Key: HIVE-1202
> URL: https://issues.apache.org/jira/browse/HIVE-1202
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.4.1
> Environment: hive-0.4.1
> hadoop 0.19.1
> Reporter: Mafish
> Fix For: 0.4.1
>
> Attachments: HIVE-1202.branch-0.4.1.patch
>
>
> Hive throws "Unknown exception : null" with query:
> select * from
> (
> select name from classes
> ) a
> join classes b
> where a.name > b.number
> After tracing the code, I found this bug will occur with following
> conditions:
> 1. It is join operation.
> 2. At least one of the source of join is physical table (right side in
> above case).
> 3. With where condition and condition(s) of where clause must include
> columns from both side of join (a.name and b.number in case)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1202) "Unknown exception : null" while join
Posted by "Mafish (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12839547#action_12839547 ]
Mafish commented on HIVE-1202:
------------------------------
Error message in hive is as title: Unknown exception : null.
And the call stack is the same as my first comment.
> "Unknown exception : null" while join
> -------------------------------------
>
> Key: HIVE-1202
> URL: https://issues.apache.org/jira/browse/HIVE-1202
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.4.1
> Environment: hive-0.4.1
> hadoop 0.19.1
> Reporter: Mafish
> Fix For: 0.4.1
>
> Attachments: HIVE-1202.branch-0.4.1.patch
>
>
> Hive throws "Unknown exception : null" with query:
> select * from
> (
> select name from classes
> ) a
> join classes b
> where a.name > b.number
> After tracing the code, I found this bug will occur with following
> conditions:
> 1. It is join operation.
> 2. At least one of the source of join is physical table (right side in
> above case).
> 3. With where condition and condition(s) of where clause must include
> columns from both side of join (a.name and b.number in case)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1202) "Unknown exception : null" while join
Posted by "Mafish (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12839515#action_12839515 ]
Mafish commented on HIVE-1202:
------------------------------
@Yongqaing
I ran the query:
select a.name, b.* from classes a join classes b on a.name = b.number where a.name > b.number
It passed.
In this case, two tables are physical.
But when I changed one of them to sub-query, error occured again, as:
select a.name, b.* from (select name from classes) a join classes b on a.name = b.number where a.name > b.number ;
Please try this case.
> "Unknown exception : null" while join
> -------------------------------------
>
> Key: HIVE-1202
> URL: https://issues.apache.org/jira/browse/HIVE-1202
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.4.1
> Environment: hive-0.4.1
> hadoop 0.19.1
> Reporter: Mafish
> Fix For: 0.4.1
>
> Attachments: HIVE-1202.branch-0.4.1.patch
>
>
> Hive throws "Unknown exception : null" with query:
> select * from
> (
> select name from classes
> ) a
> join classes b
> where a.name > b.number
> After tracing the code, I found this bug will occur with following
> conditions:
> 1. It is join operation.
> 2. At least one of the source of join is physical table (right side in
> above case).
> 3. With where condition and condition(s) of where clause must include
> columns from both side of join (a.name and b.number in case)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1202) "Unknown exception : null" while join
Posted by "Mafish (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12838743#action_12838743 ]
Mafish commented on HIVE-1202:
------------------------------
The call stack is:
org.apache.hadoop.hive.ql.session.SessionState$LogHelper.printError(SessionState.java:279) - FAILED: Unknown exception : null
java.lang.NullPointerException
at org.apache.hadoop.hive.ql.parse.QBMetaData.getTableForAlias(QBMetaData.java:76)
at org.apache.hadoop.hive.ql.parse.ASTPartitionPruner.getTableColumnDesc(ASTPartitionPruner.java:298)
at org.apache.hadoop.hive.ql.parse.ASTPartitionPruner.genExprNodeDesc(ASTPartitionPruner.java:220)
at org.apache.hadoop.hive.ql.parse.ASTPartitionPruner.genExprNodeDesc(ASTPartitionPruner.java:234)
at org.apache.hadoop.hive.ql.parse.ASTPartitionPruner.genExprNodeDesc(ASTPartitionPruner.java:234)
at org.apache.hadoop.hive.ql.parse.ASTPartitionPruner.addExpression(ASTPartitionPruner.java:397)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPartitionPruners(SemanticAnalyzer.java:624)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:4440)
at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:76)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:249)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:281)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:123)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:181)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:287)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:165)
at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
This bug occurs while hive tries to prune table b, but it takes columns in where clauses. Bug there also exists columns of table a. Thus, hive fails to find column "name" in table b.
> "Unknown exception : null" while join
> -------------------------------------
>
> Key: HIVE-1202
> URL: https://issues.apache.org/jira/browse/HIVE-1202
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.4.1
> Environment: hive-0.4.1
> hadoop 0.19.1
> Reporter: Mafish
> Fix For: 0.4.1
>
>
> Hive throws "Unknown exception : null" with query:
> select * from
> (
> select name from classes
> ) a
> join classes b
> where a.name > b.number
> After tracing the code, I found this bug will occur with following
> conditions:
> 1. It is join operation.
> 2. At least one of the source of join is physical table (right side in
> above case).
> 3. With where condition and condition(s) of where clause must include
> columns from both side of join (a.name and b.number in case)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1202) "Unknown exception : null" while join
Posted by "He Yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12839546#action_12839546 ]
He Yongqiang commented on HIVE-1202:
------------------------------------
{quote}
But when I changed one of them to sub-query, error occured again, as:
select a.name, b.* from (select name from classes) a join classes b on a.name = b.number where a.name > b.number ;
{quote}
what's the error for this query?
> "Unknown exception : null" while join
> -------------------------------------
>
> Key: HIVE-1202
> URL: https://issues.apache.org/jira/browse/HIVE-1202
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.4.1
> Environment: hive-0.4.1
> hadoop 0.19.1
> Reporter: Mafish
> Fix For: 0.4.1
>
> Attachments: HIVE-1202.branch-0.4.1.patch
>
>
> Hive throws "Unknown exception : null" with query:
> select * from
> (
> select name from classes
> ) a
> join classes b
> where a.name > b.number
> After tracing the code, I found this bug will occur with following
> conditions:
> 1. It is join operation.
> 2. At least one of the source of join is physical table (right side in
> above case).
> 3. With where condition and condition(s) of where clause must include
> columns from both side of join (a.name and b.number in case)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1202) "Unknown exception : null" while join
Posted by "Mafish (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12839552#action_12839552 ]
Mafish commented on HIVE-1202:
------------------------------
Which trunk are you using?
I'm using release 0.4.1, which is checkoed out from
http://svn.apache.org/repos/asf/hadoop/hive/branches/branch-0.4
$ svn info
Path: .
URL: http://svn.apache.org/repos/asf/hadoop/hive/branches/branch-0.4
Repository Root: http://svn.apache.org/repos/asf
Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
Revision: 916543
Node Kind: directory
Schedule: normal
Last Changed Author: nzhang
Last Changed Rev: 912061
Last Changed Date: 2010-02-20 09:44:44 +0800 (Sat, 20 Feb 2010)
> "Unknown exception : null" while join
> -------------------------------------
>
> Key: HIVE-1202
> URL: https://issues.apache.org/jira/browse/HIVE-1202
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.4.1
> Environment: hive-0.4.1
> hadoop 0.19.1
> Reporter: Mafish
> Fix For: 0.4.1
>
> Attachments: HIVE-1202.branch-0.4.1.patch
>
>
> Hive throws "Unknown exception : null" with query:
> select * from
> (
> select name from classes
> ) a
> join classes b
> where a.name > b.number
> After tracing the code, I found this bug will occur with following
> conditions:
> 1. It is join operation.
> 2. At least one of the source of join is physical table (right side in
> above case).
> 3. With where condition and condition(s) of where clause must include
> columns from both side of join (a.name and b.number in case)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.