You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Gang Tim Liu (JIRA)" <ji...@apache.org> on 2012/08/08 02:03:09 UTC
[jira] [Created] (HIVE-3343) Hive: Query misaligned result for
Group by followed by Join with filter and skip a group-by result
Gang Tim Liu created HIVE-3343:
----------------------------------
Summary: Hive: Query misaligned result for Group by followed by Join with filter and skip a group-by result
Key: HIVE-3343
URL: https://issues.apache.org/jira/browse/HIVE-3343
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Gang Tim Liu
Assignee: Gang Tim Liu
This simple Hive query would generate wrong result:
select a.key, b.k2, b.k3
from src a
join (
select key,
min(key) as k,
min(key)+1 as k1,
min(key)+2 as k2,
min(key)+3 as k3
from src
group by key
) b
on a.key=b.key and b.k1 < 5;
0 3.0 1.0
0 3.0 1.0
0 3.0 1.0
2 5.0 3.0
The right result is:
0 2.0 3.0
0 2.0 3.0
0 2.0 3.0
2 4.0 5.0
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3343) Hive: Query misaligned result for
Group by followed by Join with filter and skip a group-by result
Posted by "Gang Tim Liu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gang Tim Liu updated HIVE-3343:
-------------------------------
Attachment: hive-3343.patch
> Hive: Query misaligned result for Group by followed by Join with filter and skip a group-by result
> --------------------------------------------------------------------------------------------------
>
> Key: HIVE-3343
> URL: https://issues.apache.org/jira/browse/HIVE-3343
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Gang Tim Liu
> Assignee: Gang Tim Liu
> Attachments: hive-3343.patch
>
>
> This simple Hive query would generate wrong result:
> select a.key, b.k2, b.k3
> from src a
> join (
> select key,
> min(key) as k,
> min(key)+1 as k1,
> min(key)+2 as k2,
> min(key)+3 as k3
> from src
> group by key
> ) b
> on a.key=b.key and b.k1 < 5;
> 0 3.0 1.0
> 0 3.0 1.0
> 0 3.0 1.0
> 2 5.0 3.0
> The right result is:
> 0 2.0 3.0
> 0 2.0 3.0
> 0 2.0 3.0
> 2 4.0 5.0
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3343) Hive: Query misaligned result for
Group by followed by Join with filter and skip a group-by result
Posted by "Gang Tim Liu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430776#comment-13430776 ]
Gang Tim Liu commented on HIVE-3343:
------------------------------------
https://reviews.facebook.net/D4563
> Hive: Query misaligned result for Group by followed by Join with filter and skip a group-by result
> --------------------------------------------------------------------------------------------------
>
> Key: HIVE-3343
> URL: https://issues.apache.org/jira/browse/HIVE-3343
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Gang Tim Liu
> Assignee: Gang Tim Liu
>
> This simple Hive query would generate wrong result:
> select a.key, b.k2, b.k3
> from src a
> join (
> select key,
> min(key) as k,
> min(key)+1 as k1,
> min(key)+2 as k2,
> min(key)+3 as k3
> from src
> group by key
> ) b
> on a.key=b.key and b.k1 < 5;
> 0 3.0 1.0
> 0 3.0 1.0
> 0 3.0 1.0
> 2 5.0 3.0
> The right result is:
> 0 2.0 3.0
> 0 2.0 3.0
> 0 2.0 3.0
> 2 4.0 5.0
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3343) Hive: Query misaligned result for
Group by followed by Join with filter and skip a group-by result
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13433868#comment-13433868 ]
Namit Jain commented on HIVE-3343:
----------------------------------
+1
> Hive: Query misaligned result for Group by followed by Join with filter and skip a group-by result
> --------------------------------------------------------------------------------------------------
>
> Key: HIVE-3343
> URL: https://issues.apache.org/jira/browse/HIVE-3343
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Gang Tim Liu
> Assignee: Gang Tim Liu
> Attachments: hive-3343.patch
>
>
> This simple Hive query would generate wrong result:
> select a.key, b.k2, b.k3
> from src a
> join (
> select key,
> min(key) as k,
> min(key)+1 as k1,
> min(key)+2 as k2,
> min(key)+3 as k3
> from src
> group by key
> ) b
> on a.key=b.key and b.k1 < 5;
> 0 3.0 1.0
> 0 3.0 1.0
> 0 3.0 1.0
> 2 5.0 3.0
> The right result is:
> 0 2.0 3.0
> 0 2.0 3.0
> 0 2.0 3.0
> 2 4.0 5.0
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3343) Hive: Query misaligned result for
Group by followed by Join with filter and skip a group-by result
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434042#comment-13434042 ]
Namit Jain commented on HIVE-3343:
----------------------------------
thanks
> Hive: Query misaligned result for Group by followed by Join with filter and skip a group-by result
> --------------------------------------------------------------------------------------------------
>
> Key: HIVE-3343
> URL: https://issues.apache.org/jira/browse/HIVE-3343
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Gang Tim Liu
> Assignee: Gang Tim Liu
> Fix For: 0.10.0
>
> Attachments: hive-3343.patch
>
>
> This simple Hive query would generate wrong result:
> select a.key, b.k2, b.k3
> from src a
> join (
> select key,
> min(key) as k,
> min(key)+1 as k1,
> min(key)+2 as k2,
> min(key)+3 as k3
> from src
> group by key
> ) b
> on a.key=b.key and b.k1 < 5;
> 0 3.0 1.0
> 0 3.0 1.0
> 0 3.0 1.0
> 2 5.0 3.0
> The right result is:
> 0 2.0 3.0
> 0 2.0 3.0
> 0 2.0 3.0
> 2 4.0 5.0
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3343) Hive: Query misaligned result for
Group by followed by Join with filter and skip a group-by result
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-3343:
-----------------------------
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed. Thanks Tim
> Hive: Query misaligned result for Group by followed by Join with filter and skip a group-by result
> --------------------------------------------------------------------------------------------------
>
> Key: HIVE-3343
> URL: https://issues.apache.org/jira/browse/HIVE-3343
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Gang Tim Liu
> Assignee: Gang Tim Liu
> Attachments: hive-3343.patch
>
>
> This simple Hive query would generate wrong result:
> select a.key, b.k2, b.k3
> from src a
> join (
> select key,
> min(key) as k,
> min(key)+1 as k1,
> min(key)+2 as k2,
> min(key)+3 as k3
> from src
> group by key
> ) b
> on a.key=b.key and b.k1 < 5;
> 0 3.0 1.0
> 0 3.0 1.0
> 0 3.0 1.0
> 2 5.0 3.0
> The right result is:
> 0 2.0 3.0
> 0 2.0 3.0
> 0 2.0 3.0
> 2 4.0 5.0
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3343) Hive: Query misaligned result for
Group by followed by Join with filter and skip a group-by result
Posted by "Gang Tim Liu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13433892#comment-13433892 ]
Gang Tim Liu commented on HIVE-3343:
------------------------------------
thanks Namit
> Hive: Query misaligned result for Group by followed by Join with filter and skip a group-by result
> --------------------------------------------------------------------------------------------------
>
> Key: HIVE-3343
> URL: https://issues.apache.org/jira/browse/HIVE-3343
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Gang Tim Liu
> Assignee: Gang Tim Liu
> Attachments: hive-3343.patch
>
>
> This simple Hive query would generate wrong result:
> select a.key, b.k2, b.k3
> from src a
> join (
> select key,
> min(key) as k,
> min(key)+1 as k1,
> min(key)+2 as k2,
> min(key)+3 as k3
> from src
> group by key
> ) b
> on a.key=b.key and b.k1 < 5;
> 0 3.0 1.0
> 0 3.0 1.0
> 0 3.0 1.0
> 2 5.0 3.0
> The right result is:
> 0 2.0 3.0
> 0 2.0 3.0
> 0 2.0 3.0
> 2 4.0 5.0
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3343) Hive: Query misaligned result for
Group by followed by Join with filter and skip a group-by result
Posted by "Gang Tim Liu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gang Tim Liu updated HIVE-3343:
-------------------------------
Status: Patch Available (was: Open)
patch is submitted.
> Hive: Query misaligned result for Group by followed by Join with filter and skip a group-by result
> --------------------------------------------------------------------------------------------------
>
> Key: HIVE-3343
> URL: https://issues.apache.org/jira/browse/HIVE-3343
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Gang Tim Liu
> Assignee: Gang Tim Liu
> Attachments: hive-3343.patch
>
>
> This simple Hive query would generate wrong result:
> select a.key, b.k2, b.k3
> from src a
> join (
> select key,
> min(key) as k,
> min(key)+1 as k1,
> min(key)+2 as k2,
> min(key)+3 as k3
> from src
> group by key
> ) b
> on a.key=b.key and b.k1 < 5;
> 0 3.0 1.0
> 0 3.0 1.0
> 0 3.0 1.0
> 2 5.0 3.0
> The right result is:
> 0 2.0 3.0
> 0 2.0 3.0
> 0 2.0 3.0
> 2 4.0 5.0
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3343) Hive: Query misaligned result for
Group by followed by Join with filter and skip a group-by result
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach updated HIVE-3343:
---------------------------------
Fix Version/s: 0.10.0
Setting the fixversion field for Namit...
> Hive: Query misaligned result for Group by followed by Join with filter and skip a group-by result
> --------------------------------------------------------------------------------------------------
>
> Key: HIVE-3343
> URL: https://issues.apache.org/jira/browse/HIVE-3343
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Gang Tim Liu
> Assignee: Gang Tim Liu
> Fix For: 0.10.0
>
> Attachments: hive-3343.patch
>
>
> This simple Hive query would generate wrong result:
> select a.key, b.k2, b.k3
> from src a
> join (
> select key,
> min(key) as k,
> min(key)+1 as k1,
> min(key)+2 as k2,
> min(key)+3 as k3
> from src
> group by key
> ) b
> on a.key=b.key and b.k1 < 5;
> 0 3.0 1.0
> 0 3.0 1.0
> 0 3.0 1.0
> 2 5.0 3.0
> The right result is:
> 0 2.0 3.0
> 0 2.0 3.0
> 0 2.0 3.0
> 2 4.0 5.0
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3343) Hive: Query misaligned result for
Group by followed by Join with filter and skip a group-by result
Posted by "Gang Tim Liu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430757#comment-13430757 ]
Gang Tim Liu commented on HIVE-3343:
------------------------------------
If we remove "b.k1 < 5", it will generate right result.
"explain extended" shows Select Operator has different order of columns (_col0, _col3, _col4, _col2) from filter operator's (_col0,_col2,_col3,_col4). Here is snapshot:
Reduce Operator Tree:
Group By Operator
...
outputColumnNames: _col0, _col1
Select Operator
...
outputColumnNames: _col0, _col3, _col4, _col2
Filter Operator
...
predicate:
expr: (_col2 < 5.0)
type: boolean
File Output Operator
...
table:
...
properties:
columns _col0,_col2,_col3,_col4
columns.types string,double,double,double
escape.delim \
...
> Hive: Query misaligned result for Group by followed by Join with filter and skip a group-by result
> --------------------------------------------------------------------------------------------------
>
> Key: HIVE-3343
> URL: https://issues.apache.org/jira/browse/HIVE-3343
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Gang Tim Liu
> Assignee: Gang Tim Liu
>
> This simple Hive query would generate wrong result:
> select a.key, b.k2, b.k3
> from src a
> join (
> select key,
> min(key) as k,
> min(key)+1 as k1,
> min(key)+2 as k2,
> min(key)+3 as k3
> from src
> group by key
> ) b
> on a.key=b.key and b.k1 < 5;
> 0 3.0 1.0
> 0 3.0 1.0
> 0 3.0 1.0
> 2 5.0 3.0
> The right result is:
> 0 2.0 3.0
> 0 2.0 3.0
> 0 2.0 3.0
> 2 4.0 5.0
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3343) Hive: Query misaligned result for
Group by followed by Join with filter and skip a group-by result
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434740#comment-13434740 ]
Hudson commented on HIVE-3343:
------------------------------
Integrated in Hive-trunk-h0.21 #1605 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1605/])
HIVE-3343 Hive: Query misaligned result for Group by followed by Join with filter
and skip a group-by result (Gang Tim Liu via namit) (Revision 1372727)
Result = SUCCESS
namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1372727
Files :
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcFactory.java
* /hive/trunk/ql/src/test/queries/clientpositive/ppd_join_filter.q
* /hive/trunk/ql/src/test/results/clientpositive/lateral_view_ppd.q.out
* /hive/trunk/ql/src/test/results/clientpositive/ppd_join_filter.q.out
> Hive: Query misaligned result for Group by followed by Join with filter and skip a group-by result
> --------------------------------------------------------------------------------------------------
>
> Key: HIVE-3343
> URL: https://issues.apache.org/jira/browse/HIVE-3343
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Gang Tim Liu
> Assignee: Gang Tim Liu
> Fix For: 0.10.0
>
> Attachments: hive-3343.patch
>
>
> This simple Hive query would generate wrong result:
> select a.key, b.k2, b.k3
> from src a
> join (
> select key,
> min(key) as k,
> min(key)+1 as k1,
> min(key)+2 as k2,
> min(key)+3 as k3
> from src
> group by key
> ) b
> on a.key=b.key and b.k1 < 5;
> 0 3.0 1.0
> 0 3.0 1.0
> 0 3.0 1.0
> 2 5.0 3.0
> The right result is:
> 0 2.0 3.0
> 0 2.0 3.0
> 0 2.0 3.0
> 2 4.0 5.0
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3343) Hive: Query misaligned result for
Group by followed by Join with filter and skip a group-by result
Posted by "Gang Tim Liu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430761#comment-13430761 ]
Gang Tim Liu commented on HIVE-3343:
------------------------------------
root-cause is ColumnPrunerFilterProc doesn't perserve order of prunned column list (filterOpPrunedColLists). It causes
outputColumnNames: _col0, _col3, _col4, _col2
It should be outputColumnNames: _col0, _col2, _col3, _col4
> Hive: Query misaligned result for Group by followed by Join with filter and skip a group-by result
> --------------------------------------------------------------------------------------------------
>
> Key: HIVE-3343
> URL: https://issues.apache.org/jira/browse/HIVE-3343
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Gang Tim Liu
> Assignee: Gang Tim Liu
>
> This simple Hive query would generate wrong result:
> select a.key, b.k2, b.k3
> from src a
> join (
> select key,
> min(key) as k,
> min(key)+1 as k1,
> min(key)+2 as k2,
> min(key)+3 as k3
> from src
> group by key
> ) b
> on a.key=b.key and b.k1 < 5;
> 0 3.0 1.0
> 0 3.0 1.0
> 0 3.0 1.0
> 2 5.0 3.0
> The right result is:
> 0 2.0 3.0
> 0 2.0 3.0
> 0 2.0 3.0
> 2 4.0 5.0
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira