You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Gang Tim Liu (JIRA)" <ji...@apache.org> on 2012/08/08 02:03:09 UTC

[jira] [Created] (HIVE-3343) Hive: Query misaligned result for Group by followed by Join with filter and skip a group-by result

Gang Tim Liu created HIVE-3343:
----------------------------------

             Summary: Hive: Query misaligned result for Group by followed by Join with filter and skip a group-by result
                 Key: HIVE-3343
                 URL: https://issues.apache.org/jira/browse/HIVE-3343
             Project: Hive
          Issue Type: Bug
          Components: Query Processor
            Reporter: Gang Tim Liu
            Assignee: Gang Tim Liu


This simple Hive query would generate wrong result:

select a.key, b.k2, b.k3
from src a
join (
  select key, 
         min(key) as k,
         min(key)+1 as k1,
         min(key)+2 as k2,
         min(key)+3 as k3
  from src
  group by key
) b
on a.key=b.key and b.k1 < 5;


0       3.0     1.0
0       3.0     1.0
0       3.0     1.0
2       5.0     3.0

The right result is:
0	2.0	3.0
0	2.0	3.0
0	2.0	3.0
2	4.0	5.0

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HIVE-3343) Hive: Query misaligned result for Group by followed by Join with filter and skip a group-by result

Posted by "Gang Tim Liu (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HIVE-3343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Gang Tim Liu updated HIVE-3343:
-------------------------------

    Attachment: hive-3343.patch
    
> Hive: Query misaligned result for Group by followed by Join with filter and skip a group-by result
> --------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-3343
>                 URL: https://issues.apache.org/jira/browse/HIVE-3343
>             Project: Hive
>          Issue Type: Bug
>          Components: Query Processor
>            Reporter: Gang Tim Liu
>            Assignee: Gang Tim Liu
>         Attachments: hive-3343.patch
>
>
> This simple Hive query would generate wrong result:
> select a.key, b.k2, b.k3
> from src a
> join (
>   select key, 
>          min(key) as k,
>          min(key)+1 as k1,
>          min(key)+2 as k2,
>          min(key)+3 as k3
>   from src
>   group by key
> ) b
> on a.key=b.key and b.k1 < 5;
> 0       3.0     1.0
> 0       3.0     1.0
> 0       3.0     1.0
> 2       5.0     3.0
> The right result is:
> 0	2.0	3.0
> 0	2.0	3.0
> 0	2.0	3.0
> 2	4.0	5.0

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HIVE-3343) Hive: Query misaligned result for Group by followed by Join with filter and skip a group-by result

Posted by "Gang Tim Liu (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HIVE-3343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430776#comment-13430776 ] 

Gang Tim Liu commented on HIVE-3343:
------------------------------------

https://reviews.facebook.net/D4563
                
> Hive: Query misaligned result for Group by followed by Join with filter and skip a group-by result
> --------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-3343
>                 URL: https://issues.apache.org/jira/browse/HIVE-3343
>             Project: Hive
>          Issue Type: Bug
>          Components: Query Processor
>            Reporter: Gang Tim Liu
>            Assignee: Gang Tim Liu
>
> This simple Hive query would generate wrong result:
> select a.key, b.k2, b.k3
> from src a
> join (
>   select key, 
>          min(key) as k,
>          min(key)+1 as k1,
>          min(key)+2 as k2,
>          min(key)+3 as k3
>   from src
>   group by key
> ) b
> on a.key=b.key and b.k1 < 5;
> 0       3.0     1.0
> 0       3.0     1.0
> 0       3.0     1.0
> 2       5.0     3.0
> The right result is:
> 0	2.0	3.0
> 0	2.0	3.0
> 0	2.0	3.0
> 2	4.0	5.0

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HIVE-3343) Hive: Query misaligned result for Group by followed by Join with filter and skip a group-by result

Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HIVE-3343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13433868#comment-13433868 ] 

Namit Jain commented on HIVE-3343:
----------------------------------

+1
                
> Hive: Query misaligned result for Group by followed by Join with filter and skip a group-by result
> --------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-3343
>                 URL: https://issues.apache.org/jira/browse/HIVE-3343
>             Project: Hive
>          Issue Type: Bug
>          Components: Query Processor
>            Reporter: Gang Tim Liu
>            Assignee: Gang Tim Liu
>         Attachments: hive-3343.patch
>
>
> This simple Hive query would generate wrong result:
> select a.key, b.k2, b.k3
> from src a
> join (
>   select key, 
>          min(key) as k,
>          min(key)+1 as k1,
>          min(key)+2 as k2,
>          min(key)+3 as k3
>   from src
>   group by key
> ) b
> on a.key=b.key and b.k1 < 5;
> 0       3.0     1.0
> 0       3.0     1.0
> 0       3.0     1.0
> 2       5.0     3.0
> The right result is:
> 0	2.0	3.0
> 0	2.0	3.0
> 0	2.0	3.0
> 2	4.0	5.0

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HIVE-3343) Hive: Query misaligned result for Group by followed by Join with filter and skip a group-by result

Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HIVE-3343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434042#comment-13434042 ] 

Namit Jain commented on HIVE-3343:
----------------------------------

thanks
                
> Hive: Query misaligned result for Group by followed by Join with filter and skip a group-by result
> --------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-3343
>                 URL: https://issues.apache.org/jira/browse/HIVE-3343
>             Project: Hive
>          Issue Type: Bug
>          Components: Query Processor
>            Reporter: Gang Tim Liu
>            Assignee: Gang Tim Liu
>             Fix For: 0.10.0
>
>         Attachments: hive-3343.patch
>
>
> This simple Hive query would generate wrong result:
> select a.key, b.k2, b.k3
> from src a
> join (
>   select key, 
>          min(key) as k,
>          min(key)+1 as k1,
>          min(key)+2 as k2,
>          min(key)+3 as k3
>   from src
>   group by key
> ) b
> on a.key=b.key and b.k1 < 5;
> 0       3.0     1.0
> 0       3.0     1.0
> 0       3.0     1.0
> 2       5.0     3.0
> The right result is:
> 0	2.0	3.0
> 0	2.0	3.0
> 0	2.0	3.0
> 2	4.0	5.0

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HIVE-3343) Hive: Query misaligned result for Group by followed by Join with filter and skip a group-by result

Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HIVE-3343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Namit Jain updated HIVE-3343:
-----------------------------

      Resolution: Fixed
    Hadoop Flags: Reviewed
          Status: Resolved  (was: Patch Available)

Committed. Thanks Tim
                
> Hive: Query misaligned result for Group by followed by Join with filter and skip a group-by result
> --------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-3343
>                 URL: https://issues.apache.org/jira/browse/HIVE-3343
>             Project: Hive
>          Issue Type: Bug
>          Components: Query Processor
>            Reporter: Gang Tim Liu
>            Assignee: Gang Tim Liu
>         Attachments: hive-3343.patch
>
>
> This simple Hive query would generate wrong result:
> select a.key, b.k2, b.k3
> from src a
> join (
>   select key, 
>          min(key) as k,
>          min(key)+1 as k1,
>          min(key)+2 as k2,
>          min(key)+3 as k3
>   from src
>   group by key
> ) b
> on a.key=b.key and b.k1 < 5;
> 0       3.0     1.0
> 0       3.0     1.0
> 0       3.0     1.0
> 2       5.0     3.0
> The right result is:
> 0	2.0	3.0
> 0	2.0	3.0
> 0	2.0	3.0
> 2	4.0	5.0

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HIVE-3343) Hive: Query misaligned result for Group by followed by Join with filter and skip a group-by result

Posted by "Gang Tim Liu (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HIVE-3343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13433892#comment-13433892 ] 

Gang Tim Liu commented on HIVE-3343:
------------------------------------

thanks Namit
                
> Hive: Query misaligned result for Group by followed by Join with filter and skip a group-by result
> --------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-3343
>                 URL: https://issues.apache.org/jira/browse/HIVE-3343
>             Project: Hive
>          Issue Type: Bug
>          Components: Query Processor
>            Reporter: Gang Tim Liu
>            Assignee: Gang Tim Liu
>         Attachments: hive-3343.patch
>
>
> This simple Hive query would generate wrong result:
> select a.key, b.k2, b.k3
> from src a
> join (
>   select key, 
>          min(key) as k,
>          min(key)+1 as k1,
>          min(key)+2 as k2,
>          min(key)+3 as k3
>   from src
>   group by key
> ) b
> on a.key=b.key and b.k1 < 5;
> 0       3.0     1.0
> 0       3.0     1.0
> 0       3.0     1.0
> 2       5.0     3.0
> The right result is:
> 0	2.0	3.0
> 0	2.0	3.0
> 0	2.0	3.0
> 2	4.0	5.0

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HIVE-3343) Hive: Query misaligned result for Group by followed by Join with filter and skip a group-by result

Posted by "Gang Tim Liu (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HIVE-3343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Gang Tim Liu updated HIVE-3343:
-------------------------------

    Status: Patch Available  (was: Open)

patch is submitted.
                
> Hive: Query misaligned result for Group by followed by Join with filter and skip a group-by result
> --------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-3343
>                 URL: https://issues.apache.org/jira/browse/HIVE-3343
>             Project: Hive
>          Issue Type: Bug
>          Components: Query Processor
>            Reporter: Gang Tim Liu
>            Assignee: Gang Tim Liu
>         Attachments: hive-3343.patch
>
>
> This simple Hive query would generate wrong result:
> select a.key, b.k2, b.k3
> from src a
> join (
>   select key, 
>          min(key) as k,
>          min(key)+1 as k1,
>          min(key)+2 as k2,
>          min(key)+3 as k3
>   from src
>   group by key
> ) b
> on a.key=b.key and b.k1 < 5;
> 0       3.0     1.0
> 0       3.0     1.0
> 0       3.0     1.0
> 2       5.0     3.0
> The right result is:
> 0	2.0	3.0
> 0	2.0	3.0
> 0	2.0	3.0
> 2	4.0	5.0

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HIVE-3343) Hive: Query misaligned result for Group by followed by Join with filter and skip a group-by result

Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HIVE-3343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Carl Steinbach updated HIVE-3343:
---------------------------------

    Fix Version/s: 0.10.0

Setting the fixversion field for Namit...
                
> Hive: Query misaligned result for Group by followed by Join with filter and skip a group-by result
> --------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-3343
>                 URL: https://issues.apache.org/jira/browse/HIVE-3343
>             Project: Hive
>          Issue Type: Bug
>          Components: Query Processor
>            Reporter: Gang Tim Liu
>            Assignee: Gang Tim Liu
>             Fix For: 0.10.0
>
>         Attachments: hive-3343.patch
>
>
> This simple Hive query would generate wrong result:
> select a.key, b.k2, b.k3
> from src a
> join (
>   select key, 
>          min(key) as k,
>          min(key)+1 as k1,
>          min(key)+2 as k2,
>          min(key)+3 as k3
>   from src
>   group by key
> ) b
> on a.key=b.key and b.k1 < 5;
> 0       3.0     1.0
> 0       3.0     1.0
> 0       3.0     1.0
> 2       5.0     3.0
> The right result is:
> 0	2.0	3.0
> 0	2.0	3.0
> 0	2.0	3.0
> 2	4.0	5.0

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HIVE-3343) Hive: Query misaligned result for Group by followed by Join with filter and skip a group-by result

Posted by "Gang Tim Liu (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HIVE-3343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430757#comment-13430757 ] 

Gang Tim Liu commented on HIVE-3343:
------------------------------------

If we remove "b.k1 < 5", it will generate right result.

"explain extended" shows Select Operator has different order of columns (_col0, _col3, _col4, _col2) from filter operator's (_col0,_col2,_col3,_col4). Here is snapshot:

Reduce Operator Tree:
        Group By Operator
          ...
          outputColumnNames: _col0, _col1
          Select Operator
            ...
            outputColumnNames: _col0, _col3, _col4, _col2
            Filter Operator
              ...
              predicate:
                  expr: (_col2 < 5.0)
                  type: boolean
              File Output Operator
                ...
                table:
                    ...
                    properties:
                      columns _col0,_col2,_col3,_col4
                      columns.types string,double,double,double
                      escape.delim \
                ...
                
> Hive: Query misaligned result for Group by followed by Join with filter and skip a group-by result
> --------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-3343
>                 URL: https://issues.apache.org/jira/browse/HIVE-3343
>             Project: Hive
>          Issue Type: Bug
>          Components: Query Processor
>            Reporter: Gang Tim Liu
>            Assignee: Gang Tim Liu
>
> This simple Hive query would generate wrong result:
> select a.key, b.k2, b.k3
> from src a
> join (
>   select key, 
>          min(key) as k,
>          min(key)+1 as k1,
>          min(key)+2 as k2,
>          min(key)+3 as k3
>   from src
>   group by key
> ) b
> on a.key=b.key and b.k1 < 5;
> 0       3.0     1.0
> 0       3.0     1.0
> 0       3.0     1.0
> 2       5.0     3.0
> The right result is:
> 0	2.0	3.0
> 0	2.0	3.0
> 0	2.0	3.0
> 2	4.0	5.0

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HIVE-3343) Hive: Query misaligned result for Group by followed by Join with filter and skip a group-by result

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HIVE-3343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434740#comment-13434740 ] 

Hudson commented on HIVE-3343:
------------------------------

Integrated in Hive-trunk-h0.21 #1605 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1605/])
    HIVE-3343 Hive: Query misaligned result for Group by followed by Join with filter
and skip a group-by result (Gang Tim Liu via namit) (Revision 1372727)

     Result = SUCCESS
namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1372727
Files : 
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcFactory.java
* /hive/trunk/ql/src/test/queries/clientpositive/ppd_join_filter.q
* /hive/trunk/ql/src/test/results/clientpositive/lateral_view_ppd.q.out
* /hive/trunk/ql/src/test/results/clientpositive/ppd_join_filter.q.out

                
> Hive: Query misaligned result for Group by followed by Join with filter and skip a group-by result
> --------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-3343
>                 URL: https://issues.apache.org/jira/browse/HIVE-3343
>             Project: Hive
>          Issue Type: Bug
>          Components: Query Processor
>            Reporter: Gang Tim Liu
>            Assignee: Gang Tim Liu
>             Fix For: 0.10.0
>
>         Attachments: hive-3343.patch
>
>
> This simple Hive query would generate wrong result:
> select a.key, b.k2, b.k3
> from src a
> join (
>   select key, 
>          min(key) as k,
>          min(key)+1 as k1,
>          min(key)+2 as k2,
>          min(key)+3 as k3
>   from src
>   group by key
> ) b
> on a.key=b.key and b.k1 < 5;
> 0       3.0     1.0
> 0       3.0     1.0
> 0       3.0     1.0
> 2       5.0     3.0
> The right result is:
> 0	2.0	3.0
> 0	2.0	3.0
> 0	2.0	3.0
> 2	4.0	5.0

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HIVE-3343) Hive: Query misaligned result for Group by followed by Join with filter and skip a group-by result

Posted by "Gang Tim Liu (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HIVE-3343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430761#comment-13430761 ] 

Gang Tim Liu commented on HIVE-3343:
------------------------------------

root-cause is ColumnPrunerFilterProc doesn't perserve order of prunned column list (filterOpPrunedColLists). It causes

outputColumnNames: _col0, _col3, _col4, _col2

It should be outputColumnNames: _col0, _col2, _col3, _col4

                
> Hive: Query misaligned result for Group by followed by Join with filter and skip a group-by result
> --------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-3343
>                 URL: https://issues.apache.org/jira/browse/HIVE-3343
>             Project: Hive
>          Issue Type: Bug
>          Components: Query Processor
>            Reporter: Gang Tim Liu
>            Assignee: Gang Tim Liu
>
> This simple Hive query would generate wrong result:
> select a.key, b.k2, b.k3
> from src a
> join (
>   select key, 
>          min(key) as k,
>          min(key)+1 as k1,
>          min(key)+2 as k2,
>          min(key)+3 as k3
>   from src
>   group by key
> ) b
> on a.key=b.key and b.k1 < 5;
> 0       3.0     1.0
> 0       3.0     1.0
> 0       3.0     1.0
> 2       5.0     3.0
> The right result is:
> 0	2.0	3.0
> 0	2.0	3.0
> 0	2.0	3.0
> 2	4.0	5.0

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira