You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Yongqiang He (JIRA)" <ji...@apache.org> on 2009/06/25 11:37:07 UTC

[jira] Created: (HIVE-582) "SELECT count(1) as cnt from src x" not working for column pruning

"SELECT count(1) as cnt from src x" not working for column pruning
------------------------------------------------------------------

                 Key: HIVE-582
                 URL: https://issues.apache.org/jira/browse/HIVE-582
             Project: Hadoop Hive
          Issue Type: Bug
            Reporter: Yongqiang He
            Assignee: Yongqiang He


The query "SELECT count(1) as cnt from src x"  has some problems, even though nullgroup*.q passed in tests. 
Will check and solve it in this jira.
A wrong plan sometimes can be generated due to some unknow factor:
Wrong plan:
{noformat}
STAGE PLANS:
  Stage: Stage-1
    Map Reduce
      Alias -> Map Operator Tree:
        x 
            Select Operator
              Reduce Output Operator
                sort order: 
                tag: -1
      Reduce Operator Tree:
        Group By Operator
          aggregations:
                expr: count(VALUE._col0)
          mode: complete
          Select Operator
            expressions:
                  expr: _col0
                  type: bigint
            File Output Operator
              compressed: false
              GlobalTableId: 0
              table:
                  input format: org.apache.hadoop.mapred.TextInputFormat
                  output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat

  Stage: Stage-0
    Fetch Operator
      limit: -1
{noformat}

Without unknow factor, correct plan can be generated:
{noformat}
STAGE PLANS:
  Stage: Stage-1
    Map Reduce
      Alias -> Map Operator Tree:
        x 
            Select Operator
              Group By Operator
                aggregations:
                      expr: count(1)
                mode: hash
                Reduce Output Operator
                  sort order: 
                  tag: -1
                  value expressions:
                        expr: _col0
                        type: bigint
      Reduce Operator Tree:
        Group By Operator
          aggregations:
                expr: count(VALUE._col0)
          mode: mergepartial
          Select Operator
            expressions:
                  expr: _col0
                  type: bigint
            File Output Operator
              compressed: false
              GlobalTableId: 0
              table:
                  input format: org.apache.hadoop.mapred.TextInputFormat
                  output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
{noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HIVE-582) "SELECT count(1) as cnt from src x" not working for column pruning

Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HIVE-582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Namit Jain updated HIVE-582:
----------------------------

       Resolution: Fixed
    Fix Version/s: 0.4.0
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

Committed. Thanks Yongqiang

> "SELECT count(1) as cnt from src x" not working for column pruning
> ------------------------------------------------------------------
>
>                 Key: HIVE-582
>                 URL: https://issues.apache.org/jira/browse/HIVE-582
>             Project: Hadoop Hive
>          Issue Type: Bug
>            Reporter: Yongqiang He
>            Assignee: Yongqiang He
>             Fix For: 0.4.0
>
>         Attachments: hive-582-2009-06-25.patch
>
>
> The query "SELECT count(1) as cnt from src x"  has some problems, even though nullgroup*.q passed in tests. 
> Will check and solve it in this jira.
> A wrong plan sometimes can be generated due to some unknow factor:
> Wrong plan:
> {noformat}
> STAGE PLANS:
>   Stage: Stage-1
>     Map Reduce
>       Alias -> Map Operator Tree:
>         x 
>             Select Operator
>               Reduce Output Operator
>                 sort order: 
>                 tag: -1
>       Reduce Operator Tree:
>         Group By Operator
>           aggregations:
>                 expr: count(VALUE._col0)
>           mode: complete
>           Select Operator
>             expressions:
>                   expr: _col0
>                   type: bigint
>             File Output Operator
>               compressed: false
>               GlobalTableId: 0
>               table:
>                   input format: org.apache.hadoop.mapred.TextInputFormat
>                   output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
>   Stage: Stage-0
>     Fetch Operator
>       limit: -1
> {noformat}
> Without unknow factor, correct plan can be generated:
> {noformat}
> STAGE PLANS:
>   Stage: Stage-1
>     Map Reduce
>       Alias -> Map Operator Tree:
>         x 
>             Select Operator
>               Group By Operator
>                 aggregations:
>                       expr: count(1)
>                 mode: hash
>                 Reduce Output Operator
>                   sort order: 
>                   tag: -1
>                   value expressions:
>                         expr: _col0
>                         type: bigint
>       Reduce Operator Tree:
>         Group By Operator
>           aggregations:
>                 expr: count(VALUE._col0)
>           mode: mergepartial
>           Select Operator
>             expressions:
>                   expr: _col0
>                   type: bigint
>             File Output Operator
>               compressed: false
>               GlobalTableId: 0
>               table:
>                   input format: org.apache.hadoop.mapred.TextInputFormat
>                   output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HIVE-582) "SELECT count(1) as cnt from src x" not working for column pruning

Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HIVE-582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12725364#action_12725364 ] 

Namit Jain commented on HIVE-582:
---------------------------------

looks good 

> "SELECT count(1) as cnt from src x" not working for column pruning
> ------------------------------------------------------------------
>
>                 Key: HIVE-582
>                 URL: https://issues.apache.org/jira/browse/HIVE-582
>             Project: Hadoop Hive
>          Issue Type: Bug
>            Reporter: Yongqiang He
>            Assignee: Yongqiang He
>         Attachments: hive-582-2009-06-25.patch
>
>
> The query "SELECT count(1) as cnt from src x"  has some problems, even though nullgroup*.q passed in tests. 
> Will check and solve it in this jira.
> A wrong plan sometimes can be generated due to some unknow factor:
> Wrong plan:
> {noformat}
> STAGE PLANS:
>   Stage: Stage-1
>     Map Reduce
>       Alias -> Map Operator Tree:
>         x 
>             Select Operator
>               Reduce Output Operator
>                 sort order: 
>                 tag: -1
>       Reduce Operator Tree:
>         Group By Operator
>           aggregations:
>                 expr: count(VALUE._col0)
>           mode: complete
>           Select Operator
>             expressions:
>                   expr: _col0
>                   type: bigint
>             File Output Operator
>               compressed: false
>               GlobalTableId: 0
>               table:
>                   input format: org.apache.hadoop.mapred.TextInputFormat
>                   output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
>   Stage: Stage-0
>     Fetch Operator
>       limit: -1
> {noformat}
> Without unknow factor, correct plan can be generated:
> {noformat}
> STAGE PLANS:
>   Stage: Stage-1
>     Map Reduce
>       Alias -> Map Operator Tree:
>         x 
>             Select Operator
>               Group By Operator
>                 aggregations:
>                       expr: count(1)
>                 mode: hash
>                 Reduce Output Operator
>                   sort order: 
>                   tag: -1
>                   value expressions:
>                         expr: _col0
>                         type: bigint
>       Reduce Operator Tree:
>         Group By Operator
>           aggregations:
>                 expr: count(VALUE._col0)
>           mode: mergepartial
>           Select Operator
>             expressions:
>                   expr: _col0
>                   type: bigint
>             File Output Operator
>               compressed: false
>               GlobalTableId: 0
>               table:
>                   input format: org.apache.hadoop.mapred.TextInputFormat
>                   output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HIVE-582) "SELECT count(1) as cnt from src x" not working for column pruning

Posted by "Yongqiang He (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HIVE-582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yongqiang He updated HIVE-582:
------------------------------

    Attachment: hive-582-2009-06-25.patch

> "SELECT count(1) as cnt from src x" not working for column pruning
> ------------------------------------------------------------------
>
>                 Key: HIVE-582
>                 URL: https://issues.apache.org/jira/browse/HIVE-582
>             Project: Hadoop Hive
>          Issue Type: Bug
>            Reporter: Yongqiang He
>            Assignee: Yongqiang He
>         Attachments: hive-582-2009-06-25.patch
>
>
> The query "SELECT count(1) as cnt from src x"  has some problems, even though nullgroup*.q passed in tests. 
> Will check and solve it in this jira.
> A wrong plan sometimes can be generated due to some unknow factor:
> Wrong plan:
> {noformat}
> STAGE PLANS:
>   Stage: Stage-1
>     Map Reduce
>       Alias -> Map Operator Tree:
>         x 
>             Select Operator
>               Reduce Output Operator
>                 sort order: 
>                 tag: -1
>       Reduce Operator Tree:
>         Group By Operator
>           aggregations:
>                 expr: count(VALUE._col0)
>           mode: complete
>           Select Operator
>             expressions:
>                   expr: _col0
>                   type: bigint
>             File Output Operator
>               compressed: false
>               GlobalTableId: 0
>               table:
>                   input format: org.apache.hadoop.mapred.TextInputFormat
>                   output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
>   Stage: Stage-0
>     Fetch Operator
>       limit: -1
> {noformat}
> Without unknow factor, correct plan can be generated:
> {noformat}
> STAGE PLANS:
>   Stage: Stage-1
>     Map Reduce
>       Alias -> Map Operator Tree:
>         x 
>             Select Operator
>               Group By Operator
>                 aggregations:
>                       expr: count(1)
>                 mode: hash
>                 Reduce Output Operator
>                   sort order: 
>                   tag: -1
>                   value expressions:
>                         expr: _col0
>                         type: bigint
>       Reduce Operator Tree:
>         Group By Operator
>           aggregations:
>                 expr: count(VALUE._col0)
>           mode: mergepartial
>           Select Operator
>             expressions:
>                   expr: _col0
>                   type: bigint
>             File Output Operator
>               compressed: false
>               GlobalTableId: 0
>               table:
>                   input format: org.apache.hadoop.mapred.TextInputFormat
>                   output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HIVE-582) "SELECT count(1) as cnt from src x" not working for column pruning

Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HIVE-582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Zheng Shao updated HIVE-582:
----------------------------

    Status: Patch Available  (was: Open)

> "SELECT count(1) as cnt from src x" not working for column pruning
> ------------------------------------------------------------------
>
>                 Key: HIVE-582
>                 URL: https://issues.apache.org/jira/browse/HIVE-582
>             Project: Hadoop Hive
>          Issue Type: Bug
>            Reporter: Yongqiang He
>            Assignee: Yongqiang He
>         Attachments: hive-582-2009-06-25.patch
>
>
> The query "SELECT count(1) as cnt from src x"  has some problems, even though nullgroup*.q passed in tests. 
> Will check and solve it in this jira.
> A wrong plan sometimes can be generated due to some unknow factor:
> Wrong plan:
> {noformat}
> STAGE PLANS:
>   Stage: Stage-1
>     Map Reduce
>       Alias -> Map Operator Tree:
>         x 
>             Select Operator
>               Reduce Output Operator
>                 sort order: 
>                 tag: -1
>       Reduce Operator Tree:
>         Group By Operator
>           aggregations:
>                 expr: count(VALUE._col0)
>           mode: complete
>           Select Operator
>             expressions:
>                   expr: _col0
>                   type: bigint
>             File Output Operator
>               compressed: false
>               GlobalTableId: 0
>               table:
>                   input format: org.apache.hadoop.mapred.TextInputFormat
>                   output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
>   Stage: Stage-0
>     Fetch Operator
>       limit: -1
> {noformat}
> Without unknow factor, correct plan can be generated:
> {noformat}
> STAGE PLANS:
>   Stage: Stage-1
>     Map Reduce
>       Alias -> Map Operator Tree:
>         x 
>             Select Operator
>               Group By Operator
>                 aggregations:
>                       expr: count(1)
>                 mode: hash
>                 Reduce Output Operator
>                   sort order: 
>                   tag: -1
>                   value expressions:
>                         expr: _col0
>                         type: bigint
>       Reduce Operator Tree:
>         Group By Operator
>           aggregations:
>                 expr: count(VALUE._col0)
>           mode: mergepartial
>           Select Operator
>             expressions:
>                   expr: _col0
>                   type: bigint
>             File Output Operator
>               compressed: false
>               GlobalTableId: 0
>               table:
>                   input format: org.apache.hadoop.mapred.TextInputFormat
>                   output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HIVE-582) "SELECT count(1) as cnt from src x" not working for column pruning

Posted by "Yongqiang He (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HIVE-582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12723988#action_12723988 ] 

Yongqiang He commented on HIVE-582:
-----------------------------------

It seems that the wrong plan is generated when setting hive.map.aggr=false, and the reduce output nothing(because of the column pruning). Reducer got nothing, so the exception occured.

> "SELECT count(1) as cnt from src x" not working for column pruning
> ------------------------------------------------------------------
>
>                 Key: HIVE-582
>                 URL: https://issues.apache.org/jira/browse/HIVE-582
>             Project: Hadoop Hive
>          Issue Type: Bug
>            Reporter: Yongqiang He
>            Assignee: Yongqiang He
>
> The query "SELECT count(1) as cnt from src x"  has some problems, even though nullgroup*.q passed in tests. 
> Will check and solve it in this jira.
> A wrong plan sometimes can be generated due to some unknow factor:
> Wrong plan:
> {noformat}
> STAGE PLANS:
>   Stage: Stage-1
>     Map Reduce
>       Alias -> Map Operator Tree:
>         x 
>             Select Operator
>               Reduce Output Operator
>                 sort order: 
>                 tag: -1
>       Reduce Operator Tree:
>         Group By Operator
>           aggregations:
>                 expr: count(VALUE._col0)
>           mode: complete
>           Select Operator
>             expressions:
>                   expr: _col0
>                   type: bigint
>             File Output Operator
>               compressed: false
>               GlobalTableId: 0
>               table:
>                   input format: org.apache.hadoop.mapred.TextInputFormat
>                   output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
>   Stage: Stage-0
>     Fetch Operator
>       limit: -1
> {noformat}
> Without unknow factor, correct plan can be generated:
> {noformat}
> STAGE PLANS:
>   Stage: Stage-1
>     Map Reduce
>       Alias -> Map Operator Tree:
>         x 
>             Select Operator
>               Group By Operator
>                 aggregations:
>                       expr: count(1)
>                 mode: hash
>                 Reduce Output Operator
>                   sort order: 
>                   tag: -1
>                   value expressions:
>                         expr: _col0
>                         type: bigint
>       Reduce Operator Tree:
>         Group By Operator
>           aggregations:
>                 expr: count(VALUE._col0)
>           mode: mergepartial
>           Select Operator
>             expressions:
>                   expr: _col0
>                   type: bigint
>             File Output Operator
>               compressed: false
>               GlobalTableId: 0
>               table:
>                   input format: org.apache.hadoop.mapred.TextInputFormat
>                   output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.