You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Yongqiang He (JIRA)" <ji...@apache.org> on 2009/06/25 11:37:07 UTC
[jira] Created: (HIVE-582) "SELECT count(1) as cnt from src x" not
working for column pruning
"SELECT count(1) as cnt from src x" not working for column pruning
------------------------------------------------------------------
Key: HIVE-582
URL: https://issues.apache.org/jira/browse/HIVE-582
Project: Hadoop Hive
Issue Type: Bug
Reporter: Yongqiang He
Assignee: Yongqiang He
The query "SELECT count(1) as cnt from src x" has some problems, even though nullgroup*.q passed in tests.
Will check and solve it in this jira.
A wrong plan sometimes can be generated due to some unknow factor:
Wrong plan:
{noformat}
STAGE PLANS:
Stage: Stage-1
Map Reduce
Alias -> Map Operator Tree:
x
Select Operator
Reduce Output Operator
sort order:
tag: -1
Reduce Operator Tree:
Group By Operator
aggregations:
expr: count(VALUE._col0)
mode: complete
Select Operator
expressions:
expr: _col0
type: bigint
File Output Operator
compressed: false
GlobalTableId: 0
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
Stage: Stage-0
Fetch Operator
limit: -1
{noformat}
Without unknow factor, correct plan can be generated:
{noformat}
STAGE PLANS:
Stage: Stage-1
Map Reduce
Alias -> Map Operator Tree:
x
Select Operator
Group By Operator
aggregations:
expr: count(1)
mode: hash
Reduce Output Operator
sort order:
tag: -1
value expressions:
expr: _col0
type: bigint
Reduce Operator Tree:
Group By Operator
aggregations:
expr: count(VALUE._col0)
mode: mergepartial
Select Operator
expressions:
expr: _col0
type: bigint
File Output Operator
compressed: false
GlobalTableId: 0
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
{noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-582) "SELECT count(1) as cnt from src x" not
working for column pruning
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-582:
----------------------------
Resolution: Fixed
Fix Version/s: 0.4.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Committed. Thanks Yongqiang
> "SELECT count(1) as cnt from src x" not working for column pruning
> ------------------------------------------------------------------
>
> Key: HIVE-582
> URL: https://issues.apache.org/jira/browse/HIVE-582
> Project: Hadoop Hive
> Issue Type: Bug
> Reporter: Yongqiang He
> Assignee: Yongqiang He
> Fix For: 0.4.0
>
> Attachments: hive-582-2009-06-25.patch
>
>
> The query "SELECT count(1) as cnt from src x" has some problems, even though nullgroup*.q passed in tests.
> Will check and solve it in this jira.
> A wrong plan sometimes can be generated due to some unknow factor:
> Wrong plan:
> {noformat}
> STAGE PLANS:
> Stage: Stage-1
> Map Reduce
> Alias -> Map Operator Tree:
> x
> Select Operator
> Reduce Output Operator
> sort order:
> tag: -1
> Reduce Operator Tree:
> Group By Operator
> aggregations:
> expr: count(VALUE._col0)
> mode: complete
> Select Operator
> expressions:
> expr: _col0
> type: bigint
> File Output Operator
> compressed: false
> GlobalTableId: 0
> table:
> input format: org.apache.hadoop.mapred.TextInputFormat
> output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> Stage: Stage-0
> Fetch Operator
> limit: -1
> {noformat}
> Without unknow factor, correct plan can be generated:
> {noformat}
> STAGE PLANS:
> Stage: Stage-1
> Map Reduce
> Alias -> Map Operator Tree:
> x
> Select Operator
> Group By Operator
> aggregations:
> expr: count(1)
> mode: hash
> Reduce Output Operator
> sort order:
> tag: -1
> value expressions:
> expr: _col0
> type: bigint
> Reduce Operator Tree:
> Group By Operator
> aggregations:
> expr: count(VALUE._col0)
> mode: mergepartial
> Select Operator
> expressions:
> expr: _col0
> type: bigint
> File Output Operator
> compressed: false
> GlobalTableId: 0
> table:
> input format: org.apache.hadoop.mapred.TextInputFormat
> output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-582) "SELECT count(1) as cnt from src x"
not working for column pruning
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12725364#action_12725364 ]
Namit Jain commented on HIVE-582:
---------------------------------
looks good
> "SELECT count(1) as cnt from src x" not working for column pruning
> ------------------------------------------------------------------
>
> Key: HIVE-582
> URL: https://issues.apache.org/jira/browse/HIVE-582
> Project: Hadoop Hive
> Issue Type: Bug
> Reporter: Yongqiang He
> Assignee: Yongqiang He
> Attachments: hive-582-2009-06-25.patch
>
>
> The query "SELECT count(1) as cnt from src x" has some problems, even though nullgroup*.q passed in tests.
> Will check and solve it in this jira.
> A wrong plan sometimes can be generated due to some unknow factor:
> Wrong plan:
> {noformat}
> STAGE PLANS:
> Stage: Stage-1
> Map Reduce
> Alias -> Map Operator Tree:
> x
> Select Operator
> Reduce Output Operator
> sort order:
> tag: -1
> Reduce Operator Tree:
> Group By Operator
> aggregations:
> expr: count(VALUE._col0)
> mode: complete
> Select Operator
> expressions:
> expr: _col0
> type: bigint
> File Output Operator
> compressed: false
> GlobalTableId: 0
> table:
> input format: org.apache.hadoop.mapred.TextInputFormat
> output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> Stage: Stage-0
> Fetch Operator
> limit: -1
> {noformat}
> Without unknow factor, correct plan can be generated:
> {noformat}
> STAGE PLANS:
> Stage: Stage-1
> Map Reduce
> Alias -> Map Operator Tree:
> x
> Select Operator
> Group By Operator
> aggregations:
> expr: count(1)
> mode: hash
> Reduce Output Operator
> sort order:
> tag: -1
> value expressions:
> expr: _col0
> type: bigint
> Reduce Operator Tree:
> Group By Operator
> aggregations:
> expr: count(VALUE._col0)
> mode: mergepartial
> Select Operator
> expressions:
> expr: _col0
> type: bigint
> File Output Operator
> compressed: false
> GlobalTableId: 0
> table:
> input format: org.apache.hadoop.mapred.TextInputFormat
> output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-582) "SELECT count(1) as cnt from src x" not
working for column pruning
Posted by "Yongqiang He (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yongqiang He updated HIVE-582:
------------------------------
Attachment: hive-582-2009-06-25.patch
> "SELECT count(1) as cnt from src x" not working for column pruning
> ------------------------------------------------------------------
>
> Key: HIVE-582
> URL: https://issues.apache.org/jira/browse/HIVE-582
> Project: Hadoop Hive
> Issue Type: Bug
> Reporter: Yongqiang He
> Assignee: Yongqiang He
> Attachments: hive-582-2009-06-25.patch
>
>
> The query "SELECT count(1) as cnt from src x" has some problems, even though nullgroup*.q passed in tests.
> Will check and solve it in this jira.
> A wrong plan sometimes can be generated due to some unknow factor:
> Wrong plan:
> {noformat}
> STAGE PLANS:
> Stage: Stage-1
> Map Reduce
> Alias -> Map Operator Tree:
> x
> Select Operator
> Reduce Output Operator
> sort order:
> tag: -1
> Reduce Operator Tree:
> Group By Operator
> aggregations:
> expr: count(VALUE._col0)
> mode: complete
> Select Operator
> expressions:
> expr: _col0
> type: bigint
> File Output Operator
> compressed: false
> GlobalTableId: 0
> table:
> input format: org.apache.hadoop.mapred.TextInputFormat
> output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> Stage: Stage-0
> Fetch Operator
> limit: -1
> {noformat}
> Without unknow factor, correct plan can be generated:
> {noformat}
> STAGE PLANS:
> Stage: Stage-1
> Map Reduce
> Alias -> Map Operator Tree:
> x
> Select Operator
> Group By Operator
> aggregations:
> expr: count(1)
> mode: hash
> Reduce Output Operator
> sort order:
> tag: -1
> value expressions:
> expr: _col0
> type: bigint
> Reduce Operator Tree:
> Group By Operator
> aggregations:
> expr: count(VALUE._col0)
> mode: mergepartial
> Select Operator
> expressions:
> expr: _col0
> type: bigint
> File Output Operator
> compressed: false
> GlobalTableId: 0
> table:
> input format: org.apache.hadoop.mapred.TextInputFormat
> output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-582) "SELECT count(1) as cnt from src x" not
working for column pruning
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated HIVE-582:
----------------------------
Status: Patch Available (was: Open)
> "SELECT count(1) as cnt from src x" not working for column pruning
> ------------------------------------------------------------------
>
> Key: HIVE-582
> URL: https://issues.apache.org/jira/browse/HIVE-582
> Project: Hadoop Hive
> Issue Type: Bug
> Reporter: Yongqiang He
> Assignee: Yongqiang He
> Attachments: hive-582-2009-06-25.patch
>
>
> The query "SELECT count(1) as cnt from src x" has some problems, even though nullgroup*.q passed in tests.
> Will check and solve it in this jira.
> A wrong plan sometimes can be generated due to some unknow factor:
> Wrong plan:
> {noformat}
> STAGE PLANS:
> Stage: Stage-1
> Map Reduce
> Alias -> Map Operator Tree:
> x
> Select Operator
> Reduce Output Operator
> sort order:
> tag: -1
> Reduce Operator Tree:
> Group By Operator
> aggregations:
> expr: count(VALUE._col0)
> mode: complete
> Select Operator
> expressions:
> expr: _col0
> type: bigint
> File Output Operator
> compressed: false
> GlobalTableId: 0
> table:
> input format: org.apache.hadoop.mapred.TextInputFormat
> output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> Stage: Stage-0
> Fetch Operator
> limit: -1
> {noformat}
> Without unknow factor, correct plan can be generated:
> {noformat}
> STAGE PLANS:
> Stage: Stage-1
> Map Reduce
> Alias -> Map Operator Tree:
> x
> Select Operator
> Group By Operator
> aggregations:
> expr: count(1)
> mode: hash
> Reduce Output Operator
> sort order:
> tag: -1
> value expressions:
> expr: _col0
> type: bigint
> Reduce Operator Tree:
> Group By Operator
> aggregations:
> expr: count(VALUE._col0)
> mode: mergepartial
> Select Operator
> expressions:
> expr: _col0
> type: bigint
> File Output Operator
> compressed: false
> GlobalTableId: 0
> table:
> input format: org.apache.hadoop.mapred.TextInputFormat
> output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-582) "SELECT count(1) as cnt from src x"
not working for column pruning
Posted by "Yongqiang He (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12723988#action_12723988 ]
Yongqiang He commented on HIVE-582:
-----------------------------------
It seems that the wrong plan is generated when setting hive.map.aggr=false, and the reduce output nothing(because of the column pruning). Reducer got nothing, so the exception occured.
> "SELECT count(1) as cnt from src x" not working for column pruning
> ------------------------------------------------------------------
>
> Key: HIVE-582
> URL: https://issues.apache.org/jira/browse/HIVE-582
> Project: Hadoop Hive
> Issue Type: Bug
> Reporter: Yongqiang He
> Assignee: Yongqiang He
>
> The query "SELECT count(1) as cnt from src x" has some problems, even though nullgroup*.q passed in tests.
> Will check and solve it in this jira.
> A wrong plan sometimes can be generated due to some unknow factor:
> Wrong plan:
> {noformat}
> STAGE PLANS:
> Stage: Stage-1
> Map Reduce
> Alias -> Map Operator Tree:
> x
> Select Operator
> Reduce Output Operator
> sort order:
> tag: -1
> Reduce Operator Tree:
> Group By Operator
> aggregations:
> expr: count(VALUE._col0)
> mode: complete
> Select Operator
> expressions:
> expr: _col0
> type: bigint
> File Output Operator
> compressed: false
> GlobalTableId: 0
> table:
> input format: org.apache.hadoop.mapred.TextInputFormat
> output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> Stage: Stage-0
> Fetch Operator
> limit: -1
> {noformat}
> Without unknow factor, correct plan can be generated:
> {noformat}
> STAGE PLANS:
> Stage: Stage-1
> Map Reduce
> Alias -> Map Operator Tree:
> x
> Select Operator
> Group By Operator
> aggregations:
> expr: count(1)
> mode: hash
> Reduce Output Operator
> sort order:
> tag: -1
> value expressions:
> expr: _col0
> type: bigint
> Reduce Operator Tree:
> Group By Operator
> aggregations:
> expr: count(VALUE._col0)
> mode: mergepartial
> Select Operator
> expressions:
> expr: _col0
> type: bigint
> File Output Operator
> compressed: false
> GlobalTableId: 0
> table:
> input format: org.apache.hadoop.mapred.TextInputFormat
> output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.