You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Ning Zhang (JIRA)" <ji...@apache.org> on 2010/09/17 00:26:32 UTC
[jira] Created: (HIVE-1648) Automatically gathering stats when
reading a table/partition
Automatically gathering stats when reading a table/partition
------------------------------------------------------------
Key: HIVE-1648
URL: https://issues.apache.org/jira/browse/HIVE-1648
Project: Hadoop Hive
Issue Type: New Feature
Reporter: Ning Zhang
HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to gathering stats. This requires additional scan of the data. Stats gathering can be piggy-backed on TableScanOperator whenever a table/partition is scanned (given not LIMIT operator).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1648) Automatically gathering stats when
reading a table/partition
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12928709#action_12928709 ]
Namit Jain commented on HIVE-1648:
----------------------------------
Can you regenerate the patch - it conflicted with HIVE-1750 and may need some cleanup.
> Automatically gathering stats when reading a table/partition
> ------------------------------------------------------------
>
> Key: HIVE-1648
> URL: https://issues.apache.org/jira/browse/HIVE-1648
> Project: Hive
> Issue Type: Sub-task
> Reporter: Ning Zhang
> Attachments: HIVE-1648.patch
>
>
> HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to gathering stats. This requires additional scan of the data. Stats gathering can be piggy-backed on TableScanOperator whenever a table/partition is scanned (given not LIMIT operator).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1648) Automatically gathering stats when
reading a table/partition
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12928733#action_12928733 ]
Namit Jain commented on HIVE-1648:
----------------------------------
getOuterQueryLimit is not used correctly.
qb maintains a mapping of limit per destination
case HiveParser.TOK_LIMIT:
qbp.setDestLimit(ctx_1.dest, new Integer(ast.getChild(0).getText()));
break;
Since you are browsing the input tables, you dont care about the destination.
Go over:
private final HashMap<String, Integer> destToLimit;
in QBParseInfo.
If that is not-empty, do not gather stats
> Automatically gathering stats when reading a table/partition
> ------------------------------------------------------------
>
> Key: HIVE-1648
> URL: https://issues.apache.org/jira/browse/HIVE-1648
> Project: Hive
> Issue Type: Sub-task
> Reporter: Ning Zhang
> Attachments: HIVE-1648.patch
>
>
> HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to gathering stats. This requires additional scan of the data. Stats gathering can be piggy-backed on TableScanOperator whenever a table/partition is scanned (given not LIMIT operator).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1648) Automatically gathering stats when
reading a table/partition
Posted by "Paul Butler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Paul Butler updated HIVE-1648:
------------------------------
Attachment: HIVE-1648.2.patch
> Automatically gathering stats when reading a table/partition
> ------------------------------------------------------------
>
> Key: HIVE-1648
> URL: https://issues.apache.org/jira/browse/HIVE-1648
> Project: Hive
> Issue Type: Sub-task
> Reporter: Ning Zhang
> Assignee: Paul Butler
> Attachments: HIVE-1648.2.patch, HIVE-1648.patch
>
>
> HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to gathering stats. This requires additional scan of the data. Stats gathering can be piggy-backed on TableScanOperator whenever a table/partition is scanned (given not LIMIT operator).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HIVE-1648) Automatically gathering stats when
reading a table/partition
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Sichi reassigned HIVE-1648:
--------------------------------
Assignee: Paul Butler
> Automatically gathering stats when reading a table/partition
> ------------------------------------------------------------
>
> Key: HIVE-1648
> URL: https://issues.apache.org/jira/browse/HIVE-1648
> Project: Hive
> Issue Type: Sub-task
> Reporter: Ning Zhang
> Assignee: Paul Butler
> Attachments: HIVE-1648.patch
>
>
> HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to gathering stats. This requires additional scan of the data. Stats gathering can be piggy-backed on TableScanOperator whenever a table/partition is scanned (given not LIMIT operator).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1648) Automatically gathering stats when
reading a table/partition
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12928737#action_12928737 ]
Namit Jain commented on HIVE-1648:
----------------------------------
Also, add tests for all the cases above.
You should be able to compare the plan via a explain extended
> Automatically gathering stats when reading a table/partition
> ------------------------------------------------------------
>
> Key: HIVE-1648
> URL: https://issues.apache.org/jira/browse/HIVE-1648
> Project: Hive
> Issue Type: Sub-task
> Reporter: Ning Zhang
> Attachments: HIVE-1648.patch
>
>
> HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to gathering stats. This requires additional scan of the data. Stats gathering can be piggy-backed on TableScanOperator whenever a table/partition is scanned (given not LIMIT operator).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1648) Automatically gathering stats when
reading a table/partition
Posted by "Paul Butler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Paul Butler updated HIVE-1648:
------------------------------
Attachment: (was: hadoop-6974.2.patch)
> Automatically gathering stats when reading a table/partition
> ------------------------------------------------------------
>
> Key: HIVE-1648
> URL: https://issues.apache.org/jira/browse/HIVE-1648
> Project: Hive
> Issue Type: Sub-task
> Reporter: Ning Zhang
> Assignee: Paul Butler
> Attachments: HIVE-1648.2.patch, HIVE-1648.patch
>
>
> HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to gathering stats. This requires additional scan of the data. Stats gathering can be piggy-backed on TableScanOperator whenever a table/partition is scanned (given not LIMIT operator).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1648) Automatically gathering stats when
reading a table/partition
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12928804#action_12928804 ]
John Sichi commented on HIVE-1648:
----------------------------------
Obtained Paul Butler's account name from the JIRA change history...I'll add him as contributor.
> Automatically gathering stats when reading a table/partition
> ------------------------------------------------------------
>
> Key: HIVE-1648
> URL: https://issues.apache.org/jira/browse/HIVE-1648
> Project: Hive
> Issue Type: Sub-task
> Reporter: Ning Zhang
> Attachments: HIVE-1648.patch
>
>
> HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to gathering stats. This requires additional scan of the data. Stats gathering can be piggy-backed on TableScanOperator whenever a table/partition is scanned (given not LIMIT operator).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1648) Automatically gathering stats when
reading a table/partition
Posted by "Paul Butler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12930326#action_12930326 ]
Paul Butler commented on HIVE-1648:
-----------------------------------
I get a bunch of tests failing when I build the latest trunk, even without applying my patch. I'm trying to figure out what's wrong with those first.
> Automatically gathering stats when reading a table/partition
> ------------------------------------------------------------
>
> Key: HIVE-1648
> URL: https://issues.apache.org/jira/browse/HIVE-1648
> Project: Hive
> Issue Type: Sub-task
> Reporter: Ning Zhang
> Assignee: Paul Butler
> Attachments: HIVE-1648.2.patch, HIVE-1648.patch
>
>
> HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to gathering stats. This requires additional scan of the data. Stats gathering can be piggy-backed on TableScanOperator whenever a table/partition is scanned (given not LIMIT operator).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1648) Automatically gathering stats when
reading a table/partition
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12928706#action_12928706 ]
Namit Jain commented on HIVE-1648:
----------------------------------
Paul, do you have an apache account ?
Can you send it to me - I will add you as a hive contributor.
Also, can you 'Submit Patch' when you add a patch - this way everyone knows that the patch is ready for review
> Automatically gathering stats when reading a table/partition
> ------------------------------------------------------------
>
> Key: HIVE-1648
> URL: https://issues.apache.org/jira/browse/HIVE-1648
> Project: Hive
> Issue Type: Sub-task
> Reporter: Ning Zhang
> Attachments: HIVE-1648.patch
>
>
> HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to gathering stats. This requires additional scan of the data. Stats gathering can be piggy-backed on TableScanOperator whenever a table/partition is scanned (given not LIMIT operator).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1648) Automatically gathering stats when
reading a table/partition
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ning Zhang updated HIVE-1648:
-----------------------------
Parent: HIVE-33
Issue Type: Sub-task (was: New Feature)
> Automatically gathering stats when reading a table/partition
> ------------------------------------------------------------
>
> Key: HIVE-1648
> URL: https://issues.apache.org/jira/browse/HIVE-1648
> Project: Hadoop Hive
> Issue Type: Sub-task
> Reporter: Ning Zhang
>
> HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to gathering stats. This requires additional scan of the data. Stats gathering can be piggy-backed on TableScanOperator whenever a table/partition is scanned (given not LIMIT operator).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1648) Automatically gathering stats when
reading a table/partition
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12930321#action_12930321 ]
Namit Jain commented on HIVE-1648:
----------------------------------
Paul, any updates on the unit tests ?
> Automatically gathering stats when reading a table/partition
> ------------------------------------------------------------
>
> Key: HIVE-1648
> URL: https://issues.apache.org/jira/browse/HIVE-1648
> Project: Hive
> Issue Type: Sub-task
> Reporter: Ning Zhang
> Assignee: Paul Butler
> Attachments: HIVE-1648.2.patch, HIVE-1648.patch
>
>
> HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to gathering stats. This requires additional scan of the data. Stats gathering can be piggy-backed on TableScanOperator whenever a table/partition is scanned (given not LIMIT operator).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1648) Automatically gathering stats when
reading a table/partition
Posted by "Paul Butler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12928836#action_12928836 ]
Paul Butler commented on HIVE-1648:
-----------------------------------
Attached a new patch which applies on top of HIVE-1750. Also fixed the limits as suggested. Working on unit tests now.
-- Paul
> Automatically gathering stats when reading a table/partition
> ------------------------------------------------------------
>
> Key: HIVE-1648
> URL: https://issues.apache.org/jira/browse/HIVE-1648
> Project: Hive
> Issue Type: Sub-task
> Reporter: Ning Zhang
> Assignee: Paul Butler
> Attachments: HIVE-1648.2.patch, HIVE-1648.patch
>
>
> HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to gathering stats. This requires additional scan of the data. Stats gathering can be piggy-backed on TableScanOperator whenever a table/partition is scanned (given not LIMIT operator).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1648) Automatically gathering stats when
reading a table/partition
Posted by "Paul Butler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Paul Butler updated HIVE-1648:
------------------------------
Attachment: HIVE-1648.patch
> Automatically gathering stats when reading a table/partition
> ------------------------------------------------------------
>
> Key: HIVE-1648
> URL: https://issues.apache.org/jira/browse/HIVE-1648
> Project: Hive
> Issue Type: Sub-task
> Reporter: Ning Zhang
> Attachments: HIVE-1648.patch
>
>
> HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to gathering stats. This requires additional scan of the data. Stats gathering can be piggy-backed on TableScanOperator whenever a table/partition is scanned (given not LIMIT operator).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1648) Automatically gathering stats when
reading a table/partition
Posted by "Paul Butler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Paul Butler updated HIVE-1648:
------------------------------
Attachment: hadoop-6974.2.patch
> Automatically gathering stats when reading a table/partition
> ------------------------------------------------------------
>
> Key: HIVE-1648
> URL: https://issues.apache.org/jira/browse/HIVE-1648
> Project: Hive
> Issue Type: Sub-task
> Reporter: Ning Zhang
> Assignee: Paul Butler
> Attachments: HIVE-1648.2.patch, HIVE-1648.patch
>
>
> HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to gathering stats. This requires additional scan of the data. Stats gathering can be piggy-backed on TableScanOperator whenever a table/partition is scanned (given not LIMIT operator).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.