You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Ning Zhang (JIRA)" <ji...@apache.org> on 2010/09/17 00:26:32 UTC

[jira] Created: (HIVE-1648) Automatically gathering stats when reading a table/partition

Automatically gathering stats when reading a table/partition
------------------------------------------------------------

                 Key: HIVE-1648
                 URL: https://issues.apache.org/jira/browse/HIVE-1648
             Project: Hadoop Hive
          Issue Type: New Feature
            Reporter: Ning Zhang


HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to gathering stats. This requires additional scan of the data. Stats gathering can be piggy-backed on TableScanOperator whenever a table/partition is scanned (given not LIMIT operator). 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HIVE-1648) Automatically gathering stats when reading a table/partition

Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12928709#action_12928709 ] 

Namit Jain commented on HIVE-1648:
----------------------------------

Can you regenerate the patch - it conflicted with HIVE-1750 and may need some cleanup.

> Automatically gathering stats when reading a table/partition
> ------------------------------------------------------------
>
>                 Key: HIVE-1648
>                 URL: https://issues.apache.org/jira/browse/HIVE-1648
>             Project: Hive
>          Issue Type: Sub-task
>            Reporter: Ning Zhang
>         Attachments: HIVE-1648.patch
>
>
> HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to gathering stats. This requires additional scan of the data. Stats gathering can be piggy-backed on TableScanOperator whenever a table/partition is scanned (given not LIMIT operator). 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HIVE-1648) Automatically gathering stats when reading a table/partition

Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12928733#action_12928733 ] 

Namit Jain commented on HIVE-1648:
----------------------------------

getOuterQueryLimit is not used correctly.

qb maintains a mapping of limit per destination

      case HiveParser.TOK_LIMIT:
        qbp.setDestLimit(ctx_1.dest, new Integer(ast.getChild(0).getText()));
        break;



Since you are browsing the input tables, you dont care about the destination.
Go over:

private final HashMap<String, Integer> destToLimit;

in QBParseInfo.

If that is not-empty, do not gather stats

> Automatically gathering stats when reading a table/partition
> ------------------------------------------------------------
>
>                 Key: HIVE-1648
>                 URL: https://issues.apache.org/jira/browse/HIVE-1648
>             Project: Hive
>          Issue Type: Sub-task
>            Reporter: Ning Zhang
>         Attachments: HIVE-1648.patch
>
>
> HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to gathering stats. This requires additional scan of the data. Stats gathering can be piggy-backed on TableScanOperator whenever a table/partition is scanned (given not LIMIT operator). 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HIVE-1648) Automatically gathering stats when reading a table/partition

Posted by "Paul Butler (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Paul Butler updated HIVE-1648:
------------------------------

    Attachment: HIVE-1648.2.patch

> Automatically gathering stats when reading a table/partition
> ------------------------------------------------------------
>
>                 Key: HIVE-1648
>                 URL: https://issues.apache.org/jira/browse/HIVE-1648
>             Project: Hive
>          Issue Type: Sub-task
>            Reporter: Ning Zhang
>            Assignee: Paul Butler
>         Attachments: HIVE-1648.2.patch, HIVE-1648.patch
>
>
> HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to gathering stats. This requires additional scan of the data. Stats gathering can be piggy-backed on TableScanOperator whenever a table/partition is scanned (given not LIMIT operator). 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Assigned: (HIVE-1648) Automatically gathering stats when reading a table/partition

Posted by "John Sichi (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

John Sichi reassigned HIVE-1648:
--------------------------------

    Assignee: Paul Butler

> Automatically gathering stats when reading a table/partition
> ------------------------------------------------------------
>
>                 Key: HIVE-1648
>                 URL: https://issues.apache.org/jira/browse/HIVE-1648
>             Project: Hive
>          Issue Type: Sub-task
>            Reporter: Ning Zhang
>            Assignee: Paul Butler
>         Attachments: HIVE-1648.patch
>
>
> HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to gathering stats. This requires additional scan of the data. Stats gathering can be piggy-backed on TableScanOperator whenever a table/partition is scanned (given not LIMIT operator). 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HIVE-1648) Automatically gathering stats when reading a table/partition

Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12928737#action_12928737 ] 

Namit Jain commented on HIVE-1648:
----------------------------------

Also, add tests for all the cases above.

You should be able to compare the plan via a explain extended

> Automatically gathering stats when reading a table/partition
> ------------------------------------------------------------
>
>                 Key: HIVE-1648
>                 URL: https://issues.apache.org/jira/browse/HIVE-1648
>             Project: Hive
>          Issue Type: Sub-task
>            Reporter: Ning Zhang
>         Attachments: HIVE-1648.patch
>
>
> HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to gathering stats. This requires additional scan of the data. Stats gathering can be piggy-backed on TableScanOperator whenever a table/partition is scanned (given not LIMIT operator). 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HIVE-1648) Automatically gathering stats when reading a table/partition

Posted by "Paul Butler (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Paul Butler updated HIVE-1648:
------------------------------

    Attachment:     (was: hadoop-6974.2.patch)

> Automatically gathering stats when reading a table/partition
> ------------------------------------------------------------
>
>                 Key: HIVE-1648
>                 URL: https://issues.apache.org/jira/browse/HIVE-1648
>             Project: Hive
>          Issue Type: Sub-task
>            Reporter: Ning Zhang
>            Assignee: Paul Butler
>         Attachments: HIVE-1648.2.patch, HIVE-1648.patch
>
>
> HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to gathering stats. This requires additional scan of the data. Stats gathering can be piggy-backed on TableScanOperator whenever a table/partition is scanned (given not LIMIT operator). 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HIVE-1648) Automatically gathering stats when reading a table/partition

Posted by "John Sichi (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12928804#action_12928804 ] 

John Sichi commented on HIVE-1648:
----------------------------------

Obtained Paul Butler's account name from the JIRA change history...I'll add him as contributor.


> Automatically gathering stats when reading a table/partition
> ------------------------------------------------------------
>
>                 Key: HIVE-1648
>                 URL: https://issues.apache.org/jira/browse/HIVE-1648
>             Project: Hive
>          Issue Type: Sub-task
>            Reporter: Ning Zhang
>         Attachments: HIVE-1648.patch
>
>
> HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to gathering stats. This requires additional scan of the data. Stats gathering can be piggy-backed on TableScanOperator whenever a table/partition is scanned (given not LIMIT operator). 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HIVE-1648) Automatically gathering stats when reading a table/partition

Posted by "Paul Butler (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12930326#action_12930326 ] 

Paul Butler commented on HIVE-1648:
-----------------------------------

I get a bunch of tests failing when I build the latest trunk, even without applying my patch. I'm trying to figure out what's wrong with those first.

> Automatically gathering stats when reading a table/partition
> ------------------------------------------------------------
>
>                 Key: HIVE-1648
>                 URL: https://issues.apache.org/jira/browse/HIVE-1648
>             Project: Hive
>          Issue Type: Sub-task
>            Reporter: Ning Zhang
>            Assignee: Paul Butler
>         Attachments: HIVE-1648.2.patch, HIVE-1648.patch
>
>
> HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to gathering stats. This requires additional scan of the data. Stats gathering can be piggy-backed on TableScanOperator whenever a table/partition is scanned (given not LIMIT operator). 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HIVE-1648) Automatically gathering stats when reading a table/partition

Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12928706#action_12928706 ] 

Namit Jain commented on HIVE-1648:
----------------------------------

Paul, do you have an apache account ?
Can you send it to me - I will add you as a hive contributor.

Also, can you 'Submit Patch' when you add a patch - this way everyone knows that the patch is ready for review

> Automatically gathering stats when reading a table/partition
> ------------------------------------------------------------
>
>                 Key: HIVE-1648
>                 URL: https://issues.apache.org/jira/browse/HIVE-1648
>             Project: Hive
>          Issue Type: Sub-task
>            Reporter: Ning Zhang
>         Attachments: HIVE-1648.patch
>
>
> HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to gathering stats. This requires additional scan of the data. Stats gathering can be piggy-backed on TableScanOperator whenever a table/partition is scanned (given not LIMIT operator). 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HIVE-1648) Automatically gathering stats when reading a table/partition

Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ning Zhang updated HIVE-1648:
-----------------------------

        Parent: HIVE-33
    Issue Type: Sub-task  (was: New Feature)

> Automatically gathering stats when reading a table/partition
> ------------------------------------------------------------
>
>                 Key: HIVE-1648
>                 URL: https://issues.apache.org/jira/browse/HIVE-1648
>             Project: Hadoop Hive
>          Issue Type: Sub-task
>            Reporter: Ning Zhang
>
> HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to gathering stats. This requires additional scan of the data. Stats gathering can be piggy-backed on TableScanOperator whenever a table/partition is scanned (given not LIMIT operator). 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HIVE-1648) Automatically gathering stats when reading a table/partition

Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12930321#action_12930321 ] 

Namit Jain commented on HIVE-1648:
----------------------------------

Paul, any updates on the unit tests ?

> Automatically gathering stats when reading a table/partition
> ------------------------------------------------------------
>
>                 Key: HIVE-1648
>                 URL: https://issues.apache.org/jira/browse/HIVE-1648
>             Project: Hive
>          Issue Type: Sub-task
>            Reporter: Ning Zhang
>            Assignee: Paul Butler
>         Attachments: HIVE-1648.2.patch, HIVE-1648.patch
>
>
> HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to gathering stats. This requires additional scan of the data. Stats gathering can be piggy-backed on TableScanOperator whenever a table/partition is scanned (given not LIMIT operator). 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HIVE-1648) Automatically gathering stats when reading a table/partition

Posted by "Paul Butler (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12928836#action_12928836 ] 

Paul Butler commented on HIVE-1648:
-----------------------------------

Attached a new patch which applies on top of HIVE-1750. Also fixed the limits as suggested. Working on unit tests now.

-- Paul

> Automatically gathering stats when reading a table/partition
> ------------------------------------------------------------
>
>                 Key: HIVE-1648
>                 URL: https://issues.apache.org/jira/browse/HIVE-1648
>             Project: Hive
>          Issue Type: Sub-task
>            Reporter: Ning Zhang
>            Assignee: Paul Butler
>         Attachments: HIVE-1648.2.patch, HIVE-1648.patch
>
>
> HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to gathering stats. This requires additional scan of the data. Stats gathering can be piggy-backed on TableScanOperator whenever a table/partition is scanned (given not LIMIT operator). 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HIVE-1648) Automatically gathering stats when reading a table/partition

Posted by "Paul Butler (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Paul Butler updated HIVE-1648:
------------------------------

    Attachment: HIVE-1648.patch

> Automatically gathering stats when reading a table/partition
> ------------------------------------------------------------
>
>                 Key: HIVE-1648
>                 URL: https://issues.apache.org/jira/browse/HIVE-1648
>             Project: Hive
>          Issue Type: Sub-task
>            Reporter: Ning Zhang
>         Attachments: HIVE-1648.patch
>
>
> HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to gathering stats. This requires additional scan of the data. Stats gathering can be piggy-backed on TableScanOperator whenever a table/partition is scanned (given not LIMIT operator). 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HIVE-1648) Automatically gathering stats when reading a table/partition

Posted by "Paul Butler (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Paul Butler updated HIVE-1648:
------------------------------

    Attachment: hadoop-6974.2.patch

> Automatically gathering stats when reading a table/partition
> ------------------------------------------------------------
>
>                 Key: HIVE-1648
>                 URL: https://issues.apache.org/jira/browse/HIVE-1648
>             Project: Hive
>          Issue Type: Sub-task
>            Reporter: Ning Zhang
>            Assignee: Paul Butler
>         Attachments: HIVE-1648.2.patch, HIVE-1648.patch
>
>
> HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to gathering stats. This requires additional scan of the data. Stats gathering can be piggy-backed on TableScanOperator whenever a table/partition is scanned (given not LIMIT operator). 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.