[jira] [Updated] (IMPALA-3563) Evaluate using global opposed to per partition statistics

["Tim Armstrong (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IMPALA-3563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tim Armstrong updated IMPALA-3563:
----------------------------------
    External issue ID:   (was: IMPALA-3563)
               Parent:     (was: IMPALA-2649)
           Issue Type: Bug  (was: Sub-task)

> Evaluate using global opposed to per partition statistics
> ---------------------------------------------------------
>
>                 Key: IMPALA-3563
>                 URL: https://issues.apache.org/jira/browse/IMPALA-3563
>             Project: IMPALA
>          Issue Type: Bug
>          Components: Catalog
>    Affects Versions: Impala 2.5.0
>            Reporter: Mostafa Mokhtar
>            Priority: Major
>              Labels: catalog-server, planner, scalability
>
> Impala and other SQL on Hadoop solutions use per partition statistics which creates a metadata scalability problem which I reckon outweighs benefits of having more accurate statistics. 
> This is the proposal is for a partitioned table :
> * "Compute statistics" computes and stores per partition HLL same as before
> * Catalog merges the HLL(s) for all partitions and stores/persists global statistics 
> * Impalad(s) never request per partition statics only global stats 
> * The only time the catalog needs to read the per partition HLL is when regenerating the global stats as part of adding/removing partitions
> In other words during planning the partitioned table looks very similar to a non-partitioned table.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org