[jira] [Commented] (IMPALA-10377) Improve the accuracy of resource estimation

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-10377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17300682#comment-17300682 ] 

ASF subversion and git services commented on IMPALA-10377:
----------------------------------------------------------

Commit 1a01bfe831b548204cde8087def51a2f27b40cb4 in impala's branch refs/heads/master from liuyao
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=1a01bfe ]

IMPALA-10377: Improve the accuracy of resource estimation

PlanNode does not consider some factors when estimating memory,
this will cause a large error rate

AggregationNode
1.MemoryEstimate = Ndv * (AvgRowSize + SizeOfBucket)
2.When estimating the Ndv of merge aggregation, Ndv should be
  divided only once.
3.If there is no grouping exprs, MemoryEstimate =
  MIN_PLAIN_AGG_MEM

SortNode
1.MemoryEstimate = Cardinality * AvgRowSize. Memory used when
  there is enough memory

HashJoinNode
1.MemoryEstimate= DataRows + Buckets + DuplicateNodes,
  DataRows = RightTableCardinality * AvgRowSize,
  Buckets= roundUpToPowerOf2(RightTableCardinality) *
           SizeOfBucket,
  DuplicateNodes = (RightTableCardinality - RightNdv) *
                    SizeOfDuplicateNode

KuduScanNode
1.MemoryEstimate = Columns * BytesPerColumn * MaxScannerThreads,
  Columns are scanned in query, not all the columns of the table

UnitTest
1.CardinalityTest adds test cases to test memory estimation.
  Modify existing test cases related to memory estimation

Change-Id: Ic01db168ff2c6d6de33ee553a8175599f035d7a1
Reviewed-on: http://gerrit.cloudera.org:8080/16842
Reviewed-by: Zoltan Borok-Nagy <bo...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>


> Improve the accuracy of resource estimation
> -------------------------------------------
>
>                 Key: IMPALA-10377
>                 URL: https://issues.apache.org/jira/browse/IMPALA-10377
>             Project: IMPALA
>          Issue Type: Improvement
>          Components: Frontend
>    Affects Versions: Impala 3.4.0
>            Reporter: liuyao
>            Assignee: liuyao
>            Priority: Major
>              Labels: estimate, memory, statistics
>   Original Estimate: 120h
>  Remaining Estimate: 120h
>
> PlanNode does not consider some factors when estimating memory, this will cause a large error rate
>  
> AggregationNode
>  
> 1.The memory occupied by hash table's own data structure is not considered. Hash table inserts a new value, which will add a bucket. The size of a bucket is 16 bytes.
> 2.When estimating the NDV of merge aggregation, if there are multiple grouping exprs, it may be divided by the number of Fragment Instances several times, and it should be divided only once.
> 3.When estimating the NDV of merge aggregation, and there are multiple grouping exprs, the estimated memory is much smaller than the actual use.
> 4.If there is no grouping exprs, the estimated memory is much larger than the actual use.
> 5.If the NDV of grouping exprs is very small, the estimated memory is much larger than the actual use.
>  
> SortNode
> 1.Estimate the memory usage of external sort. the estimated memory is much smaller than the actual use.
>  
>  
> HashJoinNode
> 1.The memory occupied by hash table's own data structure is not considered.Hash Table will keep duplicate data, so the size of DuplicateNode should be considered.
> 2.Hash table will create multiple buckets in advance. The size of these buckets should be considered.
>  
> KuduScanNode
> 1.Estimate memory by scanning all columns,the estimated memory is much larger than the actual use.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org