[jira] [Commented] (IMPALA-7353) Fix bogus too-high memory estimates

["ASF subversion and git services (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-7353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16631016#comment-16631016 ] 

ASF subversion and git services commented on IMPALA-7353:
---------------------------------------------------------

Commit 1c94450ca92606fb6b708de2ea07445cc6610dbf in impala's branch refs/heads/master from [~bikram.sngh91]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=1c94450 ]

IMPALA-7353: Improve memory estimates for Hbase Scan Nodes

Currently for hbase scan nodes we use a constant estimate of 1GB which
is generally a gross over-estimation. This patch improves upon those
estimates by using huerestics based on how hbase rows are stored and
fetched and how the scanners interact with the internal memory pool.

Testing:
Added/Modified resource requirements planner test.
Added a junit test for the estimation logic.

Change-Id: I583545c3f5e454854f111871c5fbc4f108ae4bff
Reviewed-on: http://gerrit.cloudera.org:8080/11306
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>


> Fix bogus too-high memory estimates
> -----------------------------------
>
>                 Key: IMPALA-7353
>                 URL: https://issues.apache.org/jira/browse/IMPALA-7353
>             Project: IMPALA
>          Issue Type: Sub-task
>          Components: Frontend
>            Reporter: Tim Armstrong
>            Assignee: Bikramjeet Vig
>            Priority: Major
>              Labels: resource-management
>
> Some operators have bogus memory estimates that are probably way too high. We should adjust them to be a little more realistic.
> E.g. in HBaseScanNode:
> {code}
>   @Override
>   public void computeNodeResourceProfile(TQueryOptions queryOptions) {
>     // TODO: What's a good estimate of memory consumption?
>     nodeResourceProfile_ =  ResourceProfile.noReservation(1024L * 1024L * 1024L);
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org