[jira] [Created] (IMPALA-9765) Preaggregations should stop expanding if memory is scarce

["Tim Armstrong (Jira)" <ji...@apache.org>]

Tim Armstrong created IMPALA-9765:
-------------------------------------

             Summary: Preaggregations should stop expanding if memory is scarce
                 Key: IMPALA-9765
                 URL: https://issues.apache.org/jira/browse/IMPALA-9765
             Project: IMPALA
          Issue Type: Improvement
          Components: Backend
            Reporter: Tim Armstrong
            Assignee: Tim Armstrong


We've seen several workloads recently where the preaggregations expanded to consume nearly all of the memory reservation on a host, starving the merge agg. We should think about runtime mechanisms to reduce the impact of this, e.g. stop expanding once we are low on memory.

I think we'd need to do this based on reservation_limit_ - child_reservations_  - used_reservation_ on the query mem tracker. Various heuristics are possible. Even something simple like keeping 10-20% free would be beneficial.

Related: IMPALA-2945 IMPALA-9530  IMPALA-9591 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)