You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@hive.apache.org by "Panagiotis Garefalakis (Jira)" <ji...@apache.org> on 2020/04/09 15:31:00 UTC

[jira] [Created] (HIVE-23166) Protect VGB from flushing too often

Panagiotis Garefalakis created HIVE-23166:
---------------------------------------------

             Summary: Protect VGB from flushing too often
                 Key: HIVE-23166
                 URL: https://issues.apache.org/jira/browse/HIVE-23166
             Project: Hive
          Issue Type: Improvement
          Components: llap
    Affects Versions: 4.0.0
            Reporter: Panagiotis Garefalakis
            Assignee: Panagiotis Garefalakis


The existing flush logic in our VectorGroupByOperator is completely static.
 It depends on the: number of HtEntries (*hive.vectorized.groupby.maxentries*) and the MAX memory threshold (by default 90% of available memory)
 
Assuming that we are not memory constrained the periodicity of flushing is currently dictated by the static number of entries (1M by default) which can be also misconfigured to a very low value.

I am proposing along with maxHtEntries, to also take into account current memory usage, to avoid flushing too ofter as it can hurt op throughput for particular workloads.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)