[jira] [Created] (IMPALA-10193) Limit the memory usage of the whole mini-cluster

["Fifteen (Jira)" <ji...@apache.org>]

Fifteen created IMPALA-10193:
--------------------------------

             Summary: Limit the memory usage of the whole mini-cluster
                 Key: IMPALA-10193
                 URL: https://issues.apache.org/jira/browse/IMPALA-10193
             Project: IMPALA
          Issue Type: Bug
          Components: Infrastructure
    Affects Versions: Impala 3.4.0
            Reporter: Fifteen


The mini-cluster contains 3 virtual nodes, and all of them runs in a single 'Machine'. By quoting, it implies the machine can be a docker container. If the container is started with `-priviledged` and the actual memory is limited by CGROUPS, then the total memory in `top` and the actual available memory can be different! 

 

For example, in the container below, `top` tells us the total memory is 128GB, while the total memory set in CGROUPS is actually 32GB. If the acutal mem usage exceeds 32GB, process (such as impalad, hivemaster2 etc.) get killed.

!image-2020-09-28-17-08-02-604.png!

 

So we may need a way to limit the whole mini-cluster's memory usage.

   



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org