You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@impala.apache.org by "Fifteen (Jira)" <ji...@apache.org> on 2020/10/12 03:12:00 UTC

[jira] [Resolved] (IMPALA-10193) Limit the memory usage of the whole mini-cluster

     [ https://issues.apache.org/jira/browse/IMPALA-10193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Fifteen resolved IMPALA-10193.
------------------------------
    Resolution: Fixed

> Limit the memory usage of the whole mini-cluster
> ------------------------------------------------
>
>                 Key: IMPALA-10193
>                 URL: https://issues.apache.org/jira/browse/IMPALA-10193
>             Project: IMPALA
>          Issue Type: Bug
>          Components: Infrastructure
>    Affects Versions: Impala 3.4.0
>            Reporter: Fifteen
>            Assignee: Fifteen
>            Priority: Minor
>         Attachments: image-2020-09-28-17-18-15-358.png
>
>
> The mini-cluster contains 3 virtual nodes, and all of them runs in a single 'Machine'. By quoting, it implies the machine can be a docker container. If the container is started with `-priviledged` and the actual memory is limited by CGROUPS, then the total memory in `htop` and the actual available memory can be different! 
>  
> For example, in the container below, `htop` tells us the total memory is 128GB, while the total memory set in CGROUPS is actually 32GB. If the acutal mem usage exceeds 32GB, process (such as impalad, hivemaster2 etc.) get killed.
>   !image-2020-09-28-17-18-15-358.png!
>  
> So we may need a way to limit the whole mini-cluster's memory usage.
>    



--
This message was sent by Atlassian Jira
(v8.3.4#803005)