[jira] [Created] (IMPALA-5737) Constrain and reduce memory requirements of minicluster and test processes

["Matthew Jacobs (JIRA)" <ji...@apache.org>]

Matthew Jacobs created IMPALA-5737:
--------------------------------------

             Summary: Constrain and reduce memory requirements of minicluster and test processes
                 Key: IMPALA-5737
                 URL: https://issues.apache.org/jira/browse/IMPALA-5737
             Project: IMPALA
          Issue Type: Improvement
          Components: Infrastructure
    Affects Versions: Impala 2.10.0
            Reporter: Matthew Jacobs


The memory used by minicluster daemons and our test processes are not well constrained, which can lead to processes being killed by the OOM killer (e.g. in IMPALA-5733). Some processes have explicit memory limits (e.g. Kudu daemons have a 2gb limit), but Impala daemons consume an arbitrary amount (80% of memory). We need to also look at HDFS, HBase, Hive, etc. to make sure the limits are set appropriately. Further, it seems that we've never considered the memory used by test processes, e.g. python pytests, which spawn a process for each parallel worker - and this can add up especially in the minicluster stress tests. 

We need to see how we can constrain memory requirements, and look for opportunities to reduce memory requirements. This will help avoid OOMs, it may allow us to run gerrit-verify-dryrun jobs with smaller instance types, and will help reduce the requirements for dev machines which will be good for the community.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)