You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Tianyi Wang (JIRA)" <ji...@apache.org> on 2018/04/30 22:36:00 UTC
[jira] [Assigned] (IMPALA-5737) Constrain and reduce memory
requirements of minicluster and test processes
[ https://issues.apache.org/jira/browse/IMPALA-5737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tianyi Wang reassigned IMPALA-5737:
-----------------------------------
Assignee: Tianyi Wang
> Constrain and reduce memory requirements of minicluster and test processes
> --------------------------------------------------------------------------
>
> Key: IMPALA-5737
> URL: https://issues.apache.org/jira/browse/IMPALA-5737
> Project: IMPALA
> Issue Type: Improvement
> Components: Infrastructure
> Affects Versions: Impala 2.10.0
> Reporter: Matthew Jacobs
> Assignee: Tianyi Wang
> Priority: Major
>
> The memory used by minicluster daemons and our test processes are not well constrained, which can lead to processes being killed by the OOM killer (e.g. in IMPALA-5733). Some processes have explicit memory limits (e.g. Kudu daemons have a 2gb limit), but Impala daemons consume an arbitrary amount (80% of memory). We need to also look at HDFS, HBase, Hive, etc. to make sure the limits are set appropriately. Further, it seems that we've never considered the memory used by test processes, e.g. python pytests, which spawn a process for each parallel worker - and this can add up especially in the minicluster stress tests.
> We need to see how we can constrain memory requirements, and look for opportunities to reduce memory requirements. This will help avoid OOMs, it may allow us to run gerrit-verify-dryrun jobs with smaller instance types, and will help reduce the requirements for dev machines which will be good for the community.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org