You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Sidharta Seethana (JIRA)" <ji...@apache.org> on 2015/12/05 01:54:11 UTC
[jira] [Commented] (YARN-1856) cgroups based memory monitoring for
containers
[ https://issues.apache.org/jira/browse/YARN-1856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15042505#comment-15042505 ]
Sidharta Seethana commented on YARN-1856:
-----------------------------------------
Hi [~vvasudev],
Thanks for the patch. Please see below for some comments.
{code}
@VisibleForTesting
static final String HARD_LIMIT = "limit_in_bytes";
@VisibleForTesting
static final String SOFT_LIMIT = "soft_limit_in_bytes";
@VisibleForTesting
static final String SWAPPINESS = "swappiness”;
{code}
Could you move these constants to CGroupsHandler ? There are already some cgroups parameter constants defined there.
{code}
if (configuration.getBoolean(YarnConfiguration.NM_PMEM_CHECK_ENABLED,
YarnConfiguration.DEFAULT_NM_PMEM_CHECK_ENABLED)) {
LOG.warn("You have enabled the default YARN physical memory health"
+ " checker as well as the CGroups memory controller. This could"
+ " lead to unpredictable behaviour");
}
{code}
IMO, If the behavior here is unpredictable, we should simply error our here in case both are enabled.
{code}
long softLimit = (long) (container.getResource().getMemory() * 0.90f);
{code}
We should make the fraction configurable, I think. What are the implications of the soft limit?
{code}
public static final String NM_MEMORY_RESOURCE_PREFIX = NM_PREFIX
+ "resource.memory.";
public static final String NM_MEMORY_RESOURCE_ENABLED =
NM_MEMORY_RESOURCE_PREFIX + "enabled";
public static final boolean DEFAULT_NM_MEMORY_RESOURCE_ENABLED = false;
public static final String NM_MEMORY_RESOURCE_CGROUPS_SWAPPINESS =
NM_MEMORY_RESOURCE_PREFIX + "cgroups.swappiness";
public static final int DEFAULT_NM_MEMORY_RESOURCE_CGROUPS_SWAPPINESS = 0;
{code}
Since we are skipping changes to yarn-default.xml (based on changes I see in TestYarnConfigurationFields), these should be marked @Private , similar to how network/disk configs settings are annotated?
Thinking aloud here : should we add support in some form for {{memory.oom_control}} and notifications/stats?
> cgroups based memory monitoring for containers
> ----------------------------------------------
>
> Key: YARN-1856
> URL: https://issues.apache.org/jira/browse/YARN-1856
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
> Affects Versions: 2.3.0
> Reporter: Karthik Kambatla
> Assignee: Varun Vasudev
> Attachments: YARN-1856.001.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)