[jira] [Commented] (SAMZA-972) Add holistic memory monitoring for SamzaContainer

["Yi Pan (Data Infrastructure) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SAMZA-972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15396540#comment-15396540 ] 

Yi Pan (Data Infrastructure) commented on SAMZA-972:
----------------------------------------------------

Add the RB link [here|https://reviews.apache.org/r/49877/]. 

> Add holistic memory monitoring for SamzaContainer
> -------------------------------------------------
>
>                 Key: SAMZA-972
>                 URL: https://issues.apache.org/jira/browse/SAMZA-972
>             Project: Samza
>          Issue Type: Bug
>            Reporter: Jagadish
>            Assignee: Jagadish
>
> This feature introduces physical memory monitoring in SamzaContainer.
> Context:
> Often memory used by the SamzaContainer process includes 
> A. JVM Heap memory: This is where all JVM variables live.
> B. Native memory: This memory lives out of the JVM heap and is not visible to the JVM. Examples include used by RocksDb, native libraries that user code depends on etc.
> User jobs could be killed by Yarn if their total memory (A+B) exceeds the configured maximum of yarn.container.memory.mb.
> Currently, while our existing metrics provide visibility into [A] via JMX, we don't have visibility into [B]. (as it's totally external to the JVM). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)