[jira] [Created] (SPARK-31265) Add -XX:MaxDirectMemorySize jvm options in yarn mode

["wangzhun (Jira)" <ji...@apache.org>]

wangzhun created SPARK-31265:
--------------------------------

             Summary: Add -XX:MaxDirectMemorySize jvm options in yarn mode
                 Key: SPARK-31265
                 URL: https://issues.apache.org/jira/browse/SPARK-31265
             Project: Spark
          Issue Type: Improvement
          Components: YARN
    Affects Versions: 3.0.0
            Reporter: wangzhun


Current memory composition `amMemory` + `amMemoryOverhead`
{code:java}
val capability = Records.newRecord(classOf[Resource])
capability.setMemory(amMemory + amMemoryOverhead)
capability.setVirtualCores(amCores)
if (amResources.nonEmpty) {
 ResourceRequestHelper.setResourceRequests(amResources, capability)
}
logDebug(s"Created resource capability for AM request: $capability")
{code}
{code:java}
 // Add Xmx for AM memory 
javaOpts += "-Xmx" + amMemory + "m"
{code}
It is possible that the physical memory of the container exceeds the limit and is killed by yarn.
I suggest setting `-XX:MaxDirectMemorySize` here
{code:java}
// Add Xmx for AM memory
javaOpts += "-Xmx" + amMemory + "m"
javaOpts += s"-XX:MaxDirectMemorySize=${amMemoryOverhead}m"{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org