You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Terry Wang (Jira)" <ji...@apache.org> on 2024/11/26 03:33:00 UTC

[jira] [Created] (SPARK-50420) Executor memory config is incorrect when multiple stage level resource profiles works

Terry Wang created SPARK-50420:
----------------------------------

             Summary: Executor memory config is incorrect when multiple stage level resource profiles works
                 Key: SPARK-50420
                 URL: https://issues.apache.org/jira/browse/SPARK-50420
             Project: Spark
          Issue Type: Bug
          Components: Scheduler
    Affects Versions: 3.5.3, 3.4.4, 3.3.4, 3.2.4, 3.1.3
         Environment: None
            Reporter: Terry Wang


https://issues.apache.org/jira/browse/SPARK-27495 has imported a great feature to help adjust stage's level resource profile(abbv rp), we are observing some config is not as expected when multiple resource profile works. 
When one stage assigned rp2 different with default rp, the `UnifiedMemoryManager` will still get old conf as the default rp, which will cause  maxOffHeapMemory is still the old value!

{code:scala}
protected[this] val maxOffHeapMemory = conf.get(MEMORY_OFFHEAP_SIZE)
{code}





--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org