You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "jpcorreia99 (via GitHub)" <gi...@apache.org> on 2024/02/24 16:14:45 UTC
[PR] Enable user to override base overhead memory [spark]
jpcorreia99 opened a new pull request, #45240:
URL: https://github.com/apache/spark/pull/45240
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'common/utils/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
-->
### Was this patch authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47208][CORE] Allow overriding base overhead memory [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #45240:
URL: https://github.com/apache/spark/pull/45240#issuecomment-1976240642
This isn't really my area. cc @tgravescs FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47208][CORE] Allow overriding base overhead memory [spark]
Posted by "tgravescs (via GitHub)" <gi...@apache.org>.
tgravescs commented on PR #45240:
URL: https://github.com/apache/spark/pull/45240#issuecomment-1976750387
can you provide more details as to why this is needed? Its yet another configuration for memory overhead when we already have multiple. I assume you aren't trying to make the minimum lower because you say using a lot of off-heap memory. You can manually set the memoryoverhead vs setting this new config, so what use case is this trying to fix?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47208] Allow overriding base overhead memory [spark]
Posted by "jpcorreia99 (via GitHub)" <gi...@apache.org>.
jpcorreia99 commented on PR #45240:
URL: https://github.com/apache/spark/pull/45240#issuecomment-1976141153
Hey @HyukjinKwon, would you happen to have the time to give this a review or point me to someone who might?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47208][CORE] Allow overriding base overhead memory [spark]
Posted by "jpcorreia99 (via GitHub)" <gi...@apache.org>.
jpcorreia99 commented on PR #45240:
URL: https://github.com/apache/spark/pull/45240#issuecomment-1996976159
@tgravescs great! Any timeline on merging this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47208][CORE] Allow overriding base overhead memory [spark]
Posted by "jpcorreia99 (via GitHub)" <gi...@apache.org>.
jpcorreia99 commented on PR #45240:
URL: https://github.com/apache/spark/pull/45240#issuecomment-1992568472
Hey @mridulm, thanks for commenting!
> would like to understand the usecase better here - It is still unclear to me what characteristics you are shooting for by this PR.
I recognize it's a bit of a contrived scenario. We provide users the ability to write spark code while we handle the whole infrastructure provisioning. We have defaults for both the JVM but allow the users to override it. In Spark, the default rule for off heap allocation is 10% of the JVM, with this value being configurable via the `memoryOverheadFactor` config.
Let's now say that as part of the infrastructure provisioning we have to perform certain off-heap operations we know will need 500Mb of off-heap memory.
We can:
- Directly set the `memoryOverhead` config. But here we risk actually giving the user less overhead memory than he would be given otherwise (e.g. if the user requested 10GB JVM, with this conf he's only get 500MB instead of the 1Gb (10*0.1) he'd have otherwise.
- We can set the `memoryOverheadFactor` to give us the desired value, but let's imagine a scenario where the default JVM allocation is 1GB: to get 500MB of off-heap we'd need a factor of 0.5. If the user then decided to request for 10GB of JVM instead, we'd be also allocating 5Gb off off-heap memory, which could be very wasteful.
As such, that's why I'm proposing this config, as a way to let off-heap memory allocation scale as it usually does but while guaranteeing that we have *at least* a certain configurable amount of off heap memory!
> Reduction in OOM is mentioned https://github.com/apache/spark/pull/45240#issuecomment-1976891862 - but overhead does not impact that, heap memory does.
In our case the OOMs we got where K8s OOMkiller OOMs, not JVM ones. These originated from the off-heap processes consuming too much memory and k8s killing down the pod. So, in this scenario, increasing the overhead actually stopped the OOMs
> Any increase in overhead memory implies reduction in available heap memory if the max container memory has a upper limit and it is at that limit (if container memory is not limited - there is no impact of memory available to user, tune memory and overhead independently).
That's a fair concern. In our case we allocate the containers based on the memory required for the JVM + overhead instead of having a fixed memory container, so we don't run into this issue.
> The reason behind the question is, I am trying to understand if the default min (which was estimated quite a while ago) needs to be relooked or not - and understanding why your deployment needs a higher min will help.
The 387mb value does seem a bit arbitrary 😅 but it seems mostly fine. In our case we simply rely on heavy off-heap operations. Some examples would be package managers like conda + pip, performing ZSTD decompression (the jni implementation delegates to native operations), even up until spark 3.2, the `SparkContext::AddArchive` call would launch `tar` processes for decompression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47208] Allow overriding base overhead memory [spark]
Posted by "jpcorreia99 (via GitHub)" <gi...@apache.org>.
jpcorreia99 commented on code in PR #45240:
URL: https://github.com/apache/spark/pull/45240#discussion_r1505739465
##########
docs/configuration.md:
##########
@@ -202,6 +202,15 @@ of the most common options to set are:
</td>
<td>2.3.0</td>
</tr>
+<tr>
+ <td><code>spark.driver.minMemoryOverhead</code></td>
+ <td>None</td>
+ <td>
+ The minimum amount of non-heap memory to be allocated per driver process in cluster mode, in MiB unless otherwise specified, if <code>spark.driver.memoryOverhead</code> is not defined.
+ This option is currently supported on YARN and Kubernetes.
+ </td>
+ <td>3.5.2</td>
Review Comment:
What value should be added here? I could not find a roadmap of future releases for spark, but saw the latest release is '3.5.1'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47208][CORE] Allow overriding base overhead memory [spark]
Posted by "jpcorreia99 (via GitHub)" <gi...@apache.org>.
jpcorreia99 commented on code in PR #45240:
URL: https://github.com/apache/spark/pull/45240#discussion_r1522177128
##########
docs/configuration.md:
##########
@@ -202,6 +202,15 @@ of the most common options to set are:
</td>
<td>2.3.0</td>
</tr>
+<tr>
+ <td><code>spark.driver.minMemoryOverhead</code></td>
+ <td>384Mib</td>
Review Comment:
@tgravescs just want to make sure I'm being correct here.
I create this flag with " .createWithDefaultString("384m")". Is it correct for me to write here `Mib` instead of `m` for clarity to the reader or should I stick to exactly the value it contains?
##########
docs/configuration.md:
##########
@@ -202,6 +202,15 @@ of the most common options to set are:
</td>
<td>2.3.0</td>
</tr>
+<tr>
+ <td><code>spark.driver.minMemoryOverhead</code></td>
+ <td>384Mib</td>
Review Comment:
@tgravescs just want to make sure I'm being correct here.
I create this flag with `.createWithDefaultString("384m")`. Is it correct for me to write here `Mib` instead of `m` for clarity to the reader or should I stick to exactly the value it contains?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47208][CORE] Allow overriding base overhead memory [spark]
Posted by "mridulm (via GitHub)" <gi...@apache.org>.
mridulm commented on code in PR #45240:
URL: https://github.com/apache/spark/pull/45240#discussion_r1522422583
##########
core/src/main/scala/org/apache/spark/resource/ResourceProfile.scala:
##########
@@ -489,10 +489,11 @@ object ResourceProfile extends Logging {
private[spark] def calculateOverHeadMemory(
overHeadMemFromConf: Option[Long],
+ minimumOverHeadMemoryFromConf: Long,
executorMemoryMiB: Long,
overheadFactor: Double): Long = {
overHeadMemFromConf.getOrElse(math.max((overheadFactor * executorMemoryMiB).toInt,
- ResourceProfile.MEMORY_OVERHEAD_MIN_MIB))
+ minimumOverHeadMemoryFromConf))
Review Comment:
Discuss:
Do we need `MEMORY_OVERHEAD_MIN_MIB` after this PR is merged ?
If no, remove that field ?
yarn client mode AM overhead can be specified in that file, does not need to be in `ResourceProfile` ... ditto for suites.
Thoughts @tgravescs, @jpcorreia99 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47208][CORE] Allow overriding base overhead memory [spark]
Posted by "jpcorreia99 (via GitHub)" <gi...@apache.org>.
jpcorreia99 commented on PR #45240:
URL: https://github.com/apache/spark/pull/45240#issuecomment-1992666783
Hey @tgravescs I've now addressed your comments and also answered @mridulm, whenever you have the time, I'd highly appreciate if you could give it another review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47208][CORE] Allow overriding base overhead memory [spark]
Posted by "jpcorreia99 (via GitHub)" <gi...@apache.org>.
jpcorreia99 commented on code in PR #45240:
URL: https://github.com/apache/spark/pull/45240#discussion_r1522135158
##########
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:
##########
@@ -96,12 +96,21 @@ private[spark] class Client(
} else {
sparkConf.get(AM_MEMORY).toInt
}
+
+ private val driverMinimumMemoryOverhead =
+ if (isClusterMode && sparkConf.contains(DRIVER_MIN_MEMORY_OVERHEAD)) {
+ sparkConf.get(DRIVER_MIN_MEMORY_OVERHEAD).get
+ } else {
+ ResourceProfile.MEMORY_OVERHEAD_MIN_MIB
+ }
+
private val amMemoryOverhead = {
val amMemoryOverheadEntry = if (isClusterMode) DRIVER_MEMORY_OVERHEAD else AM_MEMORY_OVERHEAD
sparkConf.get(amMemoryOverheadEntry).getOrElse(
math.max((amMemoryOverheadFactor * amMemory).toLong,
- ResourceProfile.MEMORY_OVERHEAD_MIN_MIB)).toInt
+ driverMinimumMemoryOverhead)).toInt
Review Comment:
I may be misunderstanding something, but doesn't the fact that above we have
```
private val driverMinimumMemoryOverhead =
if (isClusterMode) {
sparkConf.get(DRIVER_MIN_MEMORY_OVERHEAD)
} else {
ResourceProfile.MEMORY_OVERHEAD_MIN_MIB
}
```
guarantee that the changes are only applied in cluster mode, which is case where it actually corresponds to the driver?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47208][CORE] Allow overriding base overhead memory [spark]
Posted by "tgravescs (via GitHub)" <gi...@apache.org>.
tgravescs commented on PR #45240:
URL: https://github.com/apache/spark/pull/45240#issuecomment-1994608914
Looks good, thanks @jpcorreia99
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47208] Allow overriding base overhead memory [spark]
Posted by "jpcorreia99 (via GitHub)" <gi...@apache.org>.
jpcorreia99 commented on code in PR #45240:
URL: https://github.com/apache/spark/pull/45240#discussion_r1505736480
##########
core/src/main/scala/org/apache/spark/resource/ResourceProfile.scala:
##########
@@ -489,10 +489,13 @@ object ResourceProfile extends Logging {
private[spark] def calculateOverHeadMemory(
overHeadMemFromConf: Option[Long],
+ minimumOverHeadMemoryFromConf: Option[Long],
executorMemoryMiB: Long,
overheadFactor: Double): Long = {
+ val minMemoryOverhead =
+ minimumOverHeadMemoryFromConf.getOrElse(ResourceProfile.MEMORY_OVERHEAD_MIN_MIB);
overHeadMemFromConf.getOrElse(math.max((overheadFactor * executorMemoryMiB).toInt,
- ResourceProfile.MEMORY_OVERHEAD_MIN_MIB))
+ minMemoryOverhead))
Review Comment:
I order to minimize the changes to existing logic, I chose to make this flag not have a default and instead write the logic to check if it is not set and use the existing constant.
However, from a code cleaniness and maintenance standpoint, I think the constant can be deleted and the flag value defaulted to 384Mib.
@the reviewer, what's your view here?
##########
core/src/main/scala/org/apache/spark/resource/ResourceProfile.scala:
##########
@@ -489,10 +489,13 @@ object ResourceProfile extends Logging {
private[spark] def calculateOverHeadMemory(
overHeadMemFromConf: Option[Long],
+ minimumOverHeadMemoryFromConf: Option[Long],
executorMemoryMiB: Long,
overheadFactor: Double): Long = {
+ val minMemoryOverhead =
+ minimumOverHeadMemoryFromConf.getOrElse(ResourceProfile.MEMORY_OVERHEAD_MIN_MIB);
overHeadMemFromConf.getOrElse(math.max((overheadFactor * executorMemoryMiB).toInt,
- ResourceProfile.MEMORY_OVERHEAD_MIN_MIB))
+ minMemoryOverhead))
Review Comment:
I order to minimize the changes to existing logic, I chose to make this flag not have a default and instead write the logic to check if it is not set and use the existing constant.
However, from a code cleaniness and maintenance standpoint, I think the constant can be deleted and the flag value defaulted to 384Mib.
@reviewer, what's your view here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47208][CORE] Allow overriding base overhead memory [spark]
Posted by "asfgit (via GitHub)" <gi...@apache.org>.
asfgit closed pull request #45240: [SPARK-47208][CORE] Allow overriding base overhead memory
URL: https://github.com/apache/spark/pull/45240
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47208][CORE] Allow overriding base overhead memory [spark]
Posted by "tgravescs (via GitHub)" <gi...@apache.org>.
tgravescs commented on PR #45240:
URL: https://github.com/apache/spark/pull/45240#issuecomment-1997459699
Merged to master, thanks @mridulm @jpcorreia99
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47208][CORE] Allow overriding base overhead memory [spark]
Posted by "tgravescs (via GitHub)" <gi...@apache.org>.
tgravescs commented on code in PR #45240:
URL: https://github.com/apache/spark/pull/45240#discussion_r1523328549
##########
resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala:
##########
@@ -706,6 +706,39 @@ class ClientSuite extends SparkFunSuite
assert(client.getPreloadedStatCache(sparkConf.get(JARS_TO_DISTRIBUTE), mockFsLookup).size === 2)
}
+ Seq(
+ "client",
+ "cluster"
+ ).foreach { case (deployMode) =>
+ test(s"SPARK-47208: minimum memory overhead is correctly set in ($deployMode mode)") {
+ val sparkConf = new SparkConf()
+ .set("spark.app.name", "foo-test-app")
+ .set(SUBMIT_DEPLOY_MODE, deployMode)
+ .set(DRIVER_MIN_MEMORY_OVERHEAD, 500L)
+ val args = new ClientArguments(Array())
+
+ val appContext = Records.newRecord(classOf[ApplicationSubmissionContext])
+ val getNewApplicationResponse = Records.newRecord(classOf[GetNewApplicationResponse])
+ val containerLaunchContext = Records.newRecord(classOf[ContainerLaunchContext])
+
+ val client = new Client(args, sparkConf, null)
+ client.createApplicationSubmissionContext(
+ new YarnClientApplication(getNewApplicationResponse, appContext),
+ containerLaunchContext)
+
+ appContext.getApplicationName should be("foo-test-app")
+ // flag should only work for cluster mode
+ if (deployMode=="cluster") {
Review Comment:
nit spaces around " == "
##########
resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnAllocatorSuite.scala:
##########
@@ -772,6 +773,42 @@ class YarnAllocatorSuite extends SparkFunSuite
assert(memory == (executorMemory * 1.4).toLong)
} finally {
sparkConf.set(EXECUTOR_MEMORY_OVERHEAD_FACTOR, 0.1)
+ sparkConf.remove(EXECUTOR_MEMORY_OVERHEAD)
+ }
+ }
+
+ test("SPARK-47208: User can override the minimum memory overhead of the executor") {
+ val executorMemory = sparkConf.get(EXECUTOR_MEMORY)
+ try {
+ sparkConf
+ .set(EXECUTOR_MIN_MEMORY_OVERHEAD, 500L)
+ val (handler, _) = createAllocator(maxExecutors = 1,
+ additionalConfigs = Map(EXECUTOR_MEMORY.key -> executorMemory.toString))
+ val defaultResource = handler.rpIdToYarnResource.get(defaultRPId)
+ val memory = defaultResource.getMemorySize
+ assert(memory == (executorMemory + 500))
+ } finally {
+ sparkConf
+ .remove(EXECUTOR_MIN_MEMORY_OVERHEAD)
+ }
+ }
+
+ test("SPARK-47208: Explicit overhead takes precedence over minimum overhead") {
+ val executorMemory = sparkConf.get(EXECUTOR_MEMORY)
+ try {
+ sparkConf
+ .set(EXECUTOR_MIN_MEMORY_OVERHEAD, 500L)
+ .set(EXECUTOR_MEMORY_OVERHEAD, 100L)
+ val (handler, _) = createAllocator(maxExecutors = 1,
+ additionalConfigs = Map(EXECUTOR_MEMORY.key -> executorMemory.toString))
+ val defaultResource = handler.rpIdToYarnResource.get(defaultRPId)
+ val memory = defaultResource.getMemorySize
+ assert(memory == (executorMemory + 100))
+ } finally {
+ sparkConf
+ .remove(EXECUTOR_MIN_MEMORY_OVERHEAD)
+ sparkConf
+ .remove(EXECUTOR_MIN_MEMORY_OVERHEAD)
Review Comment:
you are removing EXECUTOR_MIN_MEMORY_OVERHEAD twice, one of these should be EXECUTOR_MEMORY_OVERHEAD right?
##########
resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala:
##########
@@ -706,6 +706,39 @@ class ClientSuite extends SparkFunSuite
assert(client.getPreloadedStatCache(sparkConf.get(JARS_TO_DISTRIBUTE), mockFsLookup).size === 2)
}
+ Seq(
+ "client",
+ "cluster"
+ ).foreach { case (deployMode) =>
+ test(s"SPARK-47208: minimum memory overhead is correctly set in ($deployMode mode)") {
+ val sparkConf = new SparkConf()
+ .set("spark.app.name", "foo-test-app")
+ .set(SUBMIT_DEPLOY_MODE, deployMode)
+ .set(DRIVER_MIN_MEMORY_OVERHEAD, 500L)
+ val args = new ClientArguments(Array())
+
+ val appContext = Records.newRecord(classOf[ApplicationSubmissionContext])
+ val getNewApplicationResponse = Records.newRecord(classOf[GetNewApplicationResponse])
+ val containerLaunchContext = Records.newRecord(classOf[ContainerLaunchContext])
+
+ val client = new Client(args, sparkConf, null)
+ client.createApplicationSubmissionContext(
+ new YarnClientApplication(getNewApplicationResponse, appContext),
+ containerLaunchContext)
+
+ appContext.getApplicationName should be("foo-test-app")
+ // flag should only work for cluster mode
+ if (deployMode=="cluster") {
+ // 1Gb driver default + 500 overridden minimum default overhead
+ appContext.getResource should be(Resource.newInstance(1524L, 1))
Review Comment:
nit space after "should be " for readability
##########
resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnAllocatorSuite.scala:
##########
@@ -772,6 +773,42 @@ class YarnAllocatorSuite extends SparkFunSuite
assert(memory == (executorMemory * 1.4).toLong)
} finally {
sparkConf.set(EXECUTOR_MEMORY_OVERHEAD_FACTOR, 0.1)
+ sparkConf.remove(EXECUTOR_MEMORY_OVERHEAD)
+ }
+ }
+
+ test("SPARK-47208: User can override the minimum memory overhead of the executor") {
+ val executorMemory = sparkConf.get(EXECUTOR_MEMORY)
+ try {
+ sparkConf
+ .set(EXECUTOR_MIN_MEMORY_OVERHEAD, 500L)
+ val (handler, _) = createAllocator(maxExecutors = 1,
+ additionalConfigs = Map(EXECUTOR_MEMORY.key -> executorMemory.toString))
+ val defaultResource = handler.rpIdToYarnResource.get(defaultRPId)
+ val memory = defaultResource.getMemorySize
+ assert(memory == (executorMemory + 500))
+ } finally {
+ sparkConf
+ .remove(EXECUTOR_MIN_MEMORY_OVERHEAD)
+ }
+ }
+
+ test("SPARK-47208: Explicit overhead takes precedence over minimum overhead") {
+ val executorMemory = sparkConf.get(EXECUTOR_MEMORY)
+ try {
+ sparkConf
+ .set(EXECUTOR_MIN_MEMORY_OVERHEAD, 500L)
+ .set(EXECUTOR_MEMORY_OVERHEAD, 100L)
+ val (handler, _) = createAllocator(maxExecutors = 1,
+ additionalConfigs = Map(EXECUTOR_MEMORY.key -> executorMemory.toString))
+ val defaultResource = handler.rpIdToYarnResource.get(defaultRPId)
+ val memory = defaultResource.getMemorySize
+ assert(memory == (executorMemory + 100))
+ } finally {
+ sparkConf
Review Comment:
nit the .remove should be on same line as sparkConf as its not to long
##########
resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnAllocatorSuite.scala:
##########
@@ -772,6 +773,42 @@ class YarnAllocatorSuite extends SparkFunSuite
assert(memory == (executorMemory * 1.4).toLong)
} finally {
sparkConf.set(EXECUTOR_MEMORY_OVERHEAD_FACTOR, 0.1)
+ sparkConf.remove(EXECUTOR_MEMORY_OVERHEAD)
+ }
+ }
+
+ test("SPARK-47208: User can override the minimum memory overhead of the executor") {
+ val executorMemory = sparkConf.get(EXECUTOR_MEMORY)
+ try {
+ sparkConf
+ .set(EXECUTOR_MIN_MEMORY_OVERHEAD, 500L)
+ val (handler, _) = createAllocator(maxExecutors = 1,
+ additionalConfigs = Map(EXECUTOR_MEMORY.key -> executorMemory.toString))
+ val defaultResource = handler.rpIdToYarnResource.get(defaultRPId)
+ val memory = defaultResource.getMemorySize
+ assert(memory == (executorMemory + 500))
+ } finally {
+ sparkConf
Review Comment:
nit put the .removeConf on the same line as sparkConf
##########
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:
##########
@@ -96,12 +96,21 @@ private[spark] class Client(
} else {
sparkConf.get(AM_MEMORY).toInt
}
+
+ private val driverMinimumMemoryOverhead =
+ if (isClusterMode && sparkConf.contains(DRIVER_MIN_MEMORY_OVERHEAD)) {
+ sparkConf.get(DRIVER_MIN_MEMORY_OVERHEAD).get
+ } else {
+ ResourceProfile.MEMORY_OVERHEAD_MIN_MIB
+ }
+
private val amMemoryOverhead = {
val amMemoryOverheadEntry = if (isClusterMode) DRIVER_MEMORY_OVERHEAD else AM_MEMORY_OVERHEAD
sparkConf.get(amMemoryOverheadEntry).getOrElse(
math.max((amMemoryOverheadFactor * amMemory).toLong,
- ResourceProfile.MEMORY_OVERHEAD_MIN_MIB)).toInt
+ driverMinimumMemoryOverhead)).toInt
Review Comment:
ah yes that works
##########
docs/configuration.md:
##########
@@ -202,6 +202,15 @@ of the most common options to set are:
</td>
<td>2.3.0</td>
</tr>
+<tr>
+ <td><code>spark.driver.minMemoryOverhead</code></td>
+ <td>384Mib</td>
Review Comment:
I think I would put it at "384m" to be more consistent with other docs and be format user would specify if changing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47208][CORE] Allow overriding base overhead memory [spark]
Posted by "mridulm (via GitHub)" <gi...@apache.org>.
mridulm commented on code in PR #45240:
URL: https://github.com/apache/spark/pull/45240#discussion_r1513970260
##########
core/src/main/scala/org/apache/spark/internal/config/package.scala:
##########
@@ -117,6 +117,14 @@ package object config {
.bytesConf(ByteUnit.MiB)
.createOptional
+ private[spark] val DRIVER_MIN_MEMORY_OVERHEAD = ConfigBuilder("spark.driver.minMemoryOverhead")
+ .doc("The minimum amount of non-heap memory to be allocated per driver in cluster mode, " +
+ "in MiB unless otherwise specified. Overrides the default value of 384Mib." +
+ " This value is ignored if spark.driver.memoryOverhead is set directly.")
+ .version("3.5.2")
+ .bytesConf(ByteUnit.MiB)
+ .createOptional
Review Comment:
set the default here to 384 - and use this instead of existing 384
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47208][CORE] Allow overriding base overhead memory [spark]
Posted by "tgravescs (via GitHub)" <gi...@apache.org>.
tgravescs commented on code in PR #45240:
URL: https://github.com/apache/spark/pull/45240#discussion_r1511443512
##########
core/src/main/scala/org/apache/spark/internal/config/package.scala:
##########
@@ -117,6 +117,14 @@ package object config {
.bytesConf(ByteUnit.MiB)
.createOptional
+ private[spark] val DRIVER_MIN_MEMORY_OVERHEAD = ConfigBuilder("spark.driver.minMemoryOverhead")
+ .doc("The minimum amount of non-heap memory to be allocated per driver in cluster mode, " +
+ "in MiB unless otherwise specified. Overrides the default value of 384Mib." +
+ " This value is ignored if spark.driver.memoryOverhead is set directly.")
+ .version("3.5.2")
Review Comment:
this should be 4.0.0. This is a new feature so generally wouldn't be back ported, at least without discussion.
##########
resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnAllocatorSuite.scala:
##########
@@ -772,6 +775,40 @@ class YarnAllocatorSuite extends SparkFunSuite
assert(memory == (executorMemory * 1.4).toLong)
} finally {
sparkConf.set(EXECUTOR_MEMORY_OVERHEAD_FACTOR, 0.1)
+ sparkConf.remove(EXECUTOR_MEMORY_OVERHEAD)
+ }
+ }
+
+ test("SPARK-47208: User can override the minimum memory overhead of the executor") {
+ val executorMemory = sparkConf.get(EXECUTOR_MEMORY)
+ try {
+ sparkConf
+ .set(EXECUTOR_MIN_MEMORY_OVERHEAD, 500L)
+ val (handler, _) = createAllocator(maxExecutors = 1,
+ additionalConfigs = Map(EXECUTOR_MEMORY.key -> executorMemory.toString))
+ val defaultResource = handler.rpIdToYarnResource.get(defaultRPId)
+ val memory = defaultResource.getMemorySize
+ assert(memory == (executorMemory + 500))
+ } finally {
+ sparkConf
+ .remove(EXECUTOR_MIN_MEMORY_OVERHEAD)
+ }
+ }
+
+ test("SPARK-47208: Explicit overhead takes precedence over minimum overhead") {
+ val executorMemory = sparkConf.get(EXECUTOR_MEMORY)
+ try {
+ sparkConf
+ .set(EXECUTOR_MIN_MEMORY_OVERHEAD, 500L)
+ .set(EXECUTOR_MEMORY_OVERHEAD, 100L)
+ val (handler, _) = createAllocator(maxExecutors = 1,
+ additionalConfigs = Map(EXECUTOR_MEMORY.key -> executorMemory.toString))
+ val defaultResource = handler.rpIdToYarnResource.get(defaultRPId)
+ val memory = defaultResource.getMemorySize
+ assert(memory == (executorMemory + 100))
+ } finally {
+ sparkConf
+ .remove(EXECUTOR_MIN_MEMORY_OVERHEAD)
Review Comment:
sparkConf.remove(EXECUTOR_MEMORY_OVERHEAD)
##########
resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala:
##########
@@ -706,6 +706,39 @@ class ClientSuite extends SparkFunSuite
assert(client.getPreloadedStatCache(sparkConf.get(JARS_TO_DISTRIBUTE), mockFsLookup).size === 2)
}
+ Seq(
+ "client",
+ "cluster"
+ ).foreach { case (deployMode) =>
+ test(s"SPARK-47208: minimum memory overhead is correctly set in ($deployMode mode)") {
+ val sparkConf = new SparkConf()
+ .set("spark.app.name", "foo-test-app")
+ .set(SUBMIT_DEPLOY_MODE, deployMode)
+ .set(DRIVER_MIN_MEMORY_OVERHEAD, 500L)
+ val args = new ClientArguments(Array())
+
+ val appContext = Records.newRecord(classOf[ApplicationSubmissionContext])
+ val getNewApplicationResponse = Records.newRecord(classOf[GetNewApplicationResponse])
+ val containerLaunchContext = Records.newRecord(classOf[ContainerLaunchContext])
+
+ val client = new Client(args, sparkConf, null)
+ client.createApplicationSubmissionContext(
+ new YarnClientApplication(getNewApplicationResponse, appContext),
+ containerLaunchContext)
+
+ appContext.getApplicationName should be("foo-test-app")
+ // flag should only work for cluster mode
+ if (deployMode=="cluster") {
+ // 1Gb driver default + 500 overridden minimum default overhead
+ appContext.getResource should be(Resource.newInstance(1524L, 1))
+ } else {
+ // 512 driver default (non-cluster) + 384 overhead default
+ // that can't be changed in non cluster mode.
Review Comment:
nit extra space before "that"
##########
resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnAllocatorSuite.scala:
##########
@@ -128,6 +128,9 @@ class YarnAllocatorSuite extends SparkFunSuite
.set(EXECUTOR_CORES, 5)
.set(EXECUTOR_MEMORY, 2048L)
+ // scalastyle:off println
+ sparkConfClone.getAll.foreach(println)
Review Comment:
assume this was debugging? remove
##########
core/src/main/scala/org/apache/spark/internal/config/package.scala:
##########
@@ -358,6 +366,15 @@ package object config {
.bytesConf(ByteUnit.MiB)
.createOptional
+ private[spark] val EXECUTOR_MIN_MEMORY_OVERHEAD =
+ ConfigBuilder("spark.executor.minMemoryOverhead")
+ .doc("The minimum amount of non-heap memory to be allocated per executor " +
+ "in MiB unless otherwise specified. Overrides the default value of 384Mib." +
+ " This value is ignored if spark.executor.memoryOverhead is set directly.")
+ .version("3.5.2")
Review Comment:
4.0.0
##########
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:
##########
@@ -75,7 +75,7 @@ private[spark] class Client(
private val yarnClient = YarnClient.createYarnClient
private val hadoopConf = new YarnConfiguration(SparkHadoopUtil.newConfiguration(sparkConf))
- private val isClusterMode = sparkConf.get(SUBMIT_DEPLOY_MODE) == "cluster"
+ private val isClusterMode = sparkConf.get(SUBMIT_DEPLOY_MODE) == "cluster"
Review Comment:
remove extra spaces
##########
docs/configuration.md:
##########
@@ -202,6 +202,15 @@ of the most common options to set are:
</td>
<td>2.3.0</td>
</tr>
+<tr>
+ <td><code>spark.driver.minMemoryOverhead</code></td>
+ <td>None</td>
+ <td>
+ The minimum amount of non-heap memory to be allocated per driver process in cluster mode, in MiB unless otherwise specified, if <code>spark.driver.memoryOverhead</code> is not defined.
+ This option is currently supported on YARN and Kubernetes.
+ </td>
+ <td>3.5.2</td>
Review Comment:
oops, answered this in a different comment, should be 4.0.0
##########
core/src/main/scala/org/apache/spark/resource/ResourceProfile.scala:
##########
@@ -489,10 +489,13 @@ object ResourceProfile extends Logging {
private[spark] def calculateOverHeadMemory(
overHeadMemFromConf: Option[Long],
+ minimumOverHeadMemoryFromConf: Option[Long],
executorMemoryMiB: Long,
overheadFactor: Double): Long = {
+ val minMemoryOverhead =
+ minimumOverHeadMemoryFromConf.getOrElse(ResourceProfile.MEMORY_OVERHEAD_MIN_MIB);
overHeadMemFromConf.getOrElse(math.max((overheadFactor * executorMemoryMiB).toInt,
- ResourceProfile.MEMORY_OVERHEAD_MIN_MIB))
+ minMemoryOverhead))
Review Comment:
I think it would be cleaner just to have the defaults of the configs 384.
##########
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:
##########
@@ -96,12 +96,21 @@ private[spark] class Client(
} else {
sparkConf.get(AM_MEMORY).toInt
}
+
+ private val driverMinimumMemoryOverhead =
+ if (isClusterMode && sparkConf.contains(DRIVER_MIN_MEMORY_OVERHEAD)) {
+ sparkConf.get(DRIVER_MIN_MEMORY_OVERHEAD).get
+ } else {
+ ResourceProfile.MEMORY_OVERHEAD_MIN_MIB
+ }
+
private val amMemoryOverhead = {
val amMemoryOverheadEntry = if (isClusterMode) DRIVER_MEMORY_OVERHEAD else AM_MEMORY_OVERHEAD
sparkConf.get(amMemoryOverheadEntry).getOrElse(
math.max((amMemoryOverheadFactor * amMemory).toLong,
- ResourceProfile.MEMORY_OVERHEAD_MIN_MIB)).toInt
+ driverMinimumMemoryOverhead)).toInt
Review Comment:
driver and AM aren't necessarily the same process. In cluster mode it is but in client mode it isn't (see spark.yarn.am.memoryOverheah) - https://spark.apache.org/docs/latest/running-on-yarn.html.
Unless you have a use case for the app master doing native stuff, which I don't see, I think this should stay the same (384) or if you have a use case for it we should have a separate config for it since its a YARN specific thing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47208][CORE] Allow overriding base overhead memory [spark]
Posted by "jpcorreia99 (via GitHub)" <gi...@apache.org>.
jpcorreia99 commented on PR #45240:
URL: https://github.com/apache/spark/pull/45240#issuecomment-1976891862
@tgravescs very fair point!
We preconfigure the spark nodes we provide to users with a global config, but then allow the users to override some of the fields.
We observed a reduction in OOMs when bumping the minimum memory overhead.
The reason why we don't explicitly set the memory overhead is because we could accidentally be reducing the overall memory the user has access to.
As an example, we'd like to have at least 500Mb of overhead. User has requested a 30Gb driver. If we directly set the memory overhead to 500Mb, we'd be cutting the overhead available to the user by 2.5Gb (as overhead memory is calculated as 10% of JVM).
We also don't want to be changing the overhead factor since these are global config. If we'd like to have at least a 500Mb overhead and we have a default memory of 2Gb, we'd need to set up the factor at 0.25. But then, if the user comes and asks for a 30Gb driver, he'd also receive 7.5Gb in overhead, which is excessive.
As such, defining a global overhead memory minimum would work best if you want to have configurable memory settings and reasonable overhead defaults.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47208][CORE] Allow overriding base overhead memory [spark]
Posted by "mridulm (via GitHub)" <gi...@apache.org>.
mridulm commented on PR #45240:
URL: https://github.com/apache/spark/pull/45240#issuecomment-1980260288
I would like to understand the usecase better here - It is still unclear to me what characteristics you are shooting for by this PR.
Reduction in OOM is mentioned [as a usecase](https://github.com/apache/spark/pull/45240#issuecomment-1976891862) - but overhead does not impact that, heap memory does.
> The reason why we don't explicitly set the memory overhead is because we could accidentally be reducing the overall memory the user has access to.
Any increase in overhead memory implies reduction in available heap memory if the max container memory has a upper limit and it is at that limit (if container memory is not limited - there is no impact of memory available to user, tune memory and overhead independently).
The reason behind the question is, I am trying to understand if the default min (which was estimated quite a while ago) needs to be relooked or not - and understanding why your deployment needs a higher min will help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47208][CORE] Allow overriding base overhead memory [spark]
Posted by "jpcorreia99 (via GitHub)" <gi...@apache.org>.
jpcorreia99 commented on code in PR #45240:
URL: https://github.com/apache/spark/pull/45240#discussion_r1510949333
##########
core/src/main/scala/org/apache/spark/internal/config/package.scala:
##########
@@ -117,6 +117,14 @@ package object config {
.bytesConf(ByteUnit.MiB)
.createOptional
+ private[spark] val DRIVER_MIN_MEMORY_OVERHEAD = ConfigBuilder("spark.driver.minMemoryOverhead")
+ .doc("The minimum amount of non-heap memory to be allocated per driver in cluster mode, " +
+ "in MiB unless otherwise specified. Overrides the default value of 384Mib." +
+ " This value is ignored if spark.driver.memoryOverhead is set directly.")
+ .version("3.5.2")
Review Comment:
Should this be 4.0 instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47208][CORE] Allow overriding base overhead memory [spark]
Posted by "jpcorreia99 (via GitHub)" <gi...@apache.org>.
jpcorreia99 commented on PR #45240:
URL: https://github.com/apache/spark/pull/45240#issuecomment-1994561966
@tgravescs thank you for the comments, I've addressed them in cd03ec88bf965622c4fad3e60dc76b5a6bd78e5d
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47208][CORE] Allow overriding base overhead memory [spark]
Posted by "jpcorreia99 (via GitHub)" <gi...@apache.org>.
jpcorreia99 commented on code in PR #45240:
URL: https://github.com/apache/spark/pull/45240#discussion_r1522859546
##########
core/src/main/scala/org/apache/spark/resource/ResourceProfile.scala:
##########
@@ -489,10 +489,11 @@ object ResourceProfile extends Logging {
private[spark] def calculateOverHeadMemory(
overHeadMemFromConf: Option[Long],
+ minimumOverHeadMemoryFromConf: Long,
executorMemoryMiB: Long,
overheadFactor: Double): Long = {
overHeadMemFromConf.getOrElse(math.max((overheadFactor * executorMemoryMiB).toInt,
- ResourceProfile.MEMORY_OVERHEAD_MIN_MIB))
+ minimumOverHeadMemoryFromConf))
Review Comment:
I do agree, at this point, outside of tests, `ResourceProfile.MEMORY_OVERHEAD_MIN_MIB` has only one usage in client AM overhead, so will make that change, can revert if @tgravescs disagrees
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47208][CORE] Allow overriding base overhead memory [spark]
Posted by "jpcorreia99 (via GitHub)" <gi...@apache.org>.
jpcorreia99 commented on code in PR #45240:
URL: https://github.com/apache/spark/pull/45240#discussion_r1510949333
##########
core/src/main/scala/org/apache/spark/internal/config/package.scala:
##########
@@ -117,6 +117,14 @@ package object config {
.bytesConf(ByteUnit.MiB)
.createOptional
+ private[spark] val DRIVER_MIN_MEMORY_OVERHEAD = ConfigBuilder("spark.driver.minMemoryOverhead")
+ .doc("The minimum amount of non-heap memory to be allocated per driver in cluster mode, " +
+ "in MiB unless otherwise specified. Overrides the default value of 384Mib." +
+ " This value is ignored if spark.driver.memoryOverhead is set directly.")
+ .version("3.5.2")
Review Comment:
Should this be 4.0 instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47208][CORE] Allow overriding base overhead memory [spark]
Posted by "jpcorreia99 (via GitHub)" <gi...@apache.org>.
jpcorreia99 commented on code in PR #45240:
URL: https://github.com/apache/spark/pull/45240#discussion_r1522896795
##########
core/src/main/scala/org/apache/spark/resource/ResourceProfile.scala:
##########
@@ -489,10 +489,11 @@ object ResourceProfile extends Logging {
private[spark] def calculateOverHeadMemory(
overHeadMemFromConf: Option[Long],
+ minimumOverHeadMemoryFromConf: Long,
executorMemoryMiB: Long,
overheadFactor: Double): Long = {
overHeadMemFromConf.getOrElse(math.max((overheadFactor * executorMemoryMiB).toInt,
- ResourceProfile.MEMORY_OVERHEAD_MIN_MIB))
+ minimumOverHeadMemoryFromConf))
Review Comment:
Addressed in 3abd57efffcc3df2e2f4225b45f00f5e8af01724
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47208][CORE] Allow overriding base overhead memory [spark]
Posted by "tgravescs (via GitHub)" <gi...@apache.org>.
tgravescs commented on code in PR #45240:
URL: https://github.com/apache/spark/pull/45240#discussion_r1523337134
##########
core/src/main/scala/org/apache/spark/resource/ResourceProfile.scala:
##########
@@ -489,10 +489,11 @@ object ResourceProfile extends Logging {
private[spark] def calculateOverHeadMemory(
overHeadMemFromConf: Option[Long],
+ minimumOverHeadMemoryFromConf: Long,
executorMemoryMiB: Long,
overheadFactor: Double): Long = {
overHeadMemFromConf.getOrElse(math.max((overheadFactor * executorMemoryMiB).toInt,
- ResourceProfile.MEMORY_OVERHEAD_MIN_MIB))
+ minimumOverHeadMemoryFromConf))
Review Comment:
I agree with removing it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47208][CORE] Allow overriding base overhead memory [spark]
Posted by "jpcorreia99 (via GitHub)" <gi...@apache.org>.
jpcorreia99 commented on code in PR #45240:
URL: https://github.com/apache/spark/pull/45240#discussion_r1522135158
##########
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:
##########
@@ -96,12 +96,21 @@ private[spark] class Client(
} else {
sparkConf.get(AM_MEMORY).toInt
}
+
+ private val driverMinimumMemoryOverhead =
+ if (isClusterMode && sparkConf.contains(DRIVER_MIN_MEMORY_OVERHEAD)) {
+ sparkConf.get(DRIVER_MIN_MEMORY_OVERHEAD).get
+ } else {
+ ResourceProfile.MEMORY_OVERHEAD_MIN_MIB
+ }
+
private val amMemoryOverhead = {
val amMemoryOverheadEntry = if (isClusterMode) DRIVER_MEMORY_OVERHEAD else AM_MEMORY_OVERHEAD
sparkConf.get(amMemoryOverheadEntry).getOrElse(
math.max((amMemoryOverheadFactor * amMemory).toLong,
- ResourceProfile.MEMORY_OVERHEAD_MIN_MIB)).toInt
+ driverMinimumMemoryOverhead)).toInt
Review Comment:
I may be misunderstanding something, but doesn't the fact that above we have
```
private val driverMinimumMemoryOverhead =
if (isClusterMode) {
sparkConf.get(DRIVER_MIN_MEMORY_OVERHEAD)
} else {
ResourceProfile.MEMORY_OVERHEAD_MIN_MIB
}
```
guarantee that my changes are only applied in cluster mode, which, form that you said, seems to be the case where it actually corresponds to the driver?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org