You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/09/19 18:15:35 UTC
[PR] docs: recommend setting spark.executor.memoryOverhead alongside off-heap memory [datafusion-comet]
andygrove opened a new pull request, #6051:
URL: https://github.com/apache/datafusion-comet/pull/6051
## Which issue does this PR close?
N/A — documentation only. Related to #6050, which tracks the driver-plugin side of the same gap.
## Rationale for this change
Enabling `spark.memory.offHeap.enabled` is not sufficient on its own, and today the docs imply it
is. `spark.memory.offHeap.size` is a budget that the cluster manager already folds into the executor
container size, so the memory Comet's operators explicitly reserve has room. Everything Comet
allocates without reserving it does not: per-batch working memory in expression kernels and Arrow
array builders, decompression buffers and Parquet reader structures, object store request buffers
and the tokio runtime, Comet's JVM-side Arrow buffers, and allocator overhead. Those come from the
Rust global allocator and live in the native heap, and nothing in the container sizing accounts for
them.
`spark.executor.memoryOverhead` is the only slack the container has for that, and the JVM's own
non-heap usage is already drawing on it. Spark's default of
`max(0.1 * spark.executor.memory, 384 MiB)` leaves little behind, and `CometDriverPlugin` does not
adjust it in off-heap mode, so users get no help from either the default or the plugin.
## What changes are included in this PR?
- A new "Configuring Executor Memory Overhead" section in the tuning guide explaining why the
setting matters for Comet specifically, recommending `2g` as a starting point, and noting that
raising it grows the container on Kubernetes and YARN.
- A pointer to it from the "Memory Tuning" introduction.
- `--conf spark.executor.memoryOverhead=2g` added to the `spark-shell` examples that already set
off-heap memory: the README getting-started example, the docs home page, both examples in the
installation guide, both examples in the Iceberg guide, and the PyArrow UDF `SparkSession`
example.
## How are these changes tested?
Documentation only; no code changes. `npx prettier "**/*.md" --check` passes on the edited files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: recommend setting spark.executor.memoryOverhead alongside off-heap memory [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #6051:
URL: https://github.com/apache/datafusion-comet/pull/6051#discussion_r4054250701
##########
docs/source/user-guide/latest/tuning.md:
##########
@@ -132,6 +138,31 @@ need to spill or have a single spillable operator.
[shuffle]: #shuffle
[Advanced Memory Tuning]: #advanced-memory-tuning
+### Configuring Executor Memory Overhead
+
+Enabling off-heap memory is not sufficient on its own. It is recommended to also set
+`spark.executor.memoryOverhead` when running Comet:
+
+```
+spark.executor.memoryOverhead=2g
Review Comment:
### Correctness
[P2] Could the `2g` recommendation preserve any larger calculated or configured overhead? In Spark 3.5 and 4.0, `ResourceProfile.calculateOverHeadMemory` uses the explicit value directly, so this replaces the default allowance rather than adding headroom. For example, a 32 GiB JVM executor has 3276 MiB overhead at the 0.1 factor. An 8 GiB Kubernetes cluster-mode Python executor also has 3276 MiB at its 0.4 default, which is an exception to the formula below. Setting `2g` reduces either allowance to 2048 MiB and reduces the calculated container allowance by 1228 MiB, while keeping heap and off-heap budgets unchanged. Comet's off-heap plugin path does not increase it afterward. A reader following the OOM guidance can therefore get less container headroom. Please scope `2g` to a small-executor example and tell readers to retain larger existing/default overhead and account for the applicable factor before adding their Comet margin.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: recommend setting spark.executor.memoryOverhead alongside off-heap memory [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #6051:
URL: https://github.com/apache/datafusion-comet/pull/6051#discussion_r4054291395
##########
docs/source/user-guide/latest/tuning.md:
##########
@@ -132,6 +138,31 @@ need to spill or have a single spillable operator.
[shuffle]: #shuffle
[Advanced Memory Tuning]: #advanced-memory-tuning
+### Configuring Executor Memory Overhead
+
+Enabling off-heap memory is not sufficient on its own. It is recommended to also set
+`spark.executor.memoryOverhead` when running Comet:
+
+```
+spark.executor.memoryOverhead=2g
Review Comment:
Good catch. `calculateOverHeadMemory` is a `getOrElse`, not a max, so a flat `2g` is a reduction on anything past roughly a 20 GiB executor, and worse for a PySpark app on Kubernetes where the propagated factor is 0.4.
I've reworked the section in d334434. It now describes the derivation accurately — the 0.1 factor, the 0.4 exception for PySpark and SparkR submitted to Kubernetes in cluster mode, and the fact that the 384 MiB floor became `spark.executor.minMemoryOverhead` in 4.0 — and says outright that setting the value replaces the derived one rather than adding to it. `2g` is now scoped to small executors, with your 4 GiB (409 MiB) and 32 GiB (3276 MiB) numbers as the contrast, and anything larger is pointed at `spark.executor.memoryOverheadFactor` so the overhead keeps scaling with the executor.
I left the `2g` in the `spark-shell` examples alone since none of them set `spark.executor.memory`, so they're on the 1 GiB default and the derived overhead there is the 384 MiB floor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: recommend setting spark.executor.memoryOverhead alongside off-heap memory [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #6051:
URL: https://github.com/apache/datafusion-comet/pull/6051
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org