You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/09/28 18:27:42 UTC
[PR] docs: simplify the memory tuning page [datafusion-comet]
andygrove opened a new pull request, #6343:
URL: https://github.com/apache/datafusion-comet/pull/6343
## Which issue does this PR close?
No issue; this is documentation only.
## Rationale for this change
The memory tuning page had two answers to how much memory Comet needs. The older one, "Determining How Much Memory to Allocate", was added in March 2025, before Comet 0.8.0 was released. Its advice rests on TPC-H timings from a single executor at that time, and many memory bugs have been fixed since. The newer one, "Sizing the Overhead from the Memory Usage Log", uses the memory usage log added in 1.1.0 to measure what Comet actually uses on the user's own workload.
The page also explained untracked memory twice: once under Configuring Comet Memory, and again at the start of Configuring Executor Memory Overhead.
## What changes are included in this PR?
- Removes the "Determining How Much Memory to Allocate" section and the intro paragraph that pointed to it. The intro now points to the memory usage log section instead.
- Keeps the old section's one general point as a short paragraph under Configuring Comet Memory: a larger pool lets operators that can spill keep more in memory, and an operator that cannot spill fails the task when it cannot reserve the memory it needs. Nothing else on the page said why the pool's size matters.
- Condenses the three opening paragraphs of Configuring Executor Memory Overhead into one, since they repeated the untracked-memory list from the section above. The paragraph keeps the two points specific to the overhead: the container already has room for `spark.memory.offHeap.size`, and the JVM's own non-heap memory draws on the same overhead.
The headings that other pages link to are unchanged. The migration guide links to three of them, and the contributor guide links to the memory usage log section. Nothing linked to the removed section.
## How are these changes tested?
Documentation only, no code paths touched.
I did not build the site. A script checked that every reference link and heading anchor on the page resolves, and a search of the repository found no remaining links to the removed section. `npx prettier@latest --check` passes on the page.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: simplify the memory tuning page [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #6343:
URL: https://github.com/apache/datafusion-comet/pull/6343
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org