You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/09/25 13:39:51 UTC
(datafusion-comet) branch gh-readonly-queue/main/pr-6194-e90084420ab7acc4d1fd3a6db41f9ef9cb5a5087 created (now 092b6a6c33)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a change to branch gh-readonly-queue/main/pr-6194-e90084420ab7acc4d1fd3a6db41f9ef9cb5a5087
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
at 092b6a6c33 docs: correct which operators run separate native plans in a task (#6194)
This branch includes the following new commits:
new 092b6a6c33 docs: correct which operators run separate native plans in a task (#6194)
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
(datafusion-comet) 01/01: docs: correct which operators run separate native plans in a task (#6194)
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-6194-e90084420ab7acc4d1fd3a6db41f9ef9cb5a5087
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
commit 092b6a6c33e1fcf21f637ba789a944f6d69a1ba5
Author: Andy Grove <ag...@apache.org>
AuthorDate: Fri Sep 25 12:12:03 2026 +0000
docs: correct which operators run separate native plans in a task (#6194)
The memory management and tuning guides said that a native shuffle runs
the pre-shuffle operators and the shuffle writer as two native execution
contexts in the same task. The shuffle writer is planned together with
the native operators that feed it, so a native shuffle is one plan.
List the operators that do split a task's native work into separate
plans: union and coalesce, collect limit and take-ordered-and-project,
and native Parquet and Iceberg writes.
---
docs/source/contributor-guide/memory_management.md | 16 +++++++++++++---
docs/source/user-guide/latest/tuning.md | 7 +++----
2 files changed, 16 insertions(+), 7 deletions(-)
diff --git a/docs/source/contributor-guide/memory_management.md b/docs/source/contributor-guide/memory_management.md
index fec46087a1..3ed1120ee0 100644
--- a/docs/source/contributor-guide/memory_management.md
+++ b/docs/source/contributor-guide/memory_management.md
@@ -297,9 +297,19 @@ consumers.
### Task-shared pools and their lifetime
-A single Spark task can run more than one native plan concurrently: a shuffle runs the pre-shuffle
-operators and the shuffle writer as separate native execution contexts. If each got its own pool,
-the per-task limit would be enforced once per plan rather than once per task.
+A single Spark task can run more than one native plan at a time. A native shuffle is not one of
+these cases, because its writer is planned together with the native operators that feed it. These
+operators do split a task's native work into separate plans:
+
+- `CometUnionExec` and `CometCoalesceExec` read their children through the JVM, so the native plan
+ above them and the native plans below them are separate.
+- `CometCollectLimitExec` and `CometTakeOrderedAndProjectExec` apply their limit in a native plan
+ of their own, over the output of the plan below them.
+- A native Parquet or Iceberg write runs its writer as a native plan of its own, over the output of
+ the plan below it.
+
+If each plan got its own pool, the per-task limit would be enforced once per plan rather than once
+per task.
`acquire_task_shared_pool` (`task_shared.rs`) keeps a process-wide
`HashMap<task_attempt_id, Weak<TaskSharedMemoryPool>>`. Plans in the same task upgrade the existing
diff --git a/docs/source/user-guide/latest/tuning.md b/docs/source/user-guide/latest/tuning.md
index edfc5bc82a..429befc648 100644
--- a/docs/source/user-guide/latest/tuning.md
+++ b/docs/source/user-guide/latest/tuning.md
@@ -127,10 +127,9 @@ The valid pool types are:
- `fair_unified` (default when `spark.memory.offHeap.enabled=true` is set)
- `greedy_unified`
-Both pool types are shared across all native execution contexts within the same Spark task. When
-Comet executes a shuffle, it runs two native execution contexts concurrently (e.g. one for
-pre-shuffle operators and one for the shuffle writer). The shared pool ensures that the combined
-memory usage stays within the per-task limit.
+Both pool types are shared by all the native plans in the same Spark task. A task can run more than
+one native plan at a time, for example the native operators on either side of a union or a
+coalesce. The shared pool ensures that their combined memory usage stays within the per-task limit.
The `fair_unified` pool prevents operators from using more than an even fraction of the available memory
(i.e. `pool_size / num_consumers`, where `num_consumers` counts the memory consumers registered by all of the task's
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org