You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/09/21 06:54:21 UTC
(datafusion-comet) branch asf-site updated: Publish built docs triggered by 5ca149928f7743bfe7a96feadea5e0f9bed1412f
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 7e7766a081 Publish built docs triggered by 5ca149928f7743bfe7a96feadea5e0f9bed1412f
7e7766a081 is described below
commit 7e7766a0814465a46079fe27090697382b0a7487
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Mon Sep 21 06:54:16 2026 +0000
Publish built docs triggered by 5ca149928f7743bfe7a96feadea5e0f9bed1412f
---
.buildinfo | 2 +-
...id-013012468fa27c9e9fdb0a14196d357c619c68a8.svg | 1 +
...id-24951e49e88a2a575316ecedeb5309d49a140d7a.svg | 1 +
...id-d0bb31b07009ce49e46219a49cdd8e6531dd24ce.svg | 1 +
contributor-guide/ci.html | 16 ++++++++
contributor-guide/memory_management.html | 47 ++++++++++++++++++++++
6 files changed, 67 insertions(+), 1 deletion(-)
diff --git a/.buildinfo b/.buildinfo
index c2bc25f3aa..b2a695f0bf 100644
--- a/.buildinfo
+++ b/.buildinfo
@@ -1,4 +1,4 @@
# Sphinx build info version 1
# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
-config: e33ca1ce50b275ab64dabb2fb0347e98
+config: eb8ea1a0b32c78350beb1996fa5f6c03
tags: 645f666f9bcd5a90fca523b33c5a78b7
diff --git a/_images/mermaid-013012468fa27c9e9fdb0a14196d357c619c68a8.svg b/_images/mermaid-013012468fa27c9e9fdb0a14196d357c619c68a8.svg
new file mode 100644
index 0000000000..6bd32b44e8
--- /dev/null
+++ b/_images/mermaid-013012468fa27c9e9fdb0a14196d357c619c68a8.svg
@@ -0,0 +1 @@
+<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 942" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-svg [...]
\ No newline at end of file
diff --git a/_images/mermaid-24951e49e88a2a575316ecedeb5309d49a140d7a.svg b/_images/mermaid-24951e49e88a2a575316ecedeb5309d49a140d7a.svg
new file mode 100644
index 0000000000..92fc4294f2
--- /dev/null
+++ b/_images/mermaid-24951e49e88a2a575316ecedeb5309d49a140d7a.svg
@@ -0,0 +1 @@
+<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 1170.02px; background-color: transparent;" viewBox="0 0 1170.015625 1229" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-svg .ed [...]
\ No newline at end of file
diff --git a/_images/mermaid-d0bb31b07009ce49e46219a49cdd8e6531dd24ce.svg b/_images/mermaid-d0bb31b07009ce49e46219a49cdd8e6531dd24ce.svg
new file mode 100644
index 0000000000..f848f9b61f
--- /dev/null
+++ b/_images/mermaid-d0bb31b07009ce49e46219a49cdd8e6531dd24ce.svg
@@ -0,0 +1 @@
+<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 1661px; background-color: transparent;" viewBox="0 0 1661 660" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-svg .edge-animatio [...]
\ No newline at end of file
diff --git a/contributor-guide/ci.html b/contributor-guide/ci.html
index e4fd820df1..1c48d21a3e 100644
--- a/contributor-guide/ci.html
+++ b/contributor-guide/ci.html
@@ -559,6 +559,22 @@ mechanics behind the configuration are documented in
preflight checks first (license headers, Markdown formatting, workflow linting, the CI config
checks), computes which heavy jobs the changed files are relevant to, and fans out to those jobs.
Which jobs run also depends on the event:</p>
+<object data="../_images/mermaid-013012468fa27c9e9fdb0a14196d357c619c68a8.svg" type="image/svg+xml">
+ <p class="warning">flowchart LR
+ PR([pull request]) --> PRTIER
+ PR -. with label .-> QUEUE
+ PR -. with label .-> NIGHTLY
+ PR -. with label .-> S34
+ MQ([merge queue]) --> PRTIER
+ MQ --> QUEUE
+ CRON([schedule<br>06:00 UTC]) --> NIGHTLY
+ PUSH([push to main]) --> CACHE
+
+ PRTIER["PR tier<br>Linux build, lint, Rust tests<br>TPC-H / TPC-DS<br>Comet suites, Spark 4.1"]
+ QUEUE["Queue tier, on top of the PR tier<br>Spark SQL, Spark 4.1<br>Iceberg 1.11<br>macOS build and Comet suites<br>Benchmark check, Delta gate<br>PyArrow UDF, Spark 4.0 / 4.1 / 4.2"]
+ NIGHTLY["Nightly tier<br>Comet suites, Spark 3.4 / 3.5 / 4.0 / 4.2<br>Spark SQL, Spark 3.5 / 4.0<br>Iceberg 1.8 / 1.9 / 1.10"]
+ S34["Neither tier<br>Spark SQL, Spark 3.4"]
+ CACHE["Cache-refresh-only mode<br>the four cache-writing jobs, plus Lint"]</p></object>
<p>Suite by suite:</p>
<div class="pst-scrollable-table-container"><table class="table">
<thead>
diff --git a/contributor-guide/memory_management.html b/contributor-guide/memory_management.html
index 0452b0156d..2a4408aabb 100644
--- a/contributor-guide/memory_management.html
+++ b/contributor-guide/memory_management.html
@@ -681,6 +681,36 @@ returns <code class="docutils literal notranslate"><span class="pre">CometUnifie
<code class="docutils literal notranslate"><span class="pre">spark.memory.offHeap.size</span></code>, so shuffle pages are arbitrated against Spark’s other consumers in the
same task like any other allocation.</p>
<p>Which allocator each call site uses, and who ends up charged for the bytes:</p>
+<object data="../_images/mermaid-24951e49e88a2a575316ecedeb5309d49a140d7a.svg" type="image/svg+xml">
+ <p class="warning">flowchart LR
+ subgraph SITES["JVM Arrow allocation sites"]
+ NU["NativeUtil<br>FFI structs, imports, exports"]
+ UDF["CometUdfBridge<br>JVM UDF inputs and result"]
+ CGO["CometBatchKernelCodegenOutput<br>codegen UDF output"]
+ SR["StreamReader<br>shuffle and IPC reads"]
+ NAS["CometNativeArrowSource<br>stream and readerBatchIter"]
+ CACHE["ArrowCachedBatchSerializer"]
+ PY["CometArrowPythonRunnerBase"]
+ BC["Utils broadcast-coalesce"]
+ end
+
+ ROOT["CometArrowAllocator<br>RootAllocator, no limit<br>no allocation listener"]
+ SHUF["Comet JVM shuffle pages<br>CometUnifiedShuffleMemoryAllocator"]
+ NPOOL["Comet native memory pool<br>declared reservations only"]
+ TMM["Spark off-heap execution pool<br>TaskMemoryManager"]
+ NOBODY["accounted by nobody"]
+
+ NU --> ROOT
+ UDF --> ROOT
+ CGO --> ROOT
+ SR --> ROOT
+ NAS --> ROOT
+ CACHE --> ROOT
+ PY --> ROOT
+ BC --> ROOT
+ ROOT --> NOBODY
+ SHUF --> TMM
+ NPOOL -->|"CometTaskMemoryManager over JNI"| TMM</p></object>
<section id="constraints-on-a-comet-memory-consumer">
<h3>Constraints on a Comet memory consumer<a class="headerlink" href="#constraints-on-a-comet-memory-consumer" title="Link to this heading">#</a></h3>
<p><code class="docutils literal notranslate"><span class="pre">CometTaskMemoryManager</span></code> is the one place where Comet code acts as a Spark <code class="docutils literal notranslate"><span class="pre">MemoryConsumer</span></code>, and the
@@ -925,6 +955,23 @@ hard ceiling on the sum of everything in the container. That cgroup counts, amon
<li><p>page cache charged to the cgroup by the container’s file I/O, including spill files.</p></li>
</ul>
<p>Everything the cgroup counts, and who accounts for each part:</p>
+<object data="../_images/mermaid-d0bb31b07009ce49e46219a49cdd8e6531dd24ce.svg" type="image/svg+xml">
+ <p class="warning">flowchart TB
+ subgraph CG["pod cgroup memory.max, kernel OOM kill above this"]
+ subgraph SEEN["visible to Spark's accounting"]
+ HEAP["JVM heap<br>execution and storage<br>spark.executor.memory"]
+ TUNG["Spark Tungsten off-heap<br>TaskMemoryManager"]
+ SHUFP["Comet JVM shuffle pages<br>CometUnifiedShuffleMemoryAllocator"]
+ NATRES["Comet native heap, reserved<br>operators that call try_grow<br>declared to Spark over JNI, never measured"]
+ end
+ subgraph NONE["accounted by nobody"]
+ NATUND["Comet native heap, undeclared<br>kernels, array builders, decompression<br>Parquet metadata, object_store, tokio"]
+ ARROWR["Comet JVM Arrow<br>CometArrowAllocator, unbounded"]
+ NONHEAP["JVM non-heap<br>metaspace, code cache, thread stacks<br>GC structures, Netty direct buffers"]
+ PAGEC["page cache charged to the cgroup<br>file I/O, including spill files"]
+ FRAG["allocator overhead<br>fragmentation, padding<br>jemalloc retained and dirty pages"]
+ end
+ end</p></object>
<p>Spark’s accounting covers the first group, though not in the same sense throughout it. The JVM
heap, Tungsten pages and Comet’s shuffle pages are allocated by JVM code that reports what it
allocated. A native reservation is a number an operator declared before allocating: <code class="docutils literal notranslate"><span class="pre">try_grow</span></code>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org