You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/05/08 21:20:07 UTC

(datafusion-comet) branch asf-site updated: Publish built docs triggered by 52610a8dc675bdeb9245221381d3b08847e131bc

This is an automated email from the ASF dual-hosted git repository.

github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git


The following commit(s) were added to refs/heads/asf-site by this push:
     new 756de9a30 Publish built docs triggered by 52610a8dc675bdeb9245221381d3b08847e131bc
756de9a30 is described below

commit 756de9a30b6bb7bccbd4123d0eb6ce79dac9ff5b
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Fri May 8 21:20:02 2026 +0000

    Publish built docs triggered by 52610a8dc675bdeb9245221381d3b08847e131bc
---
 _sources/contributor-guide/debugging.md.txt | 70 +++++++++++++++++++++++++++++
 contributor-guide/debugging.html            | 63 ++++++++++++++++++++++++++
 searchindex.js                              |  2 +-
 3 files changed, 134 insertions(+), 1 deletion(-)

diff --git a/_sources/contributor-guide/debugging.md.txt b/_sources/contributor-guide/debugging.md.txt
index e5372d922..ce988cac2 100644
--- a/_sources/contributor-guide/debugging.md.txt
+++ b/_sources/contributor-guide/debugging.md.txt
@@ -220,3 +220,73 @@ Example log output:
 ```
 
 When backtraces are enabled (see earlier section) then backtraces will be included for failed allocations.
+
+### Dumping native stream output with the `debug` module
+
+The `native/core/src/debug/` module ships small wrappers that print every
+`RecordBatch` (and, at expression granularity, every `ColumnarValue`) flowing
+through a node. They are gated behind `#[cfg(debug_assertions)]` in
+`native/core/src/lib.rs`, so they exist in every `cargo check` / `cargo test`
+/ `make core` build but are stripped from `make release`. CI builds them on
+every run, so the wrappers cannot bitrot.
+
+Two types are exposed:
+
+- `DebugExecutionDataStream` — wraps an `Arc<dyn ExecutionPlan>`, prints each batch it
+  produces.
+- `DebugExecutionDataPhyExpr` — wraps an `Arc<dyn PhysicalExpr>`, prints the input
+  batch and output `ColumnarValue` for every `evaluate()` call.
+
+Output is `eprintln!` + `dbg!`, i.e. plain stderr of the executor process — the
+same place `rust-gdb` / `dbg!` output lands. It is not routed through `log4rs`,
+so `spark.comet.debug.enabled` and `COMET_LOG_LEVEL` do not affect it.
+
+#### Example: dumping window operator output
+
+In `PhysicalPlanner::create_plan` in `native/core/src/execution/planner.rs`,
+the `OpStruct::Window` arm builds a `BoundedWindowAggExec` and hands it to
+`SparkPlan::new`. Wrap the node between those two steps:
+
+```rust
+let window_agg = Arc::new(BoundedWindowAggExec::try_new(
+    window_expr?,
+    Arc::clone(&child.native_plan),
+    InputOrderMode::Sorted,
+    !partition_exprs.is_empty(),
+)?);
+
+#[cfg(debug_assertions)]
+let window_agg: Arc<dyn datafusion::physical_plan::ExecutionPlan> = {
+    use crate::debug::DebugExecutionDataStream;
+    Arc::new(DebugExecutionDataStream::new("window-output", window_agg))
+};
+
+Ok((
+    scans,
+    shuffle_scans,
+    Arc::new(SparkPlan::new(spark_plan.plan_id, window_agg, vec![child])),
+))
+```
+
+The `let` rebinding keeps the original `window_agg` typed concretely on release
+builds (where the `cfg` block is compiled out) and re-binds it to
+`Arc<dyn ExecutionPlan>` when the debug module is active. No annotation is
+needed on the first `let` because release builds use the original binding
+directly.
+
+Rebuild with `make core` and run a window test, for example:
+
+```sh
+./mvnw test -Dsuites="org.apache.comet.exec.CometWindowExecSuite window query with rangeBetween" -Dtest=none
+```
+
+Sample stderr excerpt (abbreviated):
+
+```text
+[comet-debug] DebugExecutionDataStream[window-output] execute(partition=0)
+[core/src/debug/debug_batch_stream.rs:30] batch = RecordBatch { columns: [...], row_count: 4 }
+[core/src/debug/debug_batch_stream.rs:37] col_idx = 0
+[core/src/debug/debug_batch_stream.rs:37] column = PrimitiveArray<Int32> [1, 1, 2, 2]
+[core/src/debug/debug_batch_stream.rs:37] column.nulls() = None
+...
+```
diff --git a/contributor-guide/debugging.html b/contributor-guide/debugging.html
index 17401b101..c5652f4fd 100644
--- a/contributor-guide/debugging.html
+++ b/contributor-guide/debugging.html
@@ -633,6 +633,69 @@ For example, see: <a class="reference external" href="https://github.com/apache/
 </div>
 <p>When backtraces are enabled (see earlier section) then backtraces will be included for failed allocations.</p>
 </section>
+<section id="dumping-native-stream-output-with-the-debug-module">
+<h3>Dumping native stream output with the <code class="docutils literal notranslate"><span class="pre">debug</span></code> module<a class="headerlink" href="#dumping-native-stream-output-with-the-debug-module" title="Link to this heading">#</a></h3>
+<p>The <code class="docutils literal notranslate"><span class="pre">native/core/src/debug/</span></code> module ships small wrappers that print every
+<code class="docutils literal notranslate"><span class="pre">RecordBatch</span></code> (and, at expression granularity, every <code class="docutils literal notranslate"><span class="pre">ColumnarValue</span></code>) flowing
+through a node. They are gated behind <code class="docutils literal notranslate"><span class="pre">#[cfg(debug_assertions)]</span></code> in
+<code class="docutils literal notranslate"><span class="pre">native/core/src/lib.rs</span></code>, so they exist in every <code class="docutils literal notranslate"><span class="pre">cargo</span> <span class="pre">check</span></code> / <code class="docutils literal notranslate"><span class="pre">cargo</span> <span class="pre">test</span></code>
+/ <code class="docutils literal notranslate"><span class="pre">make</span> <span class="pre">core</span></code> build but are stripped from <code class="docutils literal notranslate"><span class="pre">make</span> <span class="pre">release</span></code>. CI builds them on
+every run, so the wrappers cannot bitrot.</p>
+<p>Two types are exposed:</p>
+<ul class="simple">
+<li><p><code class="docutils literal notranslate"><span class="pre">DebugExecutionDataStream</span></code> — wraps an <code class="docutils literal notranslate"><span class="pre">Arc&lt;dyn</span> <span class="pre">ExecutionPlan&gt;</span></code>, prints each batch it
+produces.</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">DebugExecutionDataPhyExpr</span></code> — wraps an <code class="docutils literal notranslate"><span class="pre">Arc&lt;dyn</span> <span class="pre">PhysicalExpr&gt;</span></code>, prints the input
+batch and output <code class="docutils literal notranslate"><span class="pre">ColumnarValue</span></code> for every <code class="docutils literal notranslate"><span class="pre">evaluate()</span></code> call.</p></li>
+</ul>
+<p>Output is <code class="docutils literal notranslate"><span class="pre">eprintln!</span></code> + <code class="docutils literal notranslate"><span class="pre">dbg!</span></code>, i.e. plain stderr of the executor process — the
+same place <code class="docutils literal notranslate"><span class="pre">rust-gdb</span></code> / <code class="docutils literal notranslate"><span class="pre">dbg!</span></code> output lands. It is not routed through <code class="docutils literal notranslate"><span class="pre">log4rs</span></code>,
+so <code class="docutils literal notranslate"><span class="pre">spark.comet.debug.enabled</span></code> and <code class="docutils literal notranslate"><span class="pre">COMET_LOG_LEVEL</span></code> do not affect it.</p>
+<section id="example-dumping-window-operator-output">
+<h4>Example: dumping window operator output<a class="headerlink" href="#example-dumping-window-operator-output" title="Link to this heading">#</a></h4>
+<p>In <code class="docutils literal notranslate"><span class="pre">PhysicalPlanner::create_plan</span></code> in <code class="docutils literal notranslate"><span class="pre">native/core/src/execution/planner.rs</span></code>,
+the <code class="docutils literal notranslate"><span class="pre">OpStruct::Window</span></code> arm builds a <code class="docutils literal notranslate"><span class="pre">BoundedWindowAggExec</span></code> and hands it to
+<code class="docutils literal notranslate"><span class="pre">SparkPlan::new</span></code>. Wrap the node between those two steps:</p>
+<div class="highlight-rust notranslate"><div class="highlight"><pre><span></span><span class="kd">let</span><span class="w"> </span><span class="n">window_agg</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Arc</span><span class="p">::</span><span class="n">new</span><span class="p">(</span><span class="n">BoundedWindowAggExec</span><span class="p">::</span><span class="n">try_new</span><span class="p">(</span>
+<span class="w">    </span><span class="n">window_expr</span><span class="o">?</span><span class="p">,</span>
+<span class="w">    </span><span class="n">Arc</span><span class="p">::</span><span class="n">clone</span><span class="p">(</span><span class="o">&amp;</span><span class="n">child</span><span class="p">.</span><span class="n">native_plan</span><span class="p">),</span>
+<span class="w">    </span><span class="n">InputOrderMode</span><span class="p">::</span><span class="n">Sorted</span><span class="p">,</span>
+<span class="w">    </span><span class="o">!</span><span class="n">partition_exprs</span><span class="p">.</span><span class="n">is_empty</span><span class="p">(),</span>
+<span class="p">)</span><span class="o">?</span><span class="p">);</span>
+
+<span class="cp">#[cfg(debug_assertions)]</span>
+<span class="kd">let</span><span class="w"> </span><span class="n">window_agg</span><span class="p">:</span><span class="w"> </span><span class="nc">Arc</span><span class="o">&lt;</span><span class="k">dyn</span><span class="w"> </span><span class="n">datafusion</span><span class="p">::</span><span class="n">physical_plan</span><span class="p">::</span><span class="n">ExecutionPlan</span><span class="o">&gt;</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><s [...]
+<span class="w">    </span><span class="k">use</span><span class="w"> </span><span class="k">crate</span><span class="p">::</span><span class="n">debug</span><span class="p">::</span><span class="n">DebugExecutionDataStream</span><span class="p">;</span>
+<span class="w">    </span><span class="n">Arc</span><span class="p">::</span><span class="n">new</span><span class="p">(</span><span class="n">DebugExecutionDataStream</span><span class="p">::</span><span class="n">new</span><span class="p">(</span><span class="s">&quot;window-output&quot;</span><span class="p">,</span><span class="w"> </span><span class="n">window_agg</span><span class="p">))</span>
+<span class="p">};</span>
+
+<span class="nb">Ok</span><span class="p">((</span>
+<span class="w">    </span><span class="n">scans</span><span class="p">,</span>
+<span class="w">    </span><span class="n">shuffle_scans</span><span class="p">,</span>
+<span class="w">    </span><span class="n">Arc</span><span class="p">::</span><span class="n">new</span><span class="p">(</span><span class="n">SparkPlan</span><span class="p">::</span><span class="n">new</span><span class="p">(</span><span class="n">spark_plan</span><span class="p">.</span><span class="n">plan_id</span><span class="p">,</span><span class="w"> </span><span class="n">window_agg</span><span class="p">,</span><span class="w"> </span><span class="fm">vec!</span><span class=" [...]
+<span class="p">))</span>
+</pre></div>
+</div>
+<p>The <code class="docutils literal notranslate"><span class="pre">let</span></code> rebinding keeps the original <code class="docutils literal notranslate"><span class="pre">window_agg</span></code> typed concretely on release
+builds (where the <code class="docutils literal notranslate"><span class="pre">cfg</span></code> block is compiled out) and re-binds it to
+<code class="docutils literal notranslate"><span class="pre">Arc&lt;dyn</span> <span class="pre">ExecutionPlan&gt;</span></code> when the debug module is active. No annotation is
+needed on the first <code class="docutils literal notranslate"><span class="pre">let</span></code> because release builds use the original binding
+directly.</p>
+<p>Rebuild with <code class="docutils literal notranslate"><span class="pre">make</span> <span class="pre">core</span></code> and run a window test, for example:</p>
+<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>./mvnw<span class="w"> </span><span class="nb">test</span><span class="w"> </span>-Dsuites<span class="o">=</span><span class="s2">&quot;org.apache.comet.exec.CometWindowExecSuite window query with rangeBetween&quot;</span><span class="w"> </span>-Dtest<span class="o">=</span>none
+</pre></div>
+</div>
+<p>Sample stderr excerpt (abbreviated):</p>
+<div class="highlight-text notranslate"><div class="highlight"><pre><span></span>[comet-debug] DebugExecutionDataStream[window-output] execute(partition=0)
+[core/src/debug/debug_batch_stream.rs:30] batch = RecordBatch { columns: [...], row_count: 4 }
+[core/src/debug/debug_batch_stream.rs:37] col_idx = 0
+[core/src/debug/debug_batch_stream.rs:37] column = PrimitiveArray&lt;Int32&gt; [1, 1, 2, 2]
+[core/src/debug/debug_batch_stream.rs:37] column.nulls() = None
+...
+</pre></div>
+</div>
+</section>
+</section>
 </section>
 </section>
 
diff --git a/searchindex.js b/searchindex.js
index 09b582283..692888929 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"1. Format Your Code": [[13, "format-your-code"]], "1. Install Comet": [[15, "install-comet"], [23, "install-comet"]], "1. Native Operators (nativeExecs map)": [[3, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[13, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[15, "clone-iceberg-and-apply-diff"]], "2. Clone Spark and Apply Diff": [[23, "clone-spark-and-apply-diff"]], "2. Sink Operators (sinks map)": [[3, "sink-operators-sinks-m [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"1. Format Your Code": [[13, "format-your-code"]], "1. Install Comet": [[15, "install-comet"], [23, "install-comet"]], "1. Native Operators (nativeExecs map)": [[3, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[13, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[15, "clone-iceberg-and-apply-diff"]], "2. Clone Spark and Apply Diff": [[23, "clone-spark-and-apply-diff"]], "2. Sink Operators (sinks map)": [[3, "sink-operators-sinks-m [...]
\ No newline at end of file


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org