You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/09/08 20:28:43 UTC
(datafusion-comet) branch asf-site updated (1e25b8fb9e -> 5011132374)
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
discard 1e25b8fb9e Publish built docs triggered by 8c7b706a327687f7cd0abcb055a2a9308542974d
new 5011132374 Publish built docs triggered by 03875d4776a5edefbfedb5ddcb26716c29aec43c
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (1e25b8fb9e)
\
N -- N -- N refs/heads/asf-site (5011132374)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
_sources/user-guide/latest/iceberg-writes.md.txt | 9 +++++++++
searchindex.js | 2 +-
user-guide/latest/iceberg-writes.html | 9 +++++++++
3 files changed, 19 insertions(+), 1 deletion(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
(datafusion-comet) 01/01: Publish built docs triggered by 03875d4776a5edefbfedb5ddcb26716c29aec43c
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
commit 501113237494ffc019ad7d173dd0b360355d5296
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Tue Sep 8 20:28:37 2026 +0000
Publish built docs triggered by 03875d4776a5edefbfedb5ddcb26716c29aec43c
---
_sources/user-guide/latest/configs.md.txt | 1 +
_sources/user-guide/latest/iceberg-writes.md.txt | 9 ++++++
_sources/user-guide/latest/metrics.md.txt | 20 ++++++++++++
_sources/user-guide/latest/tuning.md.txt | 20 ++++++++++++
searchindex.js | 2 +-
user-guide/latest/configs.html | 26 +++++++++-------
user-guide/latest/iceberg-writes.html | 9 ++++++
user-guide/latest/metrics.html | 39 ++++++++++++++++++++++++
user-guide/latest/tuning.html | 23 +++++++++++++-
9 files changed, 136 insertions(+), 13 deletions(-)
diff --git a/_sources/user-guide/latest/configs.md.txt b/_sources/user-guide/latest/configs.md.txt
index f63289cc13..55fd65cb4b 100644
--- a/_sources/user-guide/latest/configs.md.txt
+++ b/_sources/user-guide/latest/configs.md.txt
@@ -62,6 +62,7 @@ Comet provides the following configuration settings.
| `spark.comet.exec.enabled` | Whether to enable Comet native vectorized execution for Spark. This controls whether Spark should convert operators into their Comet counterparts and execute them in native space. Note: each operator is associated with a separate config in the format of `spark.comet.exec.<operator_name>.enabled` at the moment, and both the config and this need to be turned on, in order for the operator to be executed in native. | true |
| `spark.comet.exec.forceShuffledHashJoin` | Experimental feature to force Spark to replace SortMergeJoin with ShuffledHashJoin for improved performance. This feature is not stable yet. For more information, refer to the [Comet Tuning Guide](https://datafusion.apache.org/comet/user-guide/latest/tuning.html). | false |
| `spark.comet.exec.inMemoryCache.enabled` | Whether to enable Comet native execution for in-memory cached tables. Its value at startup also decides whether CometDriverPlugin installs Comet's cache serializer, which stores cached data in Arrow format. Because spark.sql.cache.serializer is a static config, the cached format is fixed for the application, and disabling this at runtime only sends cached scans back to Spark's execution path. Relations whose schema Comet's Arrow writer does no [...]
+| `spark.comet.exec.join.dynamicFilter.enabled` | Experimental opt-in: use a completed native hash join build's key domain to prune eligible native Parquet row groups and filter probe batches before the hash probe. Supports inner joins with one direct signed integer key and one native partition per input, including both Spark build sides. The probe filter remains active when reader pruning is unavailable. Unsupported joins retain their existing execution path. Filters do not cross Spark [...]
| `spark.comet.exec.pyarrowUDF.enabled` | Experimental: whether to enable optimized execution of PyArrow UDFs (mapInArrow/mapInPandas). When enabled, Comet passes Arrow columnar data directly to Python UDFs without the intermediate Arrow-to-Row-to-Arrow conversion that Spark normally performs. Disabled by default while the feature stabilizes. | false |
| `spark.comet.exec.scalaUDF.codegen.enabled` | Whether to route Spark `ScalaUDF` expressions through Comet's Arrow-direct codegen dispatcher. When enabled, a supported ScalaUDF is compiled into a per-batch kernel that reads and writes Arrow vectors directly from native execution. When disabled, plans containing a ScalaUDF fall back to Spark for the enclosing operator. The same dispatcher backs the regex family (`rlike`, `regexp_replace`, `split`, `regexp_extract`, `regexp_extract_all`, [...]
| `spark.comet.exec.strictFloatingPoint` | When enabled, fall back to Spark for floating-point operations that may differ from Spark, such as when comparing or sorting -0.0 and 0.0. For more information, refer to the [Comet Compatibility Guide](https://datafusion.apache.org/comet/user-guide/latest/compatibility/index.html). | false |
diff --git a/_sources/user-guide/latest/iceberg-writes.md.txt b/_sources/user-guide/latest/iceberg-writes.md.txt
index 49f3500920..62a4df1f12 100644
--- a/_sources/user-guide/latest/iceberg-writes.md.txt
+++ b/_sources/user-guide/latest/iceberg-writes.md.txt
@@ -251,6 +251,15 @@ a data file but not what any reader computes from it:
differences (iceberg-java checks the target file size every 1000 rows and names files
`<partition>-<task>-<operation>-<count>`; iceberg-rust checks per batch and uses a
process-local counter).
+- Partition directory names match iceberg-java 1.8+'s `PartitionSpec.partitionToPath` for every
+ partition type except `float` and `double`, where the value is rendered with Rust's shortest
+ representation instead of `Float.toString` / `Double.toString` (`f=1` where iceberg-java writes
+ `f=1.0`). On Iceberg 1.5.x, which the Spark 3.4 profile pins, iceberg-java itself spelled
+ `timestamp` and `timestamptz` directories with `LocalDateTime.toString()` /
+ `OffsetDateTime.toString()` (`ts=1969-12-31T23:59:58.500Z`) and left the partition field name
+ unescaped; Comet uses the 1.8+ spelling on every profile. Distinct partition values still get
+ distinct directories in all cases, and no reader parses these names — files are resolved through
+ committed manifests. Iceberg deprecated float and double partitioning in 1.3.
- Compressed page bytes are implementation-defined: the codec and any explicit level are
translated, but parquet-rs and parquet-mr embed different encoder implementations and
defaults (zstd default levels, LZ4 framing), so byte-identical output is not achievable even
diff --git a/_sources/user-guide/latest/metrics.md.txt b/_sources/user-guide/latest/metrics.md.txt
index f1804e5c13..9d35536fc2 100644
--- a/_sources/user-guide/latest/metrics.md.txt
+++ b/_sources/user-guide/latest/metrics.md.txt
@@ -29,6 +29,26 @@ Comet operators report the following metrics in the Spark SQL UI.
| ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `scan time` | Total time to scan a Parquet file. This is not comparable to the same metric in Spark because Comet's scan metric is more accurate. Although both Comet and Spark measure the time in nanoseconds, Spark rounds this time to the nearest millisecond per batch and Comet does not. |
+### Hash Joins
+
+With `spark.comet.exec.join.dynamicFilter.enabled=true`, native broadcast and shuffled hash joins
+report these additional metric keys. See [Join Runtime Filters](tuning.md#join-runtime-filters) for
+eligibility and reader restrictions.
+
+| Metric | Description |
+| ---------------------------------------- | ----------------------------------------------------------------- |
+| `dynamic_filter_rows_evaluated` | Probe rows evaluated by the runtime filter. |
+| `dynamic_filter_rows_pruned` | Probe rows rejected by that filter before the hash probe. |
+| `dynamic_filter_rows_bypassed` | Probe rows passed through while the runtime filter is inactive. |
+| `dynamic_filter_eval_time` | Time evaluating the runtime filter. |
+| `dynamic_filter_reader_filters_attached` | Executions that attach their runtime filter to a native reader. |
+| `dynamic_filter_reader_filters_skipped` | Executions whose probe input is ineligible for reader attachment. |
+
+The row counters measure residual filtering of decoded probe batches. They exclude rows skipped
+by the reader. An attached filter does not guarantee that any row groups are pruned: compare the
+probe scan's `bytes_scanned` and `row_groups_pruned_statistics` with filtering disabled to assess
+reader savings. Existing join, scan, and intervening filter metrics retain their own meanings.
+
### Exchange
Comet adds some additional metrics:
diff --git a/_sources/user-guide/latest/tuning.md.txt b/_sources/user-guide/latest/tuning.md.txt
index 0ad8ec992c..ceec8b1a15 100644
--- a/_sources/user-guide/latest/tuning.md.txt
+++ b/_sources/user-guide/latest/tuning.md.txt
@@ -192,6 +192,26 @@ to test with both for your specific workloads.
To configure Comet to convert `SortMergeJoin` to `ShuffledHashJoin`, set `spark.comet.exec.forceShuffledHashJoin=true`.
+### Join Runtime Filters
+
+Set `spark.comet.exec.join.dynamicFilter.enabled=true` to try experimental native hash join runtime
+filtering. It is disabled by default. Eligible joins are inner joins with one direct signed integer
+key (`TINYINT`, `SMALLINT`, `INT`, or `BIGINT`) and one native partition per input within each task.
+Both broadcast and shuffled hash joins support either Spark build side. Unsupported joins keep
+their existing execution path.
+
+Once the build completes, its key domain filters probe batches before the hash probe. Eligible
+native Parquet readers also use the domain to prune row groups. Reader attachment can pass through
+direct-column `IS NOT NULL` checks, including conjunctions, and remaps columns when the scan itself
+projects the file schema. The original null checks and residual runtime filter remain in place.
+The original join still verifies matches, including any hash collisions admitted by the filter.
+Standalone projections, other filter expressions, and limits prevent reader attachment.
+
+Filters stay within the task's native plan and do not propagate across Spark exchanges or JVM/Arrow
+boundaries. A shuffled hash join can still filter probe batches after shuffle, but it cannot send
+its filter back to an earlier scan stage. Compare the [runtime-filter and scan metrics](metrics.md#hash-joins)
+with the setting disabled to distinguish reduced hash-probe work from reader I/O savings.
+
## Shuffle
Comet provides accelerated shuffle implementations that can be used to improve the performance of your queries.
diff --git a/searchindex.js b/searchindex.js
index a761c0b8a7..e228591c6b 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[55, "id1"]], "%": [[53, "id1"]], "&": [[43, "id1"]], "*": [[53, "id2"]], "+": [[53, "id3"]], "-": [[53, "id4"]], "/": [[53, "id5"]], "1. Format Your Code": [[40, "format-your-code"]], "1. Install Comet": [[61, "install-comet"], [71, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[40, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[61, "clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[55, "id1"]], "%": [[53, "id1"]], "&": [[43, "id1"]], "*": [[53, "id2"]], "+": [[53, "id3"]], "-": [[53, "id4"]], "/": [[53, "id5"]], "1. Format Your Code": [[40, "format-your-code"]], "1. Install Comet": [[61, "install-comet"], [71, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[40, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[61, "clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/latest/configs.html b/user-guide/latest/configs.html
index efbe1bf8dd..5e93cf21b0 100644
--- a/user-guide/latest/configs.html
+++ b/user-guide/latest/configs.html
@@ -893,47 +893,51 @@ under the License.
<td><p>Whether to enable Comet native execution for in-memory cached tables. Its value at startup also decides whether CometDriverPlugin installs Comet’s cache serializer, which stores cached data in Arrow format. Because spark.sql.cache.serializer is a static config, the cached format is fixed for the application, and disabling this at runtime only sends cached scans back to Spark’s execution path. Relations whose schema Comet’s Arrow writer does not support are always cached in Spark’s [...]
<td><p>false</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.pyarrowUDF.enabled</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.join.dynamicFilter.enabled</span></code></p></td>
+<td><p>Experimental opt-in: use a completed native hash join build’s key domain to prune eligible native Parquet row groups and filter probe batches before the hash probe. Supports inner joins with one direct signed integer key and one native partition per input, including both Spark build sides. The probe filter remains active when reader pruning is unavailable. Unsupported joins retain their existing execution path. Filters do not cross Spark exchanges or JVM/Arrow boundaries.</p></td>
+<td><p>false</p></td>
+</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.pyarrowUDF.enabled</span></code></p></td>
<td><p>Experimental: whether to enable optimized execution of PyArrow UDFs (mapInArrow/mapInPandas). When enabled, Comet passes Arrow columnar data directly to Python UDFs without the intermediate Arrow-to-Row-to-Arrow conversion that Spark normally performs. Disabled by default while the feature stabilizes.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.scalaUDF.codegen.enabled</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.scalaUDF.codegen.enabled</span></code></p></td>
<td><p>Whether to route Spark <code class="docutils literal notranslate"><span class="pre">ScalaUDF</span></code> expressions through Comet’s Arrow-direct codegen dispatcher. When enabled, a supported ScalaUDF is compiled into a per-batch kernel that reads and writes Arrow vectors directly from native execution. When disabled, plans containing a ScalaUDF fall back to Spark for the enclosing operator. The same dispatcher backs the regex family (<code class="docutils literal notranslate">< [...]
<td><p>true</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.strictFloatingPoint</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.strictFloatingPoint</span></code></p></td>
<td><p>When enabled, fall back to Spark for floating-point operations that may differ from Spark, such as when comparing or sorting -0.0 and 0.0. For more information, refer to the <a class="reference external" href="https://datafusion.apache.org/comet/user-guide/latest/compatibility/index.html">Comet Compatibility Guide</a>.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.transitionRevert.enabled</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.transitionRevert.enabled</span></code></p></td>
<td><p>When enabled, Comet reverts a query stage to Spark row-based execution if the number of columnar-to-row (C2R) transitions in the stage exceeds the configured threshold. This avoids the overhead of repeated format conversions in stages where many operators fall back to row-based execution.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.transitionRevert.maxTransitions</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.transitionRevert.maxTransitions</span></code></p></td>
<td><p>The maximum number of columnar-to-row (C2R) transitions allowed in a single query stage before Comet reverts the entire stage to Spark row-based execution. When columnar shuffle is enabled, each such C2R typically implies a corresponding row-to-columnar conversion to feed back into the columnar shuffle, so each counted C2R is a useful proxy for the conversion overhead in the stage. Set to 0 to revert any stage with transitions. Only effective when spark.comet.exec.transitionRevert [...]
<td><p>2</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.metrics.enabled</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.metrics.enabled</span></code></p></td>
<td><p>Whether to enable Comet metrics reporting through Spark’s external monitoring system. When enabled, Comet exposes metrics such as native operators, Spark operators, queries planned, transitions, and acceleration ratio. These metrics can be visualized through tools like Grafana when a metrics sink (e.g., Prometheus) is configured. Disabled by default because Spark plan traversal adds overhead and metrics require a sink to be useful. This config must be set before the SparkSession i [...]
<td><p>false</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.metrics.updateInterval</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.metrics.updateInterval</span></code></p></td>
<td><p>The interval in milliseconds to update metrics. If interval is negative, metrics will be updated upon task completion.</p></td>
<td><p>3000</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.nativeLoadRequired</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.nativeLoadRequired</span></code></p></td>
<td><p>Whether to require Comet native library to load successfully when Comet is enabled. If not, Comet will silently fallback to Spark when it fails to load the native lib. Otherwise, an error will be thrown and the Spark job will be aborted.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.operator.DataWritingCommandExec.allowIncompatible</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.operator.DataWritingCommandExec.allowIncompatible</span></code></p></td>
<td><p>Whether to allow incompatibility for operator: DataWritingCommandExec. False by default. Can be overridden with SPARK_COMET_OPERATOR_DATAWRITINGCOMMANDEXEC_ALLOWINCOMPATIBLE env variable It can be overridden by the environment variable <code class="docutils literal notranslate"><span class="pre">SPARK_COMET_OPERATOR_DATAWRITINGCOMMANDEXEC_ALLOWINCOMPATIBLE</span></code>.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.sparkToColumnar.enabled</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.sparkToColumnar.enabled</span></code></p></td>
<td><p>Whether to enable Spark to Arrow columnar conversion. When this is turned on, Comet will convert operators in <code class="docutils literal notranslate"><span class="pre">spark.comet.sparkToColumnar.supportedOperatorList</span></code> into Arrow columnar format before processing.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.sparkToColumnar.supportedOperatorList</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.sparkToColumnar.supportedOperatorList</span></code></p></td>
<td><p>A comma-separated list of operators that will be converted to Arrow columnar format when <code class="docutils literal notranslate"><span class="pre">spark.comet.sparkToColumnar.enabled</span></code> is true.</p></td>
<td><p>Range,InMemoryTableScan,RDDScan,OneRowRelation</p></td>
</tr>
diff --git a/user-guide/latest/iceberg-writes.html b/user-guide/latest/iceberg-writes.html
index 60a94a7861..1a59c15232 100644
--- a/user-guide/latest/iceberg-writes.html
+++ b/user-guide/latest/iceberg-writes.html
@@ -1031,6 +1031,15 @@ parquet-rs buffers by row count. File rolling and file naming follow the same ca
differences (iceberg-java checks the target file size every 1000 rows and names files
<code class="docutils literal notranslate"><span class="pre"><partition>-<task>-<operation>-<count></span></code>; iceberg-rust checks per batch and uses a
process-local counter).</p></li>
+<li><p>Partition directory names match iceberg-java 1.8+’s <code class="docutils literal notranslate"><span class="pre">PartitionSpec.partitionToPath</span></code> for every
+partition type except <code class="docutils literal notranslate"><span class="pre">float</span></code> and <code class="docutils literal notranslate"><span class="pre">double</span></code>, where the value is rendered with Rust’s shortest
+representation instead of <code class="docutils literal notranslate"><span class="pre">Float.toString</span></code> / <code class="docutils literal notranslate"><span class="pre">Double.toString</span></code> (<code class="docutils literal notranslate"><span class="pre">f=1</span></code> where iceberg-java writes
+<code class="docutils literal notranslate"><span class="pre">f=1.0</span></code>). On Iceberg 1.5.x, which the Spark 3.4 profile pins, iceberg-java itself spelled
+<code class="docutils literal notranslate"><span class="pre">timestamp</span></code> and <code class="docutils literal notranslate"><span class="pre">timestamptz</span></code> directories with <code class="docutils literal notranslate"><span class="pre">LocalDateTime.toString()</span></code> /
+<code class="docutils literal notranslate"><span class="pre">OffsetDateTime.toString()</span></code> (<code class="docutils literal notranslate"><span class="pre">ts=1969-12-31T23:59:58.500Z</span></code>) and left the partition field name
+unescaped; Comet uses the 1.8+ spelling on every profile. Distinct partition values still get
+distinct directories in all cases, and no reader parses these names — files are resolved through
+committed manifests. Iceberg deprecated float and double partitioning in 1.3.</p></li>
<li><p>Compressed page bytes are implementation-defined: the codec and any explicit level are
translated, but parquet-rs and parquet-mr embed different encoder implementations and
defaults (zstd default levels, LZ4 framing), so byte-identical output is not achievable even
diff --git a/user-guide/latest/metrics.html b/user-guide/latest/metrics.html
index b3fb934593..f869aede12 100644
--- a/user-guide/latest/metrics.html
+++ b/user-guide/latest/metrics.html
@@ -795,6 +795,44 @@ under the License.
</table>
</div>
</section>
+<section id="hash-joins">
+<h3>Hash Joins<a class="headerlink" href="#hash-joins" title="Link to this heading">#</a></h3>
+<p>With <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.join.dynamicFilter.enabled=true</span></code>, native broadcast and shuffled hash joins
+report these additional metric keys. See <a class="reference internal" href="tuning.html#join-runtime-filters"><span class="std std-ref">Join Runtime Filters</span></a> for
+eligibility and reader restrictions.</p>
+<div class="pst-scrollable-table-container"><table class="table">
+<thead>
+<tr class="row-odd"><th class="head"><p>Metric</p></th>
+<th class="head"><p>Description</p></th>
+</tr>
+</thead>
+<tbody>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">dynamic_filter_rows_evaluated</span></code></p></td>
+<td><p>Probe rows evaluated by the runtime filter.</p></td>
+</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">dynamic_filter_rows_pruned</span></code></p></td>
+<td><p>Probe rows rejected by that filter before the hash probe.</p></td>
+</tr>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">dynamic_filter_rows_bypassed</span></code></p></td>
+<td><p>Probe rows passed through while the runtime filter is inactive.</p></td>
+</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">dynamic_filter_eval_time</span></code></p></td>
+<td><p>Time evaluating the runtime filter.</p></td>
+</tr>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">dynamic_filter_reader_filters_attached</span></code></p></td>
+<td><p>Executions that attach their runtime filter to a native reader.</p></td>
+</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">dynamic_filter_reader_filters_skipped</span></code></p></td>
+<td><p>Executions whose probe input is ineligible for reader attachment.</p></td>
+</tr>
+</tbody>
+</table>
+</div>
+<p>The row counters measure residual filtering of decoded probe batches. They exclude rows skipped
+by the reader. An attached filter does not guarantee that any row groups are pruned: compare the
+probe scan’s <code class="docutils literal notranslate"><span class="pre">bytes_scanned</span></code> and <code class="docutils literal notranslate"><span class="pre">row_groups_pruned_statistics</span></code> with filtering disabled to assess
+reader savings. Existing join, scan, and intervening filter metrics retain their own meanings.</p>
+</section>
<section id="exchange">
<h3>Exchange<a class="headerlink" href="#exchange" title="Link to this heading">#</a></h3>
<p>Comet adds some additional metrics:</p>
@@ -983,6 +1021,7 @@ Spark’s for large files. Comet’s value reflects what the storage layer actua
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#spark-sql-metrics">Spark SQL Metrics</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#cometscanexec">CometScanExec</a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#hash-joins">Hash Joins</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#exchange">Exchange</a></li>
</ul>
</li>
diff --git a/user-guide/latest/tuning.html b/user-guide/latest/tuning.html
index ff9f700442..02122ec67e 100644
--- a/user-guide/latest/tuning.html
+++ b/user-guide/latest/tuning.html
@@ -926,6 +926,24 @@ to configure Comet to convert <code class="docutils literal notranslate"><span c
<code class="docutils literal notranslate"><span class="pre">ShuffledHashJoin</span></code> so this could result in OOM. Also, <code class="docutils literal notranslate"><span class="pre">SortMergeJoin</span></code> may still be faster in some cases. It is best
to test with both for your specific workloads.</p>
<p>To configure Comet to convert <code class="docutils literal notranslate"><span class="pre">SortMergeJoin</span></code> to <code class="docutils literal notranslate"><span class="pre">ShuffledHashJoin</span></code>, set <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.forceShuffledHashJoin=true</span></code>.</p>
+<section id="join-runtime-filters">
+<h3>Join Runtime Filters<a class="headerlink" href="#join-runtime-filters" title="Link to this heading">#</a></h3>
+<p>Set <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.join.dynamicFilter.enabled=true</span></code> to try experimental native hash join runtime
+filtering. It is disabled by default. Eligible joins are inner joins with one direct signed integer
+key (<code class="docutils literal notranslate"><span class="pre">TINYINT</span></code>, <code class="docutils literal notranslate"><span class="pre">SMALLINT</span></code>, <code class="docutils literal notranslate"><span class="pre">INT</span></code>, or <code class="docutils literal notranslate"><span class="pre">BIGINT</span></code>) and one native partition per input within each task.
+Both broadcast and shuffled hash joins support either Spark build side. Unsupported joins keep
+their existing execution path.</p>
+<p>Once the build completes, its key domain filters probe batches before the hash probe. Eligible
+native Parquet readers also use the domain to prune row groups. Reader attachment can pass through
+direct-column <code class="docutils literal notranslate"><span class="pre">IS</span> <span class="pre">NOT</span> <span class="pre">NULL</span></code> checks, including conjunctions, and remaps columns when the scan itself
+projects the file schema. The original null checks and residual runtime filter remain in place.
+The original join still verifies matches, including any hash collisions admitted by the filter.
+Standalone projections, other filter expressions, and limits prevent reader attachment.</p>
+<p>Filters stay within the task’s native plan and do not propagate across Spark exchanges or JVM/Arrow
+boundaries. A shuffled hash join can still filter probe batches after shuffle, but it cannot send
+its filter back to an earlier scan stage. Compare the <a class="reference internal" href="metrics.html#hash-joins"><span class="std std-ref">runtime-filter and scan metrics</span></a>
+with the setting disabled to distinguish reduced hash-probe work from reader I/O savings.</p>
+</section>
</section>
<section id="shuffle">
<h2>Shuffle<a class="headerlink" href="#shuffle" title="Link to this heading">#</a></h2>
@@ -1156,7 +1174,10 @@ fallback to Spark works, see <a class="reference internal" href="understanding-c
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#iceberg-scan-tuning">Iceberg Scan Tuning</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#optimizing-sorting-on-floating-point-values">Optimizing Sorting on Floating-Point Values</a></li>
-<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#optimizing-joins">Optimizing Joins</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#optimizing-joins">Optimizing Joins</a><ul class="nav section-nav flex-column">
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#join-runtime-filters">Join Runtime Filters</a></li>
+</ul>
+</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#shuffle">Shuffle</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#shuffle-implementations">Shuffle Implementations</a><ul class="nav section-nav flex-column">
<li class="toc-h4 nav-item toc-entry"><a class="reference internal nav-link" href="#native-shuffle">Native Shuffle</a></li>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org