You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/12 16:52:13 UTC
(datafusion-comet) branch asf-site updated: Publish built docs triggered by 38f394e2707814020cbf057510e25e85f07e47a8
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new a05c259ad4 Publish built docs triggered by 38f394e2707814020cbf057510e25e85f07e47a8
a05c259ad4 is described below
commit a05c259ad4522f8547266a3be3536c0cc1b246b8
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Mon Oct 12 16:52:07 2026 +0000
Publish built docs triggered by 38f394e2707814020cbf057510e25e85f07e47a8
---
...rmaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg | 2 +-
.../contributor-guide/adding_a_new_expression.md.txt | 16 +++++++++++++++-
contributor-guide/adding_a_new_expression.html | 19 +++++++++++++++++--
searchindex.js | 2 +-
4 files changed, 34 insertions(+), 5 deletions(-)
diff --git a/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg b/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg
index 9ebaf5fc00..3a6964e796 100644
--- a/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg
+++ b/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg
@@ -1 +1 @@
-<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 966" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-svg [...]
\ No newline at end of file
+<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 966" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-svg [...]
\ No newline at end of file
diff --git a/_sources/contributor-guide/adding_a_new_expression.md.txt b/_sources/contributor-guide/adding_a_new_expression.md.txt
index a246ea70bb..20bcae06e6 100644
--- a/_sources/contributor-guide/adding_a_new_expression.md.txt
+++ b/_sources/contributor-guide/adding_a_new_expression.md.txt
@@ -582,7 +582,21 @@ Spark.
If the expression you're adding has different behavior across different Spark versions, you'll need to account for that in your implementation. There are two tools at your disposal to help with this:
1. Shims that exist in `spark/src/main/spark-$SPARK_VERSION/org/apache/comet/shims/CometExprShim.scala` for each Spark version. These shims are used to provide compatibility between different Spark versions.
-2. Variables that correspond to the Spark version, such as `isSpark33Plus`, which can be used to conditionally execute code based on the Spark version.
+2. Variables that correspond to the Spark version, such as `isSpark42Plus`, which can be used to conditionally execute code based on the Spark version.
+
+#### Name the behavior, not the Spark version
+
+When native code has to match a behavior that differs between Spark versions, decide which behavior applies in Scala and pass the result to native code under a name that describes the behavior, not the Spark version that introduced it. This covers protobuf fields, native function arguments, and native function names. For example, `TruncTimestamp` carries a `wrap_second_millisecond_overflow` flag, which the serde sets from `!isSpark42Plus`. It is not called `spark_420_plus`. The `filter_v [...]
+
+Set the flag in one place on the Scala side, in a shim or in the serde, so that a fork overrides only that. A helper such as `isSpark42Plus` is enough when the behavior changed in a minor release. When Spark changed it in patch releases, compare the full version in a named predicate, because a check on the minor version is wrong for every earlier patch. `RegrSparkVersions.r2DegenerateCasesSwapped` in `serde/aggregates.scala` is the model. It resolves SPARK-55969, which shipped in 3.5.9, [...]
+
+Reasons to prefer behavior names:
+
+- Some deployments run a Spark fork that backports fixes from newer open-source releases. Such a fork changes the one Scala check that sets the flag and leaves the protobuf and native code alone. A `spark_420_plus` flag cannot describe a fork that has one 4.2 fix but not the others.
+- Spark occasionally changes behavior in a patch release or between minor releases, so a version number is a poor proxy for the behavior.
+- The native code stays free of Spark version logic, and a reader of the Rust or proto definition can tell what the flag does without looking up a release.
+
+Write comments the same way. Cite the Spark JIRA that changed the behavior, as the `TruncTimestamp` serde does with SPARK-56663, because forks track backports by JIRA rather than by release. Say "Spark 4.2 and later" for a behavior that future releases inherit, rather than "Spark 4.2". If a later release changes the behavior again, add a new parameter or shim for that change.
## Shimming to Support Different Spark Versions
diff --git a/contributor-guide/adding_a_new_expression.html b/contributor-guide/adding_a_new_expression.html
index 0dc7149cdc..87e62f9835 100644
--- a/contributor-guide/adding_a_new_expression.html
+++ b/contributor-guide/adding_a_new_expression.html
@@ -1067,8 +1067,20 @@ Spark.</p>
<p>If the expression you’re adding has different behavior across different Spark versions, you’ll need to account for that in your implementation. There are two tools at your disposal to help with this:</p>
<ol class="arabic simple">
<li><p>Shims that exist in <code class="docutils literal notranslate"><span class="pre">spark/src/main/spark-$SPARK_VERSION/org/apache/comet/shims/CometExprShim.scala</span></code> for each Spark version. These shims are used to provide compatibility between different Spark versions.</p></li>
-<li><p>Variables that correspond to the Spark version, such as <code class="docutils literal notranslate"><span class="pre">isSpark33Plus</span></code>, which can be used to conditionally execute code based on the Spark version.</p></li>
+<li><p>Variables that correspond to the Spark version, such as <code class="docutils literal notranslate"><span class="pre">isSpark42Plus</span></code>, which can be used to conditionally execute code based on the Spark version.</p></li>
</ol>
+<section id="name-the-behavior-not-the-spark-version">
+<h4>Name the behavior, not the Spark version<a class="headerlink" href="#name-the-behavior-not-the-spark-version" title="Link to this heading">#</a></h4>
+<p>When native code has to match a behavior that differs between Spark versions, decide which behavior applies in Scala and pass the result to native code under a name that describes the behavior, not the Spark version that introduced it. This covers protobuf fields, native function arguments, and native function names. For example, <code class="docutils literal notranslate"><span class="pre">TruncTimestamp</span></code> carries a <code class="docutils literal notranslate"><span class="p [...]
+<p>Set the flag in one place on the Scala side, in a shim or in the serde, so that a fork overrides only that. A helper such as <code class="docutils literal notranslate"><span class="pre">isSpark42Plus</span></code> is enough when the behavior changed in a minor release. When Spark changed it in patch releases, compare the full version in a named predicate, because a check on the minor version is wrong for every earlier patch. <code class="docutils literal notranslate"><span class="pre" [...]
+<p>Reasons to prefer behavior names:</p>
+<ul class="simple">
+<li><p>Some deployments run a Spark fork that backports fixes from newer open-source releases. Such a fork changes the one Scala check that sets the flag and leaves the protobuf and native code alone. A <code class="docutils literal notranslate"><span class="pre">spark_420_plus</span></code> flag cannot describe a fork that has one 4.2 fix but not the others.</p></li>
+<li><p>Spark occasionally changes behavior in a patch release or between minor releases, so a version number is a poor proxy for the behavior.</p></li>
+<li><p>The native code stays free of Spark version logic, and a reader of the Rust or proto definition can tell what the flag does without looking up a release.</p></li>
+</ul>
+<p>Write comments the same way. Cite the Spark JIRA that changed the behavior, as the <code class="docutils literal notranslate"><span class="pre">TruncTimestamp</span></code> serde does with SPARK-56663, because forks track backports by JIRA rather than by release. Say “Spark 4.2 and later” for a behavior that future releases inherit, rather than “Spark 4.2”. If a later release changes the behavior again, add a new parameter or shim for that change.</p>
+</section>
</section>
</section>
<section id="shimming-to-support-different-spark-versions">
@@ -1199,7 +1211,10 @@ Spark.</p>
<li class="toc-h4 nav-item toc-entry"><a class="reference internal nav-link" href="#producing-strings-from-arbitrary-bytes">Producing strings from arbitrary bytes</a></li>
</ul>
</li>
-<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#api-differences-between-spark-versions">API Differences Between Spark Versions</a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#api-differences-between-spark-versions">API Differences Between Spark Versions</a><ul class="nav section-nav flex-column">
+<li class="toc-h4 nav-item toc-entry"><a class="reference internal nav-link" href="#name-the-behavior-not-the-spark-version">Name the behavior, not the Spark version</a></li>
+</ul>
+</li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#shimming-to-support-different-spark-versions">Shimming to Support Different Spark Versions</a></li>
diff --git a/searchindex.js b/searchindex.js
index 216eb30a2e..5f0a571dec 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[59, "id1"]], "%": [[57, "id1"]], "&": [[47, "id1"]], "*": [[57, "id2"]], "+": [[57, "id3"]], "-": [[57, "id4"]], "/": [[57, "id5"]], "1. Format Your Code": [[44, "format-your-code"]], "1. Install Comet": [[66, "install-comet"], [78, "install-comet"]], "1. Native Operators (nativeExecs map)": [[30, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[44, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[66, "clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[59, "id1"]], "%": [[57, "id1"]], "&": [[47, "id1"]], "*": [[57, "id2"]], "+": [[57, "id3"]], "-": [[57, "id4"]], "/": [[57, "id5"]], "1. Format Your Code": [[44, "format-your-code"]], "1. Install Comet": [[66, "install-comet"], [78, "install-comet"]], "1. Native Operators (nativeExecs map)": [[30, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[44, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[66, "clone-iceberg-and-apply- [...]
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org