You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2025/01/31 00:56:44 UTC
(datafusion-comet) branch asf-site updated: Publish built docs triggered by 478672bd81cfdc8e109a36353149c6aeb09a1291
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 1936c8224 Publish built docs triggered by 478672bd81cfdc8e109a36353149c6aeb09a1291
1936c8224 is described below
commit 1936c8224cbf15fd8c1bc64e90f401b0001e6123
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Fri Jan 31 00:56:41 2025 +0000
Publish built docs triggered by 478672bd81cfdc8e109a36353149c6aeb09a1291
---
_sources/user-guide/compatibility.md.txt | 22 +++++++++++++-----
searchindex.js | 2 +-
user-guide/compatibility.html | 40 ++++++++++++++++++++++++--------
3 files changed, 47 insertions(+), 17 deletions(-)
diff --git a/_sources/user-guide/compatibility.md.txt b/_sources/user-guide/compatibility.md.txt
index 94fd9ea7e..893326e19 100644
--- a/_sources/user-guide/compatibility.md.txt
+++ b/_sources/user-guide/compatibility.md.txt
@@ -32,12 +32,6 @@ be used in production.
There is an [epic](https://github.com/apache/datafusion-comet/issues/313) where we are tracking the work to fully implement ANSI support.
-## Regular Expressions
-
-Comet uses the Rust regexp crate for evaluating regular expressions, and this has different behavior from Java's
-regular expression engine. Comet will fall back to Spark for patterns that are known to produce different results, but
-this can be overridden by setting `spark.comet.regexp.allowIncompatible=true`.
-
## Floating number comparison
Spark normalizes NaN and zero for floating point numbers for several cases. See `NormalizeFloatingNumbers` optimization rule in Spark.
@@ -46,6 +40,22 @@ because they are handled well in Spark (e.g., `SQLOrderingUtil.compareFloats`).
functions of arrow-rs used by DataFusion do not normalize NaN and zero (e.g., [arrow::compute::kernels::cmp::eq](https://docs.rs/arrow/latest/arrow/compute/kernels/cmp/fn.eq.html#)).
So Comet will add additional normalization expression of NaN and zero for comparison.
+## Incompatible Expressions
+
+Some Comet native expressions are not 100% compatible with Spark and are disabled by default. These expressions
+will fall back to Spark but can be enabled by setting `spark.comet.expression.allowIncompatible=true`.
+
+## Array Expressions
+
+Comet has experimental support for a number of array expressions. These are experimental and currently marked
+as incompatible and can be enabled by setting `spark.comet.expression.allowIncompatible=true`.
+
+## Regular Expressions
+
+Comet uses the Rust regexp crate for evaluating regular expressions, and this has different behavior from Java's
+regular expression engine. Comet will fall back to Spark for patterns that are known to produce different results, but
+this can be overridden by setting `spark.comet.regexp.allowIncompatible=true`.
+
## Cast
Cast operations in Comet fall into three levels of support:
diff --git a/searchindex.js b/searchindex.js
index ee41563f9..3d09921bb 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"1. Install Comet": [[9, "install-comet"]], "2. Clone Spark and Apply Diff": [[9, "clone-spark-and-apply-diff"]], "3. Run Spark SQL Tests": [[9, "run-spark-sql-tests"]], "ANSI mode": [[11, "ansi-mode"]], "API Differences Between Spark Versions": [[0, "api-differences-between-spark-versions"]], "ASF Links": [[10, null]], "Adding Spark-side Tests for the New Expression": [[0, "adding-spark-side-tests-for-the-new-expression"]], "Adding a New Expression": [[0, [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"1. Install Comet": [[9, "install-comet"]], "2. Clone Spark and Apply Diff": [[9, "clone-spark-and-apply-diff"]], "3. Run Spark SQL Tests": [[9, "run-spark-sql-tests"]], "ANSI mode": [[11, "ansi-mode"]], "API Differences Between Spark Versions": [[0, "api-differences-between-spark-versions"]], "ASF Links": [[10, null]], "Adding Spark-side Tests for the New Expression": [[0, "adding-spark-side-tests-for-the-new-expression"]], "Adding a New Expression": [[0, [...]
\ No newline at end of file
diff --git a/user-guide/compatibility.html b/user-guide/compatibility.html
index e6774270f..2a18f828b 100644
--- a/user-guide/compatibility.html
+++ b/user-guide/compatibility.html
@@ -292,13 +292,23 @@ under the License.
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#regular-expressions">
- Regular Expressions
+ <a class="reference internal nav-link" href="#floating-number-comparison">
+ Floating number comparison
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#floating-number-comparison">
- Floating number comparison
+ <a class="reference internal nav-link" href="#incompatible-expressions">
+ Incompatible Expressions
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#array-expressions">
+ Array Expressions
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#regular-expressions">
+ Regular Expressions
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
@@ -381,12 +391,6 @@ specify <code class="docutils literal notranslate"><span class="pre">spark.comet
be used in production.</p>
<p>There is an <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/313">epic</a> where we are tracking the work to fully implement ANSI support.</p>
</section>
-<section id="regular-expressions">
-<h2>Regular Expressions<a class="headerlink" href="#regular-expressions" title="Link to this heading">¶</a></h2>
-<p>Comet uses the Rust regexp crate for evaluating regular expressions, and this has different behavior from Java’s
-regular expression engine. Comet will fall back to Spark for patterns that are known to produce different results, but
-this can be overridden by setting <code class="docutils literal notranslate"><span class="pre">spark.comet.regexp.allowIncompatible=true</span></code>.</p>
-</section>
<section id="floating-number-comparison">
<h2>Floating number comparison<a class="headerlink" href="#floating-number-comparison" title="Link to this heading">¶</a></h2>
<p>Spark normalizes NaN and zero for floating point numbers for several cases. See <code class="docutils literal notranslate"><span class="pre">NormalizeFloatingNumbers</span></code> optimization rule in Spark.
@@ -395,6 +399,22 @@ because they are handled well in Spark (e.g., <code class="docutils literal notr
functions of arrow-rs used by DataFusion do not normalize NaN and zero (e.g., <a class="reference external" href="https://docs.rs/arrow/latest/arrow/compute/kernels/cmp/fn.eq.html#">arrow::compute::kernels::cmp::eq</a>).
So Comet will add additional normalization expression of NaN and zero for comparison.</p>
</section>
+<section id="incompatible-expressions">
+<h2>Incompatible Expressions<a class="headerlink" href="#incompatible-expressions" title="Link to this heading">¶</a></h2>
+<p>Some Comet native expressions are not 100% compatible with Spark and are disabled by default. These expressions
+will fall back to Spark but can be enabled by setting <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.allowIncompatible=true</span></code>.</p>
+</section>
+<section id="array-expressions">
+<h2>Array Expressions<a class="headerlink" href="#array-expressions" title="Link to this heading">¶</a></h2>
+<p>Comet has experimental support for a number of array expressions. These are experimental and currently marked
+as incompatible and can be enabled by setting <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.allowIncompatible=true</span></code>.</p>
+</section>
+<section id="regular-expressions">
+<h2>Regular Expressions<a class="headerlink" href="#regular-expressions" title="Link to this heading">¶</a></h2>
+<p>Comet uses the Rust regexp crate for evaluating regular expressions, and this has different behavior from Java’s
+regular expression engine. Comet will fall back to Spark for patterns that are known to produce different results, but
+this can be overridden by setting <code class="docutils literal notranslate"><span class="pre">spark.comet.regexp.allowIncompatible=true</span></code>.</p>
+</section>
<section id="cast">
<h2>Cast<a class="headerlink" href="#cast" title="Link to this heading">¶</a></h2>
<p>Cast operations in Comet fall into three levels of support:</p>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org