You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/05/28 23:14:29 UTC
(datafusion-comet) branch asf-site updated: Publish built docs triggered by 598a32d0dd1bc7340079640578962c6e4a5cc933
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 3fd74525ce Publish built docs triggered by 598a32d0dd1bc7340079640578962c6e4a5cc933
3fd74525ce is described below
commit 3fd74525cefb8351a5ba2f3c2b131980c8afc8f8
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Thu May 28 23:14:24 2026 +0000
Publish built docs triggered by 598a32d0dd1bc7340079640578962c6e4a5cc933
---
.../spark_expressions_support.md.txt | 22 ++++++++++
.../latest/compatibility/expressions/math.md.txt | 12 ++++++
contributor-guide/spark_expressions_support.html | 47 +++++++++++++++++++---
searchindex.js | 2 +-
.../latest/compatibility/expressions/index.html | 2 +
.../latest/compatibility/expressions/math.html | 14 +++++++
6 files changed, 93 insertions(+), 6 deletions(-)
diff --git a/_sources/contributor-guide/spark_expressions_support.md.txt b/_sources/contributor-guide/spark_expressions_support.md.txt
index 9cc37ccc29..09c2cec5ea 100644
--- a/_sources/contributor-guide/spark_expressions_support.md.txt
+++ b/_sources/contributor-guide/spark_expressions_support.md.txt
@@ -730,16 +730,34 @@
- [ ] is_variant_null
- [ ] java_method
- [x] monotonically_increasing_id
+ - Spark 3.4.3 (audited 2026-05-27): byte-for-byte identical to 4.1.1. `MonotonicallyIncreasingID() extends LeafExpression with Stateful`; produces a Long that encodes the partition id in the upper 31 bits and a per-partition row counter in the lower 33 bits. Comet emits an empty `MonotonicallyIncreasingId` proto and the native side produces the same encoding.
+ - Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.
+ - Spark 4.0.1 (audited 2026-05-27): identical to 3.4.3.
+ - Spark 4.1.1 (audited 2026-05-27): identical to 3.4.3.
- [ ] parse_json
- [ ] raise_error
- [x] rand
+ - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+ - Spark 3.5.8 (audited 2026-05-27): baseline. `Rand(child, hideSeed) extends RDG` (an `UnaryExpression with ExpectsInputTypes with Nondeterministic with ExpressionWithRandomSeed`); `child` is the seed expression, coerced to `IntegerType` or `LongType` via `ImplicitCastInputTypes`. Uses `XORShiftRandom(seed + partitionIndex)` per partition and returns `nextDouble()` in `[0, 1)`. NULL seed evaluates to `0L` (via `null.asInstanceOf[Long]`).
+ - Spark 4.0.1 (audited 2026-05-27): `RDG` is refactored from an `abstract class` into a trait, and `Rand` now extends a new `NondeterministicUnaryRDG` base. `ExpressionWithRandomSeed.expressionToSeed` is hoisted as a shared helper and throws `QueryCompilationErrors.invalidRandomSeedParameter` for non-literal seeds at analysis time. Runtime semantics unchanged.
+ - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+ - Comet limitation: the seed argument must be a literal (column-reference seeds are rejected via `getSupportLevel`). Pre-4.0 Spark would otherwise silently fail at runtime; 4.0+ rejects at analysis time before the expression reaches Comet.
- [x] randn
+ - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+ - Spark 3.5.8 (audited 2026-05-27): same base as `Rand`; differs only in the eval body (`nextGaussian()` instead of `nextDouble()`), producing values from the standard normal distribution.
+ - Spark 4.0.1 (audited 2026-05-27): same refactor as `Rand`; runtime unchanged.
+ - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+ - Comet limitation: same as `rand` — the seed argument must be a literal.
- [ ] reflect
- [ ] schema_of_avro
- [ ] schema_of_variant
- [ ] schema_of_variant_agg
- [ ] session_user
- [x] spark_partition_id
+ - Spark 3.4.3 (audited 2026-05-27): byte-for-byte identical to 4.1.1. `SparkPartitionID() extends LeafExpression with Nondeterministic`; returns the integer index of the partition being processed. Comet emits an empty `SparkPartitionId` proto.
+ - Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.
+ - Spark 4.0.1 (audited 2026-05-27): identical to 3.4.3.
+ - Spark 4.1.1 (audited 2026-05-27): identical to 3.4.3.
- [ ] st_asbinary
- [ ] st_geogfromwkb
- [ ] st_geomfromwkb
@@ -758,6 +776,10 @@
- [ ] try_variant_get
- [ ] typeof
- [x] user
+ - Spark 3.4.3 (audited 2026-05-27): `CurrentUser() extends LeafExpression with Unevaluable`; the analyzer's `ResolveCurrentLike` rule replaces it with a `StringType` literal of the current user name before Comet sees the plan. No Comet serde needed; the literal flows through `CometLiteral`.
+ - Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.
+ - Spark 4.0.1 (audited 2026-05-27): identical to 3.4.3 except the resulting literal carries the default string collation.
+ - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
- [ ] uuid
- [ ] variant_get
- [ ] version
diff --git a/_sources/user-guide/latest/compatibility/expressions/math.md.txt b/_sources/user-guide/latest/compatibility/expressions/math.md.txt
index f347f697e8..8b2aefc327 100644
--- a/_sources/user-guide/latest/compatibility/expressions/math.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/math.md.txt
@@ -26,4 +26,16 @@ under the License.
The following cases are not supported by Comet:
- Only integral, floating-point, and decimal types are supported
+
+## Rand
+
+The following cases are not supported by Comet:
+
+- The `seed` argument must be a literal value
+
+## Randn
+
+The following cases are not supported by Comet:
+
+- The `seed` argument must be a literal value
<!--END:EXPR_COMPAT-->
diff --git a/contributor-guide/spark_expressions_support.html b/contributor-guide/spark_expressions_support.html
index fecf1f8c16..dbf68ddccd 100644
--- a/contributor-guide/spark_expressions_support.html
+++ b/contributor-guide/spark_expressions_support.html
@@ -1607,17 +1607,47 @@ per-function details are documented in the
<li class="task-list-item"><p><input class="task-list-item-checkbox" disabled="disabled" type="checkbox"> input_file_name</p></li>
<li class="task-list-item"><p><input class="task-list-item-checkbox" disabled="disabled" type="checkbox"> is_variant_null</p></li>
<li class="task-list-item"><p><input class="task-list-item-checkbox" disabled="disabled" type="checkbox"> java_method</p></li>
-<li class="task-list-item"><p><input class="task-list-item-checkbox" checked="checked" disabled="disabled" type="checkbox"> monotonically_increasing_id</p></li>
+<li class="task-list-item"><p><input class="task-list-item-checkbox" checked="checked" disabled="disabled" type="checkbox"> monotonically_increasing_id</p>
+<ul>
+<li><p>Spark 3.4.3 (audited 2026-05-27): byte-for-byte identical to 4.1.1. <code class="docutils literal notranslate"><span class="pre">MonotonicallyIncreasingID()</span> <span class="pre">extends</span> <span class="pre">LeafExpression</span> <span class="pre">with</span> <span class="pre">Stateful</span></code>; produces a Long that encodes the partition id in the upper 31 bits and a per-partition row counter in the lower 33 bits. Comet emits an empty <code class="docutils literal notr [...]
+<li><p>Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.</p></li>
+<li><p>Spark 4.0.1 (audited 2026-05-27): identical to 3.4.3.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 3.4.3.</p></li>
+</ul>
+</li>
<li class="task-list-item"><p><input class="task-list-item-checkbox" disabled="disabled" type="checkbox"> parse_json</p></li>
<li class="task-list-item"><p><input class="task-list-item-checkbox" disabled="disabled" type="checkbox"> raise_error</p></li>
-<li class="task-list-item"><p><input class="task-list-item-checkbox" checked="checked" disabled="disabled" type="checkbox"> rand</p></li>
-<li class="task-list-item"><p><input class="task-list-item-checkbox" checked="checked" disabled="disabled" type="checkbox"> randn</p></li>
+<li class="task-list-item"><p><input class="task-list-item-checkbox" checked="checked" disabled="disabled" type="checkbox"> rand</p>
+<ul>
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">Rand(child,</span> <span class="pre">hideSeed)</span> <span class="pre">extends</span> <span class="pre">RDG</span></code> (an <code class="docutils literal notranslate"><span class="pre">UnaryExpression</span> <span class="pre">with</span> <span class="pre">ExpectsInputTypes</span> <span class="pre">with</span> <span class="pre">Nondeterministic</span> <span class="pre">with</ [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">RDG</span></code> is refactored from an <code class="docutils literal notranslate"><span class="pre">abstract</span> <span class="pre">class</span></code> into a trait, and <code class="docutils literal notranslate"><span class="pre">Rand</span></code> now extends a new <code class="docutils literal notranslate"><span class="pre">NondeterministicUnaryRDG</span></code> base. <code class=" [...]
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+<li><p>Comet limitation: the seed argument must be a literal (column-reference seeds are rejected via <code class="docutils literal notranslate"><span class="pre">getSupportLevel</span></code>). Pre-4.0 Spark would otherwise silently fail at runtime; 4.0+ rejects at analysis time before the expression reaches Comet.</p></li>
+</ul>
+</li>
+<li class="task-list-item"><p><input class="task-list-item-checkbox" checked="checked" disabled="disabled" type="checkbox"> randn</p>
+<ul>
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): same base as <code class="docutils literal notranslate"><span class="pre">Rand</span></code>; differs only in the eval body (<code class="docutils literal notranslate"><span class="pre">nextGaussian()</span></code> instead of <code class="docutils literal notranslate"><span class="pre">nextDouble()</span></code>), producing values from the standard normal distribution.</p></li>
+<li><p>Spark 4.0.1 (audited 2026-05-27): same refactor as <code class="docutils literal notranslate"><span class="pre">Rand</span></code>; runtime unchanged.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+<li><p>Comet limitation: same as <code class="docutils literal notranslate"><span class="pre">rand</span></code> — the seed argument must be a literal.</p></li>
+</ul>
+</li>
<li class="task-list-item"><p><input class="task-list-item-checkbox" disabled="disabled" type="checkbox"> reflect</p></li>
<li class="task-list-item"><p><input class="task-list-item-checkbox" disabled="disabled" type="checkbox"> schema_of_avro</p></li>
<li class="task-list-item"><p><input class="task-list-item-checkbox" disabled="disabled" type="checkbox"> schema_of_variant</p></li>
<li class="task-list-item"><p><input class="task-list-item-checkbox" disabled="disabled" type="checkbox"> schema_of_variant_agg</p></li>
<li class="task-list-item"><p><input class="task-list-item-checkbox" disabled="disabled" type="checkbox"> session_user</p></li>
-<li class="task-list-item"><p><input class="task-list-item-checkbox" checked="checked" disabled="disabled" type="checkbox"> spark_partition_id</p></li>
+<li class="task-list-item"><p><input class="task-list-item-checkbox" checked="checked" disabled="disabled" type="checkbox"> spark_partition_id</p>
+<ul>
+<li><p>Spark 3.4.3 (audited 2026-05-27): byte-for-byte identical to 4.1.1. <code class="docutils literal notranslate"><span class="pre">SparkPartitionID()</span> <span class="pre">extends</span> <span class="pre">LeafExpression</span> <span class="pre">with</span> <span class="pre">Nondeterministic</span></code>; returns the integer index of the partition being processed. Comet emits an empty <code class="docutils literal notranslate"><span class="pre">SparkPartitionId</span></code> prot [...]
+<li><p>Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.</p></li>
+<li><p>Spark 4.0.1 (audited 2026-05-27): identical to 3.4.3.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 3.4.3.</p></li>
+</ul>
+</li>
<li class="task-list-item"><p><input class="task-list-item-checkbox" disabled="disabled" type="checkbox"> st_asbinary</p></li>
<li class="task-list-item"><p><input class="task-list-item-checkbox" disabled="disabled" type="checkbox"> st_geogfromwkb</p></li>
<li class="task-list-item"><p><input class="task-list-item-checkbox" disabled="disabled" type="checkbox"> st_geomfromwkb</p></li>
@@ -1635,7 +1665,14 @@ per-function details are documented in the
<li class="task-list-item"><p><input class="task-list-item-checkbox" disabled="disabled" type="checkbox"> try_reflect</p></li>
<li class="task-list-item"><p><input class="task-list-item-checkbox" disabled="disabled" type="checkbox"> try_variant_get</p></li>
<li class="task-list-item"><p><input class="task-list-item-checkbox" disabled="disabled" type="checkbox"> typeof</p></li>
-<li class="task-list-item"><p><input class="task-list-item-checkbox" checked="checked" disabled="disabled" type="checkbox"> user</p></li>
+<li class="task-list-item"><p><input class="task-list-item-checkbox" checked="checked" disabled="disabled" type="checkbox"> user</p>
+<ul>
+<li><p>Spark 3.4.3 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">CurrentUser()</span> <span class="pre">extends</span> <span class="pre">LeafExpression</span> <span class="pre">with</span> <span class="pre">Unevaluable</span></code>; the analyzer’s <code class="docutils literal notranslate"><span class="pre">ResolveCurrentLike</span></code> rule replaces it with a <code class="docutils literal notranslate"><span class="pre">StringType</span></code> li [...]
+<li><p>Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.</p></li>
+<li><p>Spark 4.0.1 (audited 2026-05-27): identical to 3.4.3 except the resulting literal carries the default string collation.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
+</li>
<li class="task-list-item"><p><input class="task-list-item-checkbox" disabled="disabled" type="checkbox"> uuid</p></li>
<li class="task-list-item"><p><input class="task-list-item-checkbox" disabled="disabled" type="checkbox"> variant_get</p></li>
<li class="task-list-item"><p><input class="task-list-item-checkbox" disabled="disabled" type="checkbox"> version</p></li>
diff --git a/searchindex.js b/searchindex.js
index 7baadba108..d4e248281f 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"1. Format Your Code": [[34, "format-your-code"]], "1. Install Comet": [[36, "install-comet"], [45, "install-comet"]], "1. Native Operators (nativeExecs map)": [[23, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[34, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[36, "clone-iceberg-and-apply-diff"]], "2. Clone Spark and Apply Diff": [[45, "clone-spark-and-apply-diff"]], "2. Sink Operators (sinks map)": [[23, "sink-operators-sinks [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"1. Format Your Code": [[34, "format-your-code"]], "1. Install Comet": [[36, "install-comet"], [45, "install-comet"]], "1. Native Operators (nativeExecs map)": [[23, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[34, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[36, "clone-iceberg-and-apply-diff"]], "2. Clone Spark and Apply Diff": [[45, "clone-spark-and-apply-diff"]], "2. Sink Operators (sinks map)": [[23, "sink-operators-sinks [...]
\ No newline at end of file
diff --git a/user-guide/latest/compatibility/expressions/index.html b/user-guide/latest/compatibility/expressions/index.html
index 81dd1e86ba..61cc3191c2 100644
--- a/user-guide/latest/compatibility/expressions/index.html
+++ b/user-guide/latest/compatibility/expressions/index.html
@@ -572,6 +572,8 @@ the <a class="reference internal" href="../../expressions.html"><span class="std
</li>
<li class="toctree-l1"><a class="reference internal" href="math.html">Math Expressions</a><ul>
<li class="toctree-l2"><a class="reference internal" href="math.html#abs">Abs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="math.html#rand">Rand</a></li>
+<li class="toctree-l2"><a class="reference internal" href="math.html#randn">Randn</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="misc.html">Miscellaneous Expressions</a><ul>
diff --git a/user-guide/latest/compatibility/expressions/math.html b/user-guide/latest/compatibility/expressions/math.html
index dea7e4f5b7..3a88373a03 100644
--- a/user-guide/latest/compatibility/expressions/math.html
+++ b/user-guide/latest/compatibility/expressions/math.html
@@ -535,6 +535,20 @@ under the License.
<ul class="simple">
<li><p>Only integral, floating-point, and decimal types are supported</p></li>
</ul>
+</section>
+<section id="rand">
+<h2>Rand<a class="headerlink" href="#rand" title="Link to this heading">#</a></h2>
+<p>The following cases are not supported by Comet:</p>
+<ul class="simple">
+<li><p>The <code class="docutils literal notranslate"><span class="pre">seed</span></code> argument must be a literal value</p></li>
+</ul>
+</section>
+<section id="randn">
+<h2>Randn<a class="headerlink" href="#randn" title="Link to this heading">#</a></h2>
+<p>The following cases are not supported by Comet:</p>
+<ul class="simple">
+<li><p>The <code class="docutils literal notranslate"><span class="pre">seed</span></code> argument must be a literal value</p></li>
+</ul>
<!--END:EXPR_COMPAT-->
</section>
</section>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org