You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/20 05:18:19 UTC

(datafusion-comet) 01/01: Publish built docs triggered by cae599ace3bbc76fa2e29133e3a9504ff7b15d61

This is an automated email from the ASF dual-hosted git repository.

github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git

commit 9728d052ab03007ba7995107a68a7dcb452a4dc2
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Tue Oct 20 05:18:13 2026 +0000

    Publish built docs triggered by cae599ace3bbc76fa2e29133e3a9504ff7b15d61
---
 ...id-09341aca03757d9d3637a2eaff73472d472d0673.svg |  2 +-
 .../expression-audits/array_funcs.md.txt           |  1 +
 _sources/contributor-guide/floating_point.md.txt   |  5 +++
 .../latest/compatibility/floating-point.md.txt     | 12 ++++++
 _sources/user-guide/latest/configs.md.txt          |  2 +
 _sources/user-guide/latest/rust_udfs.md.txt        | 31 +++++++++++++--
 .../expression-audits/array_funcs.html             |  1 +
 contributor-guide/floating_point.html              |  5 +++
 searchindex.js                                     |  2 +-
 .../latest/compatibility/floating-point.html       | 12 ++++++
 user-guide/latest/configs.html                     |  8 ++++
 user-guide/latest/rust_udfs.html                   | 46 +++++++++++++++++++---
 12 files changed, 116 insertions(+), 11 deletions(-)

diff --git a/_images/mermaid-09341aca03757d9d3637a2eaff73472d472d0673.svg b/_images/mermaid-09341aca03757d9d3637a2eaff73472d472d0673.svg
index 27633b65b0..67e1a0ca38 100644
--- a/_images/mermaid-09341aca03757d9d3637a2eaff73472d472d0673.svg
+++ b/_images/mermaid-09341aca03757d9d3637a2eaff73472d472d0673.svg
@@ -1 +1 @@
-<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 1014" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-sv [...]
\ No newline at end of file
+<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 1014" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-sv [...]
\ No newline at end of file
diff --git a/_sources/contributor-guide/expression-audits/array_funcs.md.txt b/_sources/contributor-guide/expression-audits/array_funcs.md.txt
index af635270e7..87ff575307 100644
--- a/_sources/contributor-guide/expression-audits/array_funcs.md.txt
+++ b/_sources/contributor-guide/expression-audits/array_funcs.md.txt
@@ -149,6 +149,7 @@
 - Spark 3.5.8 (audited 2026-05-27): baseline. `ArraysOverlap(left, right)`; three-valued logic (TRUE if any common non-null element, NULL if a null is present and no overlap is found in non-nulls, FALSE otherwise). Comet routes via `CometArraysOverlap` to the native `spark_arrays_overlap` UDF, which implements the same three-valued logic.
 - Spark 4.0.1 (audited 2026-05-27): `NullIntolerant` -> `nullIntolerant` field refactor.
 - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+- Spark 4.0.5, 4.1.4 and 4.2.1 (release candidates, 2026-10-14): `fastEval` and its generated code normalize each flat `FLOAT` or `DOUBLE` element with `NormalizeFloatingNumbers.FLOAT_NORMALIZER` / `DOUBLE_NORMALIZER` before the `java.util.HashSet` lookup (SPARK-59602), so `-0.0` and `0.0` are one value. Spark 4.2.0 normalizes the arguments in the plan instead (SPARK-54918). Earlier releases keep the zeros apart, and every release treats all NaNs as one value. On the versions where `Arra [...]
 
 ## arrays_zip
 
diff --git a/_sources/contributor-guide/floating_point.md.txt b/_sources/contributor-guide/floating_point.md.txt
index 758350db8f..220a7e4ddb 100644
--- a/_sources/contributor-guide/floating_point.md.txt
+++ b/_sources/contributor-guide/floating_point.md.txt
@@ -62,6 +62,11 @@ it runs against:
   (before 3.5.2, unless probing merges them). Spark 4.2.0 normalizes their arguments in the plan
   (SPARK-54918). From 4.0.5, 4.1.4 and 4.2.1 they normalize while they evaluate instead
   (SPARK-59602).
+- `arrays_overlap` looks up the elements of a flat array in a `java.util.HashSet` of boxed values,
+  so it follows `Double.equals` and keeps signed zeros apart up to Spark 4.0.4 and 4.1.3. The same
+  two changes make the zeros one value: Spark 4.2.0 normalizes its arguments in the plan
+  (SPARK-54918), and 4.0.5, 4.1.4 and 4.2.1 normalize each flat `FLOAT` or `DOUBLE` element before
+  the lookup (SPARK-59602). Nested elements compare in the SQL ordering on every version.
 - Map construction (`ArrayBasedMapBuilder`) finds a duplicate `FLOAT` or `DOUBLE` key in a
   `HashMap` of boxed values, which follows `Double.equals`, and a duplicate struct or array key in a
   `TreeMap` in the SQL ordering. From Spark 4.0 it normalizes a `FLOAT` or `DOUBLE` key before the
diff --git a/_sources/user-guide/latest/compatibility/floating-point.md.txt b/_sources/user-guide/latest/compatibility/floating-point.md.txt
index 77d76f53c6..c94fc7dc68 100644
--- a/_sources/user-guide/latest/compatibility/floating-point.md.txt
+++ b/_sources/user-guide/latest/compatibility/floating-point.md.txt
@@ -148,6 +148,18 @@ differences are acceptable for your data.
 The check uses the Spark version number, not the changes a build contains. A vendor build that
 reports an older version but includes SPARK-54918 or SPARK-59602 still falls back.
 
+## `arrays_overlap`
+
+`arrays_overlap` runs natively on every Spark version and follows that version's rule for `FLOAT`
+and `DOUBLE` elements in a flat array. Spark 4.0.5+, 4.1.4+, and 4.2+ treat `-0.0` and `0.0` as one
+value (SPARK-54918, SPARK-59602), and earlier releases keep them apart. Every NaN representation is
+one value on every version, and nested arrays and structs compare their elements in Spark's order,
+in which the two zeros are equal.
+
+Comet chooses the rule by the Spark version number. On a vendor build that reports an older
+version but includes SPARK-59602, Comet returns `false` for two flat arrays whose only common value
+is a zero of opposite sign, where Spark returns `true`.
+
 ## `array_contains`, `array_remove` and `sort_array`
 
 `array_contains` compares `FLOAT` and `DOUBLE` elements as Spark does: `-0.0` equals `0.0`, and all
diff --git a/_sources/user-guide/latest/configs.md.txt b/_sources/user-guide/latest/configs.md.txt
index cc891408bb..0a50fecd12 100644
--- a/_sources/user-guide/latest/configs.md.txt
+++ b/_sources/user-guide/latest/configs.md.txt
@@ -89,6 +89,8 @@ Comet provides the following configuration settings.
 | `spark.comet.metrics.enabled` | Whether to enable Comet metrics reporting through Spark's external monitoring system. When enabled, Comet exposes metrics such as native operators, Spark operators, queries planned, transitions, and acceleration ratio. These metrics can be visualized through tools like Grafana when a metrics sink (e.g., Prometheus) is configured. Disabled by default because Spark plan traversal adds overhead and metrics require a sink to be useful. This config must be se [...]
 | `spark.comet.metrics.updateInterval` | The interval in milliseconds to update metrics. If interval is negative, metrics will be updated upon task completion. | 3000 |
 | `spark.comet.nativeLoadRequired` | Whether to require Comet native library to load successfully when Comet is enabled. If not, Comet will silently fallback to Spark when it fails to load the native lib. Otherwise, an error will be thrown and the Spark job will be aborted. | false |
+| `spark.comet.nativeUdf.allowedPaths` | A comma-separated list of directories that native UDF libraries may be loaded from. When set, a library is loaded only if its path is absolute and, after symlinks and `..` are resolved, lies under one of these directories, which are themselves resolved the same way. A path outside them fails on the driver at registration and again on the executor. When empty, any path is allowed. This limits which files Comet will load, not what the code in them d [...]
+| `spark.comet.nativeUdf.enabled` | Whether native UDF libraries may be loaded. A native UDF is a shared library that runs with the full privileges of the JVM process, so an operator who does not want any can turn the feature off. When false, `CometNativeUDF.register` fails on the driver, and a plan that names a native UDF library fails on the executor rather than loading it. | true |
 | `spark.comet.operator.DataWritingCommandExec.allowIncompatible` | Whether to allow incompatibility for operator: DataWritingCommandExec. False by default. Can be overridden with SPARK_COMET_OPERATOR_DATAWRITINGCOMMANDEXEC_ALLOWINCOMPATIBLE env variable It can be overridden by the environment variable `SPARK_COMET_OPERATOR_DATAWRITINGCOMMANDEXEC_ALLOWINCOMPATIBLE`. | false |
 | `spark.comet.operator.WriteFilesExec.allowIncompatible` | Whether to allow incompatibility for operator: WriteFilesExec. False by default. Can be overridden with SPARK_COMET_OPERATOR_WRITEFILESEXEC_ALLOWINCOMPATIBLE env variable It can be overridden by the environment variable `SPARK_COMET_OPERATOR_WRITEFILESEXEC_ALLOWINCOMPATIBLE`. | false |
 | `spark.comet.sparkToColumnar.enabled` | Whether to convert the output of the leaf operators named in `spark.comet.sparkToColumnar.supportedOperatorList` to Arrow format. Ranges, in-memory cached tables, RDDs and queries without a FROM clause have `spark.comet.convert` configs of their own. When the list is not set, this converts those four instead, which is deprecated and will be removed in a future major release. | false |
diff --git a/_sources/user-guide/latest/rust_udfs.md.txt b/_sources/user-guide/latest/rust_udfs.md.txt
index 8620a7676d..ae1a0957f1 100644
--- a/_sources/user-guide/latest/rust_udfs.md.txt
+++ b/_sources/user-guide/latest/rust_udfs.md.txt
@@ -176,9 +176,30 @@ rather than accept a function whose volatility it would go on to ignore.
 `libraryPath` is passed to the platform's dynamic loader. An absolute path is what you want in
 practice, and it is what the rest of this page assumes, but a bare library name resolves the same
 way it would for any other shared object, through `LD_LIBRARY_PATH` on Linux and
-`DYLD_LIBRARY_PATH` on macOS. That is a convenience, not a sandbox: Comet does not restrict which
-paths may be loaded, so it makes no difference to the trust decision described under
-[Limitations](#limitations).
+`DYLD_LIBRARY_PATH` on macOS. That is a convenience, not a sandbox: unless you set the configs
+below, Comet does not restrict which paths may be loaded, so it makes no difference to the trust
+decision described under [Limitations](#limitations).
+
+## Restricting which libraries can be loaded
+
+Loading a library runs its code with the privileges of the JVM, so an operator can limit it:
+
+| Config                               | Default | Effect                                                       |
+| ------------------------------------ | ------- | ------------------------------------------------------------ |
+| `spark.comet.nativeUdf.enabled`      | `true`  | When `false`, no native UDF library is loaded.               |
+| `spark.comet.nativeUdf.allowedPaths` | empty   | Comma-separated directories a library must be under, if set. |
+
+With `allowedPaths` set, a library is loaded only if its path is absolute and, once symlinks and
+`..` are resolved, lies under one of the listed directories (which are resolved the same way). A bare
+library name is refused, because the dynamic loader's search path is not something Comet can check.
+The check applies on the driver, where `register` fails with `CometNativeUdfNotAllowedException`,
+and again on each executor before it loads a library named in a plan.
+
+These are ordinary Spark configs, so they are set in the cluster's defaults and are not a defense
+against a user who can change session configuration: such a user can set them back. The allow-list
+limits which files Comet will load, not what the code in them does, and a file that is replaced
+between the check and the load is not detected. List only directories that are writable by users
+you trust.
 
 ### When Spark evaluates the call
 
@@ -308,7 +329,9 @@ This feature is at an early stage. The current limitations are:
   layout.
 - **Loading a library is loading native code.** It runs with the full privileges of the executor
   process and Comet cannot sandbox it: a bug in a UDF can corrupt memory or crash the executor.
-  Only register libraries you trust and control.
+  Only register libraries you trust and control. See
+  [Restricting which libraries can be loaded](#restricting-which-libraries-can-be-loaded) for the
+  configs that turn the feature off or limit the directories libraries may come from.
 - Once loaded, a library stays loaded for the life of the process, and `register` loads it on the
   driver as well as on the executors. To deploy a new build, write it to a new path and register
   that, or restart the executors and the driver. Never copy a new build over a library that is
diff --git a/contributor-guide/expression-audits/array_funcs.html b/contributor-guide/expression-audits/array_funcs.html
index 39c176f9d0..b5e6631685 100644
--- a/contributor-guide/expression-audits/array_funcs.html
+++ b/contributor-guide/expression-audits/array_funcs.html
@@ -724,6 +724,7 @@ under the License.
 <li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">ArraysOverlap(left,</span> <span class="pre">right)</span></code>; three-valued logic (TRUE if any common non-null element, NULL if a null is present and no overlap is found in non-nulls, FALSE otherwise). Comet routes via <code class="docutils literal notranslate"><span class="pre">CometArraysOverlap</span></code> to the native <code class="docutils literal notranslate"><span  [...]
 <li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">NullIntolerant</span></code> -&gt; <code class="docutils literal notranslate"><span class="pre">nullIntolerant</span></code> field refactor.</p></li>
 <li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+<li><p>Spark 4.0.5, 4.1.4 and 4.2.1 (release candidates, 2026-10-14): <code class="docutils literal notranslate"><span class="pre">fastEval</span></code> and its generated code normalize each flat <code class="docutils literal notranslate"><span class="pre">FLOAT</span></code> or <code class="docutils literal notranslate"><span class="pre">DOUBLE</span></code> element with <code class="docutils literal notranslate"><span class="pre">NormalizeFloatingNumbers.FLOAT_NORMALIZER</span></code> [...]
 </ul>
 </section>
 <section id="arrays-zip">
diff --git a/contributor-guide/floating_point.html b/contributor-guide/floating_point.html
index f8d504df61..8e1b6d1ff0 100644
--- a/contributor-guide/floating_point.html
+++ b/contributor-guide/floating_point.html
@@ -631,6 +631,11 @@ into <code class="docutils literal notranslate"><span class="pre">0.0</span></co
 (before 3.5.2, unless probing merges them). Spark 4.2.0 normalizes their arguments in the plan
 (SPARK-54918). From 4.0.5, 4.1.4 and 4.2.1 they normalize while they evaluate instead
 (SPARK-59602).</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">arrays_overlap</span></code> looks up the elements of a flat array in a <code class="docutils literal notranslate"><span class="pre">java.util.HashSet</span></code> of boxed values,
+so it follows <code class="docutils literal notranslate"><span class="pre">Double.equals</span></code> and keeps signed zeros apart up to Spark 4.0.4 and 4.1.3. The same
+two changes make the zeros one value: Spark 4.2.0 normalizes its arguments in the plan
+(SPARK-54918), and 4.0.5, 4.1.4 and 4.2.1 normalize each flat <code class="docutils literal notranslate"><span class="pre">FLOAT</span></code> or <code class="docutils literal notranslate"><span class="pre">DOUBLE</span></code> element before
+the lookup (SPARK-59602). Nested elements compare in the SQL ordering on every version.</p></li>
 <li><p>Map construction (<code class="docutils literal notranslate"><span class="pre">ArrayBasedMapBuilder</span></code>) finds a duplicate <code class="docutils literal notranslate"><span class="pre">FLOAT</span></code> or <code class="docutils literal notranslate"><span class="pre">DOUBLE</span></code> key in a
 <code class="docutils literal notranslate"><span class="pre">HashMap</span></code> of boxed values, which follows <code class="docutils literal notranslate"><span class="pre">Double.equals</span></code>, and a duplicate struct or array key in a
 <code class="docutils literal notranslate"><span class="pre">TreeMap</span></code> in the SQL ordering. From Spark 4.0 it normalizes a <code class="docutils literal notranslate"><span class="pre">FLOAT</span></code> or <code class="docutils literal notranslate"><span class="pre">DOUBLE</span></code> key before the
diff --git a/searchindex.js b/searchindex.js
index a85f2525bd..e12cbe243b 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[63, "id1"]], "%": [[61, "id1"]], "&": [[51, "id1"]], "*": [[61, "id2"]], "+": [[61, "id3"]], "-": [[61, "id4"]], "/": [[61, "id5"]], "1. Format Your Code": [[48, "format-your-code"]], "1. Install Comet": [[70, "install-comet"], [82, "install-comet"]], "1. Native Operators (nativeExecs map)": [[31, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[48, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[70, "clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[63, "id1"]], "%": [[61, "id1"]], "&": [[51, "id1"]], "*": [[61, "id2"]], "+": [[61, "id3"]], "-": [[61, "id4"]], "/": [[61, "id5"]], "1. Format Your Code": [[48, "format-your-code"]], "1. Install Comet": [[70, "install-comet"], [82, "install-comet"]], "1. Native Operators (nativeExecs map)": [[31, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[48, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[70, "clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/latest/compatibility/floating-point.html b/user-guide/latest/compatibility/floating-point.html
index 3fd44710b6..54b6aa4856 100644
--- a/user-guide/latest/compatibility/floating-point.html
+++ b/user-guide/latest/compatibility/floating-point.html
@@ -960,6 +960,17 @@ differences are acceptable for your data.</p>
 <p>The check uses the Spark version number, not the changes a build contains. A vendor build that
 reports an older version but includes SPARK-54918 or SPARK-59602 still falls back.</p>
 </section>
+<section id="arrays-overlap">
+<h2><code class="docutils literal notranslate"><span class="pre">arrays_overlap</span></code><a class="headerlink" href="#arrays-overlap" title="Link to this heading">#</a></h2>
+<p><code class="docutils literal notranslate"><span class="pre">arrays_overlap</span></code> runs natively on every Spark version and follows that version’s rule for <code class="docutils literal notranslate"><span class="pre">FLOAT</span></code>
+and <code class="docutils literal notranslate"><span class="pre">DOUBLE</span></code> elements in a flat array. Spark 4.0.5+, 4.1.4+, and 4.2+ treat <code class="docutils literal notranslate"><span class="pre">-0.0</span></code> and <code class="docutils literal notranslate"><span class="pre">0.0</span></code> as one
+value (SPARK-54918, SPARK-59602), and earlier releases keep them apart. Every NaN representation is
+one value on every version, and nested arrays and structs compare their elements in Spark’s order,
+in which the two zeros are equal.</p>
+<p>Comet chooses the rule by the Spark version number. On a vendor build that reports an older
+version but includes SPARK-59602, Comet returns <code class="docutils literal notranslate"><span class="pre">false</span></code> for two flat arrays whose only common value
+is a zero of opposite sign, where Spark returns <code class="docutils literal notranslate"><span class="pre">true</span></code>.</p>
+</section>
 <section id="array-contains-array-remove-and-sort-array">
 <h2><code class="docutils literal notranslate"><span class="pre">array_contains</span></code>, <code class="docutils literal notranslate"><span class="pre">array_remove</span></code> and <code class="docutils literal notranslate"><span class="pre">sort_array</span></code><a class="headerlink" href="#array-contains-array-remove-and-sort-array" title="Link to this heading">#</a></h2>
 <p><code class="docutils literal notranslate"><span class="pre">array_contains</span></code> compares <code class="docutils literal notranslate"><span class="pre">FLOAT</span></code> and <code class="docutils literal notranslate"><span class="pre">DOUBLE</span></code> elements as Spark does: <code class="docutils literal notranslate"><span class="pre">-0.0</span></code> equals <code class="docutils literal notranslate"><span class="pre">0.0</span></code>, and all
@@ -1036,6 +1047,7 @@ values don’t affect either function. The expressions’ entries in the
 <li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#ordering-nan-and-signed-zero-0-0-vs-0-0">Ordering: NaN and signed zero (<code class="docutils literal notranslate"><span class="pre">-0.0</span></code> vs <code class="docutils literal notranslate"><span class="pre">+0.0</span></code>)</a></li>
 <li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#min-max-greatest-and-least"><code class="docutils literal notranslate"><span class="pre">min</span></code>, <code class="docutils literal notranslate"><span class="pre">max</span></code>, <code class="docutils literal notranslate"><span class="pre">greatest</span></code>, and <code class="docutils literal notranslate"><span class="pre">least</span></code></a></li>
 <li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#array-distinct-and-union">Array distinct and union</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#arrays-overlap"><code class="docutils literal notranslate"><span class="pre">arrays_overlap</span></code></a></li>
 <li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#array-contains-array-remove-and-sort-array"><code class="docutils literal notranslate"><span class="pre">array_contains</span></code>, <code class="docutils literal notranslate"><span class="pre">array_remove</span></code> and <code class="docutils literal notranslate"><span class="pre">sort_array</span></code></a></li>
 <li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#map-construction">Map construction</a></li>
 </ul>
diff --git a/user-guide/latest/configs.html b/user-guide/latest/configs.html
index c401d58121..ecc96b1a5d 100644
--- a/user-guide/latest/configs.html
+++ b/user-guide/latest/configs.html
@@ -1072,6 +1072,14 @@ under the License.
 <td><p>Whether to require Comet native library to load successfully when Comet is enabled. If not, Comet will silently fallback to Spark when it fails to load the native lib. Otherwise, an error will be thrown and the Spark job will be aborted.</p></td>
 <td><p>false</p></td>
 </tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.nativeUdf.allowedPaths</span></code></p></td>
+<td><p>A comma-separated list of directories that native UDF libraries may be loaded from. When set, a library is loaded only if its path is absolute and, after symlinks and <code class="docutils literal notranslate"><span class="pre">..</span></code> are resolved, lies under one of these directories, which are themselves resolved the same way. A path outside them fails on the driver at registration and again on the executor. When empty, any path is allowed. This limits which files Comet [...]
+<td><p></p></td>
+</tr>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.nativeUdf.enabled</span></code></p></td>
+<td><p>Whether native UDF libraries may be loaded. A native UDF is a shared library that runs with the full privileges of the JVM process, so an operator who does not want any can turn the feature off. When false, <code class="docutils literal notranslate"><span class="pre">CometNativeUDF.register</span></code> fails on the driver, and a plan that names a native UDF library fails on the executor rather than loading it.</p></td>
+<td><p>true</p></td>
+</tr>
 <tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.operator.DataWritingCommandExec.allowIncompatible</span></code></p></td>
 <td><p>Whether to allow incompatibility for operator: DataWritingCommandExec. False by default. Can be overridden with SPARK_COMET_OPERATOR_DATAWRITINGCOMMANDEXEC_ALLOWINCOMPATIBLE env variable It can be overridden by the environment variable <code class="docutils literal notranslate"><span class="pre">SPARK_COMET_OPERATOR_DATAWRITINGCOMMANDEXEC_ALLOWINCOMPATIBLE</span></code>.</p></td>
 <td><p>false</p></td>
diff --git a/user-guide/latest/rust_udfs.html b/user-guide/latest/rust_udfs.html
index 7ac399243f..3fbd4b94a5 100644
--- a/user-guide/latest/rust_udfs.html
+++ b/user-guide/latest/rust_udfs.html
@@ -983,9 +983,42 @@ rather than accept a function whose volatility it would go on to ignore.</p>
 <p><code class="docutils literal notranslate"><span class="pre">libraryPath</span></code> is passed to the platform’s dynamic loader. An absolute path is what you want in
 practice, and it is what the rest of this page assumes, but a bare library name resolves the same
 way it would for any other shared object, through <code class="docutils literal notranslate"><span class="pre">LD_LIBRARY_PATH</span></code> on Linux and
-<code class="docutils literal notranslate"><span class="pre">DYLD_LIBRARY_PATH</span></code> on macOS. That is a convenience, not a sandbox: Comet does not restrict which
-paths may be loaded, so it makes no difference to the trust decision described under
-<a class="reference internal" href="#limitations">Limitations</a>.</p>
+<code class="docutils literal notranslate"><span class="pre">DYLD_LIBRARY_PATH</span></code> on macOS. That is a convenience, not a sandbox: unless you set the configs
+below, Comet does not restrict which paths may be loaded, so it makes no difference to the trust
+decision described under <a class="reference internal" href="#limitations">Limitations</a>.</p>
+</section>
+<section id="restricting-which-libraries-can-be-loaded">
+<h2>Restricting which libraries can be loaded<a class="headerlink" href="#restricting-which-libraries-can-be-loaded" title="Link to this heading">#</a></h2>
+<p>Loading a library runs its code with the privileges of the JVM, so an operator can limit it:</p>
+<div class="pst-scrollable-table-container"><table class="table">
+<thead>
+<tr class="row-odd"><th class="head"><p>Config</p></th>
+<th class="head"><p>Default</p></th>
+<th class="head"><p>Effect</p></th>
+</tr>
+</thead>
+<tbody>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.nativeUdf.enabled</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span class="pre">true</span></code></p></td>
+<td><p>When <code class="docutils literal notranslate"><span class="pre">false</span></code>, no native UDF library is loaded.</p></td>
+</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.nativeUdf.allowedPaths</span></code></p></td>
+<td><p>empty</p></td>
+<td><p>Comma-separated directories a library must be under, if set.</p></td>
+</tr>
+</tbody>
+</table>
+</div>
+<p>With <code class="docutils literal notranslate"><span class="pre">allowedPaths</span></code> set, a library is loaded only if its path is absolute and, once symlinks and
+<code class="docutils literal notranslate"><span class="pre">..</span></code> are resolved, lies under one of the listed directories (which are resolved the same way). A bare
+library name is refused, because the dynamic loader’s search path is not something Comet can check.
+The check applies on the driver, where <code class="docutils literal notranslate"><span class="pre">register</span></code> fails with <code class="docutils literal notranslate"><span class="pre">CometNativeUdfNotAllowedException</span></code>,
+and again on each executor before it loads a library named in a plan.</p>
+<p>These are ordinary Spark configs, so they are set in the cluster’s defaults and are not a defense
+against a user who can change session configuration: such a user can set them back. The allow-list
+limits which files Comet will load, not what the code in them does, and a file that is replaced
+between the check and the load is not detected. List only directories that are writable by users
+you trust.</p>
 <section id="when-spark-evaluates-the-call">
 <h3>When Spark evaluates the call<a class="headerlink" href="#when-spark-evaluates-the-call" title="Link to this heading">#</a></h3>
 <p>A call runs only inside Comet’s native execution. Spark cannot run the function, so wherever Spark
@@ -1150,7 +1183,9 @@ Downcast defensively in <code class="docutils literal notranslate"><span class="
 layout.</p></li>
 <li><p><strong>Loading a library is loading native code.</strong> It runs with the full privileges of the executor
 process and Comet cannot sandbox it: a bug in a UDF can corrupt memory or crash the executor.
-Only register libraries you trust and control.</p></li>
+Only register libraries you trust and control. See
+<a class="reference internal" href="#restricting-which-libraries-can-be-loaded">Restricting which libraries can be loaded</a> for the
+configs that turn the feature off or limit the directories libraries may come from.</p></li>
 <li><p>Once loaded, a library stays loaded for the life of the process, and <code class="docutils literal notranslate"><span class="pre">register</span></code> loads it on the
 driver as well as on the executors. To deploy a new build, write it to a new path and register
 that, or restart the executors and the driver. Never copy a new build over a library that is
@@ -1212,7 +1247,8 @@ Rebuild your UDF library when upgrading Comet.</p></li>
   <nav class="bd-toc-nav page-toc" aria-labelledby="pst-page-navigation-heading-2">
     <ul class="visible nav section-nav flex-column">
 <li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#writing-a-udf">Writing a UDF</a></li>
-<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#registering-and-calling-a-udf">Registering and calling a UDF</a><ul class="nav section-nav flex-column">
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#registering-and-calling-a-udf">Registering and calling a UDF</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#restricting-which-libraries-can-be-loaded">Restricting which libraries can be loaded</a><ul class="nav section-nav flex-column">
 <li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#when-spark-evaluates-the-call">When Spark evaluates the call</a></li>
 </ul>
 </li>


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org