You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2025/01/30 14:37:47 UTC
(datafusion-comet) branch asf-site updated: Publish built docs triggered by 443b5db107cf8a44cc317ce06ebeeccaae671995
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 0a8041001 Publish built docs triggered by 443b5db107cf8a44cc317ce06ebeeccaae671995
0a8041001 is described below
commit 0a8041001189a917e031fb97fed4637d3d53648b
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Thu Jan 30 14:37:42 2025 +0000
Publish built docs triggered by 443b5db107cf8a44cc317ce06ebeeccaae671995
---
_sources/user-guide/configs.md.txt | 5 ++--
_sources/user-guide/expressions.md.txt | 29 +++++++++++++------
searchindex.js | 2 +-
user-guide/configs.html | 38 +++++++++++++------------
user-guide/expressions.html | 51 ++++++++++++++++++++++++++++------
5 files changed, 88 insertions(+), 37 deletions(-)
diff --git a/_sources/user-guide/configs.md.txt b/_sources/user-guide/configs.md.txt
index 2bea501e5..8245e7b76 100644
--- a/_sources/user-guide/configs.md.txt
+++ b/_sources/user-guide/configs.md.txt
@@ -25,7 +25,7 @@ Comet provides the following configuration settings.
|--------|-------------|---------------|
| spark.comet.batchSize | The columnar batch size, i.e., the maximum number of rows that a batch can contain. | 8192 |
| spark.comet.caseConversion.enabled | Java uses locale-specific rules when converting strings to upper or lower case and Rust does not, so we disable upper and lower by default. | false |
-| spark.comet.cast.allowIncompatible | Comet is not currently fully compatible with Spark for all cast operations. Set this config to true to allow them anyway. See compatibility guide for more information. | false |
+| spark.comet.cast.allowIncompatible | Comet is not currently fully compatible with Spark for all cast operations. Set this config to true to allow them anyway. For more information, refer to the Comet Compatibility Guide (https://datafusion.apache.org/comet/user-guide/compatibility.html). | false |
| spark.comet.columnar.shuffle.async.enabled | Whether to enable asynchronous shuffle for Arrow-based shuffle. | false |
| spark.comet.columnar.shuffle.async.max.thread.num | Maximum number of threads on an executor used for Comet async columnar shuffle. This is the upper bound of total number of shuffle threads per executor. In other words, if the number of cores * the number of shuffle threads per task `spark.comet.columnar.shuffle.async.thread.num` is larger than this config. Comet will use this config as the number of shuffle threads per executor instead. | 100 |
| spark.comet.columnar.shuffle.async.thread.num | Number of threads used for Comet async columnar shuffle per shuffle task. Note that more threads means more memory requirement to buffer shuffle data before flushing to disk. Also, more threads may not always improve performance, and should be set based on the number of cores available. | 3 |
@@ -64,6 +64,7 @@ Comet provides the following configuration settings.
| spark.comet.explain.native.enabled | When this setting is enabled, Comet will provide a tree representation of the native query plan before execution and again after execution, with metrics. | false |
| spark.comet.explain.verbose.enabled | When this setting is enabled, Comet will provide a verbose tree representation of the extended information. | false |
| spark.comet.explainFallback.enabled | When this setting is enabled, Comet will provide logging explaining the reason(s) why a query stage cannot be executed natively. Set this to false to reduce the amount of logging. | false |
+| spark.comet.expression.allowIncompatible | Comet is not currently fully compatible with Spark for all expressions. Set this config to true to allow them anyway. For more information, refer to the Comet Compatibility Guide (https://datafusion.apache.org/comet/user-guide/compatibility.html). | false |
| spark.comet.memory.overhead.factor | Fraction of executor memory to be allocated as additional non-heap memory per executor process for Comet. | 0.2 |
| spark.comet.memory.overhead.min | Minimum amount of additional memory to be allocated per executor process for Comet, in MiB. | 402653184b |
| spark.comet.nativeLoadRequired | Whether to require Comet native library to load successfully when Comet is enabled. If not, Comet will silently fallback to Spark when it fails to load the native lib. Otherwise, an error will be thrown and the Spark job will be aborted. | false |
@@ -73,7 +74,7 @@ Comet provides the following configuration settings.
| spark.comet.parquet.read.io.mergeRanges.delta | The delta in bytes between consecutive read ranges below which the parallel reader will try to merge the ranges. The default is 8MB. | 8388608 |
| spark.comet.parquet.read.parallel.io.enabled | Whether to enable Comet's parallel reader for Parquet files. The parallel reader reads ranges of consecutive data in a file in parallel. It is faster for large files and row groups but uses more resources. | true |
| spark.comet.parquet.read.parallel.io.thread-pool.size | The maximum number of parallel threads the parallel reader will use in a single executor. For executors configured with a smaller number of cores, use a smaller number. | 16 |
-| spark.comet.regexp.allowIncompatible | Comet is not currently fully compatible with Spark for all regular expressions. Set this config to true to allow them anyway using Rust's regular expression engine. See compatibility guide for more information. | false |
+| spark.comet.regexp.allowIncompatible | Comet is not currently fully compatible with Spark for all regular expressions. Set this config to true to allow them anyway. For more information, refer to the Comet Compatibility Guide (https://datafusion.apache.org/comet/user-guide/compatibility.html). | false |
| spark.comet.scan.enabled | Whether to enable native scans. When this is turned on, Spark will use Comet to read supported data sources (currently only Parquet is supported natively). Note that to enable native vectorized execution, both this config and 'spark.comet.exec.enabled' need to be enabled. | true |
| spark.comet.scan.preFetch.enabled | Whether to enable pre-fetching feature of CometScan. | false |
| spark.comet.scan.preFetch.threadNum | The number of threads running pre-fetching for CometScan. Effective if spark.comet.scan.preFetch.enabled is enabled. Note that more pre-fetching threads means more memory requirement to store pre-fetched row groups. | 2 |
diff --git a/_sources/user-guide/expressions.md.txt b/_sources/user-guide/expressions.md.txt
index 99490f440..6125aa074 100644
--- a/_sources/user-guide/expressions.md.txt
+++ b/_sources/user-guide/expressions.md.txt
@@ -182,15 +182,26 @@ The following Spark expressions are currently available. Any known compatibility
| VariancePop | |
| VarianceSamp | |
-## Complex Types
-
-| Expression | Notes |
-| ----------------- | ----------- |
-| CreateNamedStruct | |
-| ElementAt | Arrays only |
-| GetArrayItem | |
-| GetStructField | |
-| StructsToJson | |
+## Arrays
+
+| Expression | Notes |
+|-------------------|--------------|
+| ArrayAppend | Experimental |
+| ArrayContains | Experimental |
+| ArrayIntersect | Experimental |
+| ArrayJoin | Experimental |
+| ArrayRemove | Experimental |
+| ArraysOverlap | Experimental |
+| ElementAt | Arrays only |
+| GetArrayItem | |
+
+## Structs
+
+| Expression | Notes |
+|-------------------|--------------|
+| CreateNamedStruct | |
+| GetStructField | |
+| StructsToJson | |
## Other
diff --git a/searchindex.js b/searchindex.js
index dd6f7d1bb..ee41563f9 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"1. Install Comet": [[9, "install-comet"]], "2. Clone Spark and Apply Diff": [[9, "clone-spark-and-apply-diff"]], "3. Run Spark SQL Tests": [[9, "run-spark-sql-tests"]], "ANSI mode": [[11, "ansi-mode"]], "API Differences Between Spark Versions": [[0, "api-differences-between-spark-versions"]], "ASF Links": [[10, null]], "Adding Spark-side Tests for the New Expression": [[0, "adding-spark-side-tests-for-the-new-expression"]], "Adding a New Expression": [[0, [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"1. Install Comet": [[9, "install-comet"]], "2. Clone Spark and Apply Diff": [[9, "clone-spark-and-apply-diff"]], "3. Run Spark SQL Tests": [[9, "run-spark-sql-tests"]], "ANSI mode": [[11, "ansi-mode"]], "API Differences Between Spark Versions": [[0, "api-differences-between-spark-versions"]], "ASF Links": [[10, null]], "Adding Spark-side Tests for the New Expression": [[0, "adding-spark-side-tests-for-the-new-expression"]], "Adding a New Expression": [[0, [...]
\ No newline at end of file
diff --git a/user-guide/configs.html b/user-guide/configs.html
index 3a7aa42e9..a7105b9f2 100644
--- a/user-guide/configs.html
+++ b/user-guide/configs.html
@@ -347,7 +347,7 @@ under the License.
<td><p>false</p></td>
</tr>
<tr class="row-even"><td><p>spark.comet.cast.allowIncompatible</p></td>
-<td><p>Comet is not currently fully compatible with Spark for all cast operations. Set this config to true to allow them anyway. See compatibility guide for more information.</p></td>
+<td><p>Comet is not currently fully compatible with Spark for all cast operations. Set this config to true to allow them anyway. For more information, refer to the Comet Compatibility Guide (https://datafusion.apache.org/comet/user-guide/compatibility.html).</p></td>
<td><p>false</p></td>
</tr>
<tr class="row-odd"><td><p>spark.comet.columnar.shuffle.async.enabled</p></td>
@@ -502,63 +502,67 @@ under the License.
<td><p>When this setting is enabled, Comet will provide logging explaining the reason(s) why a query stage cannot be executed natively. Set this to false to reduce the amount of logging.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-odd"><td><p>spark.comet.memory.overhead.factor</p></td>
+<tr class="row-odd"><td><p>spark.comet.expression.allowIncompatible</p></td>
+<td><p>Comet is not currently fully compatible with Spark for all expressions. Set this config to true to allow them anyway. For more information, refer to the Comet Compatibility Guide (https://datafusion.apache.org/comet/user-guide/compatibility.html).</p></td>
+<td><p>false</p></td>
+</tr>
+<tr class="row-even"><td><p>spark.comet.memory.overhead.factor</p></td>
<td><p>Fraction of executor memory to be allocated as additional non-heap memory per executor process for Comet.</p></td>
<td><p>0.2</p></td>
</tr>
-<tr class="row-even"><td><p>spark.comet.memory.overhead.min</p></td>
+<tr class="row-odd"><td><p>spark.comet.memory.overhead.min</p></td>
<td><p>Minimum amount of additional memory to be allocated per executor process for Comet, in MiB.</p></td>
<td><p>402653184b</p></td>
</tr>
-<tr class="row-odd"><td><p>spark.comet.nativeLoadRequired</p></td>
+<tr class="row-even"><td><p>spark.comet.nativeLoadRequired</p></td>
<td><p>Whether to require Comet native library to load successfully when Comet is enabled. If not, Comet will silently fallback to Spark when it fails to load the native lib. Otherwise, an error will be thrown and the Spark job will be aborted.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-even"><td><p>spark.comet.parquet.enable.directBuffer</p></td>
+<tr class="row-odd"><td><p>spark.comet.parquet.enable.directBuffer</p></td>
<td><p>Whether to use Java direct byte buffer when reading Parquet.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-odd"><td><p>spark.comet.parquet.read.io.adjust.readRange.skew</p></td>
+<tr class="row-even"><td><p>spark.comet.parquet.read.io.adjust.readRange.skew</p></td>
<td><p>In the parallel reader, if the read ranges submitted are skewed in sizes, this option will cause the reader to break up larger read ranges into smaller ranges to reduce the skew. This will result in a slightly larger number of connections opened to the file system but may give improved performance.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-even"><td><p>spark.comet.parquet.read.io.mergeRanges</p></td>
+<tr class="row-odd"><td><p>spark.comet.parquet.read.io.mergeRanges</p></td>
<td><p>When enabled the parallel reader will try to merge ranges of data that are separated by less than ‘comet.parquet.read.io.mergeRanges.delta’ bytes. Longer continuous reads are faster on cloud storage.</p></td>
<td><p>true</p></td>
</tr>
-<tr class="row-odd"><td><p>spark.comet.parquet.read.io.mergeRanges.delta</p></td>
+<tr class="row-even"><td><p>spark.comet.parquet.read.io.mergeRanges.delta</p></td>
<td><p>The delta in bytes between consecutive read ranges below which the parallel reader will try to merge the ranges. The default is 8MB.</p></td>
<td><p>8388608</p></td>
</tr>
-<tr class="row-even"><td><p>spark.comet.parquet.read.parallel.io.enabled</p></td>
+<tr class="row-odd"><td><p>spark.comet.parquet.read.parallel.io.enabled</p></td>
<td><p>Whether to enable Comet’s parallel reader for Parquet files. The parallel reader reads ranges of consecutive data in a file in parallel. It is faster for large files and row groups but uses more resources.</p></td>
<td><p>true</p></td>
</tr>
-<tr class="row-odd"><td><p>spark.comet.parquet.read.parallel.io.thread-pool.size</p></td>
+<tr class="row-even"><td><p>spark.comet.parquet.read.parallel.io.thread-pool.size</p></td>
<td><p>The maximum number of parallel threads the parallel reader will use in a single executor. For executors configured with a smaller number of cores, use a smaller number.</p></td>
<td><p>16</p></td>
</tr>
-<tr class="row-even"><td><p>spark.comet.regexp.allowIncompatible</p></td>
-<td><p>Comet is not currently fully compatible with Spark for all regular expressions. Set this config to true to allow them anyway using Rust’s regular expression engine. See compatibility guide for more information.</p></td>
+<tr class="row-odd"><td><p>spark.comet.regexp.allowIncompatible</p></td>
+<td><p>Comet is not currently fully compatible with Spark for all regular expressions. Set this config to true to allow them anyway. For more information, refer to the Comet Compatibility Guide (https://datafusion.apache.org/comet/user-guide/compatibility.html).</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-odd"><td><p>spark.comet.scan.enabled</p></td>
+<tr class="row-even"><td><p>spark.comet.scan.enabled</p></td>
<td><p>Whether to enable native scans. When this is turned on, Spark will use Comet to read supported data sources (currently only Parquet is supported natively). Note that to enable native vectorized execution, both this config and ‘spark.comet.exec.enabled’ need to be enabled.</p></td>
<td><p>true</p></td>
</tr>
-<tr class="row-even"><td><p>spark.comet.scan.preFetch.enabled</p></td>
+<tr class="row-odd"><td><p>spark.comet.scan.preFetch.enabled</p></td>
<td><p>Whether to enable pre-fetching feature of CometScan.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-odd"><td><p>spark.comet.scan.preFetch.threadNum</p></td>
+<tr class="row-even"><td><p>spark.comet.scan.preFetch.threadNum</p></td>
<td><p>The number of threads running pre-fetching for CometScan. Effective if spark.comet.scan.preFetch.enabled is enabled. Note that more pre-fetching threads means more memory requirement to store pre-fetched row groups.</p></td>
<td><p>2</p></td>
</tr>
-<tr class="row-even"><td><p>spark.comet.shuffle.preferDictionary.ratio</p></td>
+<tr class="row-odd"><td><p>spark.comet.shuffle.preferDictionary.ratio</p></td>
<td><p>The ratio of total values to distinct values in a string column to decide whether to prefer dictionary encoding when shuffling the column. If the ratio is higher than this config, dictionary encoding will be used on shuffling string column. This config is effective if it is higher than 1.0. Note that this config is only used when <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.shuffle.mode</span></code> is <code class="docutils literal notranslate"><s [...]
<td><p>10.0</p></td>
</tr>
-<tr class="row-odd"><td><p>spark.comet.sparkToColumnar.supportedOperatorList</p></td>
+<tr class="row-even"><td><p>spark.comet.sparkToColumnar.supportedOperatorList</p></td>
<td><p>A comma-separated list of operators that will be converted to Arrow columnar format when ‘spark.comet.sparkToColumnar.enabled’ is true</p></td>
<td><p>Range,InMemoryTableScan</p></td>
</tr>
diff --git a/user-guide/expressions.html b/user-guide/expressions.html
index 0200cba0d..e1c809cbb 100644
--- a/user-guide/expressions.html
+++ b/user-guide/expressions.html
@@ -347,8 +347,13 @@ under the License.
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#complex-types">
- Complex Types
+ <a class="reference internal nav-link" href="#arrays">
+ Arrays
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#structs">
+ Structs
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
@@ -852,8 +857,8 @@ under the License.
</tbody>
</table>
</section>
-<section id="complex-types">
-<h2>Complex Types<a class="headerlink" href="#complex-types" title="Link to this heading">¶</a></h2>
+<section id="arrays">
+<h2>Arrays<a class="headerlink" href="#arrays" title="Link to this heading">¶</a></h2>
<table class="table">
<thead>
<tr class="row-odd"><th class="head"><p>Expression</p></th>
@@ -861,13 +866,43 @@ under the License.
</tr>
</thead>
<tbody>
-<tr class="row-even"><td><p>CreateNamedStruct</p></td>
-<td><p></p></td>
+<tr class="row-even"><td><p>ArrayAppend</p></td>
+<td><p>Experimental</p></td>
+</tr>
+<tr class="row-odd"><td><p>ArrayContains</p></td>
+<td><p>Experimental</p></td>
</tr>
-<tr class="row-odd"><td><p>ElementAt</p></td>
+<tr class="row-even"><td><p>ArrayIntersect</p></td>
+<td><p>Experimental</p></td>
+</tr>
+<tr class="row-odd"><td><p>ArrayJoin</p></td>
+<td><p>Experimental</p></td>
+</tr>
+<tr class="row-even"><td><p>ArrayRemove</p></td>
+<td><p>Experimental</p></td>
+</tr>
+<tr class="row-odd"><td><p>ArraysOverlap</p></td>
+<td><p>Experimental</p></td>
+</tr>
+<tr class="row-even"><td><p>ElementAt</p></td>
<td><p>Arrays only</p></td>
</tr>
-<tr class="row-even"><td><p>GetArrayItem</p></td>
+<tr class="row-odd"><td><p>GetArrayItem</p></td>
+<td><p></p></td>
+</tr>
+</tbody>
+</table>
+</section>
+<section id="structs">
+<h2>Structs<a class="headerlink" href="#structs" title="Link to this heading">¶</a></h2>
+<table class="table">
+<thead>
+<tr class="row-odd"><th class="head"><p>Expression</p></th>
+<th class="head"><p>Notes</p></th>
+</tr>
+</thead>
+<tbody>
+<tr class="row-even"><td><p>CreateNamedStruct</p></td>
<td><p></p></td>
</tr>
<tr class="row-odd"><td><p>GetStructField</p></td>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org