You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2025/04/23 12:14:23 UTC
(datafusion-comet) branch asf-site updated: Publish built docs triggered by ea125f56f838674c2c307984b22d06f31171c97b
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new a987858f2 Publish built docs triggered by ea125f56f838674c2c307984b22d06f31171c97b
a987858f2 is described below
commit a987858f207b9488a893cde3c6e18299db31367d
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Wed Apr 23 12:14:19 2025 +0000
Publish built docs triggered by ea125f56f838674c2c307984b22d06f31171c97b
---
_sources/user-guide/configs.md.txt | 1 +
searchindex.js | 2 +-
user-guide/configs.html | 6 +++++-
3 files changed, 7 insertions(+), 2 deletions(-)
diff --git a/_sources/user-guide/configs.md.txt b/_sources/user-guide/configs.md.txt
index 529103d8b..517ce960c 100644
--- a/_sources/user-guide/configs.md.txt
+++ b/_sources/user-guide/configs.md.txt
@@ -88,4 +88,5 @@ Comet provides the following configuration settings.
| spark.comet.scan.preFetch.enabled | Whether to enable pre-fetching feature of CometScan. | false |
| spark.comet.scan.preFetch.threadNum | The number of threads running pre-fetching for CometScan. Effective if spark.comet.scan.preFetch.enabled is enabled. Note that more pre-fetching threads means more memory requirement to store pre-fetched row groups. | 2 |
| spark.comet.shuffle.preferDictionary.ratio | The ratio of total values to distinct values in a string column to decide whether to prefer dictionary encoding when shuffling the column. If the ratio is higher than this config, dictionary encoding will be used on shuffling string column. This config is effective if it is higher than 1.0. Note that this config is only used when `spark.comet.exec.shuffle.mode` is `jvm`. | 10.0 |
+| spark.comet.shuffle.sizeInBytesMultiplier | Comet reports smaller sizes for shuffle due to using Arrow's columnar memory format and this can result in Spark choosing a different join strategy due to the estimated size of the exchange being smaller. Comet will multiple sizeInBytes by this amount to avoid regressions in join strategy. | 1.0 |
| spark.comet.sparkToColumnar.supportedOperatorList | A comma-separated list of operators that will be converted to Arrow columnar format when 'spark.comet.sparkToColumnar.enabled' is true | Range,InMemoryTableScan |
diff --git a/searchindex.js b/searchindex.js
index 96e10d2f9..7bfd5588a 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"1. Install Comet": [[10, "install-comet"]], "2. Clone Spark and Apply Diff": [[10, "clone-spark-and-apply-diff"]], "3. Run Spark SQL Tests": [[10, "run-spark-sql-tests"]], "ANSI mode": [[12, "ansi-mode"]], "API Differences Between Spark Versions": [[0, "api-differences-between-spark-versions"]], "ASF Links": [[11, null]], "Adding Spark-side Tests for the New Expression": [[0, "adding-spark-side-tests-for-the-new-expression"]], "Adding a New Expression": [[ [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"1. Install Comet": [[10, "install-comet"]], "2. Clone Spark and Apply Diff": [[10, "clone-spark-and-apply-diff"]], "3. Run Spark SQL Tests": [[10, "run-spark-sql-tests"]], "ANSI mode": [[12, "ansi-mode"]], "API Differences Between Spark Versions": [[0, "api-differences-between-spark-versions"]], "ASF Links": [[11, null]], "Adding Spark-side Tests for the New Expression": [[0, "adding-spark-side-tests-for-the-new-expression"]], "Adding a New Expression": [[ [...]
\ No newline at end of file
diff --git a/user-guide/configs.html b/user-guide/configs.html
index 89239836b..562afeb70 100644
--- a/user-guide/configs.html
+++ b/user-guide/configs.html
@@ -579,7 +579,11 @@ TO MODIFY THIS CONTENT MAKE SURE THAT YOU MAKE YOUR CHANGES TO THE TEMPLATE FILE
<td><p>The ratio of total values to distinct values in a string column to decide whether to prefer dictionary encoding when shuffling the column. If the ratio is higher than this config, dictionary encoding will be used on shuffling string column. This config is effective if it is higher than 1.0. Note that this config is only used when <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.shuffle.mode</span></code> is <code class="docutils literal notranslate"><s [...]
<td><p>10.0</p></td>
</tr>
-<tr class="row-odd"><td><p>spark.comet.sparkToColumnar.supportedOperatorList</p></td>
+<tr class="row-odd"><td><p>spark.comet.shuffle.sizeInBytesMultiplier</p></td>
+<td><p>Comet reports smaller sizes for shuffle due to using Arrow’s columnar memory format and this can result in Spark choosing a different join strategy due to the estimated size of the exchange being smaller. Comet will multiple sizeInBytes by this amount to avoid regressions in join strategy.</p></td>
+<td><p>1.0</p></td>
+</tr>
+<tr class="row-even"><td><p>spark.comet.sparkToColumnar.supportedOperatorList</p></td>
<td><p>A comma-separated list of operators that will be converted to Arrow columnar format when ‘spark.comet.sparkToColumnar.enabled’ is true</p></td>
<td><p>Range,InMemoryTableScan</p></td>
</tr>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org