You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/05/06 17:09:31 UTC
[PR] docs: refresh Spark version support, OS coverage, and version-pinned examples [datafusion-comet]
andygrove opened a new pull request, #4244:
URL: https://github.com/apache/datafusion-comet/pull/4244
## Which issue does this PR close?
Closes #.
## Rationale for this change
Several user-facing docs are out of date ahead of the 0.16 release:
- The Spark version compatibility, installation, and Gluten comparison pages still describe Spark 4.1 as experimental even though it now runs in full CI alongside Spark SQL tests.
- The installation page lists Apple macOS (Intel and Apple Silicon) under supported operating systems without noting that the published Maven jars only ship Linux native binaries; Intel macOS is also not exercised by CI any more.
- Several recently-added math expressions are still marked `[ ]` in the contributor-guide tracking page.
- `spark.comet.scan.enabled` is documented as a user-facing scan toggle, but in practice it exists to let Comet's own tests selectively disable native scans.
- The Iceberg and Kubernetes guides hard-code older Comet versions (`0.14.0` and `0.7.0`) in the example commands and YAML, and the Iceberg example also redundantly sets `spark.sql.extensions=org.apache.comet.CometSparkSessionExtensions`, which `CometPlugin` already registers.
## What changes are included in this PR?
- Promote Spark 4.1 to fully supported in `compatibility/spark-versions.md`, `installation.md`, and `about/gluten_comparison.md`. The experimental table now lists only Spark 4.2.0-preview4. Spark 4.1.1 is shown as running on JDK 17/21.
- Restructure the Supported Operating Systems section in `installation.md` into a table that distinguishes published Maven jars (Linux amd64/arm64) from build-from-source coverage. Drop the Intel macOS row.
- Flip seven `math_funcs` entries (`acosh`, `asinh`, `atanh`, `cbrt`, `degrees`, `pi`, `radians`) from `[ ]` to `[x]` in `docs/source/contributor-guide/spark_expressions_support.md` to reflect commits a4f022939, 1b4b26f31, e5351f4bc, and 356dd944a.
- Move `spark.comet.scan.enabled` from `CATEGORY_SCAN` to `CATEGORY_TESTING` in `CometConf.scala` and rewrite the doc string to make clear it is intended for Comet's own test suites. Remove the corresponding paragraph from `datasources.md`.
- Replace the hard-coded `0.14.0` references in `iceberg.md` and `0.7.0` references in `kubernetes.md` with the `$COMET_VERSION` placeholder used elsewhere in the docs. Drop the redundant `spark.sql.extensions=org.apache.comet.CometSparkSessionExtensions` conf from both Iceberg spark-shell examples.
## How are these changes tested?
Documentation-only changes apart from the `CometConf` doc string and category move; the `CONFIG_TABLE[testing]` section in `configs.md` is regenerated from `CometConf` by `GenerateDocs.scala` on merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: refresh Spark version support, OS coverage, and version-pinned examples [WIP] [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #4244:
URL: https://github.com/apache/datafusion-comet/pull/4244#discussion_r3198443861
##########
common/src/main/scala/org/apache/comet/CometConf.scala:
##########
@@ -94,12 +94,9 @@ object CometConf extends ShimCometConf {
.createWithEnvVarOrDefault("ENABLE_COMET", true)
val COMET_NATIVE_SCAN_ENABLED: ConfigEntry[Boolean] = conf("spark.comet.scan.enabled")
- .category(CATEGORY_SCAN)
- .doc(
- "Whether to enable native scans. When this is turned on, Spark will use Comet to " +
- "read supported data sources (currently only Parquet is supported natively). Note " +
- "that to enable native vectorized execution, both this config and " +
- "`spark.comet.exec.enabled` need to be enabled.")
+ .category(CATEGORY_TESTING)
+ .doc("Whether to enable native scans. Intended for use in Comet's own test suites to " +
+ "selectively disable native scans; not intended for production use.")
Review Comment:
this is unrelated to 0.16 / spark 4 changes, but we don't want users using this config
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: Documentation updates in preparation for 0.16 release [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #4244:
URL: https://github.com/apache/datafusion-comet/pull/4244
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: refresh Spark version support, OS coverage, and version-pinned examples [WIP] [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #4244:
URL: https://github.com/apache/datafusion-comet/pull/4244#discussion_r3198444658
##########
dev/release/build-release-comet.sh:
##########
@@ -202,7 +202,10 @@ LOCAL_REPO=$(mktemp -d /tmp/comet-staging-repo-XXXXX)
./mvnw "-Dmaven.repo.local=${LOCAL_REPO}" -P spark-3.4 -P scala-2.13 -DskipTests install
./mvnw "-Dmaven.repo.local=${LOCAL_REPO}" -P spark-3.5 -P scala-2.12 -DskipTests install
./mvnw "-Dmaven.repo.local=${LOCAL_REPO}" -P spark-3.5 -P scala-2.13 -DskipTests install
-./mvnw "-Dmaven.repo.local=${LOCAL_REPO}" -P spark-4.0 -P scala-2.13 -DskipTests install
+# The spark-4.x profiles pin their own Scala 2.13.x patch versions to match the
+# corresponding Spark release, so the scala-2.13 profile is not used here.
+./mvnw "-Dmaven.repo.local=${LOCAL_REPO}" -P spark-4.0 -DskipTests install
+./mvnw "-Dmaven.repo.local=${LOCAL_REPO}" -P spark-4.1 -DskipTests install
Review Comment:
Ship some new jars
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: refresh Spark version support, OS coverage, and version-pinned examples [WIP] [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #4244:
URL: https://github.com/apache/datafusion-comet/pull/4244#discussion_r3198464068
##########
docs/source/user-guide/latest/iceberg.md:
##########
@@ -25,21 +25,20 @@ Comet's native Iceberg reader relies on reflection to extract `FileScanTask`s fr
then serialized to Comet's native execution engine (see
[PR #2528](https://github.com/apache/datafusion-comet/pull/2528)).
-The example below uses Spark's package downloader to retrieve Comet 0.14.0 and Iceberg
+The example below uses Spark's package downloader to retrieve Comet $COMET_VERSION and Iceberg
1.8.1, but Comet has been tested with Iceberg 1.5, 1.7, 1.8, 1.9, and 1.10. The native Iceberg
reader is enabled by default. To disable it, set `spark.comet.scan.icebergNative.enabled=false`.
```shell
$SPARK_HOME/bin/spark-shell \
- --packages org.apache.datafusion:comet-spark-spark4.1_2.13:0.14.0,org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.8.1,org.apache.iceberg:iceberg-core:1.8.1 \
+ --packages org.apache.datafusion:comet-spark-spark3.5_2.12:$COMET_VERSION,org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.8.1,org.apache.iceberg:iceberg-core:1.8.1 \
Review Comment:
3.5 also supports 2.13
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: refresh Spark version support, OS coverage, and version-pinned examples [WIP] [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #4244:
URL: https://github.com/apache/datafusion-comet/pull/4244#discussion_r3198447295
##########
docs/source/contributor-guide/benchmark-results/comet-0.15.0-tpch-hashjoin.json:
##########
@@ -14,70 +14,26 @@
"scale_factor": "1000",
"spark_profile": "3.5_2.12"
},
- "1": [
- 12.007
- ],
- "2": [
- 24.2505
- ],
- "3": [
- 16.8625
- ],
- "4": [
- 14.686499999999999
- ],
- "5": [
- 32.248999999999995
- ],
- "6": [
- 0.7415
- ],
- "7": [
- 15.712499999999999
- ],
- "8": [
- 39.1135
- ],
- "9": [
- 47.789
- ],
- "10": [
- 28.569
- ],
- "11": [
- 14.600999999999999
- ],
- "12": [
- 7.8575
- ],
- "13": [
- 10.9605
- ],
- "14": [
- 2.607
- ],
- "15": [
- 11.962
- ],
- "16": [
- 15.587499999999999
- ],
- "17": [
- 36.6875
- ],
- "18": [
- 76.5035
- ],
- "19": [
- 9.379
- ],
- "20": [
- 8.841000000000001
- ],
- "21": [
- 102.9005
- ],
- "22": [
- 10.042
- ]
-}
\ No newline at end of file
+ "1": [12.007],
Review Comment:
its much nicer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org