You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/10/08 17:49:30 UTC
[PR] docs: expand the FAQ and refresh the Gluten comparison [datafusion-comet]
andygrove opened a new pull request, #6783:
URL: https://github.com/apache/datafusion-comet/pull/6783
## Which issue does this PR close?
No issue. This follows up on #6778, which added the FAQ page with a single entry.
## Rationale for this change
Some questions come up repeatedly in GitHub Discussions and in issues users file, and the docs answer them only in passing or not at all. Examples:
- `ClassNotFoundException` for `CometShuffleManager` (#864, 13 comments)
- vendor Spark builds failing with `NoSuchMethodError` (#414, #1576)
- no speedup on TPC-H or TPC-DS (#1450, #588)
- the published jars not loading on older glibc (#2504, #1499)
- how Comet compares with Gluten and other accelerators (discussion #994)
The Gluten comparison page had also gone stale. Some of its claims about Gluten no longer match Gluten's own docs and code.
## What changes are included in this PR?
**FAQ (`about/faq.md`).** Fourteen new entries in four sections. The existing vectorized Java/Scala UDF entry moves into Feature Support unchanged.
- Choosing Comet:
- how Comet compares to other open-source Spark accelerators (Gluten, Auron, the NVIDIA cuDF plugin)
- whether results match Spark
- supported Spark and Java versions
- vendor distributions of Spark
- platforms the published jars support (Linux amd64/arm64, glibc 2.31 or newer, CPU baselines)
- Running Comet:
- the `CometShuffleManager` classpath error
- memory
- why a query isn't faster
- Feature Support:
- Delta Lake, Hudi and Paimon
- PySpark and Python UDFs
- Structured Streaming
- remote shuffle services
- Project and Community:
- the roadmap
- where to ask questions and report bugs
**Gluten comparison (`about/gluten_comparison.md`).** These claims were checked against Gluten's v1.7.0 docs and code:
- Partial projection: Gluten has had this on by default since 1.3.0. It runs only the unsupported expressions in a projection, including Scala and Hive UDFs, in Spark. The codegen dispatch and UDF sections said Gluten had no equivalent and fell back. They now describe partial projection and note that Comet doesn't cover Hive UDFs.
- Regular expressions: Gluten falls back for patterns RE2 can't compile and has a setting that sends all regex functions to Spark. The section no longer says it "cannot fully match Spark's regex semantics regardless of configuration".
- ANSI: mentions `spark.gluten.sql.ansiFallback.enabled` and Gluten's note that results in that mode don't yet match Spark.
- Iceberg:
- Comet: now spec v3, deletion vectors, encrypted v3 tables, Iceberg 1.11 and experimental native writes. The page still said writes go through Spark.
- Gluten: what its Iceberg guide documents (unpartitioned reads offloaded, partitioned tables mostly fall back, equality deletes and writes fall back).
- Versions:
- Delta: 2.3 through 4.1 depending on the Spark version, not 2.0 through 4.0.
- Spark and backends: pinned to Gluten 1.7.0, with Bolt mentioned as unreleased.
- The "timestamp encodings" note is dropped, since Gluten removed it from its limitations page.
- Performance: adds AWS Labs' separate Comet 1.0.0 (1.57×) and Gluten 1.6.0 (1.63×) runs, which use the same Spark baseline. The Comet 0.16.0 vs Gluten 1.6.0 head-to-head is still AWS Labs' latest comparison of the two.
**About index.** Updated the descriptions of both pages.
## How are these changes tested?
Documentation only.
- `prettier --check` passes on the changed files.
- A script checked that every relative link and `#anchor` resolves to an existing page and heading.
- The github.io and apache.org links couldn't be fetched from my machine. The two new AWS Labs URLs follow the routing that the AWS Labs site's own index uses.
- The docs site builds only on pushes to main, so the pages haven't been rendered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: expand the FAQ and refresh the Gluten comparison [datafusion-comet]
Posted by "kazuyukitanimura (via GitHub)" <gi...@apache.org>.
kazuyukitanimura commented on code in PR #6783:
URL: https://github.com/apache/datafusion-comet/pull/6783#discussion_r4223229289
##########
docs/source/about/faq.md:
##########
@@ -19,7 +19,144 @@
# Frequently Asked Questions
-## Does Comet plan to add an API for vectorized Java/Scala UDFs similar to the Rust UDF API?
+Answers to common questions about choosing, installing, and running Comet. If your question is not
+answered here, see [Where can I ask questions or report bugs?](#where-can-i-ask-questions-or-report-bugs)
+
+## Choosing Comet
+
+### How does Comet compare to other open-source Spark accelerators?
+
+Several open-source projects speed up Spark by running query plans in native code. They differ in
+the engine they build on and the hardware they need.
+
+- [Apache Gluten](https://gluten.apache.org/) is the closest to Comet in design: a Spark plugin that
+ passes the parts of a plan it supports to a native engine, such as the C++ engine Velox. See
+ [Comparison of Comet and Gluten](gluten_comparison.md).
+- [Apache Auron (incubating)](https://auron.apache.org/), formerly Blaze, is also a Spark plugin
+ built on DataFusion. Comet is developed within the DataFusion project itself.
+- The [NVIDIA cuDF plugin for Apache Spark](https://github.com/NVIDIA/cudf-spark), formerly the
+ RAPIDS Accelerator, runs queries on NVIDIA GPUs. Comet runs on the CPUs your cluster already has.
+
+Gluten and Auron are also adding Flink support, while Comet focuses on Spark alone. Which
+accelerator is fastest depends on the workload, so we recommend benchmarking your own jobs.
+
+### Will Comet return the same results as Spark?
+
+That is the goal, and Comet runs Apache Spark's own SQL test suite in CI to check it. By default,
+Comet uses implementations that match Spark, including running Spark's own generated code for
+expressions whose native implementations behave differently. Those native implementations are
+opt-in, one expression at a time. The known differences that remain, such as the handling of strings
+that contain invalid UTF-8, are listed in the
+[Compatibility Guide](../user-guide/latest/compatibility/index.md). If you find a difference that is
+not documented, please [file an issue](https://github.com/apache/datafusion-comet/issues).
+
+### Which Spark and Java versions does Comet support?
+
+Spark 3.4, 3.5, 4.0, 4.1, and 4.2, on Java 17 or later. Spark 3.4 is deprecated. The
+[installation guide](../user-guide/latest/installation.md#supported-spark-versions) lists the exact
+Spark, Java, and Scala versions, and the
+[Spark Version Compatibility](../user-guide/latest/compatibility/spark-versions.md) page lists known
+issues for each Spark version.
+
+Comet drops a Spark version some time after the Spark project stops maintaining it. Spark 3.3 and
+earlier are no longer supported, and Comet 1.1.0 dropped Java 11. Java 8 cannot be supported because
+Apache Arrow's Java library no longer supports it. If you cannot upgrade Spark yet, stay on an earlier
+Comet release that supports your version, as the
+[versioning policy](versioning_policy.md#apache-spark-version-support) describes.
+
+### Does Comet work with vendor distributions of Spark?
+
+Comet is built and tested against open-source Apache Spark only. Spark distributions from cloud
+providers and other vendors can change the internal Spark APIs that Comet calls and add operators
+that Comet does not recognize, so Comet may fail with errors such as `NoSuchMethodError`, or leave
+much of the work to Spark. To use Comet in the cloud, run open-source Apache Spark there, for example
+on Kubernetes as described in the [Kubernetes guide](../user-guide/latest/kubernetes.md).
+
+### Which platforms do the published jars support?
+
+The jars in Maven Central include native libraries for Linux on amd64 and arm64, and need glibc 2.31
+or newer. They do not load on distributions with an older glibc, such as RHEL 7 and CentOS 7. On
+those systems and on macOS, [build Comet from source](../user-guide/latest/source.md).
+
+For performance, the libraries target CPUs that are common in data centers: `x86-64-v3`, which
+includes AVX2, on amd64, and Arm Neoverse N1 or newer cores on arm64. If Comet crashes with `SIGILL`
+(illegal instruction), the CPU is older than that. Build Comet from source for it, and please
+[open an issue](https://github.com/apache/datafusion-comet/issues) describing your environment.
+
+A jar you build yourself is smaller than the published one because it contains the native library
+for one platform only.
+
+## Running Comet
+
+### Why do I get a `ClassNotFoundException` for `CometShuffleManager`?
+
+Spark could not find the Comet jar when it created the shuffle manager. On Spark 3.4 and 3.5,
+executors create the shuffle manager before they load jars passed with `--jars` or `--packages`, so
+the jar must be on the startup classpath. Either copy it into `$SPARK_HOME/jars` on every machine, or
+set `spark.driver.extraClassPath` and `spark.executor.extraClassPath` to its path, as the
+[installation guide](../user-guide/latest/installation.md#run-spark-shell-with-comet-enabled) shows.
+
+These two settings take a local file path, not an `hdfs://` or other URL. The JVM silently ignores
+classpath entries that do not exist, so the jar must be at that path on every machine that runs a
+driver or an executor.
+
+### How much memory does Comet need?
+
+Comet needs Spark's off-heap memory: without `spark.memory.offHeap.enabled=true`, Comet disables
+itself and logs a warning. Comet's native operators share the off-heap pool, sized by
+`spark.memory.offHeap.size`, with Spark. When the pool runs out, operators that can spill, such as
+sorts, aggregations, and shuffle writes, spill to disk, and operators that cannot spill fail the
+task.
+
+Comet also uses memory that the pool does not track, which has to fit in
+`spark.executor.memoryOverhead`. If the cluster manager kills executors for exceeding their memory
+limit, raise the overhead rather than the pool. Each executor logs its native memory use while Comet
+runs, which you can use to size these settings. See
+[Memory Tuning](../user-guide/latest/tuning/memory.md).
+
+### Why isn't my query faster with Comet?
+
+First check how much of the query Comet runs. Set `spark.comet.explain.fallback.enabled=true` to log
+the reasons that parts of each query run in Spark, and see
+[Understanding Comet Plans](../user-guide/latest/understanding-comet-plans.md) to read the plan. To
+estimate how much of a workload Comet would accelerate without changing how it runs, set
+`spark.comet.explain.planOnly.enabled=true`.
+
+Common causes of a small speedup are:
+
+- **Fallbacks inside a stage.** Each switch between Comet and Spark converts data between columnar
+ and row formats, which can cost more than Comet saves. See
+ [Reducing Row/Columnar Conversion Overhead](../user-guide/latest/tuning/transitions.md).
+- **Comet shuffle is not enabled.** Unless `spark.shuffle.manager` names Comet's shuffle manager,
+ shuffles run in Spark. See [Shuffle Tuning](../user-guide/latest/tuning/shuffle.md).
+- **Too little memory.** Native operators spill to disk when the off-heap pool is too small. See
+ [Memory Tuning](../user-guide/latest/tuning/memory.md).
+- **Little computation to accelerate.** Queries that spend most of their time reading from storage,
+ listing files, or planning, or that process little data, have less to gain.
+
+## Feature Support
+
+### Does Comet support Delta Lake, Apache Hudi, or Apache Paimon tables?
+
+Not yet. Comet accelerates [Apache Iceberg](../user-guide/latest/iceberg.md) tables, with a native
+reader that is enabled by default and
+[experimental native writes](../user-guide/latest/iceberg-writes.md). It does not accelerate scans of
+Delta Lake, Hudi, or Paimon tables, so Spark reads those. Native Delta Lake reads are in development;
+see the [roadmap](../contributor-guide/roadmap.md#delta-lake-support). Hudi and Paimon are not on the
+roadmap.
+
+### Does Comet accelerate PySpark jobs and Python UDFs?
+
+PySpark DataFrame and SQL queries produce the same physical plans as Scala, so Comet accelerates them
+the same way. Set Comet up as the [installation guide](../user-guide/latest/installation.md)
+describes. There is no pip package.
+
+For Python UDFs, Comet has experimental support for `mapInArrow` and `mapInPandas` that keeps their
Review Comment:
pyarrow-udfs.md says this needs Spark 4.0 or later and does nothing on 3.4 and 3.5 even when enabled. Since the FAQ lists 3.4 and 3.5 as supported, could this say "on Spark 4.0 and later" and name spark.comet.exec.pyarrowUDF.enabled?
##########
docs/source/about/faq.md:
##########
@@ -19,7 +19,144 @@
# Frequently Asked Questions
-## Does Comet plan to add an API for vectorized Java/Scala UDFs similar to the Rust UDF API?
+Answers to common questions about choosing, installing, and running Comet. If your question is not
+answered here, see [Where can I ask questions or report bugs?](#where-can-i-ask-questions-or-report-bugs)
+
+## Choosing Comet
+
+### How does Comet compare to other open-source Spark accelerators?
+
+Several open-source projects speed up Spark by running query plans in native code. They differ in
+the engine they build on and the hardware they need.
+
+- [Apache Gluten](https://gluten.apache.org/) is the closest to Comet in design: a Spark plugin that
+ passes the parts of a plan it supports to a native engine, such as the C++ engine Velox. See
+ [Comparison of Comet and Gluten](gluten_comparison.md).
+- [Apache Auron (incubating)](https://auron.apache.org/), formerly Blaze, is also a Spark plugin
+ built on DataFusion. Comet is developed within the DataFusion project itself.
+- The [NVIDIA cuDF plugin for Apache Spark](https://github.com/NVIDIA/cudf-spark), formerly the
+ RAPIDS Accelerator, runs queries on NVIDIA GPUs. Comet runs on the CPUs your cluster already has.
+
+Gluten and Auron are also adding Flink support, while Comet focuses on Spark alone. Which
+accelerator is fastest depends on the workload, so we recommend benchmarking your own jobs.
+
+### Will Comet return the same results as Spark?
+
+That is the goal, and Comet runs Apache Spark's own SQL test suite in CI to check it. By default,
+Comet uses implementations that match Spark, including running Spark's own generated code for
+expressions whose native implementations behave differently. Those native implementations are
+opt-in, one expression at a time. The known differences that remain, such as the handling of strings
+that contain invalid UTF-8, are listed in the
+[Compatibility Guide](../user-guide/latest/compatibility/index.md). If you find a difference that is
+not documented, please [file an issue](https://github.com/apache/datafusion-comet/issues).
+
+### Which Spark and Java versions does Comet support?
+
+Spark 3.4, 3.5, 4.0, 4.1, and 4.2, on Java 17 or later. Spark 3.4 is deprecated. The
+[installation guide](../user-guide/latest/installation.md#supported-spark-versions) lists the exact
+Spark, Java, and Scala versions, and the
+[Spark Version Compatibility](../user-guide/latest/compatibility/spark-versions.md) page lists known
+issues for each Spark version.
+
+Comet drops a Spark version some time after the Spark project stops maintaining it. Spark 3.3 and
+earlier are no longer supported, and Comet 1.1.0 dropped Java 11. Java 8 cannot be supported because
+Apache Arrow's Java library no longer supports it. If you cannot upgrade Spark yet, stay on an earlier
+Comet release that supports your version, as the
+[versioning policy](versioning_policy.md#apache-spark-version-support) describes.
+
+### Does Comet work with vendor distributions of Spark?
+
+Comet is built and tested against open-source Apache Spark only. Spark distributions from cloud
+providers and other vendors can change the internal Spark APIs that Comet calls and add operators
+that Comet does not recognize, so Comet may fail with errors such as `NoSuchMethodError`, or leave
+much of the work to Spark. To use Comet in the cloud, run open-source Apache Spark there, for example
+on Kubernetes as described in the [Kubernetes guide](../user-guide/latest/kubernetes.md).
+
+### Which platforms do the published jars support?
+
+The jars in Maven Central include native libraries for Linux on amd64 and arm64, and need glibc 2.31
+or newer. They do not load on distributions with an older glibc, such as RHEL 7 and CentOS 7. On
+those systems and on macOS, [build Comet from source](../user-guide/latest/source.md).
+
+For performance, the libraries target CPUs that are common in data centers: `x86-64-v3`, which
+includes AVX2, on amd64, and Arm Neoverse N1 or newer cores on arm64. If Comet crashes with `SIGILL`
+(illegal instruction), the CPU is older than that. Build Comet from source for it, and please
+[open an issue](https://github.com/apache/datafusion-comet/issues) describing your environment.
+
+A jar you build yourself is smaller than the published one because it contains the native library
+for one platform only.
+
+## Running Comet
+
+### Why do I get a `ClassNotFoundException` for `CometShuffleManager`?
+
+Spark could not find the Comet jar when it created the shuffle manager. On Spark 3.4 and 3.5,
+executors create the shuffle manager before they load jars passed with `--jars` or `--packages`, so
+the jar must be on the startup classpath. Either copy it into `$SPARK_HOME/jars` on every machine, or
+set `spark.driver.extraClassPath` and `spark.executor.extraClassPath` to its path, as the
+[installation guide](../user-guide/latest/installation.md#run-spark-shell-with-comet-enabled) shows.
+
+These two settings take a local file path, not an `hdfs://` or other URL. The JVM silently ignores
+classpath entries that do not exist, so the jar must be at that path on every machine that runs a
+driver or an executor.
+
+### How much memory does Comet need?
+
+Comet needs Spark's off-heap memory: without `spark.memory.offHeap.enabled=true`, Comet disables
+itself and logs a warning. Comet's native operators share the off-heap pool, sized by
+`spark.memory.offHeap.size`, with Spark. When the pool runs out, operators that can spill, such as
+sorts, aggregations, and shuffle writes, spill to disk, and operators that cannot spill fail the
+task.
+
+Comet also uses memory that the pool does not track, which has to fit in
+`spark.executor.memoryOverhead`. If the cluster manager kills executors for exceeding their memory
+limit, raise the overhead rather than the pool. Each executor logs its native memory use while Comet
+runs, which you can use to size these settings. See
+[Memory Tuning](../user-guide/latest/tuning/memory.md).
+
+### Why isn't my query faster with Comet?
+
+First check how much of the query Comet runs. Set `spark.comet.explain.fallback.enabled=true` to log
+the reasons that parts of each query run in Spark, and see
+[Understanding Comet Plans](../user-guide/latest/understanding-comet-plans.md) to read the plan. To
+estimate how much of a workload Comet would accelerate without changing how it runs, set
+`spark.comet.explain.planOnly.enabled=true`.
+
+Common causes of a small speedup are:
+
+- **Fallbacks inside a stage.** Each switch between Comet and Spark converts data between columnar
+ and row formats, which can cost more than Comet saves. See
+ [Reducing Row/Columnar Conversion Overhead](../user-guide/latest/tuning/transitions.md).
+- **Comet shuffle is not enabled.** Unless `spark.shuffle.manager` names Comet's shuffle manager,
Review Comment:
spark.comet.shuffle.enabled defaults to true, and when spark.shuffle.manager isn't Comet's, CometSparkSessionExtensions logs a warning and disables Comet entirely. So the symptom is "nothing runs in Comet",
not "shuffles run in Spark". Could this say something like "If spark.shuffle.manager does not name Comet's shuffle manager, Comet disables itself and logs a warning, unless spark.comet.shuffle.enabled=false, in which case Comet runs but every shuffle runs in Spark"?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: expand the FAQ and refresh the Gluten comparison [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #6783:
URL: https://github.com/apache/datafusion-comet/pull/6783#discussion_r4223456108
##########
docs/source/about/faq.md:
##########
@@ -19,7 +19,144 @@
# Frequently Asked Questions
-## Does Comet plan to add an API for vectorized Java/Scala UDFs similar to the Rust UDF API?
+Answers to common questions about choosing, installing, and running Comet. If your question is not
+answered here, see [Where can I ask questions or report bugs?](#where-can-i-ask-questions-or-report-bugs)
+
+## Choosing Comet
+
+### How does Comet compare to other open-source Spark accelerators?
+
+Several open-source projects speed up Spark by running query plans in native code. They differ in
+the engine they build on and the hardware they need.
+
+- [Apache Gluten](https://gluten.apache.org/) is the closest to Comet in design: a Spark plugin that
+ passes the parts of a plan it supports to a native engine, such as the C++ engine Velox. See
+ [Comparison of Comet and Gluten](gluten_comparison.md).
+- [Apache Auron (incubating)](https://auron.apache.org/), formerly Blaze, is also a Spark plugin
+ built on DataFusion. Comet is developed within the DataFusion project itself.
+- The [NVIDIA cuDF plugin for Apache Spark](https://github.com/NVIDIA/cudf-spark), formerly the
+ RAPIDS Accelerator, runs queries on NVIDIA GPUs. Comet runs on the CPUs your cluster already has.
+
+Gluten and Auron are also adding Flink support, while Comet focuses on Spark alone. Which
+accelerator is fastest depends on the workload, so we recommend benchmarking your own jobs.
+
+### Will Comet return the same results as Spark?
+
+That is the goal, and Comet runs Apache Spark's own SQL test suite in CI to check it. By default,
+Comet uses implementations that match Spark, including running Spark's own generated code for
+expressions whose native implementations behave differently. Those native implementations are
+opt-in, one expression at a time. The known differences that remain, such as the handling of strings
+that contain invalid UTF-8, are listed in the
+[Compatibility Guide](../user-guide/latest/compatibility/index.md). If you find a difference that is
+not documented, please [file an issue](https://github.com/apache/datafusion-comet/issues).
+
+### Which Spark and Java versions does Comet support?
+
+Spark 3.4, 3.5, 4.0, 4.1, and 4.2, on Java 17 or later. Spark 3.4 is deprecated. The
+[installation guide](../user-guide/latest/installation.md#supported-spark-versions) lists the exact
+Spark, Java, and Scala versions, and the
+[Spark Version Compatibility](../user-guide/latest/compatibility/spark-versions.md) page lists known
+issues for each Spark version.
+
+Comet drops a Spark version some time after the Spark project stops maintaining it. Spark 3.3 and
+earlier are no longer supported, and Comet 1.1.0 dropped Java 11. Java 8 cannot be supported because
+Apache Arrow's Java library no longer supports it. If you cannot upgrade Spark yet, stay on an earlier
+Comet release that supports your version, as the
+[versioning policy](versioning_policy.md#apache-spark-version-support) describes.
+
+### Does Comet work with vendor distributions of Spark?
+
+Comet is built and tested against open-source Apache Spark only. Spark distributions from cloud
+providers and other vendors can change the internal Spark APIs that Comet calls and add operators
+that Comet does not recognize, so Comet may fail with errors such as `NoSuchMethodError`, or leave
+much of the work to Spark. To use Comet in the cloud, run open-source Apache Spark there, for example
+on Kubernetes as described in the [Kubernetes guide](../user-guide/latest/kubernetes.md).
+
+### Which platforms do the published jars support?
+
+The jars in Maven Central include native libraries for Linux on amd64 and arm64, and need glibc 2.31
+or newer. They do not load on distributions with an older glibc, such as RHEL 7 and CentOS 7. On
+those systems and on macOS, [build Comet from source](../user-guide/latest/source.md).
+
+For performance, the libraries target CPUs that are common in data centers: `x86-64-v3`, which
+includes AVX2, on amd64, and Arm Neoverse N1 or newer cores on arm64. If Comet crashes with `SIGILL`
+(illegal instruction), the CPU is older than that. Build Comet from source for it, and please
+[open an issue](https://github.com/apache/datafusion-comet/issues) describing your environment.
+
+A jar you build yourself is smaller than the published one because it contains the native library
+for one platform only.
+
+## Running Comet
+
+### Why do I get a `ClassNotFoundException` for `CometShuffleManager`?
+
+Spark could not find the Comet jar when it created the shuffle manager. On Spark 3.4 and 3.5,
+executors create the shuffle manager before they load jars passed with `--jars` or `--packages`, so
+the jar must be on the startup classpath. Either copy it into `$SPARK_HOME/jars` on every machine, or
+set `spark.driver.extraClassPath` and `spark.executor.extraClassPath` to its path, as the
+[installation guide](../user-guide/latest/installation.md#run-spark-shell-with-comet-enabled) shows.
+
+These two settings take a local file path, not an `hdfs://` or other URL. The JVM silently ignores
+classpath entries that do not exist, so the jar must be at that path on every machine that runs a
+driver or an executor.
+
+### How much memory does Comet need?
+
+Comet needs Spark's off-heap memory: without `spark.memory.offHeap.enabled=true`, Comet disables
+itself and logs a warning. Comet's native operators share the off-heap pool, sized by
+`spark.memory.offHeap.size`, with Spark. When the pool runs out, operators that can spill, such as
+sorts, aggregations, and shuffle writes, spill to disk, and operators that cannot spill fail the
+task.
+
+Comet also uses memory that the pool does not track, which has to fit in
+`spark.executor.memoryOverhead`. If the cluster manager kills executors for exceeding their memory
+limit, raise the overhead rather than the pool. Each executor logs its native memory use while Comet
+runs, which you can use to size these settings. See
+[Memory Tuning](../user-guide/latest/tuning/memory.md).
+
+### Why isn't my query faster with Comet?
+
+First check how much of the query Comet runs. Set `spark.comet.explain.fallback.enabled=true` to log
+the reasons that parts of each query run in Spark, and see
+[Understanding Comet Plans](../user-guide/latest/understanding-comet-plans.md) to read the plan. To
+estimate how much of a workload Comet would accelerate without changing how it runs, set
+`spark.comet.explain.planOnly.enabled=true`.
+
+Common causes of a small speedup are:
+
+- **Fallbacks inside a stage.** Each switch between Comet and Spark converts data between columnar
+ and row formats, which can cost more than Comet saves. See
+ [Reducing Row/Columnar Conversion Overhead](../user-guide/latest/tuning/transitions.md).
+- **Comet shuffle is not enabled.** Unless `spark.shuffle.manager` names Comet's shuffle manager,
Review Comment:
I had the symptom wrong. The bullet now says that Comet disables itself and logs a warning, unless `spark.comet.shuffle.enabled=false`, in which case Comet runs and every shuffle runs in Spark. I also renamed the bullet, since "Comet shuffle is not enabled" no longer described it. Fixed in ba7212bacc.
##########
docs/source/about/faq.md:
##########
@@ -19,7 +19,144 @@
# Frequently Asked Questions
-## Does Comet plan to add an API for vectorized Java/Scala UDFs similar to the Rust UDF API?
+Answers to common questions about choosing, installing, and running Comet. If your question is not
+answered here, see [Where can I ask questions or report bugs?](#where-can-i-ask-questions-or-report-bugs)
+
+## Choosing Comet
+
+### How does Comet compare to other open-source Spark accelerators?
+
+Several open-source projects speed up Spark by running query plans in native code. They differ in
+the engine they build on and the hardware they need.
+
+- [Apache Gluten](https://gluten.apache.org/) is the closest to Comet in design: a Spark plugin that
+ passes the parts of a plan it supports to a native engine, such as the C++ engine Velox. See
+ [Comparison of Comet and Gluten](gluten_comparison.md).
+- [Apache Auron (incubating)](https://auron.apache.org/), formerly Blaze, is also a Spark plugin
+ built on DataFusion. Comet is developed within the DataFusion project itself.
+- The [NVIDIA cuDF plugin for Apache Spark](https://github.com/NVIDIA/cudf-spark), formerly the
+ RAPIDS Accelerator, runs queries on NVIDIA GPUs. Comet runs on the CPUs your cluster already has.
+
+Gluten and Auron are also adding Flink support, while Comet focuses on Spark alone. Which
+accelerator is fastest depends on the workload, so we recommend benchmarking your own jobs.
+
+### Will Comet return the same results as Spark?
+
+That is the goal, and Comet runs Apache Spark's own SQL test suite in CI to check it. By default,
+Comet uses implementations that match Spark, including running Spark's own generated code for
+expressions whose native implementations behave differently. Those native implementations are
+opt-in, one expression at a time. The known differences that remain, such as the handling of strings
+that contain invalid UTF-8, are listed in the
+[Compatibility Guide](../user-guide/latest/compatibility/index.md). If you find a difference that is
+not documented, please [file an issue](https://github.com/apache/datafusion-comet/issues).
+
+### Which Spark and Java versions does Comet support?
+
+Spark 3.4, 3.5, 4.0, 4.1, and 4.2, on Java 17 or later. Spark 3.4 is deprecated. The
+[installation guide](../user-guide/latest/installation.md#supported-spark-versions) lists the exact
+Spark, Java, and Scala versions, and the
+[Spark Version Compatibility](../user-guide/latest/compatibility/spark-versions.md) page lists known
+issues for each Spark version.
+
+Comet drops a Spark version some time after the Spark project stops maintaining it. Spark 3.3 and
+earlier are no longer supported, and Comet 1.1.0 dropped Java 11. Java 8 cannot be supported because
+Apache Arrow's Java library no longer supports it. If you cannot upgrade Spark yet, stay on an earlier
+Comet release that supports your version, as the
+[versioning policy](versioning_policy.md#apache-spark-version-support) describes.
+
+### Does Comet work with vendor distributions of Spark?
+
+Comet is built and tested against open-source Apache Spark only. Spark distributions from cloud
+providers and other vendors can change the internal Spark APIs that Comet calls and add operators
+that Comet does not recognize, so Comet may fail with errors such as `NoSuchMethodError`, or leave
+much of the work to Spark. To use Comet in the cloud, run open-source Apache Spark there, for example
+on Kubernetes as described in the [Kubernetes guide](../user-guide/latest/kubernetes.md).
+
+### Which platforms do the published jars support?
+
+The jars in Maven Central include native libraries for Linux on amd64 and arm64, and need glibc 2.31
+or newer. They do not load on distributions with an older glibc, such as RHEL 7 and CentOS 7. On
+those systems and on macOS, [build Comet from source](../user-guide/latest/source.md).
+
+For performance, the libraries target CPUs that are common in data centers: `x86-64-v3`, which
+includes AVX2, on amd64, and Arm Neoverse N1 or newer cores on arm64. If Comet crashes with `SIGILL`
+(illegal instruction), the CPU is older than that. Build Comet from source for it, and please
+[open an issue](https://github.com/apache/datafusion-comet/issues) describing your environment.
+
+A jar you build yourself is smaller than the published one because it contains the native library
+for one platform only.
+
+## Running Comet
+
+### Why do I get a `ClassNotFoundException` for `CometShuffleManager`?
+
+Spark could not find the Comet jar when it created the shuffle manager. On Spark 3.4 and 3.5,
+executors create the shuffle manager before they load jars passed with `--jars` or `--packages`, so
+the jar must be on the startup classpath. Either copy it into `$SPARK_HOME/jars` on every machine, or
+set `spark.driver.extraClassPath` and `spark.executor.extraClassPath` to its path, as the
+[installation guide](../user-guide/latest/installation.md#run-spark-shell-with-comet-enabled) shows.
+
+These two settings take a local file path, not an `hdfs://` or other URL. The JVM silently ignores
+classpath entries that do not exist, so the jar must be at that path on every machine that runs a
+driver or an executor.
+
+### How much memory does Comet need?
+
+Comet needs Spark's off-heap memory: without `spark.memory.offHeap.enabled=true`, Comet disables
+itself and logs a warning. Comet's native operators share the off-heap pool, sized by
+`spark.memory.offHeap.size`, with Spark. When the pool runs out, operators that can spill, such as
+sorts, aggregations, and shuffle writes, spill to disk, and operators that cannot spill fail the
+task.
+
+Comet also uses memory that the pool does not track, which has to fit in
+`spark.executor.memoryOverhead`. If the cluster manager kills executors for exceeding their memory
+limit, raise the overhead rather than the pool. Each executor logs its native memory use while Comet
+runs, which you can use to size these settings. See
+[Memory Tuning](../user-guide/latest/tuning/memory.md).
+
+### Why isn't my query faster with Comet?
+
+First check how much of the query Comet runs. Set `spark.comet.explain.fallback.enabled=true` to log
+the reasons that parts of each query run in Spark, and see
+[Understanding Comet Plans](../user-guide/latest/understanding-comet-plans.md) to read the plan. To
+estimate how much of a workload Comet would accelerate without changing how it runs, set
+`spark.comet.explain.planOnly.enabled=true`.
+
+Common causes of a small speedup are:
+
+- **Fallbacks inside a stage.** Each switch between Comet and Spark converts data between columnar
+ and row formats, which can cost more than Comet saves. See
+ [Reducing Row/Columnar Conversion Overhead](../user-guide/latest/tuning/transitions.md).
+- **Comet shuffle is not enabled.** Unless `spark.shuffle.manager` names Comet's shuffle manager,
+ shuffles run in Spark. See [Shuffle Tuning](../user-guide/latest/tuning/shuffle.md).
+- **Too little memory.** Native operators spill to disk when the off-heap pool is too small. See
+ [Memory Tuning](../user-guide/latest/tuning/memory.md).
+- **Little computation to accelerate.** Queries that spend most of their time reading from storage,
+ listing files, or planning, or that process little data, have less to gain.
+
+## Feature Support
+
+### Does Comet support Delta Lake, Apache Hudi, or Apache Paimon tables?
+
+Not yet. Comet accelerates [Apache Iceberg](../user-guide/latest/iceberg.md) tables, with a native
+reader that is enabled by default and
+[experimental native writes](../user-guide/latest/iceberg-writes.md). It does not accelerate scans of
+Delta Lake, Hudi, or Paimon tables, so Spark reads those. Native Delta Lake reads are in development;
+see the [roadmap](../contributor-guide/roadmap.md#delta-lake-support). Hudi and Paimon are not on the
+roadmap.
+
+### Does Comet accelerate PySpark jobs and Python UDFs?
+
+PySpark DataFrame and SQL queries produce the same physical plans as Scala, so Comet accelerates them
+the same way. Set Comet up as the [installation guide](../user-guide/latest/installation.md)
+describes. There is no pip package.
+
+For Python UDFs, Comet has experimental support for `mapInArrow` and `mapInPandas` that keeps their
Review Comment:
Changed to "on Spark 4.0 and later", and the paragraph now names `spark.comet.exec.pyarrowUDF.enabled`. Fixed in ba7212bacc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: expand the FAQ and refresh the Gluten comparison [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #6783:
URL: https://github.com/apache/datafusion-comet/pull/6783#issuecomment-6068006433
Thanks for the review @kazuyukitanimura and for catching those errors. I pushed fixes and will go ahead and merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: expand the FAQ and refresh the Gluten comparison [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #6783:
URL: https://github.com/apache/datafusion-comet/pull/6783
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org