You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "1cbyc (via GitHub)" <gi...@apache.org> on 2026/10/13 23:21:25 UTC
[PR] docs: run Comet SQL in Kubernetes example [datafusion-comet]
1cbyc opened a new pull request, #6949:
URL: https://github.com/apache/datafusion-comet/pull/6949
## Which issue does this PR close?
Closes #6190.
## Rationale for this change
The Kubernetes example currently runs `SparkPi`, whose RDD workload does not exercise Comet. A successful run therefore cannot confirm that Comet executes a query.
## What changes are included in this PR?
The example now uses Spark's bundled SQL CLI driver to explain and execute a self-contained range aggregation. The guide also updates the application names and explains which Comet operators and result to expect in the driver log.
## How are these changes tested?
- Parsed the embedded `SparkApplication` manifest with PyYAML
- Verified the configured main class exists in the Spark 4.1.3 `spark-hive-thriftserver` artifact
- Verified the documented aggregate result independently
- Ran Prettier 3.6.2 against the changed page
- Rendered the changed page in the full Sphinx HTML build; the source checkout reports existing missing generated-version and Mermaid-tool warnings unrelated to this page
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: run Comet SQL in Kubernetes example [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #6949:
URL: https://github.com/apache/datafusion-comet/pull/6949#discussion_r4266146462
##########
docs/source/user-guide/latest/kubernetes.md:
##########
@@ -61,21 +61,24 @@ REVISION: 1
TEST SUITE: None
```
-Create example Spark application file `spark-pi.yaml`
+Create an example Spark SQL application file named `comet-dataframe-pi.yaml`. This application uses
+Spark's DataFrame API, so it exercises Comet without requiring an external data source.
```bash
apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
- name: spark-pi
+ name: comet-dataframe-pi
namespace: default
spec:
type: Scala
mode: cluster
image: apache/datafusion-comet
imagePullPolicy: IfNotPresent
- mainClass: org.apache.spark.examples.SparkPi
+ mainClass: org.apache.spark.examples.sql.SparkDataFramePi
Review Comment:
[P2] Enable native range execution for the replacement application. `SparkDataFramePi` starts with `spark.range`, but both `spark.comet.exec.range.enabled` and `spark.comet.convert.range.enabled` default to `false` and the manifest enables neither. Its projection, filter, and partial count therefore stay on Spark. Comet also declines the final count over Spark’s partial aggregate, leaving the whole query on Spark. Users get a Pi result but no native plan, contrary to the new verification instructions. Add `"spark.comet.exec.range.enabled": "true"` and validate native execution.
Evidence: A local Spark 4.1.3 probe of the example’s query, using the manifest settings and recompiled exact-head planner/config/operator sources over cached dependencies, printed `RANGE_DEFAULTS=false,false`. Its completed plan contained only Spark `HashAggregate`, `Exchange`, `Project`, `Filter`, and `Range`, with zero `Comet native query plan:` messages. Changing only `spark.comet.exec.range.enabled` to `true` produced `CometRange`, `CometProject`, `CometFilter`, `CometHashAggregate`, native shuffle, and 11 native-plan messages. The defaults are in `CometConf.scala:221-228` and `355-363`. The exclusion of Spark-partial/native-final COUNT is documented in `CometAggregateExpressionSerde.scala:85-91`.
##########
docs/source/user-guide/latest/kubernetes.md:
##########
@@ -110,41 +114,45 @@ Refer to [Comet builds](#comet-docker-images)
Run Apache Spark application with Comet enabled
```bash
-kubectl apply -f spark-pi.yaml
-sparkapplication.sparkoperator.k8s.io/spark-pi created
+kubectl apply -f comet-dataframe-pi.yaml
+sparkapplication.sparkoperator.k8s.io/comet-dataframe-pi created
```
Check application status
```bash
-kubectl get sparkapp spark-pi
+kubectl get sparkapp comet-dataframe-pi
-NAME STATUS ATTEMPTS START FINISH AGE
-spark-pi RUNNING 1 2025-03-18T21:19:48Z <no value> 65s
+NAME STATUS ATTEMPTS START FINISH AGE
+comet-dataframe-pi COMPLETED 1 ... ... ...
```
To check more runtime details
```bash
-kubectl describe sparkapplication spark-pi
+kubectl describe sparkapplication comet-dataframe-pi
....
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
- Normal SparkApplicationSubmitted 8m15s spark-application-controller SparkApplication spark-pi was submitted successfully
- Normal SparkDriverRunning 7m18s spark-application-controller Driver spark-pi-driver is running
- Normal SparkExecutorPending 7m11s spark-application-controller Executor [spark-pi-68732195ab217303-exec-1] is pending
- Normal SparkExecutorRunning 7m10s spark-application-controller Executor [spark-pi-68732195ab217303-exec-1] is running
- Normal SparkExecutorCompleted 7m5s spark-application-controller Executor [spark-pi-68732195ab217303-exec-1] completed
- Normal SparkDriverCompleted 7m4s spark-application-controller Driver spark-pi-driver completed
+ Normal SparkApplicationSubmitted 8m15s spark-application-controller SparkApplication comet-dataframe-pi was submitted successfully
+ Normal SparkDriverRunning 7m18s spark-application-controller Driver comet-dataframe-pi-driver is running
+ Normal SparkExecutorPending 7m11s spark-application-controller Executor [comet-dataframe-pi-68732195ab217303-exec-1] is pending
+ Normal SparkExecutorRunning 7m10s spark-application-controller Executor [comet-dataframe-pi-68732195ab217303-exec-1] is running
+ Normal SparkExecutorCompleted 7m5s spark-application-controller Executor [comet-dataframe-pi-68732195ab217303-exec-1] completed
+ Normal SparkDriverCompleted 7m4s spark-application-controller Driver comet-dataframe-pi-driver completed
```
Get Driver Logs
```bash
-kubectl logs spark-pi-driver
+kubectl logs comet-dataframe-pi-driver
```
+The driver log contains the computed approximation (`Pi is roughly ...`). To confirm that Comet
+executed the DataFrame query, inspect the executor log for the native plan emitted by
Review Comment:
[P2] Keep the executor logs available for the documented verification step. Following the guide in order reaches `COMPLETED` before inspecting logs, but Spark defaults `spark.kubernetes.executor.deleteOnTermination` to `true`. On a cluster without centralized log collection, the executor pods and their retrievable logs are therefore gone by this step. Even after enabling native range execution, users cannot obtain the promised execution evidence. Set this option to `false` for the example or document capturing executor logs while the application runs.
Evidence: Spark v4.1.3 `SparkDataFramePi.scala:40` calls `spark.stop()` after printing Pi. Kubernetes `Config.scala:695-701` defaults executor deletion to `true`. `KubernetesClusterSchedulerBackend.scala:159-161` consequently invokes `podAllocator.stop(applicationId())`, and `ExecutorPodsAllocator.scala:535-542` deletes the application's executor pods. The manifest supplies no override or log-retention mechanism. This establishes the default cleanup path before post-completion inspection. No live Kubernetes reproduction was performed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: run Comet SQL in Kubernetes example [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #6949:
URL: https://github.com/apache/datafusion-comet/pull/6949
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: run Comet SQL in Kubernetes example [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #6949:
URL: https://github.com/apache/datafusion-comet/pull/6949#discussion_r4265720336
##########
docs/source/user-guide/latest/kubernetes.md:
##########
@@ -61,21 +61,26 @@ REVISION: 1
TEST SUITE: None
```
-Create example Spark application file `spark-pi.yaml`
+Create an example Spark SQL application file named `comet-sql.yaml`. The query uses a generated
+range, so it does not require an external data source. Its `EXPLAIN` output also makes it possible
+to verify that Comet is executing the query.
```bash
apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
- name: spark-pi
+ name: comet-sql
namespace: default
spec:
type: Scala
mode: cluster
image: apache/datafusion-comet
imagePullPolicy: IfNotPresent
- mainClass: org.apache.spark.examples.SparkPi
- mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.13-4.1.3.jar
+ mainClass: org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver
Review Comment:
[P2] Use a main class that supports cluster deployment. The manifest still sets `mode: cluster`, which the Spark Operator forwards as `--deploy-mode cluster`. Spark explicitly identifies `SparkSQLCLIDriver` as the SQL shell and rejects this combination, including when `-e` supplies a noninteractive query. Following the example therefore fails before creating the driver instead of reaching `COMPLETED` and producing the documented Comet plan/result. Could this use a small cluster-compatible SQL/DataFrame application instead?
Evidence: Reproduced with stock Spark 4.1.3: `spark-submit --master k8s://https://127.0.0.1:65534 --deploy-mode cluster --class org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver local:///opt/spark/jars/spark-hive-thriftserver_2.13-4.1.3.jar -e 'EXPLAIN SELECT sum(id * id) FROM range(1000000); SELECT sum(id * id) FROM range(1000000);'` exits 1 with `Cluster deploy mode is not applicable to Spark SQL shell.` before contacting Kubernetes. Spark v4.1.3 `core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala:309-310` implements the rejection, and lines 1197-1198 identify this exact main class. The SQL succeeds separately in client mode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: run Comet SQL in Kubernetes example [datafusion-comet]
Posted by "1cbyc (via GitHub)" <gi...@apache.org>.
1cbyc commented on PR #6949:
URL: https://github.com/apache/datafusion-comet/pull/6949#issuecomment-6158715234
Replaced the SQL CLI with Spark’s cluster-compatible bundled `SparkDataFramePi` application in signed commit e53f5e2. The example now enables native-plan logging so the executor output can confirm Comet execution. Validation passed: embedded YAML parse, Prettier 3.6.2, `git diff --check`, and verification against the Spark 4.1.3 example source.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: run Comet SQL in Kubernetes example [datafusion-comet]
Posted by "1cbyc (via GitHub)" <gi...@apache.org>.
1cbyc commented on code in PR #6949:
URL: https://github.com/apache/datafusion-comet/pull/6949#discussion_r4266414398
##########
docs/source/user-guide/latest/kubernetes.md:
##########
@@ -61,21 +61,24 @@ REVISION: 1
TEST SUITE: None
```
-Create example Spark application file `spark-pi.yaml`
+Create an example Spark SQL application file named `comet-dataframe-pi.yaml`. This application uses
+Spark's DataFrame API, so it exercises Comet without requiring an external data source.
```bash
apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
- name: spark-pi
+ name: comet-dataframe-pi
namespace: default
spec:
type: Scala
mode: cluster
image: apache/datafusion-comet
imagePullPolicy: IfNotPresent
- mainClass: org.apache.spark.examples.SparkPi
+ mainClass: org.apache.spark.examples.sql.SparkDataFramePi
Review Comment:
Enabled `spark.comet.exec.range.enabled` in 4a1bb1c so the bundled DataFrame workload enters the native range, projection, filter, and aggregation path.
##########
docs/source/user-guide/latest/kubernetes.md:
##########
@@ -110,41 +114,45 @@ Refer to [Comet builds](#comet-docker-images)
Run Apache Spark application with Comet enabled
```bash
-kubectl apply -f spark-pi.yaml
-sparkapplication.sparkoperator.k8s.io/spark-pi created
+kubectl apply -f comet-dataframe-pi.yaml
+sparkapplication.sparkoperator.k8s.io/comet-dataframe-pi created
```
Check application status
```bash
-kubectl get sparkapp spark-pi
+kubectl get sparkapp comet-dataframe-pi
-NAME STATUS ATTEMPTS START FINISH AGE
-spark-pi RUNNING 1 2025-03-18T21:19:48Z <no value> 65s
+NAME STATUS ATTEMPTS START FINISH AGE
+comet-dataframe-pi COMPLETED 1 ... ... ...
```
To check more runtime details
```bash
-kubectl describe sparkapplication spark-pi
+kubectl describe sparkapplication comet-dataframe-pi
....
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
- Normal SparkApplicationSubmitted 8m15s spark-application-controller SparkApplication spark-pi was submitted successfully
- Normal SparkDriverRunning 7m18s spark-application-controller Driver spark-pi-driver is running
- Normal SparkExecutorPending 7m11s spark-application-controller Executor [spark-pi-68732195ab217303-exec-1] is pending
- Normal SparkExecutorRunning 7m10s spark-application-controller Executor [spark-pi-68732195ab217303-exec-1] is running
- Normal SparkExecutorCompleted 7m5s spark-application-controller Executor [spark-pi-68732195ab217303-exec-1] completed
- Normal SparkDriverCompleted 7m4s spark-application-controller Driver spark-pi-driver completed
+ Normal SparkApplicationSubmitted 8m15s spark-application-controller SparkApplication comet-dataframe-pi was submitted successfully
+ Normal SparkDriverRunning 7m18s spark-application-controller Driver comet-dataframe-pi-driver is running
+ Normal SparkExecutorPending 7m11s spark-application-controller Executor [comet-dataframe-pi-68732195ab217303-exec-1] is pending
+ Normal SparkExecutorRunning 7m10s spark-application-controller Executor [comet-dataframe-pi-68732195ab217303-exec-1] is running
+ Normal SparkExecutorCompleted 7m5s spark-application-controller Executor [comet-dataframe-pi-68732195ab217303-exec-1] completed
+ Normal SparkDriverCompleted 7m4s spark-application-controller Driver comet-dataframe-pi-driver completed
```
Get Driver Logs
```bash
-kubectl logs spark-pi-driver
+kubectl logs comet-dataframe-pi-driver
```
+The driver log contains the computed approximation (`Pi is roughly ...`). To confirm that Comet
+executed the DataFrame query, inspect the executor log for the native plan emitted by
Review Comment:
Set `spark.kubernetes.executor.deleteOnTermination` to `false` in 4a1bb1c, documented why the executor is retained, and added the cleanup command.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org