You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "1cbyc (via GitHub)" <gi...@apache.org> on 2026/10/13 23:21:25 UTC

[PR] docs: run Comet SQL in Kubernetes example [datafusion-comet]

1cbyc opened a new pull request, #6949:
URL: https://github.com/apache/datafusion-comet/pull/6949

   ## Which issue does this PR close?
   
   Closes #6190.
   
   ## Rationale for this change
   
   The Kubernetes example currently runs `SparkPi`, whose RDD workload does not exercise Comet. A successful run therefore cannot confirm that Comet executes a query.
   
   ## What changes are included in this PR?
   
   The example now uses Spark's bundled SQL CLI driver to explain and execute a self-contained range aggregation. The guide also updates the application names and explains which Comet operators and result to expect in the driver log.
   
   ## How are these changes tested?
   
   - Parsed the embedded `SparkApplication` manifest with PyYAML
   - Verified the configured main class exists in the Spark 4.1.3 `spark-hive-thriftserver` artifact
   - Verified the documented aggregate result independently
   - Ran Prettier 3.6.2 against the changed page
   - Rendered the changed page in the full Sphinx HTML build; the source checkout reports existing missing generated-version and Mermaid-tool warnings unrelated to this page


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] docs: run Comet SQL in Kubernetes example [datafusion-comet]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #6949:
URL: https://github.com/apache/datafusion-comet/pull/6949#discussion_r4266146462


##########
docs/source/user-guide/latest/kubernetes.md:
##########
@@ -61,21 +61,24 @@ REVISION: 1
 TEST SUITE: None
 ```
 
-Create example Spark application file `spark-pi.yaml`
+Create an example Spark SQL application file named `comet-dataframe-pi.yaml`. This application uses
+Spark's DataFrame API, so it exercises Comet without requiring an external data source.
 
 ```bash
 apiVersion: sparkoperator.k8s.io/v1beta2
 kind: SparkApplication
 metadata:
-  name: spark-pi
+  name: comet-dataframe-pi
   namespace: default
 spec:
   type: Scala
   mode: cluster
   image: apache/datafusion-comet
   imagePullPolicy: IfNotPresent
-  mainClass: org.apache.spark.examples.SparkPi
+  mainClass: org.apache.spark.examples.sql.SparkDataFramePi

Review Comment:
   [P2] Enable native range execution for the replacement application. `SparkDataFramePi` starts with `spark.range`, but both `spark.comet.exec.range.enabled` and `spark.comet.convert.range.enabled` default to `false` and the manifest enables neither. Its projection, filter, and partial count therefore stay on Spark. Comet also declines the final count over Spark’s partial aggregate, leaving the whole query on Spark. Users get a Pi result but no native plan, contrary to the new verification instructions. Add `"spark.comet.exec.range.enabled": "true"` and validate native execution.
   
   Evidence: A local Spark 4.1.3 probe of the example’s query, using the manifest settings and recompiled exact-head planner/config/operator sources over cached dependencies, printed `RANGE_DEFAULTS=false,false`. Its completed plan contained only Spark `HashAggregate`, `Exchange`, `Project`, `Filter`, and `Range`, with zero `Comet native query plan:` messages. Changing only `spark.comet.exec.range.enabled` to `true` produced `CometRange`, `CometProject`, `CometFilter`, `CometHashAggregate`, native shuffle, and 11 native-plan messages. The defaults are in `CometConf.scala:221-228` and `355-363`. The exclusion of Spark-partial/native-final COUNT is documented in `CometAggregateExpressionSerde.scala:85-91`.



##########
docs/source/user-guide/latest/kubernetes.md:
##########
@@ -110,41 +114,45 @@ Refer to [Comet builds](#comet-docker-images)
 Run Apache Spark application with Comet enabled
 
 ```bash
-kubectl apply -f spark-pi.yaml
-sparkapplication.sparkoperator.k8s.io/spark-pi created
+kubectl apply -f comet-dataframe-pi.yaml
+sparkapplication.sparkoperator.k8s.io/comet-dataframe-pi created
 ```
 
 Check application status
 
 ```bash
-kubectl get sparkapp spark-pi
+kubectl get sparkapp comet-dataframe-pi
 
-NAME       STATUS    ATTEMPTS   START                  FINISH       AGE
-spark-pi   RUNNING   1          2025-03-18T21:19:48Z   <no value>   65s
+NAME                 STATUS      ATTEMPTS   START   FINISH   AGE
+comet-dataframe-pi   COMPLETED   1          ...     ...      ...
 ```
 
 To check more runtime details
 
 ```bash
-kubectl describe sparkapplication spark-pi
+kubectl describe sparkapplication comet-dataframe-pi
 
 ....
 Events:
   Type    Reason                     Age    From                          Message
   ----    ------                     ----   ----                          -------
-  Normal  SparkApplicationSubmitted  8m15s  spark-application-controller  SparkApplication spark-pi was submitted successfully
-  Normal  SparkDriverRunning         7m18s  spark-application-controller  Driver spark-pi-driver is running
-  Normal  SparkExecutorPending       7m11s  spark-application-controller  Executor [spark-pi-68732195ab217303-exec-1] is pending
-  Normal  SparkExecutorRunning       7m10s  spark-application-controller  Executor [spark-pi-68732195ab217303-exec-1] is running
-  Normal  SparkExecutorCompleted     7m5s   spark-application-controller  Executor [spark-pi-68732195ab217303-exec-1] completed
-  Normal  SparkDriverCompleted       7m4s   spark-application-controller  Driver spark-pi-driver completed
+  Normal  SparkApplicationSubmitted  8m15s  spark-application-controller  SparkApplication comet-dataframe-pi was submitted successfully
+  Normal  SparkDriverRunning         7m18s  spark-application-controller  Driver comet-dataframe-pi-driver is running
+  Normal  SparkExecutorPending       7m11s  spark-application-controller  Executor [comet-dataframe-pi-68732195ab217303-exec-1] is pending
+  Normal  SparkExecutorRunning       7m10s  spark-application-controller  Executor [comet-dataframe-pi-68732195ab217303-exec-1] is running
+  Normal  SparkExecutorCompleted     7m5s   spark-application-controller  Executor [comet-dataframe-pi-68732195ab217303-exec-1] completed
+  Normal  SparkDriverCompleted       7m4s   spark-application-controller  Driver comet-dataframe-pi-driver completed
 
 ```
 
 Get Driver Logs
 
 ```bash
-kubectl logs spark-pi-driver
+kubectl logs comet-dataframe-pi-driver
 ```
 
+The driver log contains the computed approximation (`Pi is roughly ...`). To confirm that Comet
+executed the DataFrame query, inspect the executor log for the native plan emitted by

Review Comment:
   [P2] Keep the executor logs available for the documented verification step. Following the guide in order reaches `COMPLETED` before inspecting logs, but Spark defaults `spark.kubernetes.executor.deleteOnTermination` to `true`. On a cluster without centralized log collection, the executor pods and their retrievable logs are therefore gone by this step. Even after enabling native range execution, users cannot obtain the promised execution evidence. Set this option to `false` for the example or document capturing executor logs while the application runs.
   
   Evidence: Spark v4.1.3 `SparkDataFramePi.scala:40` calls `spark.stop()` after printing Pi. Kubernetes `Config.scala:695-701` defaults executor deletion to `true`. `KubernetesClusterSchedulerBackend.scala:159-161` consequently invokes `podAllocator.stop(applicationId())`, and `ExecutorPodsAllocator.scala:535-542` deletes the application's executor pods. The manifest supplies no override or log-retention mechanism. This establishes the default cleanup path before post-completion inspection. No live Kubernetes reproduction was performed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] docs: run Comet SQL in Kubernetes example [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #6949:
URL: https://github.com/apache/datafusion-comet/pull/6949


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] docs: run Comet SQL in Kubernetes example [datafusion-comet]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #6949:
URL: https://github.com/apache/datafusion-comet/pull/6949#discussion_r4265720336


##########
docs/source/user-guide/latest/kubernetes.md:
##########
@@ -61,21 +61,26 @@ REVISION: 1
 TEST SUITE: None
 ```
 
-Create example Spark application file `spark-pi.yaml`
+Create an example Spark SQL application file named `comet-sql.yaml`. The query uses a generated
+range, so it does not require an external data source. Its `EXPLAIN` output also makes it possible
+to verify that Comet is executing the query.
 
 ```bash
 apiVersion: sparkoperator.k8s.io/v1beta2
 kind: SparkApplication
 metadata:
-  name: spark-pi
+  name: comet-sql
   namespace: default
 spec:
   type: Scala
   mode: cluster
   image: apache/datafusion-comet
   imagePullPolicy: IfNotPresent
-  mainClass: org.apache.spark.examples.SparkPi
-  mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.13-4.1.3.jar
+  mainClass: org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver

Review Comment:
   [P2] Use a main class that supports cluster deployment. The manifest still sets `mode: cluster`, which the Spark Operator forwards as `--deploy-mode cluster`. Spark explicitly identifies `SparkSQLCLIDriver` as the SQL shell and rejects this combination, including when `-e` supplies a noninteractive query. Following the example therefore fails before creating the driver instead of reaching `COMPLETED` and producing the documented Comet plan/result. Could this use a small cluster-compatible SQL/DataFrame application instead?
   
   Evidence: Reproduced with stock Spark 4.1.3: `spark-submit --master k8s://https://127.0.0.1:65534 --deploy-mode cluster --class org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver local:///opt/spark/jars/spark-hive-thriftserver_2.13-4.1.3.jar -e 'EXPLAIN SELECT sum(id * id) FROM range(1000000); SELECT sum(id * id) FROM range(1000000);'` exits 1 with `Cluster deploy mode is not applicable to Spark SQL shell.` before contacting Kubernetes. Spark v4.1.3 `core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala:309-310` implements the rejection, and lines 1197-1198 identify this exact main class. The SQL succeeds separately in client mode.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] docs: run Comet SQL in Kubernetes example [datafusion-comet]

Posted by "1cbyc (via GitHub)" <gi...@apache.org>.
1cbyc commented on PR #6949:
URL: https://github.com/apache/datafusion-comet/pull/6949#issuecomment-6158715234

   Replaced the SQL CLI with Spark’s cluster-compatible bundled `SparkDataFramePi` application in signed commit e53f5e2. The example now enables native-plan logging so the executor output can confirm Comet execution. Validation passed: embedded YAML parse, Prettier 3.6.2, `git diff --check`, and verification against the Spark 4.1.3 example source.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] docs: run Comet SQL in Kubernetes example [datafusion-comet]

Posted by "1cbyc (via GitHub)" <gi...@apache.org>.
1cbyc commented on code in PR #6949:
URL: https://github.com/apache/datafusion-comet/pull/6949#discussion_r4266414398


##########
docs/source/user-guide/latest/kubernetes.md:
##########
@@ -61,21 +61,24 @@ REVISION: 1
 TEST SUITE: None
 ```
 
-Create example Spark application file `spark-pi.yaml`
+Create an example Spark SQL application file named `comet-dataframe-pi.yaml`. This application uses
+Spark's DataFrame API, so it exercises Comet without requiring an external data source.
 
 ```bash
 apiVersion: sparkoperator.k8s.io/v1beta2
 kind: SparkApplication
 metadata:
-  name: spark-pi
+  name: comet-dataframe-pi
   namespace: default
 spec:
   type: Scala
   mode: cluster
   image: apache/datafusion-comet
   imagePullPolicy: IfNotPresent
-  mainClass: org.apache.spark.examples.SparkPi
+  mainClass: org.apache.spark.examples.sql.SparkDataFramePi

Review Comment:
   Enabled `spark.comet.exec.range.enabled` in 4a1bb1c so the bundled DataFrame workload enters the native range, projection, filter, and aggregation path.



##########
docs/source/user-guide/latest/kubernetes.md:
##########
@@ -110,41 +114,45 @@ Refer to [Comet builds](#comet-docker-images)
 Run Apache Spark application with Comet enabled
 
 ```bash
-kubectl apply -f spark-pi.yaml
-sparkapplication.sparkoperator.k8s.io/spark-pi created
+kubectl apply -f comet-dataframe-pi.yaml
+sparkapplication.sparkoperator.k8s.io/comet-dataframe-pi created
 ```
 
 Check application status
 
 ```bash
-kubectl get sparkapp spark-pi
+kubectl get sparkapp comet-dataframe-pi
 
-NAME       STATUS    ATTEMPTS   START                  FINISH       AGE
-spark-pi   RUNNING   1          2025-03-18T21:19:48Z   <no value>   65s
+NAME                 STATUS      ATTEMPTS   START   FINISH   AGE
+comet-dataframe-pi   COMPLETED   1          ...     ...      ...
 ```
 
 To check more runtime details
 
 ```bash
-kubectl describe sparkapplication spark-pi
+kubectl describe sparkapplication comet-dataframe-pi
 
 ....
 Events:
   Type    Reason                     Age    From                          Message
   ----    ------                     ----   ----                          -------
-  Normal  SparkApplicationSubmitted  8m15s  spark-application-controller  SparkApplication spark-pi was submitted successfully
-  Normal  SparkDriverRunning         7m18s  spark-application-controller  Driver spark-pi-driver is running
-  Normal  SparkExecutorPending       7m11s  spark-application-controller  Executor [spark-pi-68732195ab217303-exec-1] is pending
-  Normal  SparkExecutorRunning       7m10s  spark-application-controller  Executor [spark-pi-68732195ab217303-exec-1] is running
-  Normal  SparkExecutorCompleted     7m5s   spark-application-controller  Executor [spark-pi-68732195ab217303-exec-1] completed
-  Normal  SparkDriverCompleted       7m4s   spark-application-controller  Driver spark-pi-driver completed
+  Normal  SparkApplicationSubmitted  8m15s  spark-application-controller  SparkApplication comet-dataframe-pi was submitted successfully
+  Normal  SparkDriverRunning         7m18s  spark-application-controller  Driver comet-dataframe-pi-driver is running
+  Normal  SparkExecutorPending       7m11s  spark-application-controller  Executor [comet-dataframe-pi-68732195ab217303-exec-1] is pending
+  Normal  SparkExecutorRunning       7m10s  spark-application-controller  Executor [comet-dataframe-pi-68732195ab217303-exec-1] is running
+  Normal  SparkExecutorCompleted     7m5s   spark-application-controller  Executor [comet-dataframe-pi-68732195ab217303-exec-1] completed
+  Normal  SparkDriverCompleted       7m4s   spark-application-controller  Driver comet-dataframe-pi-driver completed
 
 ```
 
 Get Driver Logs
 
 ```bash
-kubectl logs spark-pi-driver
+kubectl logs comet-dataframe-pi-driver
 ```
 
+The driver log contains the computed approximation (`Pi is roughly ...`). To confirm that Comet
+executed the DataFrame query, inspect the executor log for the native plan emitted by

Review Comment:
   Set `spark.kubernetes.executor.deleteOnTermination` to `false` in 4a1bb1c, documented why the executor is retained, and added the cleanup command.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org