You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "dongjoon-hyun (via GitHub)" <gi...@apache.org> on 2026/10/23 00:48:24 UTC

[PR] [SPARK-60494][CONNECT][SDP] Defer SQL with DataFrame arguments in a pipeline flow to flow analysis [spark]

dongjoon-hyun opened a new pull request, #59735:
URL: https://github.com/apache/spark/pull/59735

   ### What changes were proposed in this pull request?
   
   When `PipelinesHandler` handles a `DefineFlow` command, it transforms the relation of the flow with `SparkConnectPlanner.transformRelation`. `spark.sql(query, df=...)` with DataFrame arguments sends a SQL-with-references `WITH_RELATIONS` relation, and `transformRelation` handles it with `executeSQLWithRefs`. That method analyzes each reference, registers it as a temporary view, and runs the root SQL with `session.sql`, all at flow registration.
   
   This PR adds `SparkConnectPlanner.transformPipelineFlowRelation` and passes it to `PipelinesHandler` instead of `transformRelation`. While it transforms the relation of a flow, `transformSqlWithRefs` does not execute a valid SQL-with-references `WITH_RELATIONS`. It parses the root SQL and binds each reference to its alias as a CTE relation (`UnresolvedWith`). `FlowAnalysis` already substitutes CTEs before it resolves `UnresolvedRelation`s as pipeline inputs, so it handles a dataset that a DataFrame argument reads in the same way as a dataset in the SQL text.
   
   `handleSqlCommand` and all other Spark Connect requests still use `executeSQLWithRefs`, so `spark.sql` with DataFrame arguments outside of a flow definition is unchanged.
   
   ### Why are the changes needed?
   
   Inside a flow function, `spark.sql` does not run the SQL. The server returns the relation as is (SPARK-54452), so that the flow analysis can resolve it later. With DataFrame arguments, however, `DefineFlow` analyzes it right away:
   
   1. The flow fails to register when a DataFrame argument reads a dataset of the same pipeline that does not exist in the catalog: a table or materialized view on the first run, or a temporary view on every run.
   
       ```python
       @dp.materialized_view
       def upstream():
           return spark.range(5)
   
       @dp.materialized_view
       def downstream():
           return spark.sql("SELECT * FROM {df}", df=spark.read.table("upstream"))
       ```
   
       ```
       pyspark.errors.exceptions.connect.AnalysisException: [TABLE_OR_VIEW_NOT_FOUND] The table or view `upstream` cannot be found. ...
       ```
   
   2. When the table already exists, for example from a previous run, the flow registers an analyzed plan. The flow analysis then does not see the read of `upstream` and does not record it as an input of `downstream`. Without the dependency, `downstream` can run before `upstream` is refreshed and read stale data, without an error.
   
   3. A command at the root that the SDP guard allows, such as `SHOW TABLES`, is executed at flow registration.
   
   The same query without DataFrame arguments, `spark.sql("SELECT * FROM upstream")`, works because its relation is only parsed at flow registration.
   
   In Spark 4.1.0 and 4.2.0, the SDP guard in `handleSqlCommand` also analyzed the relation, so the error in (1) was raised at the `spark.sql` call in the flow function. SPARK-57189 (#59425) moved it to flow registration.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes. In a flow function, `spark.sql` with DataFrame arguments can read datasets of the same pipeline, and the flow analysis records them as inputs of the flow. The SQL is analyzed during graph resolution instead of flow registration, so an analysis error is reported as a flow resolution failure, as for `spark.sql` without DataFrame arguments.
   
   In a flow definition, DataFrame arguments are now bound as CTE relations instead of temporary views. A statement that refers to a DataFrame argument other than as a table in a query, such as `DESCRIBE TABLE {df}`, can no longer resolve it when a flow function returns the statement.
   
   ### How was this patch tested?
   
   Added two tests to `PythonPipelineSuite`. Both fail without this fix.
   
   - `SPARK-60494: referencing internal datasets through DataFrame arguments of spark.sql`: a new pipeline reads a materialized view with batch and streaming reads, and a temporary view with a batch read, all through DataFrame arguments. The test checks the inputs and streaming-ness of the flows, runs the pipeline, and checks the results. Without the fix, registration of the first flow fails with `TABLE_OR_VIEW_NOT_FOUND`.
   - `SPARK-60494: referencing an internal dataset through DataFrame arguments of spark.sql tracks the dependency when its table already exists`: Without the fix, the inputs of the flow are empty.
   
   Also passed the existing suites in `org.apache.spark.sql.connect.pipelines`, including `PythonPipelineSuite` and `SparkDeclarativePipelinesServerSuite`, as well as `SparkConnectPlannerSuite` and `SparkConnectProtoSuite`:
   
   ```
   build/sbt 'connect/testOnly org.apache.spark.sql.connect.pipelines.* *SparkConnectPlannerSuite *SparkConnectProtoSuite'
   ```
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: Claude Opus 5.5


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-60494][CONNECT][SDP] Defer SQL with DataFrame arguments in a pipeline flow to flow analysis [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #59735:
URL: https://github.com/apache/spark/pull/59735#issuecomment-6366173116

   Thank you for checking the guard path. I'd like to keep `blockUnsupportedSqlCommandIn` unchanged in this PR. Its eager fallback is the known limitation that SPARK-57189 documented, and this PR only changes how `DefineFlow` transforms the relation of a flow. I added the shape to the PR description as a remaining limitation: if a flow function passes a further-transformed `spark.sql(..., df=...)` result as a DataFrame argument of another `spark.sql` call, that call still fails on the first run.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-60494][CONNECT][SDP] Defer SQL with DataFrame arguments in a pipeline flow to flow analysis [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #59735:
URL: https://github.com/apache/spark/pull/59735#issuecomment-6404225700

   **Merge Summary:**
   - merged into master https://github.com/apache/spark/commit/f86fc93cc4cb644f020ef8e78c31e70b052b8e1b
   - merged into branch-4.x https://github.com/apache/spark/commit/4ecc99ec857cc9c27c3a0c36d108c21570497c25
   
   *Posted by `merge_spark_pr.py`*


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-60494][CONNECT][SDP] Defer SQL with DataFrame arguments in a pipeline flow to flow analysis [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #59735:
URL: https://github.com/apache/spark/pull/59735#discussion_r4360117405


##########
sql/connect/server/src/test/scala/org/apache/spark/sql/connect/pipelines/PythonPipelineSuite.scala:
##########
@@ -404,6 +404,65 @@ class PythonPipelineSuite
         Set(graphIdentifier("b"), graphIdentifier("d")))
   }
 
+  test("SPARK-60494: referencing internal datasets through DataFrame arguments of spark.sql") {

Review Comment:
   Thank you. I added both.
   
   - `referencing internal datasets failed` now has a flow that reads `src` through a DataFrame argument. It fails graph resolution with `TABLE_OR_VIEW_NOT_FOUND` for `` `spark_catalog`.`default`.`src` ``, not for the generated alias.
   - The positive test now has `d`, which applies `where("id > 0")` to the result, and `e`, which passes an untransformed result as a DataFrame argument of another `spark.sql` call. On a new pipeline, both list `src` among their inputs and produce the expected rows.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-60494][CONNECT][SDP] Defer SQL with DataFrame arguments in a pipeline flow to flow analysis [spark]

Posted by "spark-test (via GitHub)" <gi...@apache.org>.
spark-test commented on code in PR #59735:
URL: https://github.com/apache/spark/pull/59735#discussion_r4355439633


##########
sql/connect/server/src/test/scala/org/apache/spark/sql/connect/pipelines/PythonPipelineSuite.scala:
##########
@@ -404,6 +404,65 @@ class PythonPipelineSuite
         Set(graphIdentifier("b"), graphIdentifier("d")))
   }
 
+  test("SPARK-60494: referencing internal datasets through DataFrame arguments of spark.sql") {

Review Comment:
   **Nit (P3):** These tests return the `spark.sql(..., df=...)` result directly, with one direct read per argument and datasets that resolve. That leaves two behaviors this PR introduces without a failing signal:
   
   - An analysis error in a DataFrame argument is now reported as a flow resolution failure. `referencing internal datasets failed` covers `read.table`, `table`, `readStream.table`, `spark.sql`, and `STREAM`, but not the DataFrame-argument form. If that failure were lost, or reported against the generated `_pyspark_connect_temp_view_<uuid>` alias instead of `src`, no assertion would change.
   - Deferral also applies to SQL-with-references below the flow's root relation, because `deferSqlWithRefs` stays set for the whole `transformRelation` call. Two flows depend on that: one that returns `spark.sql("SELECT * FROM {df}", df=spark.read.table("src")).where("id > 0")`, and one that passes an untransformed `spark.sql(..., df=...)` result as a DataFrame argument of another `spark.sql` call. If deferral regressed to the root relation only, the first run would fail at `DefineFlow` with `TABLE_OR_VIEW_NOT_FOUND`, and a re-run would silently drop the dependency on `src`. Both new tests would still pass.
   
   To close both gaps, add a DataFrame-argument flow to `referencing internal datasets failed` and add those two shapes to the new positive test. This leaves out a flow that passes a further-transformed result into another `spark.sql` call, because that shape still fails at the flow-function SQL-command guard. See the decision challenge below.
   
   **Verification:**
   - **Behavior:** On a fresh pipeline, a flow that reads an undefined dataset through a spark.sql DataFrame argument is reported as a flow resolution failure whose message is TABLE_OR_VIEW_NOT_FOUND for the user's dataset name, not for the generated DataFrame-argument alias.
   - **Regression:** On a fresh pipeline, a flow that applies a further DataFrame operation to a spark.sql-with-DataFrame-arguments result before returning it, and a flow that passes an untransformed spark.sql-with-DataFrame-arguments result as a DataFrame argument of another spark.sql call, register successfully, list the upstream pipeline dataset among their inputs, and produce the expected rows. These assertions must fail if deferral applied only to a top-level SQL-with-references relation.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-60494][CONNECT][SDP] Defer SQL with DataFrame arguments in a pipeline flow to flow analysis [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #59735: [SPARK-60494][CONNECT][SDP] Defer SQL with DataFrame arguments in a pipeline flow to flow analysis
URL: https://github.com/apache/spark/pull/59735


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-60494][CONNECT][SDP] Defer SQL with DataFrame arguments in a pipeline flow to flow analysis [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #59735:
URL: https://github.com/apache/spark/pull/59735#issuecomment-6404199823

   Thank you, @HyukjinKwon !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org