You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "shrirangmhalgi (via GitHub)" <gi...@apache.org> on 2026/06/17 21:56:16 UTC
[PR] [SPARK-57352][SDP] Detect resolved table references in pipeline flow lineage [spark]
shrirangmhalgi opened a new pull request, #56578:
URL: https://github.com/apache/spark/pull/56578
### What changes were proposed in this pull request?
After `FlowAnalysis.analyze` resolves the plan via `transformWithSubqueries`, scan for `LogicalRelation` / `DataSourceV2Relation` / `HiveTableRelation` nodes whose table name matches a known pipeline input. Record these as `externalInputs` so the DAG scheduler correctly orders execution.
### Why are the changes needed?
`FlowAnalysis.analyze` only intercepts `UnresolvedRelation` nodes when building the pipeline dependency graph. If a flow's plan contains a pre-resolved table reference (e.g., from `spark.table()` in classic mode or a plan that was analyzed before reaching `FlowAnalysis`), the dependency is invisible to the scheduler. This causes downstream flows to execute before their upstream dependencies, resulting in either `TABLE_OR_VIEW_NOT_FOUND` errors or stale data reads.
### Does this PR introduce _any_ user-facing change?
Yes. Pipeline flows that reference upstream datasets via `spark.table()` now correctly infer the dependency, ensuring correct execution order.
### How was this patch tested?
New SparkTableLineageSuite with two tests:
- Resolved table reference (LogicalRelation) detected as pipeline input (was invisible before fix)
- Control test confirming readFlowFunc (the UnresolvedRelation path) continues to work
- Full pipeline test suite passes (378 tests, 0 failures).
### Was this patch authored or co-authored using generative AI tooling?
Yes. Authored using Claude Opus 4.6.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57352][SDP] Detect resolved table references in pipeline flow lineage [spark]
Posted by "shrirangmhalgi (via GitHub)" <gi...@apache.org>.
shrirangmhalgi commented on PR #56578:
URL: https://github.com/apache/spark/pull/56578#issuecomment-4754733302
@szehon-ho / @hvanhovell - Could you please review when you get a chance? This fixes a user-reported pipeline execution ordering issue (SPARK-57352) by detecting resolved table references in flow lineage. The key design decision is including requestedInputs in FlowFunctionResult.inputs so the DAG scheduler sees the dependency.
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57352][SDP] Detect resolved table references in pipeline flow lineage [spark]
Posted by "yadavay-amzn (via GitHub)" <gi...@apache.org>.
yadavay-amzn commented on code in PR #56578:
URL: https://github.com/apache/spark/pull/56578#discussion_r3432297708
##########
sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/FlowAnalysis.scala:
##########
@@ -127,7 +127,40 @@ object FlowAnalysis {
resolved.mergeTagsFrom(u)
resolved
}
- Dataset.ofRows(spark, resolvedPlan)
+ val result = Dataset.ofRows(spark, resolvedPlan)
+
+ // SPARK-57352: Detect inputs that were resolved directly by the user (e.g., via
+ // spark.table()) rather than through the pipeline's UnresolvedRelation path above.
+ // Scan the resolved plan for table relations that match known pipeline inputs and
+ // record them as external inputs so the DAG scheduler orders them correctly.
+ resolvedPlan.foreach {
+ case r: org.apache.spark.sql.catalyst.analysis.UnresolvedRelation =>
+ // Already handled above
+ case node =>
+ val tableIdOpt = node match {
+ case r: org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation =>
+ r.identifier.map(id => TableIdentifier(id.name(),
+ Option(id.namespace()).filter(_.nonEmpty).map(_.last)))
+ case r: org.apache.spark.sql.catalyst.catalog.HiveTableRelation =>
+ Some(r.tableMeta.identifier)
+ case r: org.apache.spark.sql.execution.datasources.LogicalRelation =>
+ r.catalogTable.map(_.identifier)
+ case _ => None
+ }
+ tableIdOpt.foreach { tableId =>
+ // Match by table name, allowing for different catalog/database qualifications
+ val matchesInput = context.allInputs.exists(input =>
+ input.table == tableId.table
+ )
+ if (matchesInput &&
+ !context.batchInputs.exists(_.input.identifier.table == tableId.table) &&
+ !context.streamingInputs.exists(_.input.identifier.table == tableId.table)) {
+ context.externalInputs += tableId
+ }
+ }
+ }
+
+ result
Review Comment:
I think this records the dependency in a field the scheduler doesn't read for ordering, so the matched table may not end up affecting execution order. Tracing it through: `context.externalInputs` becomes `FlowFunctionResult.usedExternalInputs`, but the DAG dependency set is `ResolvedFlow.inputs`, which (in Flow.scala) is `(batchInputs ++ streamingInputs).map(_.input.identifier)` — `usedExternalInputs` isn't included. The consumers that order/validate the graph all read `.inputs`: `GraphOperations.upstreamFlows`, `GraphValidations` (topo order), `DatasetManager` (materialization deps), and `CoreDataflowNodeProcessor`.
`externalInputs` also semantically means "tables outside the pipeline" — it's only written today by `readExternalBatchInput`/`readExternalStreamInput`, which are precisely the non-pipeline paths. So when the matched relation *is* a pipeline dataset, I think we'd want it to flow through the internal-input path (landing in `batchInputs`/`streamingInputs`, which is what `.inputs` and the DAG consume) rather than `externalInputs`. Otherwise the ordering the PR is targeting may not change. Could you double-check this against how you saw the scheduler pick it up? Happy to pair on it.
##########
sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/FlowAnalysis.scala:
##########
@@ -127,7 +127,40 @@ object FlowAnalysis {
resolved.mergeTagsFrom(u)
resolved
}
- Dataset.ofRows(spark, resolvedPlan)
+ val result = Dataset.ofRows(spark, resolvedPlan)
+
+ // SPARK-57352: Detect inputs that were resolved directly by the user (e.g., via
+ // spark.table()) rather than through the pipeline's UnresolvedRelation path above.
+ // Scan the resolved plan for table relations that match known pipeline inputs and
+ // record them as external inputs so the DAG scheduler orders them correctly.
+ resolvedPlan.foreach {
+ case r: org.apache.spark.sql.catalyst.analysis.UnresolvedRelation =>
+ // Already handled above
+ case node =>
+ val tableIdOpt = node match {
+ case r: org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation =>
+ r.identifier.map(id => TableIdentifier(id.name(),
+ Option(id.namespace()).filter(_.nonEmpty).map(_.last)))
+ case r: org.apache.spark.sql.catalyst.catalog.HiveTableRelation =>
+ Some(r.tableMeta.identifier)
+ case r: org.apache.spark.sql.execution.datasources.LogicalRelation =>
+ r.catalogTable.map(_.identifier)
+ case _ => None
+ }
+ tableIdOpt.foreach { tableId =>
+ // Match by table name, allowing for different catalog/database qualifications
+ val matchesInput = context.allInputs.exists(input =>
+ input.table == tableId.table
+ )
+ if (matchesInput &&
+ !context.batchInputs.exists(_.input.identifier.table == tableId.table) &&
+ !context.streamingInputs.exists(_.input.identifier.table == tableId.table)) {
+ context.externalInputs += tableId
+ }
+ }
Review Comment:
The match is on `tableId.table` only (unqualified name), so a genuinely-external table that happens to share an unqualified name with a pipeline dataset in a different catalog/database would also match here and get recorded. The comment just above says "allowing for different catalog/database qualifications" — in a multi-catalog pipeline, name-only matching could introduce a dependency on the wrong table. Would it be worth qualifying the comparison by catalog + database (the `TableIdentifier` carries them) so only the intended dataset matches?
##########
sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/FlowAnalysis.scala:
##########
@@ -127,7 +127,40 @@ object FlowAnalysis {
resolved.mergeTagsFrom(u)
resolved
}
- Dataset.ofRows(spark, resolvedPlan)
+ val result = Dataset.ofRows(spark, resolvedPlan)
+
+ // SPARK-57352: Detect inputs that were resolved directly by the user (e.g., via
+ // spark.table()) rather than through the pipeline's UnresolvedRelation path above.
+ // Scan the resolved plan for table relations that match known pipeline inputs and
+ // record them as external inputs so the DAG scheduler orders them correctly.
+ resolvedPlan.foreach {
+ case r: org.apache.spark.sql.catalyst.analysis.UnresolvedRelation =>
+ // Already handled above
+ case node =>
+ val tableIdOpt = node match {
+ case r: org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation =>
+ r.identifier.map(id => TableIdentifier(id.name(),
+ Option(id.namespace()).filter(_.nonEmpty).map(_.last)))
+ case r: org.apache.spark.sql.catalyst.catalog.HiveTableRelation =>
+ Some(r.tableMeta.identifier)
+ case r: org.apache.spark.sql.execution.datasources.LogicalRelation =>
+ r.catalogTable.map(_.identifier)
Review Comment:
`Option(id.namespace()).filter(_.nonEmpty).map(_.last)` takes only the last namespace element as the database and drops the catalog (and any earlier namespace levels). For a multi-level / 3-part identifier this loses the catalog, which feeds into the name-only matching concern below. If we end up qualifying the match by catalog+db, we'd want to preserve the full namespace here too.
##########
sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/FlowAnalysis.scala:
##########
@@ -127,7 +127,40 @@ object FlowAnalysis {
resolved.mergeTagsFrom(u)
resolved
}
- Dataset.ofRows(spark, resolvedPlan)
+ val result = Dataset.ofRows(spark, resolvedPlan)
+
+ // SPARK-57352: Detect inputs that were resolved directly by the user (e.g., via
+ // spark.table()) rather than through the pipeline's UnresolvedRelation path above.
+ // Scan the resolved plan for table relations that match known pipeline inputs and
+ // record them as external inputs so the DAG scheduler orders them correctly.
+ resolvedPlan.foreach {
+ case r: org.apache.spark.sql.catalyst.analysis.UnresolvedRelation =>
+ // Already handled above
+ case node =>
Review Comment:
A few smaller things on this detection block:
- `resolvedPlan.foreach` doesn't descend into subquery expressions, whereas the resolver above uses `transformWithSubqueries`. A resolved table reference inside a subquery (e.g. `WHERE x IN (SELECT ... FROM pipeline_tbl)`) would be missed. If the goal is parity with the `UnresolvedRelation` path, a subquery-aware traversal would match it.
- The relation types are referenced by fully-qualified name inline (`org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation`, etc.) — could add imports to match the file's style.
- The `case r: UnresolvedRelation => // Already handled` arm: by this point the plan is resolved, so the pipeline's `UnresolvedRelation`s are already gone — this arm looks like it never fires.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57352][SDP] Detect resolved table references in pipeline flow lineage [spark]
Posted by "AnishMahto (via GitHub)" <gi...@apache.org>.
AnishMahto commented on PR #56578:
URL: https://github.com/apache/spark/pull/56578#issuecomment-4863168380
Something else always worth considering is the "do nothing" option.
Or more specifically, explicitly disallow `.load(<identifier>)` reads within a dataflow graph. We can throw an exception that provides an actionabe alternative - just use `spark.table(<identifier>)` instead.
I can't think of a meaningful case today where a user would specifically need to do a `.load(<identifier>)` over `.table(<identifier>)`, so I think this is actually a reasonable option.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57352][SDP] Detect resolved table references in pipeline flow lineage [spark]
Posted by "AnishMahto (via GitHub)" <gi...@apache.org>.
AnishMahto commented on code in PR #56578:
URL: https://github.com/apache/spark/pull/56578#discussion_r3465474815
##########
sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/FlowAnalysis.scala:
##########
@@ -127,7 +130,56 @@ object FlowAnalysis {
resolved.mergeTagsFrom(u)
resolved
}
- Dataset.ofRows(spark, resolvedPlan)
+ val result = Dataset.ofRows(spark, resolvedPlan)
+
+ // SPARK-57352: Detect inputs that were resolved directly by the user (e.g., via
+ // spark.table()) rather than through the pipeline's UnresolvedRelation path above.
+ // Scan the resolved plan (including subquery expressions) for table relations whose
+ // fully-qualified identifier matches a known pipeline input. Record them in
+ // requestedInputs so the DAG scheduler orders execution correctly.
+ def scanForResolvedInputs(plan: LogicalPlan): Unit = {
+ plan.foreach { node =>
+ val tableIdOpt: Option[TableIdentifier] = node match {
+ case r: DataSourceV2Relation =>
+ r.identifier.map { id =>
+ val ns = Option(id.namespace()).getOrElse(Array.empty[String])
+ TableIdentifier(
+ table = id.name(),
+ database = if (ns.nonEmpty) Some(ns.last) else None,
+ catalog = if (ns.length > 1) Some(ns.head) else None)
+ }
+ case r: HiveTableRelation =>
+ Some(r.tableMeta.identifier)
+ case r: LogicalRelation =>
+ r.catalogTable.map(_.identifier)
+ case _ => None
Review Comment:
I'm a little worried this approach might be brittle. We're assuming that all resolved data sources materialize as one of `DataSourceV2Relation`, `HiveTableRelation`, or `LogicalRelation`.
This may or may not be true today (ex. if we're load from a cached dataframe, I think it can resolve as an `InMemoryRelation`), but I'm more concerned about if in the future new resolved relation types are introduced - this match list will become difficult to maintain.
Another high level concern is we're taking a bit of a round about approach here - we let data sources fully resolve in logical plans before mapping them back to table read/loads.
I think we can do something more direct where we directly intercept identifiers from SC Read protos and load them in the dataflow graph, rather than try to parse out those same identifiers from the analyzed logical plan. Giving my proposal in my other review comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57352][SDP] Detect resolved table references in pipeline flow lineage [spark]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on PR #56578:
URL: https://github.com/apache/spark/pull/56578#issuecomment-4784105421
@anishm-db @anew fyi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57352][SDP] Detect resolved table references in pipeline flow lineage [spark]
Posted by "AnishMahto (via GitHub)" <gi...@apache.org>.
AnishMahto commented on PR #56578:
URL: https://github.com/apache/spark/pull/56578#issuecomment-4863130626
That's a good question, the biggest difficulty with (b) would be how do you reliably identify whether a path is really just an identifier or actually a file path, so I would probably prefer (a).
But also, let me back up for a second. There's two orthogonal problems I'm realizing we need to solve:
1. For identifier based reads using `.load()`, we need to solve the eager analysis problem + we need to register the input in the dataflow graph, which could be either an internal or external read.
2. For path based reads using `.load()`, eager analysis doesn't matter. Path based reads must sldo always be to tables external to the pipeline, which we maybe we should track within the dataflow graph just for consistency reasons, but it doesn't actually affect topological execution order.
Anyway I think to solve both problems (delay eager analysis and register identifier reads in the dataflow graph), we'll have to:
1. Delay analysis for all `.load()` branches in `SparkConnectPlanner` when inside a pipelines context. By delaying analysis, an unanalyzed `spark.load()` should always produce an `UnresolvedDataSource` logical plan. Notably still not an `UnresolvedRelation`, since `UnresolvedRelation` doesn't represent things like `format` that `UnresolvedDataSource` does.
2. In SDP flow analysis, in addition to inspecting `UnresolvedRelation` for identifiers and registering them in the dataflow graph, we'll also have to do the same for `UnresolvedDataSource`. This is actually similar to what you're already doing, but we'll be extracting identifiers from the canonical unresolved logical plan for data sources, rather than the analyzed plan.
For step (2) what's not 100% clear to me yet and will require further thought is whether we can really use `readStreamInput`/`readBatchInput` as the drop in substitution for `UnresolvedDataSource` nodes, the way that we do for `UnresolvedRelation`.
Tangentially in reading the existing `readStreamInput`/`readBatchInput` paths for `UnresolvedRelation`, I'm realizing there might be a bug in respecting read options on internal dataset reads. But that's unrelated to this discussion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57352][SDP] Detect resolved table references in pipeline flow lineage [spark]
Posted by "shrirangmhalgi (via GitHub)" <gi...@apache.org>.
shrirangmhalgi commented on PR #56578:
URL: https://github.com/apache/spark/pull/56578#issuecomment-4754722860
Thanks @yadavay-amzn for the thorough review! All four points addressed in the latest commit:
- **Scheduler visibility**: Changed from `externalInputs` to `requestedInputs`, and included `requestedInputs` in `FlowFunctionResult.inputs` - the DAG scheduler now sees the dependency.
- **Qualification**: Match requires both table name AND database (when both are specified) to avoid multi-catalog false positives.
- **Subquery traversal**: Recursive `scanForResolvedInputs` descends into `SubqueryExpression` nodes for parity with `transformWithSubqueries`.
- **Namespace**: Full namespace preserved — catalog from first element, database from last. Inline FQN references replaced with proper imports.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57352][SDP] Detect resolved table references in pipeline flow lineage [spark]
Posted by "shrirangmhalgi (via GitHub)" <gi...@apache.org>.
shrirangmhalgi commented on PR #56578:
URL: https://github.com/apache/spark/pull/56578#issuecomment-4856784201
@AnishMahto Thanks for the detailed analysis and apologies for the late reply. - I agree with your approach. I traced through the code paths and your diagnosis is correct.
### What I verified:
1. `spark.table("x")` **is NOT the issue** - it produces a `NamedTable` Read proto → `transformReadRel` (line 1663 returns `UnresolvedRelation(multipartIdentifier`) → FlowAnalysis already intercepts this. Confirmed the `JIRA's spark.table()` repro claim is inaccurate.
2. `spark.read.format(...).load(identifier)` **IS the issue** - it produces a `DATA_SOURCE` Read proto → `transformReadRel` line 1690 calls `reader.load().queryExecution.analyzed` → plan arrives fully resolved → `FlowAnalysis.analyze` never sees an `UnresolvedRelation` → dependency invisible to the DAG scheduler.
3. **Existing pipeline infrastructure confirms your design direction:**
- `PipelineAnalysisContextUtils.hasPipelineAnalysisContext(userContext)` is already used at line 3028 to block unsupported SQL commands in pipeline context
- `isInsidePipelineFlowFunction(userContext)` at line 3034 defers `spark.sql()` analysis - same pattern we'd apply to `DATA_SOURCE`
- `SparkConnectPlanExecution` line 73 already throws `ATTEMPT_ANALYSIS_IN_PIPELINE_QUERY_FUNCTION` for unsupported eager execution inside flows
- `userContext` is accessible in `transformReadRel` via `executeHolderOpt.get.request.getUserContext`
4. **Why the planner-level approach is better:**
- **Complete** - intercepts at the source rather than pattern-matching a subset of resolved relation types
- **Prevents the eager analysis failure** - if the upstream table doesn't exist yet, `reader.load()` throws `TABLE_OR_VIEW_NOT_FOUND` before we can even track the dependency; deferring analysis avoids this entirely
- **Single interception point** - no maintenance burden from new relation types
### Design question before I implement:
The `DATA_SOURCE` proto's `paths` field can be either a table identifier (e.g., `catalog.db.table`) or a file path (e.g., `/data/parquet/`). When `isInsidePipelineFlowFunction` is true, should I:
(a) Return an `UnresolvedRelation(paths)` unconditionally and let `FlowAnalysis` determine whether it matches a known pipeline dataset (non-matching identifiers would fail at analysis time as usual), or
(b) Only defer analysis for cases where the path looks like a multipart identifier (no `/`, no scheme like `s3://`), and eagerly analyze file-path loads as before?
I'm leaning toward (a) since `FlowAnalysis` already handles the "not a pipeline dataset" case by letting it through for normal resolution. But wanted your input before proceeding.
Happy to rework on the PR with the `SparkConnectPlanner`-level fix once we confirm the direction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org