You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "zhengruifeng (via GitHub)" <gi...@apache.org> on 2026/04/24 08:02:07 UTC
[PR] [SQL] Add config for strict DataFrame column resolution [spark]
zhengruifeng opened a new pull request, #55531:
URL: https://github.com/apache/spark/pull/55531
### What changes were proposed in this pull request?
Add a new internal SQL config `spark.sql.analyzer.strictDataFrameColumnResolution` (default `false`) that controls how `UnresolvedAttribute`s carrying a `PLAN_ID_TAG` (DataFrame columns coming through Spark Connect) are resolved in `ColumnResolutionHelper`:
- When `true`:
- In `resolveDataFrameColumnRecursively`, if the target plan node is found but the column cannot be resolved on it, fail immediately with `CANNOT_RESOLVE_DATAFRAME_COLUMN` instead of delaying the failure for a later analyzer iteration.
- In the main `resolveExpression` path, allow name-based resolution as a fallback for tagged attributes as well.
- When `false` (default): preserve the current behavior, i.e. delay the failure so that later analyzer iterations / rules still have a chance to resolve the column, and keep skipping name-based resolution for tagged attributes.
### Why are the changes needed?
Give users an opt-in mode that surfaces DataFrame column resolution failures earlier, matching the semantics used internally in Databricks Runtime (databricks-eng/runtime@4452ee5ecb96e2bdc6184c867d4cbada6f2c13fe). The default preserves existing Apache Spark behavior so there is no user-facing change unless the flag is flipped.
### Does this PR introduce _any_ user-facing change?
No behavior change by default. A new internal config is added.
### How was this patch tested?
- New unit test in `SparkSessionExtensionSuite` that enables the config and verifies `CANNOT_RESOLVE_DATAFRAME_COLUMN` is raised immediately, alongside the existing "inject analyzer rule - hidden column" test which still exercises the default delayed-failure path.
- `build/sbt 'catalyst/testOnly *AnalysisSuite'`
- `build/sbt 'sql/testOnly *SparkSessionExtensionSuite *ColumnNodeToExpressionConverterSuite *ResolverGuardSuite *PlanResolutionSuite'`
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Anthropic), claude-opus-4-7
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56614][SQL][CONNECT] Add config for strict DataFrame column resolution [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #55531: [SPARK-56614][SQL][CONNECT] Add config for strict DataFrame column resolution
URL: https://github.com/apache/spark/pull/55531
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56614][SQL][CONNECT] Add config for strict DataFrame column resolution [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #55531:
URL: https://github.com/apache/spark/pull/55531#issuecomment-4323239789
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org