You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "zhengruifeng (via GitHub)" <gi...@apache.org> on 2026/05/13 09:12:51 UTC
[PR] [DOCS][CONNECT] Document DataFrame column resolution behavior in spark-connect-gotchas [spark]
zhengruifeng opened a new pull request, #55848:
URL: https://github.com/apache/spark/pull/55848
### What changes were proposed in this pull request?
Add a new gotcha section to `docs/spark-connect-gotchas.md` describing how Spark Connect resolves DataFrame column references (`df["col"]`) via plan-id tagging, and how this diverges from Spark Classic once a column has been shadowed by `withColumn` or `select`.
The section covers:
- Why `df.withColumn("col", ...).select(df["col"])` fails on both Spark Classic (`MISSING_ATTRIBUTES.RESOLVED_ATTRIBUTE_APPEAR_IN_OPERATION`) and Spark Connect (`CANNOT_RESOLVE_DATAFRAME_COLUMN`).
- Why users may have observed this query succeeding on older Spark Connect builds (lenient name-based fallback when plan-id resolution does not match a tagged ancestor).
- The recommended fix: use an untagged `F.col("col")` reference after column shadowing.
- The opt-in escape hatch: `spark.sql.analyzer.strictDataFrameColumnResolution=false` (introduced in SPARK-56614 / apache/spark#55531) to re-enable the lenient fallback.
Also adds a "DataFrame column references" row to the summary table at the end of the document.
### Why are the changes needed?
The plan-id-based column resolution path is a Spark Connect-specific contract that is not documented anywhere user-facing. Users migrating workloads to Spark Connect have encountered surprises when patterns that previously "worked" stop resolving, with an error class (`CANNOT_RESOLVE_DATAFRAME_COLUMN`) and a config (`strictDataFrameColumnResolution`) whose connection to their code is not obvious. This adds explicit guidance and a code-level mitigation alongside the other Connect-vs-Classic gotchas already documented in this file.
### Does this PR introduce _any_ user-facing change?
No. Documentation-only change.
### How was this patch tested?
Documentation-only change; no automated tests. Verified the markdown renders correctly and is consistent with the existing four-gotcha layout in `docs/spark-connect-gotchas.md`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Anthropic), claude-opus-4-7
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56846][DOCS][CONNECT] Document DataFrame column resolution behavior in spark-connect-gotchas [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #55848:
URL: https://github.com/apache/spark/pull/55848#discussion_r3241530942
##########
docs/spark-connect-gotchas.md:
##########
@@ -418,6 +418,46 @@ println(structColumnFields)
This approach is significantly faster when dealing with a large number of columns because it avoids creating and analyzing numerous DataFrames.
+## 5. DataFrame column references after column shadowing
+
+In Spark Connect, a DataFrame column reference such as `df["c"]` is tagged with the plan id of `df`. At analysis time the server resolves the reference by looking for the tagged ancestor in the plan and pulling the matching attribute from it. Spark Classic does not use plan ids; it resolves column references against the immediate child's output by attribute id and name.
+
+The two resolution strategies diverge once a column has been shadowed by another operator that produces an attribute with the same name:
+
+```python
+import pyspark.sql.functions as sf
+
+df = spark.sql("SELECT 1 AS c")
+df.withColumn("c", sf.col("c").cast("string")).select(df["c"]).collect()
+```
+
+`withColumn("c", ...)` does not mutate `df`; it returns a new DataFrame whose `c` is a new attribute that hides the original. The trailing `df["c"]` still refers to the *original* `c` attribute, which is no longer in the projection list.
+
+* **Spark Classic** has always rejected this query at analysis time with `MISSING_ATTRIBUTES.RESOLVED_ATTRIBUTE_APPEAR_IN_OPERATION`, because the original attribute is not present in the operator's child output.
+* **Spark Connect** rejects it with `CANNOT_RESOLVE_DATAFRAME_COLUMN` by default. The plan-id-tagged reference does not match any attribute in the current plan. But when the SQL config `spark.sql.analyzer.strictDataFrameColumnResolution` (added in Spark 4.2.0, default `true`) is set to `false`, the analyzer still tries plan-id-based resolution first, and only when that fails does it fall back to name-based resolution: the tagged `df["c"]` is then resolved by name against the projected `c` from `withColumn`, and the query succeeds.
+
+### Recommended way
+
+If you hit any of the confusing failures mentioned above, it is recommended to switch to `sf.col` first. `sf.col("c")` is an untagged name reference that resolves against the most recent projection or `withColumn`, rather than `df["c"]` which is a tagged reference to `df`'s original column:
Review Comment:
`sf.col` is confusing here as we didn't mention `import pyspark.sql.functions as sf` before
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56846][DOCS][CONNECT] Document DataFrame column resolution behavior in spark-connect-gotchas [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #55848:
URL: https://github.com/apache/spark/pull/55848#discussion_r3255908276
##########
docs/spark-connect-gotchas.md:
##########
@@ -73,7 +73,7 @@ Unlike query execution, Spark Classic and Spark Connect differ in when schema an
# Common Gotchas (with Mitigations)
-If you are not careful about the difference between lazy vs. eager analysis, there are four key gotchas to be aware of: 1) overwriting temporary view names, 2) capturing external variables in UDFs, 3) delayed error detection, and 4) excessive schema access on new DataFrames.
+If you are not careful about the difference between lazy vs. eager analysis, there are five key gotchas to be aware of: 1) overwriting temporary view names, 2) capturing external variables in UDFs, 3) delayed error detection, 4) excessive schema access on new DataFrames, and 5) DataFrame column references after a column is shadowed.
Review Comment:
Good point — dropped the count and the explicit enumeration in 9483d8c4cbf since the section headers already list the topics.
##########
docs/spark-connect-gotchas.md:
##########
@@ -418,6 +418,46 @@ println(structColumnFields)
This approach is significantly faster when dealing with a large number of columns because it avoids creating and analyzing numerous DataFrames.
+## 5. DataFrame column references after column shadowing
+
+In Spark Connect, a DataFrame column reference such as `df["c"]` is tagged with the plan id of `df`. At analysis time the server resolves the reference by looking for the tagged ancestor in the plan and pulling the matching attribute from it. Spark Classic does not use plan ids; it resolves column references against the immediate child's output by attribute id and name.
+
+The two resolution strategies diverge once a column has been shadowed by another operator that produces an attribute with the same name:
+
+```python
+import pyspark.sql.functions as sf
+
+df = spark.sql("SELECT 1 AS c")
+df.withColumn("c", sf.col("c").cast("string")).select(df["c"]).collect()
+```
+
+`withColumn("c", ...)` does not mutate `df`; it returns a new DataFrame whose `c` is a new attribute that hides the original. The trailing `df["c"]` still refers to the *original* `c` attribute, which is no longer in the projection list.
+
+* **Spark Classic** has always rejected this query at analysis time with `MISSING_ATTRIBUTES.RESOLVED_ATTRIBUTE_APPEAR_IN_OPERATION`, because the original attribute is not present in the operator's child output.
+* **Spark Connect** rejects it with `CANNOT_RESOLVE_DATAFRAME_COLUMN` by default. The plan-id-tagged reference does not match any attribute in the current plan. But when the SQL config `spark.sql.analyzer.strictDataFrameColumnResolution` (added in Spark 4.2.0, default `true`) is set to `false`, the analyzer still tries plan-id-based resolution first, and only when that fails does it fall back to name-based resolution: the tagged `df["c"]` is then resolved by name against the projected `c` from `withColumn`, and the query succeeds.
+
+### Recommended way
+
+If you hit any of the confusing failures mentioned above, it is recommended to switch to `sf.col` first. `sf.col("c")` is an untagged name reference that resolves against the most recent projection or `withColumn`, rather than `df["c"]` which is a tagged reference to `df`'s original column:
Review Comment:
Updated in 9483d8c4cbf to refer to `col` from `pyspark.sql.functions` in the prose so the reference is self-contained without depending on the earlier code block.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56846][DOCS][CONNECT] Document DataFrame column resolution behavior in spark-connect-gotchas [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #55848:
URL: https://github.com/apache/spark/pull/55848#issuecomment-4473470016
thanks, merged into master/4.x
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56846][DOCS][CONNECT] Document DataFrame column resolution behavior in spark-connect-gotchas [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #55848:
URL: https://github.com/apache/spark/pull/55848#discussion_r3241519051
##########
docs/spark-connect-gotchas.md:
##########
@@ -73,7 +73,7 @@ Unlike query execution, Spark Classic and Spark Connect differ in when schema an
# Common Gotchas (with Mitigations)
-If you are not careful about the difference between lazy vs. eager analysis, there are four key gotchas to be aware of: 1) overwriting temporary view names, 2) capturing external variables in UDFs, 3) delayed error detection, and 4) excessive schema access on new DataFrames.
+If you are not careful about the difference between lazy vs. eager analysis, there are five key gotchas to be aware of: 1) overwriting temporary view names, 2) capturing external variables in UDFs, 3) delayed error detection, 4) excessive schema access on new DataFrames, and 5) DataFrame column references after a column is shadowed.
Review Comment:
keeping a counter here is error-prone, can we just `there are several key gotchas ....`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56846][DOCS][CONNECT] Document DataFrame column resolution behavior in spark-connect-gotchas [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng closed pull request #55848: [SPARK-56846][DOCS][CONNECT] Document DataFrame column resolution behavior in spark-connect-gotchas
URL: https://github.com/apache/spark/pull/55848
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org