You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "jiwen624 (via GitHub)" <gi...@apache.org> on 2026/06/05 22:51:18 UTC
[PR] [SPARK-57223][SQL] Fix/Quote verbatim column/field names containing dots in table-write error messages [spark]
jiwen624 opened a new pull request, #56351:
URL: https://github.com/apache/spark/pull/56351
### What changes were proposed in this pull request?
A surgical fix focused on customer-facing error messages only: the three `TableOutputResolver` sites that render a column/field name in `INCOMPATIBLE_DATA_FOR_TABLE` errors now pass it as `toSQLId(Seq(name))`, so a dotted name like `b.c` shows as `` `b.c` `` rather than `` `b`.`c` ``. The root cause is `toSQLId(String)` splitting names on dots, but fixing that shared primitive would touch ~130 call sites and isn't worth the churn/risk, so this targets only the affected, user-visible write-path renders.
Affected errors: `INCOMPATIBLE_DATA_FOR_TABLE.{EXTRA_COLUMNS, EXTRA_STRUCT_FIELDS, STRUCT_MISSING_FIELDS}`.
### Why are the changes needed?
A column or struct field whose name contains a dot is misreported in V2 write /INSERT error messages — `` `b`.`c` `` reads as two nested identifiers rather than one name `b.c`, confusing users.
### Does this PR introduce _any_ user-facing change?
Yes, error-message text only (no behavior change). For a dotted name, e.g. `EXTRA_COLUMNS` now renders `` `b.c` `` instead of `` `b`.`c` ``.
### How was this patch tested?
New tests in `DataFrameWriterV2Suite` covering the three error classes (append and by-position `INSERT`); each fails before the change and passes after.
### Was this patch authored or co-authored using generative AI tooling?
Yes. Claude Code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57223][SQL] INCOMPATIBLE_DATA_FOR_TABLE errors mis-quote a column/field name that contains a dot [spark]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho closed pull request #56351: [SPARK-57223][SQL] INCOMPATIBLE_DATA_FOR_TABLE errors mis-quote a column/field name that contains a dot
URL: https://github.com/apache/spark/pull/56351
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57223][SQL] INCOMPATIBLE_DATA_FOR_TABLE errors mis-quote a column/field name that contains a dot [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #56351:
URL: https://github.com/apache/spark/pull/56351#issuecomment-4787674183
> other 6 call sites are ignored as they seem to be dead code/unreachable
This part of the PR description might be partly inaccurate. 4 of the 6 (132/405/487/548) are reachable under different error classes (CANNOT_FIND_DATA, top-level arity) and would mis-quote dotted names too; only 590/657 are genuinely unreachable (guarded by the recursion-implied non-empty colPath on the array/map element path).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57223][SQL] INCOMPATIBLE_DATA_FOR_TABLE errors mis-quote a column/field name that contains a dot [spark]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on PR #56351:
URL: https://github.com/apache/spark/pull/56351#issuecomment-4938772411
Thanks, merged to master/4.x!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57223][SQL] INCOMPATIBLE_DATA_FOR_TABLE errors mis-quote a column/field name that contains a dot [spark]
Posted by "jiwen624 (via GitHub)" <gi...@apache.org>.
jiwen624 commented on PR #56351:
URL: https://github.com/apache/spark/pull/56351#issuecomment-4804139205
@uros-b @szehon-ho Thanks for the review! I addressed the comments with the following changes:
- changed L132/L405/L487 of TableOutputResolver and kept L548 as is due to it's genuinely unreachable - also updated the PR description for the unreachable code path.
- removed the redundant tring-interpolation wrapper for L391
- Updated the PR description to drop MERGE to make it more accurate.
Please let me know anything else is missing. 🙇
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57223][SQL] INCOMPATIBLE_DATA_FOR_TABLE errors mis-quote a column/field name that contains a dot [spark]
Posted by "jiwen624 (via GitHub)" <gi...@apache.org>.
jiwen624 commented on PR #56351:
URL: https://github.com/apache/spark/pull/56351#issuecomment-4637695846
Hi @szehon-ho, since you’ve worked on/reviewed PRs in this area, could you take a look and see if this fix makes sense? 🙇
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57223][SQL] INCOMPATIBLE_DATA_FOR_TABLE errors mis-quote a column/field name that contains a dot [spark]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on PR #56351:
URL: https://github.com/apache/spark/pull/56351#issuecomment-4802640215
(repeated part of @uros-b 's comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57223][SQL] INCOMPATIBLE_DATA_FOR_TABLE errors mis-quote a column/field name that contains a dot [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on code in PR #56351:
URL: https://github.com/apache/spark/pull/56351#discussion_r3465896981
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TableOutputResolver.scala:
##########
@@ -388,7 +388,7 @@ object TableOutputResolver extends SQLConfHelper with Logging {
if (reordered.length == expectedCols.length) {
if (matchedCols.size < inputCols.length) {
val extraCols = inputCols.filterNot(col => matchedCols.contains(col.name))
- .map(col => s"${toSQLId(col.name)}").mkString(", ")
+ .map(col => s"${toSQLId(Seq(col.name))}").mkString(", ")
Review Comment:
Seems like a redundant s"${...}" string-interpolation wrapper around a single expression, we could simplify?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57223][SQL] INCOMPATIBLE_DATA_FOR_TABLE errors mis-quote a column/field name that contains a dot [spark]
Posted by "jiwen624 (via GitHub)" <gi...@apache.org>.
jiwen624 commented on PR #56351:
URL: https://github.com/apache/spark/pull/56351#issuecomment-4929084694
@szehon-ho rebased to resolve conflicts with latest master. Please let me know if you have other concern. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org