You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "ArnavBalyan (via GitHub)" <gi...@apache.org> on 2026/08/26 10:38:09 UTC
[PR] [spark] Fix nested struct values wrong order for insert columns [paimon]
ArnavBalyan opened a new pull request, #9415:
URL: https://github.com/apache/paimon/pull/9415
### Purpose
- Insert column lists should order both columns and nested struct fields by position.
- Paimon maps nested struct fields by name, causing values to be reordered when the source and target field orders differ.
- This causes the query results to differ, ensure we can fix by preserving the mapping recursively for structs inside arrays and maps.
### Tests
- UT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Fix nested struct values wrong order for insert columns [paimon]
Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on code in PR #9415:
URL: https://github.com/apache/paimon/pull/9415#discussion_r3957152493
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/catalyst/analysis/PaimonAnalysis.scala:
##########
@@ -110,7 +110,16 @@ class PaimonAnalysis(session: SparkSession) extends Rule[LogicalPlan] {
table: DataSourceV2Relation,
options: Options,
mergeSchemaEnabled: Boolean): LogicalPlan = {
- val query = stripHiveDynamicPartitionMarker(v2WriteCommand.query)
+ val queryWithoutMarker = stripHiveDynamicPartitionMarker(v2WriteCommand.query)
+ val query =
+ if (
+ v2WriteCommand.isByName &&
Review Comment:
Thanks @JingsongLi I've addressed this by deriving the column list from COLUMN_LIST_WRITE being dependent on the spark value
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Fix nested struct values wrong order for insert columns [paimon]
Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on PR #9415:
URL: https://github.com/apache/paimon/pull/9415#issuecomment-5424107473
cc @JingsongLi thanks :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Fix nested struct values wrong order for insert columns [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #9415:
URL: https://github.com/apache/paimon/pull/9415#discussion_r3930388775
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/catalyst/analysis/PaimonAnalysis.scala:
##########
@@ -110,7 +110,16 @@ class PaimonAnalysis(session: SparkSession) extends Rule[LogicalPlan] {
table: DataSourceV2Relation,
options: Options,
mergeSchemaEnabled: Boolean): LogicalPlan = {
- val query = stripHiveDynamicPartitionMarker(v2WriteCommand.query)
+ val queryWithoutMarker = stripHiveDynamicPartitionMarker(v2WriteCommand.query)
+ val query =
+ if (
+ v2WriteCommand.isByName &&
Review Comment:
[P1] Do not gate column-list semantics on Spark\x27s version-dependent `isByName`
The current head fails its own new `Paimon Insert: column list resolves unequal nested structs positionally` test on both Spark 3.2 and 3.3 with Scala 2.13. The CI stack reaches `resolveColumnsByPosition` at `arr.element` and rejects the 2-field source vs 3-field target; Spark 3.4/3.5 pass. On the older plans this `isByName` gate does not select the column-list rewrite/merge-schema path, even though the parser marker identifies the same SQL syntax.
Please derive the effective column-list mode from the explicit `COLUMN_LIST_WRITE` marker independently of Spark\x27s `V2WriteCommand.isByName`, and use that effective mode consistently for nested renaming, expected-output calculation, and missing-field resolution. The existing test already exposes the regression; it needs to pass under every supported Spark 3 profile before merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Fix nested struct values wrong order for insert columns [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #9415:
URL: https://github.com/apache/paimon/pull/9415#discussion_r4052061926
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/catalyst/analysis/PaimonOutputResolver.scala:
##########
@@ -58,6 +58,35 @@ object PaimonOutputResolver extends SQLConfHelper {
import MissingFieldBehavior._
+ def renameNestedFieldsByPosition(
+ query: LogicalPlan,
+ expected: Seq[Attribute],
+ queryOutputByName: Boolean): LogicalPlan = {
+ val renamed = query.output.zipWithIndex.map {
+ case (input, index) =>
+ val target = if (queryOutputByName) {
+ expected.find(target => conf.resolver(input.name, target.name))
+ } else {
+ expected.lift(index)
Review Comment:
[P1] Partial column lists silently misplace values on Spark 3.2/3.3
`expected.lift(index)` assumes `query.output` is already aligned with the table columns, which only holds on Spark 3.4+ (there `addColumnListOnQuery` fills the omitted columns, SPARK-42521). On Spark 3.2/3.3 `ResolveUserSpecifiedColumns.addColumnListOnQuery` keeps only the listed columns, re-ordered into table order (`tableOutput.flatMap { nameToQueryExpr.get(_).orElse(None) }`), so the ordinal no longer matches the table schema whenever the column list skips a column that precedes a listed one. `applyColumnMetadata` then aliases the value to the wrong table column, and `effectiveByName = true` NULL-fills the real target.
Reproduced on the current head (00955d7) with `INSERT ... SELECT`:
```
CREATE TABLE p1 (a INT, b INT, c INT);
INSERT INTO p1 (a, c) SELECT 1, 3; -- 3.2/3.3 store [1, 3, null], expected [1, null, 3]
CREATE TABLE p6 (a INT, b INT, c INT);
INSERT INTO p6 (c) SELECT 3; -- 3.2/3.3 store [3, null, null], expected [null, null, 3]
```
and with a gap column of a different type the insert fails with a misleading error:
```
CREATE TABLE p4 (a INT, s STRUCT<x: INT, y: INT>, c INT);
INSERT INTO p4 (a, c) SELECT 5, 7; -- AnalysisException: cannot resolve 's' due to data type mismatch: cannot cast int to struct<x:int,y:int>
```
Spark 3.4/3.5 handle the same statements correctly (`[1, null, 3]`, `[null, null, 3]`, `[5, null, 7]`).
This is a regression: on Spark 3.2/3.3 these statements used to be rejected loudly with `Cannot write to \`p1\`, the number of data columns (2) doesn't match the table schema's (3).` (the existing test comment in `InsertOverwriteTableTestBase` — "In Spark 3.3 or earlier, these commands would have failed" — documents that intent). The PR converts a clear failure into silently corrupted rows, and `INSERT INTO p1 (a, c) VALUES (1, 3)` still raises the old error on 3.2/3.3 while the `SELECT` form corrupts, so the semantics now depend on the query shape. The added tests all use full column lists, so nothing covers this.
Suggested fix: only take the positional nested-field path when the query output is known to be full width and in table order — e.g. guard the call with `query.output.size == table.output.size` and keep the previous by-position behavior (loud error) otherwise, or map each input column to the column it was actually listed as instead of `expected.lift(index)`. A regression test with a partial column list on Spark 3.2/3.3 would pin this down.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Fix nested struct values wrong order for insert columns [paimon]
Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on PR #9415:
URL: https://github.com/apache/paimon/pull/9415#issuecomment-5468603733
Hi @JingsongLi thanks for the review, have addressed the comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Fix nested struct values wrong order for insert columns [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #9415:
URL: https://github.com/apache/paimon/pull/9415#discussion_r3879081319
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/catalyst/analysis/PaimonOutputResolver.scala:
##########
@@ -478,6 +501,36 @@ object PaimonOutputResolver extends SQLConfHelper {
attr.withDataType(CharVarcharUtils.getRawType(attr.metadata).getOrElse(attr.dataType))
}
+ private def renameFieldsInStruct(input: StructType, expected: StructType): StructType = {
+ if (input.length == expected.length) {
Review Comment:
[P1] Preserve positional mapping when nested field counts differ
Returning `input` here falls back to the downstream by-name resolver whenever merge-schema fills a missing nested field. For target `ARRAY<STRUCT<x: INT, y: INT, z: INT>>`, a column-list write whose input struct fields are `(y=20, x=10)` stores `[10, 20, null]` instead of the positional `[20, 10, null]`. I reproduced this on Spark 3.5 with `spark.paimon.write.merge-schema=true`. Please rename the common ordinal prefix even when the lengths differ, preserve unmatched input fields, and let the existing strict/merge-schema handling process missing or extra fields. A regression test with unequal nested field counts would cover this path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org