[jira] [Created] (SPARK-58816) INSERT with a column list resolves structs inside arrays and maps by name instead of position

["Eames Trinh (Jira)" <ji...@apache.org>]

Eames Trinh created SPARK-58816:
-----------------------------------

             Summary: INSERT with a column list resolves structs inside arrays and maps by name instead of position
                 Key: SPARK-58816
                 URL: https://issues.apache.org/jira/browse/SPARK-58816
             Project: Spark
          Issue Type: Bug
          Components: Bug
    Affects Versions: 3.3.0
            Reporter: Eames Trinh


h3. Problem

For {{{}INSERT INTO table (column_list){}}}, nested structs should be resolved by position.

Spark currently behaves inconsistently:
 * Direct structs are resolved by position.
 * Structs inside arrays and maps are resolved by name.

This can silently write values into different fields depending only on where the struct is nested.
h3. Reproduction
{code:java}
CREATE TABLE target (
  s   STRUCT<x: INT, y: INT>,
  arr ARRAY<STRUCT<x: INT, y: INT>>,
  m   MAP<STRING, STRUCT<x: INT, y: INT>>
) USING parquet;

INSERT INTO target (s, arr, m)
SELECT
  named_struct('y', 20, 'x', 10),
  array(named_struct('y', 20, 'x', 10)),
  map('k', named_struct('y', 20, 'x', 10));

SELECT
  to_json(s) AS direct_struct,
  to_json(arr[0]) AS array_struct,
  to_json(m['k']) AS map_struct
FROM target; {code}
h3. Expected result

All three structs should be resolved by position: {{{"x":20,"y":10}}}
h3. Actual result

Only the direct struct is resolved by position. For the array and map structs we get: {{{"x":10,"y":20}}}
h2. Root cause

{{ResolveInsertionBase.createProjectForByNameQuery}} adds a projection that renames direct struct fields according to their positions. However, {{renameFieldsInStruct}} only recurses through {{{}StructType{}}}; it does not descend into {{ArrayType}} or {{{}MapType{}}}.

Consequently, field names inside arrays and maps remain unchanged, and {{TableOutputResolver}} later resolves those nested structs by name.
h2. Proposed behavior

For {{{}INSERT INTO table (column_list){}}}, struct fields should be resolved positionally at every nesting level, including:
 * Array elements
 * Map keys
 * Map values

The projection’s field-renaming logic should recursively traverse arrays and maps, or the resolver should otherwise preserve positional semantics throughout the nested type.
h2. Impact

Queries can silently write values into the wrong nested fields when source and target struct field names differ in order. The behavior is also inconsistent with direct structs and with the positional semantics of {{INSERT INTO}} column lists.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org