You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "voonhous (via GitHub)" <gi...@apache.org> on 2026/08/28 11:01:33 UTC

[PR] fix(variant): project PushVariantIntoScan struct paths in the file-group reader and pin nested-shredded internal reads [hudi]

voonhous opened a new pull request, #19783:
URL: https://github.com/apache/hudi/pull/19783

   ### Change Logs
   
   Closes #19775. Part of #18937. Stacked on #19777 (the first commit is that PR's; review the second commit): the legs need its
   test scaffolds and its Avro-path nested write parity, and it merges first.
   
   Pins that a variant shredded below the top level reads correctly through Hudi's Spark-native internal read paths on Spark 4.1+,
   corrects the four comment blocks that deferred the nested case with the wrong mechanism, and fixes the one real defect the legs
   found: the file-group reader applied a PushVariantIntoScan projection to top-level fields only.
   
   **The defect (issue outcome B, shredding-independent).** Spark 4.1+'s `PushVariantIntoScan` rewrites a variant reached through
   a struct path (`cast(s.inner as string)`, `variant_get(s.inner, ...)`) into a projection struct nested inside the scan's required
   schema (`s: struct<inner: struct<"0": string>>`). `SparkFileFormatInternalRowReaderContext` overlaid and projected such structs at
   the top level only, so on a MOR table with logs the merged row still carried a raw `VariantVal` at `s.inner` while the plan read
   that memory as `s.inner.0`: JVM SIGBUS / `InternalError` / OOM or a silent null, on native parquet logs and avro blocks, both record
   types, shredded or not. Minimal repro: `create table t (id int, s struct<inner: variant>, ts long) using hudi tblproperties
   (type='mor', hoodie.index.type='INMEMORY', ...)`, one insert, `select id, cast(s.inner as string) from t`. COW was unaffected
   because the whole catalyst schema goes to Spark's parquet reader, which handles a projection struct at any depth; with
   `spark.sql.variant.pushVariantIntoScan=false` everything worked.
   
   Fix: `overlayVariantProjections` / `shouldProjectVariants` recurse into struct members, and `buildVariantProjector` (hoisted out of
   the identical `Spark4_1Adapter` / `Spark4_2Adapter` copies into `BaseSpark4Adapter`) rebuilds a projection struct below a struct
   path with `VariantGet` children, recreating enclosing structs null-preservingly and passing untouched subtrees through by
   reference. Struct paths only, mirroring `VariantInRelation.rewriteType`; arrays and maps stay native on both sides.
   
   **What was verified before writing anything** (all on Spark 4.1.1, the pinned version):
   
   - The parquet row reader reconstructs a shredded variant for a NATIVE `VariantType` request at any depth: top-level, struct member
     and array element. `ParquetReadSupport.clipParquetType` passes a `VariantType` leaf's group through unchanged (its
     `clipVariantSchema` is a no-op TODO), `ParquetToSparkSchemaConverter.convertField` carries the catalyst target type into struct
     members, list elements and map values, and `ParquetRowConverter` builds a `ParquetVariantConverter` for it. `PushVariantIntoScan`
     itself only rewrites struct paths; arrays and maps stay native even for user queries.
   - What does read a shredded group as `value=null` is a request in the physical `{metadata, value}` struct shape. The only
     producer of that shape is the schema-on-read internal-schema branch, which #19687 guards.
   - The full-variant projection rewrite (`SparkAdapter.buildFullVariantReadSchema`) is therefore a contract, not a workaround:
     with it disabled (patched class shadowed on the test classpath, class origin verified), `TestVariantDataType` is 20/20 and
     `TestVariantShreddingMixedLayouts` 21/22 -- the one red is the schema-on-read fail-fast leg, where the projection shape is what
     lets `validateNoShreddedVariants` name the route instead of a `ClassCastException` inside the read. The same holds at the #19558
     fix commit itself (rebuilt in a worktree): the #19556 clustering repro passes there with the rewrite disabled and the logged
     parquet requested schema keeps `typed_value`. With `HoodieMergeOnReadRDDV2.shouldRerouteVariantSplit` forced off, the legacy
     streaming shredded leg passes too.
   
   So this is issue #19775's outcome A: the rewrite stays top-level and the code is unchanged; the comments now say what the shape
   is for and why nested variants need none.
   
   **Tests** (`TestVariantShreddingMixedLayouts`, section F; every table has ONLY the nested variant so nothing top-level can mask the
   nested path -- a new `withNestedOnlyVariantTable` scaffold in `VariantShreddingTestSupport`):
   
   - MOR: nested-shredded base + nested-shredded log; snapshot merge with the vectorized reader on and off, read-optimized, the legacy
     RDD path (`hoodie.file.group.reader.enabled=false`) over a base + log slice and over the compacted log-free slice, compaction under
     the forced DDL (base re-shredded on both record types, the AVRO one through #19777's Avro-path parity) and under `Unshredded`
     (typed_value stripped at depth), clustering under the forced DDL for both `row.writer.enable` values.
   - CDC (`hudi_table_changes(..., 'cdc', 'earliest')`, both supplemental logging modes): insert and update images of `s.inner`
     across a layout flip.
   - `variant_get` projection and filter on `s.inner` across a typed file and a residual file, `pushVariantIntoScan` on and off: on
     COW, and on MOR over native parquet log files (SPARK record type) and table-version-9 avro data blocks (AVRO record type, pinned
     through the log block types). Red before the fix as a JVM abort (SIGSEGV/SIGBUS in an unsafe copy stub).
   - `TestBaseSpark4AdapterVariantMethods`: the recursive projector as a unit (nested extraction, sibling preserved, null struct
     preserved, no projection -> None).
   - `TestStreamingSource`'s legacy RDD leg (the only remaining `hoodie.file.group.reader.enabled=false` consumer; batch reads no
     longer consult that key) gains a nested-shredded column on every split plus a footer pin on the compacted base.
   - `array<variant>` shredded at the element through a declared `hoodie.write.schema` (the only way a Hudi write produces that layout;
     SPARK record type, since the AVRO record-type insert cannot take a write-schema override of a different shape): snapshot read with
     the vectorized reader on and off, `variant_get` on an element, a log update, compaction under the declared schema (element
     re-shredded) and without it (element unshredded).
   
   **Not done here, on purpose:** removing the projection rewrite and the legacy re-route now that both are shown to be contract rather
   than correctness. That is a behaviour change (it would put variant tables back on the skip-merging fast path) and belongs in its
   own PR if wanted.
   
   ### Impact
   
   Reads of a struct-nested variant through the Spark file-group reader with `pushVariantIntoScan` on (the default) no longer crash
   or return nulls on MOR tables with log files. No config changes.
   
   ### Risk level
   
   Low
   
   ### Documentation Update
   
   None
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Change Logs and Impact were stated clearly
   - [x] Adequate tests were added if applicable
   - [ ] CI passed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(variant): project PushVariantIntoScan struct paths in the ... [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19783:
URL: https://github.com/apache/hudi/pull/19783#issuecomment-5476099486

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e268d40c1df977ced91f3197fa97fef1d0473ac0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16667",
       "triggerID" : "e268d40c1df977ced91f3197fa97fef1d0473ac0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "115b202afb5a10df2382ea1644f37a97ff2b933e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16671",
       "triggerID" : "115b202afb5a10df2382ea1644f37a97ff2b933e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a51b75e41a66ba8982da4aff12edd236dbfe692a",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16673",
       "triggerID" : "a51b75e41a66ba8982da4aff12edd236dbfe692a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4e1524d2a0e3e7ea61348dc237e336bf9ff7e19e",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16714",
       "triggerID" : "4e1524d2a0e3e7ea61348dc237e336bf9ff7e19e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * a51b75e41a66ba8982da4aff12edd236dbfe692a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16673) 
   * 4e1524d2a0e3e7ea61348dc237e336bf9ff7e19e Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16714) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(variant): project PushVariantIntoScan struct paths in the ... [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19783:
URL: https://github.com/apache/hudi/pull/19783#discussion_r3893261852


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieMergeOnReadRDDV2.scala:
##########
@@ -149,13 +149,14 @@ class HoodieMergeOnReadRDDV2(@transient sc: SparkContext,
     }
   }
 
-  // The plain skip-merging reader cannot read a SHREDDED variant base file: it requests native
-  // VariantType, which clips the shredded group to {metadata, value} and reads value=null (the
-  // #19556 defect family). Such splits take the file-group reader below, whose reader context
-  // requests the full-variant projection shape instead (#19578). Keyed off the adapter building
-  // that shape rather than the mere presence of a variant column: it is None below Spark 4.1,
-  // where the file-group reader would read the same nulls, so re-routing there would cost the
-  // fast path for nothing.
+  // A split whose required schema has a top-level variant column takes the file-group reader
+  // below, whose reader context requests the full-variant projection shape for parquet base
+  // files (#19578), so a SHREDDED base file is read on this legacy path through the same
+  // contract as everywhere else. The skip-merging reader's native VariantType request is
+  // reconstructed by the Spark 4.1+ row reader as well (pinned by TestStreamingSource), so this

Review Comment:
   Added the nested-only leg rather than dropping the claim. `TestStreamingSource` now runs the legacy shredded-variant stream twice, with and without the top-level `v`. Without it `shouldRerouteVariantSplit` is false, so the base-only split goes through `requiredSchemaReaderSkipMerging` and Spark's own row reader rebuilds `s.inner` from the shredded group; the footer pin asserts `s.inner` is shredded and that the file carries no `v`. This comment and the "Not swept here" note in `TestVariantShreddingMixedLayouts` now name that leg.
   



##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestStreamingSource.scala:
##########
@@ -441,29 +446,52 @@ class TestStreamingSource extends StreamTest {
         true
       }
 
-      addVariantData("""select 1 as id, parse_json('{"key":"v1"}') as v, 1000L as ts""", compact = false)
-      addVariantData("""select 2 as id, parse_json('{"key":"v2"}') as v, 1000L as ts""", compact = true)
+      addVariantData("""select 1 as id, parse_json('{"key":"v1"}') as v,
+                       | named_struct('inner', parse_json('{"key":"n1"}')) as s, 1000L as ts""".stripMargin, compact = false)
+      addVariantData("""select 2 as id, parse_json('{"key":"v2"}') as v,
+                       | named_struct('inner', parse_json('{"key":"n2"}')) as s, 1000L as ts""".stripMargin, compact = true)
+      // Pin that the compacted base file is shredded at both depths: without it the streams below
+      // would pass just the same over an unshredded base and pin nothing about shredded reads.
+      val conf = spark.sessionState.newHadoopConf()
+      val baseFiles = new Path(tablePath).getFileSystem(conf).listStatus(new Path(tablePath))

Review Comment:
   Done: the listing filters with `FSUtils.isBaseFile`, and the message says log files are excluded.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(variant): project PushVariantIntoScan struct paths in the ... [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19783:
URL: https://github.com/apache/hudi/pull/19783#issuecomment-5488184456

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e268d40c1df977ced91f3197fa97fef1d0473ac0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16667",
       "triggerID" : "e268d40c1df977ced91f3197fa97fef1d0473ac0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "115b202afb5a10df2382ea1644f37a97ff2b933e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16671",
       "triggerID" : "115b202afb5a10df2382ea1644f37a97ff2b933e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a51b75e41a66ba8982da4aff12edd236dbfe692a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16673",
       "triggerID" : "a51b75e41a66ba8982da4aff12edd236dbfe692a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4e1524d2a0e3e7ea61348dc237e336bf9ff7e19e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16714",
       "triggerID" : "4e1524d2a0e3e7ea61348dc237e336bf9ff7e19e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6b4ffe79db9a996c398df1e4888f93a9937cb98e",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16717",
       "triggerID" : "6b4ffe79db9a996c398df1e4888f93a9937cb98e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2ea92de1fcf8cd8a83abe3968d79a303d21d3140",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16729",
       "triggerID" : "2ea92de1fcf8cd8a83abe3968d79a303d21d3140",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 6b4ffe79db9a996c398df1e4888f93a9937cb98e Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16717) 
   * 2ea92de1fcf8cd8a83abe3968d79a303d21d3140 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16729) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(variant): project PushVariantIntoScan struct paths in the ... [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19783:
URL: https://github.com/apache/hudi/pull/19783#issuecomment-5490640649

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e268d40c1df977ced91f3197fa97fef1d0473ac0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16667",
       "triggerID" : "e268d40c1df977ced91f3197fa97fef1d0473ac0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "115b202afb5a10df2382ea1644f37a97ff2b933e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16671",
       "triggerID" : "115b202afb5a10df2382ea1644f37a97ff2b933e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a51b75e41a66ba8982da4aff12edd236dbfe692a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16673",
       "triggerID" : "a51b75e41a66ba8982da4aff12edd236dbfe692a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4e1524d2a0e3e7ea61348dc237e336bf9ff7e19e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16714",
       "triggerID" : "4e1524d2a0e3e7ea61348dc237e336bf9ff7e19e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6b4ffe79db9a996c398df1e4888f93a9937cb98e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16717",
       "triggerID" : "6b4ffe79db9a996c398df1e4888f93a9937cb98e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2ea92de1fcf8cd8a83abe3968d79a303d21d3140",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16729",
       "triggerID" : "2ea92de1fcf8cd8a83abe3968d79a303d21d3140",
       "triggerType" : "PUSH"
     }, {
       "hash" : "42b6ff809b4af335e5becdafa97de014912d3fd6",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16735",
       "triggerID" : "42b6ff809b4af335e5becdafa97de014912d3fd6",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 2ea92de1fcf8cd8a83abe3968d79a303d21d3140 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16729) 
   * 42b6ff809b4af335e5becdafa97de014912d3fd6 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16735) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(variant): project PushVariantIntoScan struct paths in the ... [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19783:
URL: https://github.com/apache/hudi/pull/19783#issuecomment-5451952392

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e268d40c1df977ced91f3197fa97fef1d0473ac0",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16667",
       "triggerID" : "e268d40c1df977ced91f3197fa97fef1d0473ac0",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e268d40c1df977ced91f3197fa97fef1d0473ac0 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16667) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(variant): project PushVariantIntoScan struct paths in the ... [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19783:
URL: https://github.com/apache/hudi/pull/19783#discussion_r3901766247


##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/schema/TestVariantShreddingMixedLayouts.scala:
##########
@@ -818,6 +871,474 @@ class TestVariantShreddingMixedLayouts extends HoodieSparkSqlTestBase with Varia
     }
   }
 
+  test("MOR merge, compaction and clustering carry a nested-shredded base through the internal reader") {
+    assume(HoodieSparkUtils.gteqSpark4_1, SPARK_4_1_GATE)
+
+    // The nested variant is the ONLY variant in the table, so no top-level rewrite carries it (see
+    // withNestedOnlyVariantTable): every read below has to resolve s.inner on its own - through the
+    // nested projection struct PushVariantIntoScan pushes into the scan for the user queries, and
+    // natively for the internal reads that have no catalyst schema (compaction, clustering, the
+    // legacy RDD) - which is what makes these paths say anything about nested shredding at all.
+    // The projected arm crashed the JVM before #19775 (the projection was applied at the top level
+    // only, so the merged row still held a raw variant where the plan read a struct).
+    val mergedRows = Seq(
+      Seq(1, """{"k":"n1"}"""), Seq(2, """{"k":"n2b"}"""), Seq(3, """{"k":"n3"}"""))
+    val finalRows = Seq(
+      Seq(1, """{"k":"n1"}"""), Seq(2, """{"k":"n2b"}"""), Seq(3, """{"k":"n3c"}"""))
+
+    Seq(true, false).foreach { rowWriter =>
+      // No INMEMORY index: the first insert creates a base file, the updates go to log files.
+      withNestedOnlyVariantTable(s"mor nested rowWriter=$rowWriter", "mor",
+        props = Seq("hoodie.compact.inline = 'false'"),
+        recordTypes = clusteringRecordTypes(rowWriter)) { (tableName, tablePath, leg) =>
+        val snapshotQuery = s"select id, cast(s.inner as string) from $tableName order by id"
+        val readOptimizedQuery = s"select id, cast(s.inner as string) from " +
+          s"hudi_query('$tableName', 'read_optimized') order by id"
+
+        withWriteLayout(Forced("k string")) {
+          spark.sql(s"insert into $tableName values " +
+            """(1, named_struct('inner', parse_json('{"k":"n1"}')), 1000), """ +
+            """(2, named_struct('inner', parse_json('{"k":"n2"}')), 1000), """ +
+            """(3, named_struct('inner', parse_json('{"k":"n3"}')), 1000)""")
+        }
+        val baseFiles = listDataParquetFiles(tablePath)
+        assert(baseFiles.size == 1, s"[$leg] expected one base file, got $baseFiles")
+        assertVariantLayout(tablePath, shredded = true, leg, column = "s.inner")
+        val innerGroup = variantGroupOf(baseFiles.head, "s.inner")
+        assert(getFieldAsGroup(innerGroup, "typed_value").containsField("k"),
+          s"[$leg] nested typed_value should carry k:\n$innerGroup")
+
+        // A nested-shredded native log on top of the nested-shredded base.
+        withWriteLayout(Forced("k string")) {
+          spark.sql(s"update $tableName set " +
+            """s = named_struct('inner', parse_json('{"k":"n2b"}')), ts = 1001 where id = 2""")
+        }
+        assert(listDataParquetFiles(tablePath).exists(_.endsWith(".log.parquet")),
+          s"[$leg] the update should have written a native parquet log file")
+
+        // The conf only says what the reader MAY do - supportBatch vetoes vectorization for a
+        // variant at any depth (pinned in F2) - so the sweep is the control if that guard is ever
+        // narrowed back to top-level columns.
+        Seq("true", "false").foreach { vectorizedReader =>
+          withSQLConf("spark.sql.parquet.enableVectorizedReader" -> vectorizedReader) {
+            checkAnswer(snapshotQuery)(mergedRows: _*)
+          }
+        }
+
+        // Not swept here: hoodie.file.group.reader.enabled=false, which no longer routes a batch
+        // read anywhere (only the streaming sources consult it). The legacy RDD path over a
+        // nested-shredded base - HoodieMergeOnReadRDDV2, whose shouldRerouteVariantSplit stays
+        // false without a top-level variant - is pinned by TestStreamingSource's nested-only
+        // legacy leg (the leg with a top-level variant beside it is re-routed to the file-group
+        // reader instead).
+
+        // Read-optimized serves the base file alone: id 2 is still the pre-update value.
+        checkAnswer(readOptimizedQuery)(
+          Seq(1, """{"k":"n1"}"""), Seq(2, """{"k":"n2"}"""), Seq(3, """{"k":"n3"}"""))
+
+        // Compaction merges the nested-shredded log onto the nested-shredded base and re-derives
+        // the layout from the forced DDL. On the AVRO record type that write goes through
+        // HoodieAvroWriteSupport, whose nested forced hook is #19689's parity fix.
+        withWriteLayout(Forced("k string")) {
+          runCompaction(tableName)
+        }
+        assertCompactionCount(tablePath, 1, leg)
+        assertNestedBaseLayout(tablePath, latestCompletedInstant(tablePath), shredded = true, leg)
+        checkAnswer(snapshotQuery)(mergedRows: _*)
+
+        // Unshredded round: the update and the compaction both run with shredding off, so
+        // typed_value has to be stripped at depth on the way out.
+        withWriteLayout(Unshredded) {
+          spark.sql(s"update $tableName set " +
+            """s = named_struct('inner', parse_json('{"k":"n3c"}')), ts = 1002 where id = 3""")
+          runCompaction(tableName)
+        }
+        assertCompactionCount(tablePath, 2, leg)
+        assertNestedBaseLayout(tablePath, latestCompletedInstant(tablePath), shredded = false, leg)
+        checkAnswer(snapshotQuery)(finalRows: _*)
+
+        // Clustering re-derives the nested layout from the forced DDL over that unshredded input:
+        // the row-writer path when rowWriter is true, the record writers otherwise.
+        withWriteLayout(Forced("k string")) {
+          runClustering(tableName, rowWriter)
+        }
+        val clusteringInstant = completedClusteringInstant(tablePath, leg)
+        assertNestedBaseLayout(tablePath, clusteringInstant, shredded = true, leg)
+        checkAnswer(snapshotQuery)(finalRows: _*)
+        checkAnswer(readOptimizedQuery)(finalRows: _*)
+      }
+    }
+  }
+
+  test("CDC images carry a nested-shredded variant") {
+    assume(HoodieSparkUtils.gteqSpark4_1, SPARK_4_1_GATE)
+
+    // The nested twin of TestVariantDataType's CDC test. OP_KEY_ONLY reconstructs both images by
+    // reading the file slices; DATA_BEFORE_AFTER (the default) reads the update images from the cdc
+    // log instead. The insert leg takes BASE_FILE_INSERT in both modes, which reads the new base
+    // file directly rather than through the reader context.
+    Seq("OP_KEY_ONLY", "DATA_BEFORE_AFTER").foreach { loggingMode =>
+      // SPARK pinned: a cdc-enabled table always writes through FileGroupReaderBasedMergeHandle and
+      // the merger's record type picks that handle's reader context; AVRO would route the update's
+      // base-file read through HoodieAvroParquetReader, the separate defect tracked as #19567.
+      withNestedOnlyVariantTable(s"cdc nested $loggingMode", "cow", props = Seq(
+        "'hoodie.table.cdc.enabled' = 'true'",
+        s"'hoodie.table.cdc.supplemental.logging.mode' = '$loggingMode'",
+        "hoodie.index.type = 'INMEMORY'"),
+        recordTypes = Seq(HoodieRecordType.SPARK)) { (tableName, tablePath, leg) =>
+        withWriteLayout(Forced("k string")) {
+          spark.sql(s"insert into $tableName values " +
+            """(1, named_struct('inner', parse_json('{"k":"c1"}')), 1000)""")
+        }
+        assertVariantLayout(tablePath, shredded = true, leg, column = "s.inner")
+
+        withWriteLayout(Forced("k string")) {
+          spark.sql(s"update $tableName set " +
+            """s = named_struct('inner', parse_json('{"k":"c2"}')), ts = 1001 where id = 1""")
+        }
+        // Layout flip: the second update rewrites the file unshredded, so the images below span
+        // both physical slots.
+        withWriteLayout(Unshredded) {
+          spark.sql(s"update $tableName set " +
+            """s = named_struct('inner', parse_json('{"k":"c3"}')), ts = 1002 where id = 1""")
+        }
+        assertNestedBaseLayout(tablePath, latestCompletedInstant(tablePath), shredded = false, leg)
+
+        val cdc = spark.sql(s"select op, get_json_object(before, '$$.s.inner.k') as before_k, " +
+          s"get_json_object(after, '$$.s.inner.k') as after_k " +
+          s"from hudi_table_changes('$tableName', 'cdc', 'earliest')")
+        val insertRows = cdc.where("op = 'i'").collect()
+        assert(insertRows.length == 1, s"[$leg] expected exactly one insert cdc row")
+        assert(insertRows(0).getString(2) == "c1",
+          s"[$leg] insert after-image lost the nested variant payload: ${insertRows(0)}")
+
+        val updateRows = cdc.where("op = 'u'").orderBy("after_k").collect()
+        assert(updateRows.length == 2, s"[$leg] expected two update cdc rows")
+        assert(updateRows(0).getString(1) == "c1" && updateRows(0).getString(2) == "c2",
+          s"[$leg] first update images lost the nested variant payload: ${updateRows(0)}")
+        assert(updateRows(1).getString(1) == "c2" && updateRows(1).getString(2) == "c3",
+          s"[$leg] second (layout-flipped) update images lost the nested variant payload: ${updateRows(1)}")
+      }
+    }
+  }
+
+  test("variant_get projections and filters resolve a nested-shredded variant") {
+    assume(HoodieSparkUtils.gteqSpark4_1, SPARK_4_1_GATE)
+
+    // The nested twin of the mixed-layout variant_get test above; SPARK pinned for the same reason.
+    // pushVariantIntoScan is swept because the two arms reach the file differently: on, Spark
+    // rewrites the s.inner struct path into its own projection struct and pushes it into the scan;
+    // off, the whole variant is read and variant_get evaluates on top of it.
+    def nestedRowsSql(lo: Int, hi: Int): String =
+      s"""select cast(id as int) as id,
+         | named_struct('inner', parse_json(concat('{"k":"x', id, '"}'))) as s,
+         | 1000L as ts from range($lo, $hi, 1, 1)""".stripMargin
+
+    Seq("true", "false").foreach { pushIntoScan =>
+      withSQLConf("spark.sql.variant.pushVariantIntoScan" -> pushIntoScan) {
+        withNestedOnlyVariantTable(s"nested pushVariantIntoScan=$pushIntoScan", "cow",
+          props = Seq(NEW_FILE_GROUP_PER_COMMIT), recordTypes = Seq(HoodieRecordType.SPARK)) {
+          (tableName, tablePath, leg) =>
+          withWriteLayout(Forced("k string")) {
+            spark.sql(s"insert into $tableName ${nestedRowsSql(0, 5)}")
+          }
+          val shreddedInstant = latestCompletedInstant(tablePath)
+          withWriteLayout(Unshredded) {
+            spark.sql(s"insert into $tableName ${nestedRowsSql(5, 10)}")
+          }
+          val unshreddedInstant = latestCompletedInstant(tablePath)
+          // One file per commit (small.file.limit = 0) and one layout each, so $.k is typed in the
+          // first file and residual in the second: both slots answer the queries below.
+          assertNestedBaseLayout(tablePath, shreddedInstant, shredded = true, leg)
+          assertNestedBaseLayout(tablePath, unshreddedInstant, shredded = false, leg)
+
+          checkAnswer(s"select id, variant_get(s.inner, '$$.k', 'string') from $tableName order by id")(
+            (0 until 10).map(id => Seq(id, s"x$id")): _*)
+          // A filter served out of the residual file, and one out of the typed file.
+          checkAnswer(s"select id from $tableName where variant_get(s.inner, '$$.k', 'string') = 'x7'")(
+            Seq(7))
+          checkAnswer(s"select id from $tableName where variant_get(s.inner, '$$.k', 'string') = 'x2'")(
+            Seq(2))
+          checkAnswer(s"select count(*) from $tableName " +
+            s"where try_variant_get(s.inner, '$$.missing', 'string') is null")(Seq(10))
+          checkAnswer(s"select count(*) from $tableName where s.inner is null")(Seq(0))
+          checkAnswer(s"select id, cast(s.inner as string) from $tableName order by id")(
+            (0 until 10).map(id => Seq(id, s"""{"k":"x$id"}""")): _*)
+          // Both arms expect the very same rows, so this is what tells them apart: whether the
+          // rule actually rewrote s.inner into a projection struct inside the scan.
+          val pushed = pushIntoScan.toBoolean
+          val verdict = if (pushed) "should have" else "must not have"
+          assert(variantProjectionPushedIntoScan(
+            s"select id, variant_get(s.inner, '$$.k', 'string') from $tableName") == pushed,
+            s"[$leg] PushVariantIntoScan $verdict rewritten s.inner into a projection struct")
+        }
+      }
+    }
+
+    // MOR: a nested-shredded base under an unshredded log update, so the merged read has to serve
+    // ids 0-2 from the base file and ids 3-4 from the log. Before #19775 the pushIntoScan=true arms
+    // here crashed the JVM (SIGSEGV/SIGBUS inside an unsafe copy stub, or a java.lang.InternalError
+    // about a fault in a recent unsafe memory access): the internal reader applied the
+    // PushVariantIntoScan projection to TOP-LEVEL fields only, so the merged row still held a raw
+    // VariantVal at s.inner while the plan read that memory as the projected struct s.inner.0.
+    def seedNestedMor(tableName: String, tablePath: String, leg: String): Unit = {
+      withWriteLayout(Forced("k string")) {
+        spark.sql(s"insert into $tableName ${nestedRowsSql(0, 5)}")
+      }
+      assertNestedBaseLayout(tablePath, latestCompletedInstant(tablePath), shredded = true, leg)
+      // id 4's variant is nulled out on the way through the log: before #19775 the avro-path
+      // projector emitted a non-null struct of nulls for it rather than a NULL struct, which is
+      // what the `s.inner is null` assertion below catches.
+      withWriteLayout(Unshredded) {
+        spark.sql(s"update $tableName set " +
+          """s = named_struct('inner', parse_json(case when id = 4 then cast(null as string) """ +
+          """else concat('{"k":"y', id, '"}') end)), ts = 1001 """ +
+          "where id >= 3")
+      }
+    }
+    def assertNestedMorReads(tableName: String, leg: String, pushed: Boolean): Unit = {
+      // ids 0-2 come off the base file, id 3 off the log, and id 4's variant is the log's NULL.
+      def merged(id: Int): String = if (id < 3) s"x$id" else if (id == 3) "y3" else null
+      def mergedJson(id: Int): String = Option(merged(id)).map(k => s"""{"k":"$k"}""").orNull
+      checkAnswer(s"select id, variant_get(s.inner, '$$.k', 'string') from $tableName order by id")(
+        (0 until 5).map(id => Seq(id, merged(id))): _*)
+      // A filter served out of the log row, then one out of the base row.
+      checkAnswer(s"select id from $tableName where variant_get(s.inner, '$$.k', 'string') = 'y3'")(
+        Seq(3))
+      checkAnswer(s"select id from $tableName where variant_get(s.inner, '$$.k', 'string') = 'x1'")(
+        Seq(1))
+      checkAnswer(s"select id, cast(s.inner as string) from $tableName order by id")(
+        (0 until 5).map(id => Seq(id, mergedJson(id))): _*)
+      // PushVariantIntoScan rewrites `s.inner is null` onto the projection struct itself, so a
+      // null variant projected as a struct OF nulls rather than a NULL struct made this row
+      // disappear: the avro-block leg with the rule on is the one that failed before the fix.
+      checkAnswer(s"select id from $tableName where s.inner is null")(Seq(4))
+      checkAnswer(s"select count(*) from $tableName where s.inner is not null")(Seq(4))
+      // The whole struct: no extraction, so nothing is rewritten even with pushVariantIntoScan on,
+      // and the merge is read through a plain native VariantType at depth. `s` itself is never
+      // null - only its `inner` member is, and only for id 4.
+      val wholeStruct = spark.sql(s"select id, s from $tableName order by id").collect()
+      assert(wholeStruct.length == 5, s"[$leg] whole-struct read should return 5 rows")
+      assert(wholeStruct.forall(!_.isNullAt(1)), s"[$leg] whole-struct read nulled out s")

Review Comment:
   Done: the whole-struct read now checks `s.inner` per row against `mergedJson` (`VariantVal.toString` is its JSON), null for id 4, alongside the non-null `s` check.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(variant): project PushVariantIntoScan struct paths in the ... [hudi]

Posted by "wombatu-kun (via GitHub)" <gi...@apache.org>.
wombatu-kun commented on code in PR #19783:
URL: https://github.com/apache/hudi/pull/19783#discussion_r3888224040


##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/SparkFileFormatInternalRowReaderContext.scala:
##########
@@ -93,28 +93,41 @@ class SparkFileFormatInternalRowReaderContext(baseFileReader: SparkColumnarFileR
   private lazy val allFilters = filters ++ requiredFilters
 
   // For each field of `target`, replace its dataType with the matching field's projected
-  // variant struct from `source` (when present). Non-matching fields pass through. Why a parallel
-  // `sparkRequiredSchema` overlay exists at all is documented on that constructor parameter.
+  // variant struct from `source` (when present), recursing into struct members so a variant
+  // reached through a struct path is overlaid too. Fields are matched by name (findFieldByName);
+  // non-matching fields pass through. The recursion mirrors PushVariantIntoScan's
+  // VariantInRelation.rewriteType, which rewrites variants at the root of the relation output
+  // and below STRUCT paths only, so an array element or a map value is never overlaid here
+  // either (#19775). Why a parallel `sparkRequiredSchema` overlay exists at all is documented on
+  // that constructor parameter.
   private def overlayVariantProjections(target: StructType, source: StructType): StructType = {
     StructType(target.fields.map { f =>
-      SparkFileFormatInternalRowReaderContext.findFieldByName(source, f.name).map(_.dataType) match {
-        case Some(projStruct: StructType) if sparkAdapter.isVariantProjectionStruct(projStruct) =>
+      (f.dataType, SparkFileFormatInternalRowReaderContext.findFieldByName(source, f.name).map(_.dataType)) match {
+        case (_, Some(projStruct: StructType)) if sparkAdapter.isVariantProjectionStruct(projStruct) =>
           f.copy(dataType = projStruct)
+        case (targetStruct: StructType, Some(sourceStruct: StructType)) =>

Review Comment:
   `SparkSchemaTransformUtils.addMissingFields` has no variant-projection case, so once any sibling member of the enclosing struct triggers an implicit type change the nested projection installed here is replaced by the file's `VariantType` in the read schema. Should it get the same special case `isDataTypeEqualForPhysicalSchema` already has, or is implicit evolution alongside a nested variant out of scope for this PR?



##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestStreamingSource.scala:
##########
@@ -441,29 +446,52 @@ class TestStreamingSource extends StreamTest {
         true
       }
 
-      addVariantData("""select 1 as id, parse_json('{"key":"v1"}') as v, 1000L as ts""", compact = false)
-      addVariantData("""select 2 as id, parse_json('{"key":"v2"}') as v, 1000L as ts""", compact = true)
+      addVariantData("""select 1 as id, parse_json('{"key":"v1"}') as v,
+                       | named_struct('inner', parse_json('{"key":"n1"}')) as s, 1000L as ts""".stripMargin, compact = false)
+      addVariantData("""select 2 as id, parse_json('{"key":"v2"}') as v,
+                       | named_struct('inner', parse_json('{"key":"n2"}')) as s, 1000L as ts""".stripMargin, compact = true)
+      // Pin that the compacted base file is shredded at both depths: without it the streams below
+      // would pass just the same over an unshredded base and pin nothing about shredded reads.
+      val conf = spark.sessionState.newHadoopConf()
+      val baseFiles = new Path(tablePath).getFileSystem(conf).listStatus(new Path(tablePath))

Review Comment:
   A native parquet log file is named `<fileId>_<token>_<instant>_<v>.log.parquet` with no dot prefix, so this listing also picks up the two deltacommit logs and "expected a compacted base file" is satisfied even if inline compaction never ran. Filter with `FSUtils.isBaseFile` the way `assertNestedBaseLayout` does.



##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieMergeOnReadRDDV2.scala:
##########
@@ -149,13 +149,14 @@ class HoodieMergeOnReadRDDV2(@transient sc: SparkContext,
     }
   }
 
-  // The plain skip-merging reader cannot read a SHREDDED variant base file: it requests native
-  // VariantType, which clips the shredded group to {metadata, value} and reads value=null (the
-  // #19556 defect family). Such splits take the file-group reader below, whose reader context
-  // requests the full-variant projection shape instead (#19578). Keyed off the adapter building
-  // that shape rather than the mere presence of a variant column: it is None below Spark 4.1,
-  // where the file-group reader would read the same nulls, so re-routing there would cost the
-  // fast path for nothing.
+  // A split whose required schema has a top-level variant column takes the file-group reader
+  // below, whose reader context requests the full-variant projection shape for parquet base
+  // files (#19578), so a SHREDDED base file is read on this legacy path through the same
+  // contract as everywhere else. The skip-merging reader's native VariantType request is
+  // reconstructed by the Spark 4.1+ row reader as well (pinned by TestStreamingSource), so this

Review Comment:
   TestStreamingSource's table also carries a top-level `v`, so `shouldRerouteVariantSplit` is true there and the base-only split never reaches `requiredSchemaReaderSkipMerging` - that leg is not what the test pins. Drop the reference here and in the matching "Not swept here" note in TestVariantShreddingMixedLayouts, or add a nested-only leg that actually takes the branch?



##########
hudi-spark-datasource/hudi-spark4-common/src/main/scala/org/apache/spark/sql/adapter/BaseSpark4Adapter.scala:
##########
@@ -320,6 +324,89 @@ abstract class BaseSpark4Adapter extends SparkAdapter with Logging {
     if (rewritten) Some(StructType(fields)) else None
   }
 
+  /**
+   * Shared implementation behind [[SparkAdapter#buildVariantProjector]] for the 4.x adapters
+   * whose planner rewrites variants into projection structs (4.1+).
+   *
+   * Recurses into struct members, mirroring PushVariantIntoScan's `VariantInRelation.rewriteType`:
+   * a variant is rewritten at the root of the relation output or below a STRUCT path, while
+   * arrays and maps keep their native VariantType, so nothing under a collection is projected
+   * here either. Before #19775 this walked top-level fields only, and a projection struct sitting
+   * one struct member down was left holding a raw variant that the plan then read as its
+   * projected children.
+   */
+  protected final def buildVariantProjectorForStructPaths(
+      sparkDataSchema: StructType,
+      sparkRequiredSchema: StructType): Option[InternalRow => InternalRow] = {
+    // Quick check: does any required field carry a variant projection struct, at any depth?
+    if (!sparkRequiredSchema.fields.exists(f => containsVariantProjection(f.dataType))) {
+      None
+    } else {
+      // Surface mismatched schemas with both field lists rather than Spark's bare
+      // IllegalArgumentException from fieldIndex. `path` is the dotted field path of `name`.
+      def lookupDataField(dataStruct: StructType, requiredStruct: StructType,
+                          name: String, path: String): (Int, StructField) = {
+        val idx = dataStruct.getFieldIndex(name).getOrElse(
+          throw new IllegalStateException(
+            s"Required field '$path' is absent from sparkDataSchema; " +
+              s"required=${requiredStruct.fieldNames.mkString("[", ",", "]")}, " +
+              s"data=${dataStruct.fieldNames.mkString("[", ",", "]")}"))
+        (idx, dataStruct.fields(idx))
+      }
+
+      // `ref` reads the data-schema value of type `dataType`; the result has type `requiredType`.
+      def projectionExpr(ref: Expression, dataType: DataType, requiredType: DataType,
+                         path: String): Expression = requiredType match {
+        case projectedStruct: StructType if VariantMetadata.isVariantStruct(projectedStruct) =>
+          require(isVariantType(dataType),
+            s"Expected VariantType for field '$path' in data schema, got $dataType")
+          val childExprs: Seq[Expression] = projectedStruct.fields.toSeq.flatMap { child =>
+            val vm = VariantMetadata.fromMetadata(child.metadata)
+            val pathLit = Literal(UTF8String.fromString(vm.path), StringType)
+            val variantGet: Expression =
+              VariantGet(ref, pathLit, child.dataType, vm.failOnError, Option(vm.timeZoneId))
+            Seq(Literal(UTF8String.fromString(child.name), StringType), variantGet)
+          }
+          CreateNamedStruct(childExprs)
+        case requiredStruct: StructType =>
+          dataType match {
+            // Rebuild the struct member by member only when something below it is projected;
+            // otherwise the reference is already in the required shape and is cheaper untouched.
+            case dataStruct: StructType if containsVariantProjection(requiredStruct) =>
+              val childExprs: Seq[Expression] = requiredStruct.fields.toSeq.flatMap { rf =>
+                val childPath = s"$path.${rf.name}"
+                val (childIdx, childField) = lookupDataField(dataStruct, requiredStruct, rf.name, childPath)
+                val childRef = GetStructField(ref, childIdx, Some(rf.name))
+                Seq(Literal(UTF8String.fromString(rf.name), StringType),
+                  projectionExpr(childRef, childField.dataType, rf.dataType, childPath))
+              }
+              val rebuilt = CreateNamedStruct(childExprs)
+              // CreateNamedStruct is never null, so a null struct would come back as a struct of
+              // nulls without this guard.
+              If(IsNull(ref), Literal(null, rebuilt.dataType), rebuilt)
+            case _ => ref
+          }
+        case _ => ref
+      }
+
+      val exprs: Array[Expression] = sparkRequiredSchema.fields.map { rf =>
+        val (dataIdx, dataField) = lookupDataField(sparkDataSchema, sparkRequiredSchema, rf.name, rf.name)
+        val ref: Expression = BoundReference(dataIdx, dataField.dataType, dataField.nullable)
+        projectionExpr(ref, dataField.dataType, rf.dataType, rf.name)
+      }
+
+      val projection = UnsafeProjection.create(exprs.toIndexedSeq, DataTypeUtils.toAttributes(sparkDataSchema))
+      Some(row => projection(row))
+    }
+  }
+
+  /** True when `dataType` is a variant projection struct or holds one below a struct path. */
+  private def containsVariantProjection(dataType: DataType): Boolean = dataType match {

Review Comment:
   `containsVariantProjection` is now defined identically here and in `SparkFileFormatInternalRowReaderContext`. `SparkAdapter` already hosts `isVariantProjectionStruct`, so a default method there would give both call sites one definition.



##########
hudi-spark-datasource/hudi-spark4-common/src/main/scala/org/apache/spark/sql/adapter/BaseSpark4Adapter.scala:
##########
@@ -320,6 +324,89 @@ abstract class BaseSpark4Adapter extends SparkAdapter with Logging {
     if (rewritten) Some(StructType(fields)) else None
   }
 
+  /**
+   * Shared implementation behind [[SparkAdapter#buildVariantProjector]] for the 4.x adapters
+   * whose planner rewrites variants into projection structs (4.1+).
+   *
+   * Recurses into struct members, mirroring PushVariantIntoScan's `VariantInRelation.rewriteType`:
+   * a variant is rewritten at the root of the relation output or below a STRUCT path, while
+   * arrays and maps keep their native VariantType, so nothing under a collection is projected
+   * here either. Before #19775 this walked top-level fields only, and a projection struct sitting
+   * one struct member down was left holding a raw variant that the plan then read as its
+   * projected children.
+   */
+  protected final def buildVariantProjectorForStructPaths(
+      sparkDataSchema: StructType,
+      sparkRequiredSchema: StructType): Option[InternalRow => InternalRow] = {
+    // Quick check: does any required field carry a variant projection struct, at any depth?
+    if (!sparkRequiredSchema.fields.exists(f => containsVariantProjection(f.dataType))) {
+      None
+    } else {
+      // Surface mismatched schemas with both field lists rather than Spark's bare
+      // IllegalArgumentException from fieldIndex. `path` is the dotted field path of `name`.
+      def lookupDataField(dataStruct: StructType, requiredStruct: StructType,
+                          name: String, path: String): (Int, StructField) = {
+        val idx = dataStruct.getFieldIndex(name).getOrElse(
+          throw new IllegalStateException(
+            s"Required field '$path' is absent from sparkDataSchema; " +
+              s"required=${requiredStruct.fieldNames.mkString("[", ",", "]")}, " +
+              s"data=${dataStruct.fieldNames.mkString("[", ",", "]")}"))
+        (idx, dataStruct.fields(idx))
+      }
+
+      // `ref` reads the data-schema value of type `dataType`; the result has type `requiredType`.
+      def projectionExpr(ref: Expression, dataType: DataType, requiredType: DataType,
+                         path: String): Expression = requiredType match {
+        case projectedStruct: StructType if VariantMetadata.isVariantStruct(projectedStruct) =>
+          require(isVariantType(dataType),
+            s"Expected VariantType for field '$path' in data schema, got $dataType")
+          val childExprs: Seq[Expression] = projectedStruct.fields.toSeq.flatMap { child =>
+            val vm = VariantMetadata.fromMetadata(child.metadata)
+            val pathLit = Literal(UTF8String.fromString(vm.path), StringType)
+            val variantGet: Expression =
+              VariantGet(ref, pathLit, child.dataType, vm.failOnError, Option(vm.timeZoneId))
+            Seq(Literal(UTF8String.fromString(child.name), StringType), variantGet)
+          }
+          CreateNamedStruct(childExprs)

Review Comment:
   A null variant in an avro log record is projected into a non-null struct of nulls here, while the parquet paths leave the field null - and `PushVariantIntoScan` rewrites `IsNull(v)` / `IsNotNull(v)` directly onto that struct. Wrap this in the same `If(IsNull(...))` guard the struct rebuild below already carries?



##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/schema/TestVariantShreddingMixedLayouts.scala:
##########
@@ -751,35 +806,519 @@ class TestVariantShreddingMixedLayouts extends HoodieSparkSqlTestBase with Varia
       assert(getFieldAsGroup(innerGroup, "typed_value").containsField("k"),
         s"[$leg] nested typed_value should carry k:\n$innerGroup")
 
-      // #18605 history: the batch-disabling guards in HoodieFileGroupReaderBasedFileFormat are
-      // top-level-only, so a NESTED variant still reaches the vectorized reader. The session conf
-      // does pick the reader - HoodieFileGroupReaderBasedFileFormat.supportBatch reads
-      // sparkSession.sessionState.conf and ParquetUtils.isBatchReadSupportedForSchema gates on
-      // spark.sql.parquet.enableVectorizedReader, and only afterwards does
-      // buildReaderWithPartitionValues write that decision back into the conf - so sweep it and
-      // pin both readers. Every nested-variant read bug so far (HUDI-7190, HUDI-8803, #18605) is
-      // vectorized-only, which leaves the row-based leg as the control.
-      Seq("true", "false").foreach { vectorizedReader =>
-        withSQLConf("spark.sql.parquet.enableVectorizedReader" -> vectorizedReader) {
-          checkAnswer(s"select id, cast(s.inner as string) from $tableName")(
-            Seq(1, """{"k":"x1"}""")
-          )
-        }
-      }
+      // One read, whatever spark.sql.parquet.enableVectorizedReader says: supportBatch vetoes
+      // batch reads for a variant at any depth before that conf is consulted, and section F2 pins
+      // that decision directly.
+      checkAnswer(s"select id, cast(s.inner as string) from $tableName")(
+        Seq(1, """{"k":"x1"}""")
+      )
 
       // A plain insert bin-packs into the same file group: the small-file merge must read the
       // nested-shredded base back (nested reconstruction on the AVRO record type leg).
       withWriteLayout(Forced("k string")) {
         spark.sql(s"""insert into $tableName values (2, named_struct('inner', parse_json('{"k":"x2"}')), 1000)""")
       }
       assertSingleFileGroup(tablePath, leg)
+      // The merge rewrote the whole file group under the INCOMING layout, so the layout below is
+      // the record writer's, not the row writer's seed. On the AVRO leg that write goes through
+      // HoodieAvroWriteSupport, whose forced hook now reaches the nested member; before the #19689
+      // fix it silently rewrote s.inner unshredded and nothing here noticed. It pins the
+      // small-file MERGE handle's write, while the record-writer test below pins a fresh INSERT
+      // handle, so the two are not interchangeable.
+      assertVariantLayout(tablePath, shredded = true, leg, column = "s.inner")
       checkAnswer(s"select id, cast(s.inner as string) from $tableName order by id")(
         Seq(1, """{"k":"x1"}"""),
         Seq(2, """{"k":"x2"}""")
       )
     }
   }
 
+  test("Both record types force-shred a nested variant through the record writer") {
+    assume(HoodieSparkUtils.gteqSpark4_1, SPARK_4_1_GATE)
+
+    // A plain insert, so the write goes through the record writers rather than the row writer: the
+    // AVRO leg is HoodieAvroWriteSupport end to end, the SPARK leg HoodieRowParquetWriteSupport,
+    // and the #19689 fix is what made their forced hooks agree. The DDL reaches a variant that is
+    // a record MEMBER at any depth - s.inner, and the inner of the struct element of items - but
+    // never one that is directly a collection element, which is why arr stays unshredded on both
+    // paths.
+    withVariantTable("record-writer nested forced shredding", "cow",
+      extraCols = "s struct<inner: variant>, items array<struct<inner: variant>>, arr array<variant>") {
+      (tableName, tablePath, leg) =>
+      withWriteLayout(Forced("k string")) {
+        spark.sql(
+          s"""insert into $tableName values (1, parse_json('{"k":"top"}'),
+             | named_struct('inner', parse_json('{"k":"nested"}')),
+             | array(named_struct('inner', parse_json('{"k":"element"}'))),
+             | array(parse_json('{"k":"bare"}')), 1000)""".stripMargin)
+      }
+
+      val files = listDataParquetFiles(tablePath)
+      assert(files.size == 1, s"[$leg] expected one base file, got $files")
+      Seq("s.inner", "items.inner").foreach { column =>
+        assertVariantLayout(tablePath, shredded = true, leg, column = column)
+        val group = variantGroupOf(files.head, column)
+        assert(getFieldAsGroup(group, "typed_value").containsField("k"),
+          s"[$leg] typed_value of $column should carry k:\n$group")
+      }
+      assertVariantLayout(tablePath, shredded = false, leg, column = "arr")
+
+      checkAnswer(s"select id, cast(v as string), cast(s.inner as string), " +
+        s"cast(items[0].inner as string), cast(arr[0] as string) from $tableName")(
+        Seq(1, """{"k":"top"}""", """{"k":"nested"}""", """{"k":"element"}""", """{"k":"bare"}""")
+      )
+    }
+  }
+
+  test("MOR merge, compaction and clustering carry a nested-shredded base through the internal reader") {
+    assume(HoodieSparkUtils.gteqSpark4_1, SPARK_4_1_GATE)
+
+    // The nested variant is the ONLY variant in the table, so no top-level rewrite carries it (see
+    // withNestedOnlyVariantTable): every read below has to resolve s.inner on its own - through the
+    // nested projection struct PushVariantIntoScan pushes into the scan for the user queries, and
+    // natively for the internal reads that have no catalyst schema (compaction, clustering, the
+    // legacy RDD) - which is what makes these paths say anything about nested shredding at all.
+    // The projected arm crashed the JVM before #19775 (the projection was applied at the top level
+    // only, so the merged row still held a raw variant where the plan read a struct).
+    val mergedRows = Seq(
+      Seq(1, """{"k":"n1"}"""), Seq(2, """{"k":"n2b"}"""), Seq(3, """{"k":"n3"}"""))
+    val finalRows = Seq(
+      Seq(1, """{"k":"n1"}"""), Seq(2, """{"k":"n2b"}"""), Seq(3, """{"k":"n3c"}"""))
+
+    Seq(true, false).foreach { rowWriter =>
+      // No INMEMORY index: the first insert creates a base file, the updates go to log files.
+      withNestedOnlyVariantTable(s"mor nested rowWriter=$rowWriter", "mor",
+        props = Seq("hoodie.compact.inline = 'false'"),
+        recordTypes = clusteringRecordTypes(rowWriter)) { (tableName, tablePath, leg) =>
+        val snapshotQuery = s"select id, cast(s.inner as string) from $tableName order by id"
+        val readOptimizedQuery = s"select id, cast(s.inner as string) from " +
+          s"hudi_query('$tableName', 'read_optimized') order by id"
+
+        withWriteLayout(Forced("k string")) {
+          spark.sql(s"insert into $tableName values " +
+            """(1, named_struct('inner', parse_json('{"k":"n1"}')), 1000), """ +
+            """(2, named_struct('inner', parse_json('{"k":"n2"}')), 1000), """ +
+            """(3, named_struct('inner', parse_json('{"k":"n3"}')), 1000)""")
+        }
+        val baseFiles = listDataParquetFiles(tablePath)
+        assert(baseFiles.size == 1, s"[$leg] expected one base file, got $baseFiles")
+        assertVariantLayout(tablePath, shredded = true, leg, column = "s.inner")
+        val innerGroup = variantGroupOf(baseFiles.head, "s.inner")
+        assert(getFieldAsGroup(innerGroup, "typed_value").containsField("k"),
+          s"[$leg] nested typed_value should carry k:\n$innerGroup")
+
+        // A nested-shredded native log on top of the nested-shredded base.
+        withWriteLayout(Forced("k string")) {
+          spark.sql(s"update $tableName set " +
+            """s = named_struct('inner', parse_json('{"k":"n2b"}')), ts = 1001 where id = 2""")
+        }
+        assert(listDataParquetFiles(tablePath).exists(_.endsWith(".log.parquet")),
+          s"[$leg] the update should have written a native parquet log file")
+
+        // The conf only says what the reader MAY do - supportBatch vetoes vectorization for a
+        // variant at any depth (pinned in F2) - so the sweep is the control if that guard is ever
+        // narrowed back to top-level columns.
+        Seq("true", "false").foreach { vectorizedReader =>
+          withSQLConf("spark.sql.parquet.enableVectorizedReader" -> vectorizedReader) {
+            checkAnswer(snapshotQuery)(mergedRows: _*)
+          }
+        }
+
+        // Not swept here: hoodie.file.group.reader.enabled=false, which no longer routes a batch
+        // read anywhere (only the streaming sources consult it). The legacy RDD path over a
+        // nested-shredded base - HoodieMergeOnReadRDDV2, whose shouldRerouteVariantSplit stays
+        // false without a top-level variant - is pinned by TestStreamingSource's legacy leg.
+
+        // Read-optimized serves the base file alone: id 2 is still the pre-update value.
+        checkAnswer(readOptimizedQuery)(
+          Seq(1, """{"k":"n1"}"""), Seq(2, """{"k":"n2"}"""), Seq(3, """{"k":"n3"}"""))
+
+        // Compaction merges the nested-shredded log onto the nested-shredded base and re-derives
+        // the layout from the forced DDL. On the AVRO record type that write goes through
+        // HoodieAvroWriteSupport, whose nested forced hook is #19689's parity fix.
+        withWriteLayout(Forced("k string")) {
+          runCompaction(tableName)
+        }
+        assertCompactionCount(tablePath, 1, leg)
+        assertNestedBaseLayout(tablePath, latestCompletedInstant(tablePath), shredded = true, leg)
+        checkAnswer(snapshotQuery)(mergedRows: _*)
+
+        // Unshredded round: the update and the compaction both run with shredding off, so
+        // typed_value has to be stripped at depth on the way out.
+        withWriteLayout(Unshredded) {
+          spark.sql(s"update $tableName set " +
+            """s = named_struct('inner', parse_json('{"k":"n3c"}')), ts = 1002 where id = 3""")
+          runCompaction(tableName)
+        }
+        assertCompactionCount(tablePath, 2, leg)
+        assertNestedBaseLayout(tablePath, latestCompletedInstant(tablePath), shredded = false, leg)
+        checkAnswer(snapshotQuery)(finalRows: _*)
+
+        // Clustering re-derives the nested layout from the forced DDL over that unshredded input:
+        // the row-writer path when rowWriter is true, the record writers otherwise.
+        withWriteLayout(Forced("k string")) {
+          runClustering(tableName, rowWriter)
+        }
+        val clusteringInstant = completedClusteringInstant(tablePath, leg)
+        assertNestedBaseLayout(tablePath, clusteringInstant, shredded = true, leg)
+        checkAnswer(snapshotQuery)(finalRows: _*)
+        checkAnswer(readOptimizedQuery)(finalRows: _*)
+      }
+    }
+  }
+
+  test("CDC images carry a nested-shredded variant") {
+    assume(HoodieSparkUtils.gteqSpark4_1, SPARK_4_1_GATE)
+
+    // The nested twin of TestVariantDataType's CDC test. OP_KEY_ONLY reconstructs both images by
+    // reading the file slices; DATA_BEFORE_AFTER (the default) reads the update images from the cdc
+    // log instead. The insert leg takes BASE_FILE_INSERT in both modes, which reads the new base
+    // file directly rather than through the reader context.
+    Seq("OP_KEY_ONLY", "DATA_BEFORE_AFTER").foreach { loggingMode =>
+      // SPARK pinned: a cdc-enabled table always writes through FileGroupReaderBasedMergeHandle and
+      // the merger's record type picks that handle's reader context; AVRO would route the update's
+      // base-file read through HoodieAvroParquetReader, the separate defect tracked as #19567.
+      withNestedOnlyVariantTable(s"cdc nested $loggingMode", "cow", props = Seq(
+        "'hoodie.table.cdc.enabled' = 'true'",
+        s"'hoodie.table.cdc.supplemental.logging.mode' = '$loggingMode'",
+        "hoodie.index.type = 'INMEMORY'"),
+        recordTypes = Seq(HoodieRecordType.SPARK)) { (tableName, tablePath, leg) =>
+        withWriteLayout(Forced("k string")) {
+          spark.sql(s"insert into $tableName values " +
+            """(1, named_struct('inner', parse_json('{"k":"c1"}')), 1000)""")
+        }
+        assertVariantLayout(tablePath, shredded = true, leg, column = "s.inner")
+
+        withWriteLayout(Forced("k string")) {
+          spark.sql(s"update $tableName set " +
+            """s = named_struct('inner', parse_json('{"k":"c2"}')), ts = 1001 where id = 1""")
+        }
+        // Layout flip: the second update rewrites the file unshredded, so the images below span
+        // both physical slots.
+        withWriteLayout(Unshredded) {
+          spark.sql(s"update $tableName set " +
+            """s = named_struct('inner', parse_json('{"k":"c3"}')), ts = 1002 where id = 1""")
+        }
+        assertNestedBaseLayout(tablePath, latestCompletedInstant(tablePath), shredded = false, leg)
+
+        val cdc = spark.sql(s"select op, get_json_object(before, '$$.s.inner.k') as before_k, " +
+          s"get_json_object(after, '$$.s.inner.k') as after_k " +
+          s"from hudi_table_changes('$tableName', 'cdc', 'earliest')")
+        val insertRows = cdc.where("op = 'i'").collect()
+        assert(insertRows.length == 1, s"[$leg] expected exactly one insert cdc row")
+        assert(insertRows(0).getString(2) == "c1",
+          s"[$leg] insert after-image lost the nested variant payload: ${insertRows(0)}")
+
+        val updateRows = cdc.where("op = 'u'").orderBy("after_k").collect()
+        assert(updateRows.length == 2, s"[$leg] expected two update cdc rows")
+        assert(updateRows(0).getString(1) == "c1" && updateRows(0).getString(2) == "c2",
+          s"[$leg] first update images lost the nested variant payload: ${updateRows(0)}")
+        assert(updateRows(1).getString(1) == "c2" && updateRows(1).getString(2) == "c3",
+          s"[$leg] second (layout-flipped) update images lost the nested variant payload: ${updateRows(1)}")
+      }
+    }
+  }
+
+  test("variant_get projections and filters resolve a nested-shredded variant") {
+    assume(HoodieSparkUtils.gteqSpark4_1, SPARK_4_1_GATE)
+
+    // The nested twin of the mixed-layout variant_get test above; SPARK pinned for the same reason.
+    // pushVariantIntoScan is swept because the two arms reach the file differently: on, Spark
+    // rewrites the s.inner struct path into its own projection struct and pushes it into the scan;
+    // off, the whole variant is read and variant_get evaluates on top of it.
+    def nestedRowsSql(lo: Int, hi: Int): String =
+      s"""select cast(id as int) as id,
+         | named_struct('inner', parse_json(concat('{"k":"x', id, '"}'))) as s,
+         | 1000L as ts from range($lo, $hi, 1, 1)""".stripMargin
+
+    Seq("true", "false").foreach { pushIntoScan =>
+      withSQLConf("spark.sql.variant.pushVariantIntoScan" -> pushIntoScan) {

Review Comment:
   Nothing in these legs asserts that `PushVariantIntoScan` actually fired, and both arms expect the same rows, so if the rule ever stops matching `HoodieFileGroupReaderBasedFileFormat` the projected arm silently becomes a duplicate of the other one. Worth pinning the plan in the true arm the way `assertLegacyRddPlan` pins its own path in TestStreamingSource - follow-up, not a blocker.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(variant): project PushVariantIntoScan struct paths in the ... [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19783:
URL: https://github.com/apache/hudi/pull/19783#issuecomment-5452793505

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e268d40c1df977ced91f3197fa97fef1d0473ac0",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16667",
       "triggerID" : "e268d40c1df977ced91f3197fa97fef1d0473ac0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "115b202afb5a10df2382ea1644f37a97ff2b933e",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16671",
       "triggerID" : "115b202afb5a10df2382ea1644f37a97ff2b933e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a51b75e41a66ba8982da4aff12edd236dbfe692a",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a51b75e41a66ba8982da4aff12edd236dbfe692a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e268d40c1df977ced91f3197fa97fef1d0473ac0 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16667) 
   * 115b202afb5a10df2382ea1644f37a97ff2b933e Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16671) 
   * a51b75e41a66ba8982da4aff12edd236dbfe692a UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(variant): project PushVariantIntoScan struct paths in the ... [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19783:
URL: https://github.com/apache/hudi/pull/19783#issuecomment-5490570095

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e268d40c1df977ced91f3197fa97fef1d0473ac0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16667",
       "triggerID" : "e268d40c1df977ced91f3197fa97fef1d0473ac0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "115b202afb5a10df2382ea1644f37a97ff2b933e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16671",
       "triggerID" : "115b202afb5a10df2382ea1644f37a97ff2b933e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a51b75e41a66ba8982da4aff12edd236dbfe692a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16673",
       "triggerID" : "a51b75e41a66ba8982da4aff12edd236dbfe692a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4e1524d2a0e3e7ea61348dc237e336bf9ff7e19e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16714",
       "triggerID" : "4e1524d2a0e3e7ea61348dc237e336bf9ff7e19e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6b4ffe79db9a996c398df1e4888f93a9937cb98e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16717",
       "triggerID" : "6b4ffe79db9a996c398df1e4888f93a9937cb98e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2ea92de1fcf8cd8a83abe3968d79a303d21d3140",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16729",
       "triggerID" : "2ea92de1fcf8cd8a83abe3968d79a303d21d3140",
       "triggerType" : "PUSH"
     }, {
       "hash" : "42b6ff809b4af335e5becdafa97de014912d3fd6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "42b6ff809b4af335e5becdafa97de014912d3fd6",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 2ea92de1fcf8cd8a83abe3968d79a303d21d3140 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16729) 
   * 42b6ff809b4af335e5becdafa97de014912d3fd6 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(variant): project PushVariantIntoScan struct paths in the ... [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19783:
URL: https://github.com/apache/hudi/pull/19783#issuecomment-5451888155

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e268d40c1df977ced91f3197fa97fef1d0473ac0",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e268d40c1df977ced91f3197fa97fef1d0473ac0",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e268d40c1df977ced91f3197fa97fef1d0473ac0 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(variant): project PushVariantIntoScan struct paths in the ... [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19783:
URL: https://github.com/apache/hudi/pull/19783#issuecomment-5453163004

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e268d40c1df977ced91f3197fa97fef1d0473ac0",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16667",
       "triggerID" : "e268d40c1df977ced91f3197fa97fef1d0473ac0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "115b202afb5a10df2382ea1644f37a97ff2b933e",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16671",
       "triggerID" : "115b202afb5a10df2382ea1644f37a97ff2b933e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a51b75e41a66ba8982da4aff12edd236dbfe692a",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16673",
       "triggerID" : "a51b75e41a66ba8982da4aff12edd236dbfe692a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e268d40c1df977ced91f3197fa97fef1d0473ac0 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16667) 
   * 115b202afb5a10df2382ea1644f37a97ff2b933e Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16671) 
   * a51b75e41a66ba8982da4aff12edd236dbfe692a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16673) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(variant): project PushVariantIntoScan struct paths in the ... [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19783:
URL: https://github.com/apache/hudi/pull/19783#issuecomment-5476156410

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e268d40c1df977ced91f3197fa97fef1d0473ac0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16667",
       "triggerID" : "e268d40c1df977ced91f3197fa97fef1d0473ac0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "115b202afb5a10df2382ea1644f37a97ff2b933e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16671",
       "triggerID" : "115b202afb5a10df2382ea1644f37a97ff2b933e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a51b75e41a66ba8982da4aff12edd236dbfe692a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16673",
       "triggerID" : "a51b75e41a66ba8982da4aff12edd236dbfe692a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4e1524d2a0e3e7ea61348dc237e336bf9ff7e19e",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16714",
       "triggerID" : "4e1524d2a0e3e7ea61348dc237e336bf9ff7e19e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 4e1524d2a0e3e7ea61348dc237e336bf9ff7e19e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16714) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(variant): project PushVariantIntoScan struct paths in the ... [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19783:
URL: https://github.com/apache/hudi/pull/19783#issuecomment-5477734274

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e268d40c1df977ced91f3197fa97fef1d0473ac0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16667",
       "triggerID" : "e268d40c1df977ced91f3197fa97fef1d0473ac0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "115b202afb5a10df2382ea1644f37a97ff2b933e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16671",
       "triggerID" : "115b202afb5a10df2382ea1644f37a97ff2b933e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a51b75e41a66ba8982da4aff12edd236dbfe692a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16673",
       "triggerID" : "a51b75e41a66ba8982da4aff12edd236dbfe692a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4e1524d2a0e3e7ea61348dc237e336bf9ff7e19e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16714",
       "triggerID" : "4e1524d2a0e3e7ea61348dc237e336bf9ff7e19e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6b4ffe79db9a996c398df1e4888f93a9937cb98e",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16717",
       "triggerID" : "6b4ffe79db9a996c398df1e4888f93a9937cb98e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 6b4ffe79db9a996c398df1e4888f93a9937cb98e Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16717) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(variant): project PushVariantIntoScan struct paths in the ... [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19783:
URL: https://github.com/apache/hudi/pull/19783#issuecomment-5488123402

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e268d40c1df977ced91f3197fa97fef1d0473ac0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16667",
       "triggerID" : "e268d40c1df977ced91f3197fa97fef1d0473ac0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "115b202afb5a10df2382ea1644f37a97ff2b933e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16671",
       "triggerID" : "115b202afb5a10df2382ea1644f37a97ff2b933e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a51b75e41a66ba8982da4aff12edd236dbfe692a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16673",
       "triggerID" : "a51b75e41a66ba8982da4aff12edd236dbfe692a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4e1524d2a0e3e7ea61348dc237e336bf9ff7e19e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16714",
       "triggerID" : "4e1524d2a0e3e7ea61348dc237e336bf9ff7e19e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6b4ffe79db9a996c398df1e4888f93a9937cb98e",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16717",
       "triggerID" : "6b4ffe79db9a996c398df1e4888f93a9937cb98e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2ea92de1fcf8cd8a83abe3968d79a303d21d3140",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2ea92de1fcf8cd8a83abe3968d79a303d21d3140",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 6b4ffe79db9a996c398df1e4888f93a9937cb98e Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16717) 
   * 2ea92de1fcf8cd8a83abe3968d79a303d21d3140 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(variant): project PushVariantIntoScan struct paths in the ... [hudi]

Posted by "wombatu-kun (via GitHub)" <gi...@apache.org>.
wombatu-kun commented on code in PR #19783:
URL: https://github.com/apache/hudi/pull/19783#discussion_r3900146754


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieMergeOnReadRDDV2.scala:
##########
@@ -149,13 +149,15 @@ class HoodieMergeOnReadRDDV2(@transient sc: SparkContext,
     }
   }
 
-  // The plain skip-merging reader cannot read a SHREDDED variant base file: it requests native
-  // VariantType, which clips the shredded group to {metadata, value} and reads value=null (the
-  // #19556 defect family). Such splits take the file-group reader below, whose reader context
-  // requests the full-variant projection shape instead (#19578). Keyed off the adapter building
-  // that shape rather than the mere presence of a variant column: it is None below Spark 4.1,
-  // where the file-group reader would read the same nulls, so re-routing there would cost the
-  // fast path for nothing.
+  // A split whose required schema has a top-level variant column takes the file-group reader
+  // below, whose reader context requests the full-variant projection shape for parquet base
+  // files (#19578), so a SHREDDED base file is read on this legacy path through the same
+  // contract as everywhere else. Without a top-level variant the base-only split stays on
+  // requiredSchemaReaderSkipMerging, whose native VariantType request the Spark 4.1+ row reader

Review Comment:
   `Spark41LegacyHoodieParquetFileFormat` extends `ParquetFileFormat` without overriding `supportBatch`, and `VariantType` is an `AtomicType`, so this split reads through the vectorized reader at stock settings - the variant veto lives only in `HoodieFileGroupReaderBasedFileFormat`. Name the parquet reader rather than the row reader here and in `TestStreamingSource`'s scaladoc?



##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/schema/TestVariantDataType.scala:
##########
@@ -22,13 +22,11 @@ package org.apache.spark.sql.hudi.dml.schema
 import org.apache.hudi.{DataSourceReadOptions, HoodieSparkUtils}
 import org.apache.hudi.common.avro.VariantShreddingRuntime
 import org.apache.hudi.common.fs.FSUtils
-import org.apache.hudi.common.model.HoodieLogFile
 import org.apache.hudi.common.model.HoodieRecord.HoodieRecordType
 import org.apache.hudi.common.model.WriteOperationType
 import org.apache.hudi.common.schema.HoodieSchema
 import org.apache.hudi.common.schema.internal.HoodieSchemaException
 import org.apache.hudi.common.table.TableSchemaResolver
-import org.apache.hudi.common.table.log.HoodieLogFormat
 import org.apache.hudi.common.table.log.block.HoodieLogBlock.HoodieLogBlockType
 import org.apache.hudi.common.testutils.HoodieTestUtils
 import org.apache.hudi.common.util.StringUtils

Review Comment:
   Moving `listLogBlockTypes` to `VariantShreddingTestSupport` left `scala.collection.JavaConverters._` and `getMetaClientAndFileSystemView` imported here with no remaining use. Drop both.



##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/schema/TestVariantShreddingMixedLayouts.scala:
##########
@@ -818,6 +871,474 @@ class TestVariantShreddingMixedLayouts extends HoodieSparkSqlTestBase with Varia
     }
   }
 
+  test("MOR merge, compaction and clustering carry a nested-shredded base through the internal reader") {
+    assume(HoodieSparkUtils.gteqSpark4_1, SPARK_4_1_GATE)
+
+    // The nested variant is the ONLY variant in the table, so no top-level rewrite carries it (see
+    // withNestedOnlyVariantTable): every read below has to resolve s.inner on its own - through the
+    // nested projection struct PushVariantIntoScan pushes into the scan for the user queries, and
+    // natively for the internal reads that have no catalyst schema (compaction, clustering, the
+    // legacy RDD) - which is what makes these paths say anything about nested shredding at all.
+    // The projected arm crashed the JVM before #19775 (the projection was applied at the top level
+    // only, so the merged row still held a raw variant where the plan read a struct).
+    val mergedRows = Seq(
+      Seq(1, """{"k":"n1"}"""), Seq(2, """{"k":"n2b"}"""), Seq(3, """{"k":"n3"}"""))
+    val finalRows = Seq(
+      Seq(1, """{"k":"n1"}"""), Seq(2, """{"k":"n2b"}"""), Seq(3, """{"k":"n3c"}"""))
+
+    Seq(true, false).foreach { rowWriter =>
+      // No INMEMORY index: the first insert creates a base file, the updates go to log files.
+      withNestedOnlyVariantTable(s"mor nested rowWriter=$rowWriter", "mor",
+        props = Seq("hoodie.compact.inline = 'false'"),
+        recordTypes = clusteringRecordTypes(rowWriter)) { (tableName, tablePath, leg) =>
+        val snapshotQuery = s"select id, cast(s.inner as string) from $tableName order by id"
+        val readOptimizedQuery = s"select id, cast(s.inner as string) from " +
+          s"hudi_query('$tableName', 'read_optimized') order by id"
+
+        withWriteLayout(Forced("k string")) {
+          spark.sql(s"insert into $tableName values " +
+            """(1, named_struct('inner', parse_json('{"k":"n1"}')), 1000), """ +
+            """(2, named_struct('inner', parse_json('{"k":"n2"}')), 1000), """ +
+            """(3, named_struct('inner', parse_json('{"k":"n3"}')), 1000)""")
+        }
+        val baseFiles = listDataParquetFiles(tablePath)
+        assert(baseFiles.size == 1, s"[$leg] expected one base file, got $baseFiles")
+        assertVariantLayout(tablePath, shredded = true, leg, column = "s.inner")
+        val innerGroup = variantGroupOf(baseFiles.head, "s.inner")
+        assert(getFieldAsGroup(innerGroup, "typed_value").containsField("k"),
+          s"[$leg] nested typed_value should carry k:\n$innerGroup")
+
+        // A nested-shredded native log on top of the nested-shredded base.
+        withWriteLayout(Forced("k string")) {
+          spark.sql(s"update $tableName set " +
+            """s = named_struct('inner', parse_json('{"k":"n2b"}')), ts = 1001 where id = 2""")
+        }
+        assert(listDataParquetFiles(tablePath).exists(_.endsWith(".log.parquet")),
+          s"[$leg] the update should have written a native parquet log file")
+
+        // The conf only says what the reader MAY do - supportBatch vetoes vectorization for a
+        // variant at any depth (pinned in F2) - so the sweep is the control if that guard is ever
+        // narrowed back to top-level columns.
+        Seq("true", "false").foreach { vectorizedReader =>
+          withSQLConf("spark.sql.parquet.enableVectorizedReader" -> vectorizedReader) {
+            checkAnswer(snapshotQuery)(mergedRows: _*)
+          }
+        }
+
+        // Not swept here: hoodie.file.group.reader.enabled=false, which no longer routes a batch
+        // read anywhere (only the streaming sources consult it). The legacy RDD path over a
+        // nested-shredded base - HoodieMergeOnReadRDDV2, whose shouldRerouteVariantSplit stays
+        // false without a top-level variant - is pinned by TestStreamingSource's nested-only
+        // legacy leg (the leg with a top-level variant beside it is re-routed to the file-group
+        // reader instead).
+
+        // Read-optimized serves the base file alone: id 2 is still the pre-update value.
+        checkAnswer(readOptimizedQuery)(
+          Seq(1, """{"k":"n1"}"""), Seq(2, """{"k":"n2"}"""), Seq(3, """{"k":"n3"}"""))
+
+        // Compaction merges the nested-shredded log onto the nested-shredded base and re-derives
+        // the layout from the forced DDL. On the AVRO record type that write goes through
+        // HoodieAvroWriteSupport, whose nested forced hook is #19689's parity fix.
+        withWriteLayout(Forced("k string")) {
+          runCompaction(tableName)
+        }
+        assertCompactionCount(tablePath, 1, leg)
+        assertNestedBaseLayout(tablePath, latestCompletedInstant(tablePath), shredded = true, leg)
+        checkAnswer(snapshotQuery)(mergedRows: _*)
+
+        // Unshredded round: the update and the compaction both run with shredding off, so
+        // typed_value has to be stripped at depth on the way out.
+        withWriteLayout(Unshredded) {
+          spark.sql(s"update $tableName set " +
+            """s = named_struct('inner', parse_json('{"k":"n3c"}')), ts = 1002 where id = 3""")
+          runCompaction(tableName)
+        }
+        assertCompactionCount(tablePath, 2, leg)
+        assertNestedBaseLayout(tablePath, latestCompletedInstant(tablePath), shredded = false, leg)
+        checkAnswer(snapshotQuery)(finalRows: _*)
+
+        // Clustering re-derives the nested layout from the forced DDL over that unshredded input:
+        // the row-writer path when rowWriter is true, the record writers otherwise.
+        withWriteLayout(Forced("k string")) {
+          runClustering(tableName, rowWriter)
+        }
+        val clusteringInstant = completedClusteringInstant(tablePath, leg)
+        assertNestedBaseLayout(tablePath, clusteringInstant, shredded = true, leg)
+        checkAnswer(snapshotQuery)(finalRows: _*)
+        checkAnswer(readOptimizedQuery)(finalRows: _*)
+      }
+    }
+  }
+
+  test("CDC images carry a nested-shredded variant") {
+    assume(HoodieSparkUtils.gteqSpark4_1, SPARK_4_1_GATE)
+
+    // The nested twin of TestVariantDataType's CDC test. OP_KEY_ONLY reconstructs both images by
+    // reading the file slices; DATA_BEFORE_AFTER (the default) reads the update images from the cdc
+    // log instead. The insert leg takes BASE_FILE_INSERT in both modes, which reads the new base
+    // file directly rather than through the reader context.
+    Seq("OP_KEY_ONLY", "DATA_BEFORE_AFTER").foreach { loggingMode =>
+      // SPARK pinned: a cdc-enabled table always writes through FileGroupReaderBasedMergeHandle and
+      // the merger's record type picks that handle's reader context; AVRO would route the update's
+      // base-file read through HoodieAvroParquetReader, the separate defect tracked as #19567.
+      withNestedOnlyVariantTable(s"cdc nested $loggingMode", "cow", props = Seq(
+        "'hoodie.table.cdc.enabled' = 'true'",
+        s"'hoodie.table.cdc.supplemental.logging.mode' = '$loggingMode'",
+        "hoodie.index.type = 'INMEMORY'"),
+        recordTypes = Seq(HoodieRecordType.SPARK)) { (tableName, tablePath, leg) =>
+        withWriteLayout(Forced("k string")) {
+          spark.sql(s"insert into $tableName values " +
+            """(1, named_struct('inner', parse_json('{"k":"c1"}')), 1000)""")
+        }
+        assertVariantLayout(tablePath, shredded = true, leg, column = "s.inner")
+
+        withWriteLayout(Forced("k string")) {
+          spark.sql(s"update $tableName set " +
+            """s = named_struct('inner', parse_json('{"k":"c2"}')), ts = 1001 where id = 1""")
+        }
+        // Layout flip: the second update rewrites the file unshredded, so the images below span
+        // both physical slots.
+        withWriteLayout(Unshredded) {
+          spark.sql(s"update $tableName set " +
+            """s = named_struct('inner', parse_json('{"k":"c3"}')), ts = 1002 where id = 1""")
+        }
+        assertNestedBaseLayout(tablePath, latestCompletedInstant(tablePath), shredded = false, leg)
+
+        val cdc = spark.sql(s"select op, get_json_object(before, '$$.s.inner.k') as before_k, " +
+          s"get_json_object(after, '$$.s.inner.k') as after_k " +
+          s"from hudi_table_changes('$tableName', 'cdc', 'earliest')")
+        val insertRows = cdc.where("op = 'i'").collect()
+        assert(insertRows.length == 1, s"[$leg] expected exactly one insert cdc row")
+        assert(insertRows(0).getString(2) == "c1",
+          s"[$leg] insert after-image lost the nested variant payload: ${insertRows(0)}")
+
+        val updateRows = cdc.where("op = 'u'").orderBy("after_k").collect()
+        assert(updateRows.length == 2, s"[$leg] expected two update cdc rows")
+        assert(updateRows(0).getString(1) == "c1" && updateRows(0).getString(2) == "c2",
+          s"[$leg] first update images lost the nested variant payload: ${updateRows(0)}")
+        assert(updateRows(1).getString(1) == "c2" && updateRows(1).getString(2) == "c3",
+          s"[$leg] second (layout-flipped) update images lost the nested variant payload: ${updateRows(1)}")
+      }
+    }
+  }
+
+  test("variant_get projections and filters resolve a nested-shredded variant") {
+    assume(HoodieSparkUtils.gteqSpark4_1, SPARK_4_1_GATE)
+
+    // The nested twin of the mixed-layout variant_get test above; SPARK pinned for the same reason.
+    // pushVariantIntoScan is swept because the two arms reach the file differently: on, Spark
+    // rewrites the s.inner struct path into its own projection struct and pushes it into the scan;
+    // off, the whole variant is read and variant_get evaluates on top of it.
+    def nestedRowsSql(lo: Int, hi: Int): String =
+      s"""select cast(id as int) as id,
+         | named_struct('inner', parse_json(concat('{"k":"x', id, '"}'))) as s,
+         | 1000L as ts from range($lo, $hi, 1, 1)""".stripMargin
+
+    Seq("true", "false").foreach { pushIntoScan =>
+      withSQLConf("spark.sql.variant.pushVariantIntoScan" -> pushIntoScan) {
+        withNestedOnlyVariantTable(s"nested pushVariantIntoScan=$pushIntoScan", "cow",
+          props = Seq(NEW_FILE_GROUP_PER_COMMIT), recordTypes = Seq(HoodieRecordType.SPARK)) {
+          (tableName, tablePath, leg) =>
+          withWriteLayout(Forced("k string")) {
+            spark.sql(s"insert into $tableName ${nestedRowsSql(0, 5)}")
+          }
+          val shreddedInstant = latestCompletedInstant(tablePath)
+          withWriteLayout(Unshredded) {
+            spark.sql(s"insert into $tableName ${nestedRowsSql(5, 10)}")
+          }
+          val unshreddedInstant = latestCompletedInstant(tablePath)
+          // One file per commit (small.file.limit = 0) and one layout each, so $.k is typed in the
+          // first file and residual in the second: both slots answer the queries below.
+          assertNestedBaseLayout(tablePath, shreddedInstant, shredded = true, leg)
+          assertNestedBaseLayout(tablePath, unshreddedInstant, shredded = false, leg)
+
+          checkAnswer(s"select id, variant_get(s.inner, '$$.k', 'string') from $tableName order by id")(
+            (0 until 10).map(id => Seq(id, s"x$id")): _*)
+          // A filter served out of the residual file, and one out of the typed file.
+          checkAnswer(s"select id from $tableName where variant_get(s.inner, '$$.k', 'string') = 'x7'")(
+            Seq(7))
+          checkAnswer(s"select id from $tableName where variant_get(s.inner, '$$.k', 'string') = 'x2'")(
+            Seq(2))
+          checkAnswer(s"select count(*) from $tableName " +
+            s"where try_variant_get(s.inner, '$$.missing', 'string') is null")(Seq(10))
+          checkAnswer(s"select count(*) from $tableName where s.inner is null")(Seq(0))
+          checkAnswer(s"select id, cast(s.inner as string) from $tableName order by id")(
+            (0 until 10).map(id => Seq(id, s"""{"k":"x$id"}""")): _*)
+          // Both arms expect the very same rows, so this is what tells them apart: whether the
+          // rule actually rewrote s.inner into a projection struct inside the scan.
+          val pushed = pushIntoScan.toBoolean
+          val verdict = if (pushed) "should have" else "must not have"
+          assert(variantProjectionPushedIntoScan(
+            s"select id, variant_get(s.inner, '$$.k', 'string') from $tableName") == pushed,
+            s"[$leg] PushVariantIntoScan $verdict rewritten s.inner into a projection struct")
+        }
+      }
+    }
+
+    // MOR: a nested-shredded base under an unshredded log update, so the merged read has to serve
+    // ids 0-2 from the base file and ids 3-4 from the log. Before #19775 the pushIntoScan=true arms
+    // here crashed the JVM (SIGSEGV/SIGBUS inside an unsafe copy stub, or a java.lang.InternalError
+    // about a fault in a recent unsafe memory access): the internal reader applied the
+    // PushVariantIntoScan projection to TOP-LEVEL fields only, so the merged row still held a raw
+    // VariantVal at s.inner while the plan read that memory as the projected struct s.inner.0.
+    def seedNestedMor(tableName: String, tablePath: String, leg: String): Unit = {
+      withWriteLayout(Forced("k string")) {
+        spark.sql(s"insert into $tableName ${nestedRowsSql(0, 5)}")
+      }
+      assertNestedBaseLayout(tablePath, latestCompletedInstant(tablePath), shredded = true, leg)
+      // id 4's variant is nulled out on the way through the log: before #19775 the avro-path
+      // projector emitted a non-null struct of nulls for it rather than a NULL struct, which is
+      // what the `s.inner is null` assertion below catches.
+      withWriteLayout(Unshredded) {
+        spark.sql(s"update $tableName set " +
+          """s = named_struct('inner', parse_json(case when id = 4 then cast(null as string) """ +
+          """else concat('{"k":"y', id, '"}') end)), ts = 1001 """ +
+          "where id >= 3")
+      }
+    }
+    def assertNestedMorReads(tableName: String, leg: String, pushed: Boolean): Unit = {
+      // ids 0-2 come off the base file, id 3 off the log, and id 4's variant is the log's NULL.
+      def merged(id: Int): String = if (id < 3) s"x$id" else if (id == 3) "y3" else null
+      def mergedJson(id: Int): String = Option(merged(id)).map(k => s"""{"k":"$k"}""").orNull
+      checkAnswer(s"select id, variant_get(s.inner, '$$.k', 'string') from $tableName order by id")(
+        (0 until 5).map(id => Seq(id, merged(id))): _*)
+      // A filter served out of the log row, then one out of the base row.
+      checkAnswer(s"select id from $tableName where variant_get(s.inner, '$$.k', 'string') = 'y3'")(
+        Seq(3))
+      checkAnswer(s"select id from $tableName where variant_get(s.inner, '$$.k', 'string') = 'x1'")(
+        Seq(1))
+      checkAnswer(s"select id, cast(s.inner as string) from $tableName order by id")(
+        (0 until 5).map(id => Seq(id, mergedJson(id))): _*)
+      // PushVariantIntoScan rewrites `s.inner is null` onto the projection struct itself, so a
+      // null variant projected as a struct OF nulls rather than a NULL struct made this row
+      // disappear: the avro-block leg with the rule on is the one that failed before the fix.
+      checkAnswer(s"select id from $tableName where s.inner is null")(Seq(4))
+      checkAnswer(s"select count(*) from $tableName where s.inner is not null")(Seq(4))
+      // The whole struct: no extraction, so nothing is rewritten even with pushVariantIntoScan on,
+      // and the merge is read through a plain native VariantType at depth. `s` itself is never
+      // null - only its `inner` member is, and only for id 4.
+      val wholeStruct = spark.sql(s"select id, s from $tableName order by id").collect()
+      assert(wholeStruct.length == 5, s"[$leg] whole-struct read should return 5 rows")
+      assert(wholeStruct.forall(!_.isNullAt(1)), s"[$leg] whole-struct read nulled out s")

Review Comment:
   This pins only the row count and that `s` is non-null, so `s.inner` coming back null or stale for every row would still pass. Check the nested payload here too, the way `mergedJson` already does for the rewritten reads?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(variant): project PushVariantIntoScan struct paths in the ... [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19783:
URL: https://github.com/apache/hudi/pull/19783#discussion_r3901763668


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieMergeOnReadRDDV2.scala:
##########
@@ -149,13 +149,15 @@ class HoodieMergeOnReadRDDV2(@transient sc: SparkContext,
     }
   }
 
-  // The plain skip-merging reader cannot read a SHREDDED variant base file: it requests native
-  // VariantType, which clips the shredded group to {metadata, value} and reads value=null (the
-  // #19556 defect family). Such splits take the file-group reader below, whose reader context
-  // requests the full-variant projection shape instead (#19578). Keyed off the adapter building
-  // that shape rather than the mere presence of a variant column: it is None below Spark 4.1,
-  // where the file-group reader would read the same nulls, so re-routing there would cost the
-  // fast path for nothing.
+  // A split whose required schema has a top-level variant column takes the file-group reader
+  // below, whose reader context requests the full-variant projection shape for parquet base
+  // files (#19578), so a SHREDDED base file is read on this legacy path through the same
+  // contract as everywhere else. Without a top-level variant the base-only split stays on
+  // requiredSchemaReaderSkipMerging, whose native VariantType request the Spark 4.1+ row reader

Review Comment:
   Right: the legacy file format takes `enableVectorizedReader` straight from the inherited `ParquetFileFormat.supportBatch`, and `VariantType` is atomic, so that split reads through the vectorized reader unless the user turns it off. The comment here and the scaladoc in `TestStreamingSource` now say "parquet reader (the vectorized one at stock settings)" and name where the veto actually lives.
   



##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/schema/TestVariantDataType.scala:
##########
@@ -22,13 +22,11 @@ package org.apache.spark.sql.hudi.dml.schema
 import org.apache.hudi.{DataSourceReadOptions, HoodieSparkUtils}
 import org.apache.hudi.common.avro.VariantShreddingRuntime
 import org.apache.hudi.common.fs.FSUtils
-import org.apache.hudi.common.model.HoodieLogFile
 import org.apache.hudi.common.model.HoodieRecord.HoodieRecordType
 import org.apache.hudi.common.model.WriteOperationType
 import org.apache.hudi.common.schema.HoodieSchema
 import org.apache.hudi.common.schema.internal.HoodieSchemaException
 import org.apache.hudi.common.table.TableSchemaResolver
-import org.apache.hudi.common.table.log.HoodieLogFormat
 import org.apache.hudi.common.table.log.block.HoodieLogBlock.HoodieLogBlockType
 import org.apache.hudi.common.testutils.HoodieTestUtils
 import org.apache.hudi.common.util.StringUtils

Review Comment:
   Dropped both.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(variant): project PushVariantIntoScan struct paths in the ... [hudi]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19783:
URL: https://github.com/apache/hudi/pull/19783#issuecomment-5451918521

   ## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19783?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :x: Patch coverage is `87.64045%` with `22 lines` in your changes missing coverage. Please review.
   :white_check_mark: Project coverage is 78.13%. Comparing base ([`efe02e1`](https://app.codecov.io/gh/apache/hudi/commit/efe02e1621c17aa39a61716371a1857b4c086f79?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`e268d40`](https://app.codecov.io/gh/apache/hudi/commit/e268d40c1df977ced91f3197fa97fef1d0473ac0?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 1 commits behind head on master.
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19783?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [...a/org/apache/hudi/avro/HoodieAvroWriteSupport.java](https://app.codecov.io/gh/apache/hudi/pull/19783?src=pr&el=tree&filepath=hudi-hadoop-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Favro%2FHoodieAvroWriteSupport.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1oYWRvb3AtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2F2cm8vSG9vZGllQXZyb1dyaXRlU3VwcG9ydC5qYXZh) | 75.00% | [14 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19783?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...rg/apache/hudi/common/avro/VariantSchemaUtils.java](https://app.codecov.io/gh/apache/hudi/pull/19783?src=pr&el=tree&filepath=hudi-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fcommon%2Favro%2FVariantSchemaUtils.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2F2cm8vVmFyaWFudFNjaGVtYVV0aWxzLmphdmE=) | 98.85% | [0 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19783?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...parquet/HoodieFileGroupReaderBasedFileFormat.scala](https://app.codecov.io/gh/apache/hudi/pull/19783?src=pr&el=tree&filepath=hudi-spark-datasource%2Fhudi-spark-common%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fspark%2Fsql%2Fexecution%2Fdatasources%2Fparquet%2FHoodieFileGroupReaderBasedFileFormat.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3BhcmstY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvc3Bhcmsvc3FsL2V4ZWN1dGlvbi9kYXRhc291cmNlcy9wYXJxdWV0L0hvb2RpZUZpbGVHcm91cFJlYWRlckJhc2VkRmlsZUZvcm1hdC5zY2FsYQ==) | 90.90% | [0 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19783?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@             Coverage Diff              @@
   ##             master   #19783      +/-   ##
   ============================================
   + Coverage     78.11%   78.13%   +0.01%     
   - Complexity    33673    33705      +32     
   ============================================
     Files          2540     2540              
     Lines        141413   141447      +34     
     Branches      17123    17142      +19     
   ============================================
   + Hits         110467   110517      +50     
   + Misses        23250    23243       -7     
   + Partials       7696     7687       -9     
   ```
   
   | [Components](https://app.codecov.io/gh/apache/hudi/pull/19783/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [hudi-common](https://app.codecov.io/gh/apache/hudi/pull/19783/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.60% <98.85%> (+0.03%)` | :arrow_up: |
   | [hudi-client](https://app.codecov.io/gh/apache/hudi/pull/19783/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.15% <ø> (+0.03%)` | :arrow_up: |
   | [hudi-flink](https://app.codecov.io/gh/apache/hudi/pull/19783/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `85.66% <ø> (ø)` | |
   | [hudi-spark-datasource](https://app.codecov.io/gh/apache/hudi/pull/19783/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `72.55% <90.90%> (+0.01%)` | :arrow_up: |
   | [hudi-utilities](https://app.codecov.io/gh/apache/hudi/pull/19783/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `74.54% <ø> (+0.04%)` | :arrow_up: |
   | [hudi-cli](https://app.codecov.io/gh/apache/hudi/pull/19783/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `15.06% <ø> (ø)` | |
   | [hudi-hadoop](https://app.codecov.io/gh/apache/hudi/pull/19783/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.00% <75.00%> (-0.10%)` | :arrow_down: |
   | [hudi-sync](https://app.codecov.io/gh/apache/hudi/pull/19783/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `75.56% <ø> (-0.08%)` | :arrow_down: |
   | [hudi-io](https://app.codecov.io/gh/apache/hudi/pull/19783/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `79.76% <ø> (ø)` | |
   | [hudi-timeline-service](https://app.codecov.io/gh/apache/hudi/pull/19783/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.44% <ø> (ø)` | |
   | [hudi-cloud](https://app.codecov.io/gh/apache/hudi/pull/19783/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `65.81% <ø> (ø)` | |
   | [hudi-kafka-connect](https://app.codecov.io/gh/apache/hudi/pull/19783/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `53.20% <ø> (ø)` | |
   
   | [Flag](https://app.codecov.io/gh/apache/hudi/pull/19783/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19783/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.02% <30.53%> (+<0.01%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19783?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [...hudi/SparkFileFormatInternalRowReaderContext.scala](https://app.codecov.io/gh/apache/hudi/pull/19783?src=pr&el=tree&filepath=hudi-client%2Fhudi-spark-client%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fhudi%2FSparkFileFormatInternalRowReaderContext.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1zcGFyay1jbGllbnQvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9odWRpL1NwYXJrRmlsZUZvcm1hdEludGVybmFsUm93UmVhZGVyQ29udGV4dC5zY2FsYQ==) | `74.29% <ø> (+0.32%)` | :arrow_up: |
   | [...scala/org/apache/spark/sql/hudi/SparkAdapter.scala](https://app.codecov.io/gh/apache/hudi/pull/19783?src=pr&el=tree&filepath=hudi-client%2Fhudi-spark-client%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fspark%2Fsql%2Fhudi%2FSparkAdapter.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1zcGFyay1jbGllbnQvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9zcGFyay9zcWwvaHVkaS9TcGFya0FkYXB0ZXIuc2NhbGE=) | `55.55% <ø> (ø)` | |
   | [...apache/hudi/common/config/HoodieStorageConfig.java](https://app.codecov.io/gh/apache/hudi/pull/19783?src=pr&el=tree&filepath=hudi-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fcommon%2Fconfig%2FHoodieStorageConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2NvbmZpZy9Ib29kaWVTdG9yYWdlQ29uZmlnLmphdmE=) | `89.72% <ø> (ø)` | |
   | [.../apache/hudi/common/table/TableSchemaResolver.java](https://app.codecov.io/gh/apache/hudi/pull/19783?src=pr&el=tree&filepath=hudi-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fcommon%2Ftable%2FTableSchemaResolver.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL1RhYmxlU2NoZW1hUmVzb2x2ZXIuamF2YQ==) | `88.10% <ø> (ø)` | |
   | [...io/storage/hadoop/HoodieVariantReconstruction.java](https://app.codecov.io/gh/apache/hudi/pull/19783?src=pr&el=tree&filepath=hudi-hadoop-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fio%2Fstorage%2Fhadoop%2FHoodieVariantReconstruction.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1oYWRvb3AtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL3N0b3JhZ2UvaGFkb29wL0hvb2RpZVZhcmlhbnRSZWNvbnN0cnVjdGlvbi5qYXZh) | `92.55% <ø> (ø)` | |
   | [...rg/apache/hudi/hadoop/HiveHoodieReaderContext.java](https://app.codecov.io/gh/apache/hudi/pull/19783?src=pr&el=tree&filepath=hudi-hadoop-mr%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhadoop%2FHiveHoodieReaderContext.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL0hpdmVIb29kaWVSZWFkZXJDb250ZXh0LmphdmE=) | `76.87% <ø> (ø)` | |
   | [...scala/org/apache/hudi/HoodieMergeOnReadRDDV2.scala](https://app.codecov.io/gh/apache/hudi/pull/19783?src=pr&el=tree&filepath=hudi-spark-datasource%2Fhudi-spark-common%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fhudi%2FHoodieMergeOnReadRDDV2.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3BhcmstY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvaHVkaS9Ib29kaWVNZXJnZU9uUmVhZFJERFYyLnNjYWxh) | `79.53% <ø> (ø)` | |
   | [...ala/org/apache/hudi/cdc/CDCFileGroupIterator.scala](https://app.codecov.io/gh/apache/hudi/pull/19783?src=pr&el=tree&filepath=hudi-spark-datasource%2Fhudi-spark-common%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fhudi%2Fcdc%2FCDCFileGroupIterator.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3BhcmstY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvaHVkaS9jZGMvQ0RDRmlsZUdyb3VwSXRlcmF0b3Iuc2NhbGE=) | `79.07% <ø> (ø)` | |
   | [...g/apache/spark/sql/adapter/BaseSpark4Adapter.scala](https://app.codecov.io/gh/apache/hudi/pull/19783?src=pr&el=tree&filepath=hudi-spark-datasource%2Fhudi-spark4-common%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fspark%2Fsql%2Fadapter%2FBaseSpark4Adapter.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3Bhcms0LWNvbW1vbi9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL3NwYXJrL3NxbC9hZGFwdGVyL0Jhc2VTcGFyazRBZGFwdGVyLnNjYWxh) | `81.81% <ø> (ø)` | |
   | [...org/apache/spark/sql/adapter/Spark4\_2Adapter.scala](https://app.codecov.io/gh/apache/hudi/pull/19783?src=pr&el=tree&filepath=hudi-spark-datasource%2Fhudi-spark4.2.x%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fspark%2Fsql%2Fadapter%2FSpark4_2Adapter.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3Bhcms0LjIueC9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL3NwYXJrL3NxbC9hZGFwdGVyL1NwYXJrNF8yQWRhcHRlci5zY2FsYQ==) | `67.18% <ø> (-0.60%)` | :arrow_down: |
   | ... and [3 more](https://app.codecov.io/gh/apache/hudi/pull/19783?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | |
   
   ... and [16 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/19783/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   </details>
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(variant): project PushVariantIntoScan struct paths in the ... [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19783:
URL: https://github.com/apache/hudi/pull/19783#issuecomment-5452737283

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e268d40c1df977ced91f3197fa97fef1d0473ac0",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16667",
       "triggerID" : "e268d40c1df977ced91f3197fa97fef1d0473ac0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "115b202afb5a10df2382ea1644f37a97ff2b933e",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16671",
       "triggerID" : "115b202afb5a10df2382ea1644f37a97ff2b933e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a51b75e41a66ba8982da4aff12edd236dbfe692a",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a51b75e41a66ba8982da4aff12edd236dbfe692a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e268d40c1df977ced91f3197fa97fef1d0473ac0 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16667) 
   * 115b202afb5a10df2382ea1644f37a97ff2b933e Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16671) 
   * a51b75e41a66ba8982da4aff12edd236dbfe692a UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(variant): project PushVariantIntoScan struct paths in the ... [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19783:
URL: https://github.com/apache/hudi/pull/19783#issuecomment-5452050180

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e268d40c1df977ced91f3197fa97fef1d0473ac0",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16667",
       "triggerID" : "e268d40c1df977ced91f3197fa97fef1d0473ac0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "115b202afb5a10df2382ea1644f37a97ff2b933e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "115b202afb5a10df2382ea1644f37a97ff2b933e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e268d40c1df977ced91f3197fa97fef1d0473ac0 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16667) 
   * 115b202afb5a10df2382ea1644f37a97ff2b933e UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(variant): project PushVariantIntoScan struct paths in the ... [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19783:
URL: https://github.com/apache/hudi/pull/19783#issuecomment-5452516349

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e268d40c1df977ced91f3197fa97fef1d0473ac0",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16667",
       "triggerID" : "e268d40c1df977ced91f3197fa97fef1d0473ac0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "115b202afb5a10df2382ea1644f37a97ff2b933e",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16671",
       "triggerID" : "115b202afb5a10df2382ea1644f37a97ff2b933e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e268d40c1df977ced91f3197fa97fef1d0473ac0 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16667) 
   * 115b202afb5a10df2382ea1644f37a97ff2b933e Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16671) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(variant): project PushVariantIntoScan struct paths in the ... [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19783:
URL: https://github.com/apache/hudi/pull/19783#issuecomment-5454379136

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e268d40c1df977ced91f3197fa97fef1d0473ac0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16667",
       "triggerID" : "e268d40c1df977ced91f3197fa97fef1d0473ac0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "115b202afb5a10df2382ea1644f37a97ff2b933e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16671",
       "triggerID" : "115b202afb5a10df2382ea1644f37a97ff2b933e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a51b75e41a66ba8982da4aff12edd236dbfe692a",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16673",
       "triggerID" : "a51b75e41a66ba8982da4aff12edd236dbfe692a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * a51b75e41a66ba8982da4aff12edd236dbfe692a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16673) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(variant): project PushVariantIntoScan struct paths in the ... [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19783:
URL: https://github.com/apache/hudi/pull/19783#issuecomment-5476045192

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e268d40c1df977ced91f3197fa97fef1d0473ac0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16667",
       "triggerID" : "e268d40c1df977ced91f3197fa97fef1d0473ac0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "115b202afb5a10df2382ea1644f37a97ff2b933e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16671",
       "triggerID" : "115b202afb5a10df2382ea1644f37a97ff2b933e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a51b75e41a66ba8982da4aff12edd236dbfe692a",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16673",
       "triggerID" : "a51b75e41a66ba8982da4aff12edd236dbfe692a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4e1524d2a0e3e7ea61348dc237e336bf9ff7e19e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "4e1524d2a0e3e7ea61348dc237e336bf9ff7e19e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * a51b75e41a66ba8982da4aff12edd236dbfe692a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16673) 
   * 4e1524d2a0e3e7ea61348dc237e336bf9ff7e19e UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(variant): project PushVariantIntoScan struct paths in the ... [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19783:
URL: https://github.com/apache/hudi/pull/19783#discussion_r3893264891


##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/schema/TestVariantShreddingMixedLayouts.scala:
##########
@@ -751,35 +806,519 @@ class TestVariantShreddingMixedLayouts extends HoodieSparkSqlTestBase with Varia
       assert(getFieldAsGroup(innerGroup, "typed_value").containsField("k"),
         s"[$leg] nested typed_value should carry k:\n$innerGroup")
 
-      // #18605 history: the batch-disabling guards in HoodieFileGroupReaderBasedFileFormat are
-      // top-level-only, so a NESTED variant still reaches the vectorized reader. The session conf
-      // does pick the reader - HoodieFileGroupReaderBasedFileFormat.supportBatch reads
-      // sparkSession.sessionState.conf and ParquetUtils.isBatchReadSupportedForSchema gates on
-      // spark.sql.parquet.enableVectorizedReader, and only afterwards does
-      // buildReaderWithPartitionValues write that decision back into the conf - so sweep it and
-      // pin both readers. Every nested-variant read bug so far (HUDI-7190, HUDI-8803, #18605) is
-      // vectorized-only, which leaves the row-based leg as the control.
-      Seq("true", "false").foreach { vectorizedReader =>
-        withSQLConf("spark.sql.parquet.enableVectorizedReader" -> vectorizedReader) {
-          checkAnswer(s"select id, cast(s.inner as string) from $tableName")(
-            Seq(1, """{"k":"x1"}""")
-          )
-        }
-      }
+      // One read, whatever spark.sql.parquet.enableVectorizedReader says: supportBatch vetoes
+      // batch reads for a variant at any depth before that conf is consulted, and section F2 pins
+      // that decision directly.
+      checkAnswer(s"select id, cast(s.inner as string) from $tableName")(
+        Seq(1, """{"k":"x1"}""")
+      )
 
       // A plain insert bin-packs into the same file group: the small-file merge must read the
       // nested-shredded base back (nested reconstruction on the AVRO record type leg).
       withWriteLayout(Forced("k string")) {
         spark.sql(s"""insert into $tableName values (2, named_struct('inner', parse_json('{"k":"x2"}')), 1000)""")
       }
       assertSingleFileGroup(tablePath, leg)
+      // The merge rewrote the whole file group under the INCOMING layout, so the layout below is
+      // the record writer's, not the row writer's seed. On the AVRO leg that write goes through
+      // HoodieAvroWriteSupport, whose forced hook now reaches the nested member; before the #19689
+      // fix it silently rewrote s.inner unshredded and nothing here noticed. It pins the
+      // small-file MERGE handle's write, while the record-writer test below pins a fresh INSERT
+      // handle, so the two are not interchangeable.
+      assertVariantLayout(tablePath, shredded = true, leg, column = "s.inner")
       checkAnswer(s"select id, cast(s.inner as string) from $tableName order by id")(
         Seq(1, """{"k":"x1"}"""),
         Seq(2, """{"k":"x2"}""")
       )
     }
   }
 
+  test("Both record types force-shred a nested variant through the record writer") {
+    assume(HoodieSparkUtils.gteqSpark4_1, SPARK_4_1_GATE)
+
+    // A plain insert, so the write goes through the record writers rather than the row writer: the
+    // AVRO leg is HoodieAvroWriteSupport end to end, the SPARK leg HoodieRowParquetWriteSupport,
+    // and the #19689 fix is what made their forced hooks agree. The DDL reaches a variant that is
+    // a record MEMBER at any depth - s.inner, and the inner of the struct element of items - but
+    // never one that is directly a collection element, which is why arr stays unshredded on both
+    // paths.
+    withVariantTable("record-writer nested forced shredding", "cow",
+      extraCols = "s struct<inner: variant>, items array<struct<inner: variant>>, arr array<variant>") {
+      (tableName, tablePath, leg) =>
+      withWriteLayout(Forced("k string")) {
+        spark.sql(
+          s"""insert into $tableName values (1, parse_json('{"k":"top"}'),
+             | named_struct('inner', parse_json('{"k":"nested"}')),
+             | array(named_struct('inner', parse_json('{"k":"element"}'))),
+             | array(parse_json('{"k":"bare"}')), 1000)""".stripMargin)
+      }
+
+      val files = listDataParquetFiles(tablePath)
+      assert(files.size == 1, s"[$leg] expected one base file, got $files")
+      Seq("s.inner", "items.inner").foreach { column =>
+        assertVariantLayout(tablePath, shredded = true, leg, column = column)
+        val group = variantGroupOf(files.head, column)
+        assert(getFieldAsGroup(group, "typed_value").containsField("k"),
+          s"[$leg] typed_value of $column should carry k:\n$group")
+      }
+      assertVariantLayout(tablePath, shredded = false, leg, column = "arr")
+
+      checkAnswer(s"select id, cast(v as string), cast(s.inner as string), " +
+        s"cast(items[0].inner as string), cast(arr[0] as string) from $tableName")(
+        Seq(1, """{"k":"top"}""", """{"k":"nested"}""", """{"k":"element"}""", """{"k":"bare"}""")
+      )
+    }
+  }
+
+  test("MOR merge, compaction and clustering carry a nested-shredded base through the internal reader") {
+    assume(HoodieSparkUtils.gteqSpark4_1, SPARK_4_1_GATE)
+
+    // The nested variant is the ONLY variant in the table, so no top-level rewrite carries it (see
+    // withNestedOnlyVariantTable): every read below has to resolve s.inner on its own - through the
+    // nested projection struct PushVariantIntoScan pushes into the scan for the user queries, and
+    // natively for the internal reads that have no catalyst schema (compaction, clustering, the
+    // legacy RDD) - which is what makes these paths say anything about nested shredding at all.
+    // The projected arm crashed the JVM before #19775 (the projection was applied at the top level
+    // only, so the merged row still held a raw variant where the plan read a struct).
+    val mergedRows = Seq(
+      Seq(1, """{"k":"n1"}"""), Seq(2, """{"k":"n2b"}"""), Seq(3, """{"k":"n3"}"""))
+    val finalRows = Seq(
+      Seq(1, """{"k":"n1"}"""), Seq(2, """{"k":"n2b"}"""), Seq(3, """{"k":"n3c"}"""))
+
+    Seq(true, false).foreach { rowWriter =>
+      // No INMEMORY index: the first insert creates a base file, the updates go to log files.
+      withNestedOnlyVariantTable(s"mor nested rowWriter=$rowWriter", "mor",
+        props = Seq("hoodie.compact.inline = 'false'"),
+        recordTypes = clusteringRecordTypes(rowWriter)) { (tableName, tablePath, leg) =>
+        val snapshotQuery = s"select id, cast(s.inner as string) from $tableName order by id"
+        val readOptimizedQuery = s"select id, cast(s.inner as string) from " +
+          s"hudi_query('$tableName', 'read_optimized') order by id"
+
+        withWriteLayout(Forced("k string")) {
+          spark.sql(s"insert into $tableName values " +
+            """(1, named_struct('inner', parse_json('{"k":"n1"}')), 1000), """ +
+            """(2, named_struct('inner', parse_json('{"k":"n2"}')), 1000), """ +
+            """(3, named_struct('inner', parse_json('{"k":"n3"}')), 1000)""")
+        }
+        val baseFiles = listDataParquetFiles(tablePath)
+        assert(baseFiles.size == 1, s"[$leg] expected one base file, got $baseFiles")
+        assertVariantLayout(tablePath, shredded = true, leg, column = "s.inner")
+        val innerGroup = variantGroupOf(baseFiles.head, "s.inner")
+        assert(getFieldAsGroup(innerGroup, "typed_value").containsField("k"),
+          s"[$leg] nested typed_value should carry k:\n$innerGroup")
+
+        // A nested-shredded native log on top of the nested-shredded base.
+        withWriteLayout(Forced("k string")) {
+          spark.sql(s"update $tableName set " +
+            """s = named_struct('inner', parse_json('{"k":"n2b"}')), ts = 1001 where id = 2""")
+        }
+        assert(listDataParquetFiles(tablePath).exists(_.endsWith(".log.parquet")),
+          s"[$leg] the update should have written a native parquet log file")
+
+        // The conf only says what the reader MAY do - supportBatch vetoes vectorization for a
+        // variant at any depth (pinned in F2) - so the sweep is the control if that guard is ever
+        // narrowed back to top-level columns.
+        Seq("true", "false").foreach { vectorizedReader =>
+          withSQLConf("spark.sql.parquet.enableVectorizedReader" -> vectorizedReader) {
+            checkAnswer(snapshotQuery)(mergedRows: _*)
+          }
+        }
+
+        // Not swept here: hoodie.file.group.reader.enabled=false, which no longer routes a batch
+        // read anywhere (only the streaming sources consult it). The legacy RDD path over a
+        // nested-shredded base - HoodieMergeOnReadRDDV2, whose shouldRerouteVariantSplit stays
+        // false without a top-level variant - is pinned by TestStreamingSource's legacy leg.
+
+        // Read-optimized serves the base file alone: id 2 is still the pre-update value.
+        checkAnswer(readOptimizedQuery)(
+          Seq(1, """{"k":"n1"}"""), Seq(2, """{"k":"n2"}"""), Seq(3, """{"k":"n3"}"""))
+
+        // Compaction merges the nested-shredded log onto the nested-shredded base and re-derives
+        // the layout from the forced DDL. On the AVRO record type that write goes through
+        // HoodieAvroWriteSupport, whose nested forced hook is #19689's parity fix.
+        withWriteLayout(Forced("k string")) {
+          runCompaction(tableName)
+        }
+        assertCompactionCount(tablePath, 1, leg)
+        assertNestedBaseLayout(tablePath, latestCompletedInstant(tablePath), shredded = true, leg)
+        checkAnswer(snapshotQuery)(mergedRows: _*)
+
+        // Unshredded round: the update and the compaction both run with shredding off, so
+        // typed_value has to be stripped at depth on the way out.
+        withWriteLayout(Unshredded) {
+          spark.sql(s"update $tableName set " +
+            """s = named_struct('inner', parse_json('{"k":"n3c"}')), ts = 1002 where id = 3""")
+          runCompaction(tableName)
+        }
+        assertCompactionCount(tablePath, 2, leg)
+        assertNestedBaseLayout(tablePath, latestCompletedInstant(tablePath), shredded = false, leg)
+        checkAnswer(snapshotQuery)(finalRows: _*)
+
+        // Clustering re-derives the nested layout from the forced DDL over that unshredded input:
+        // the row-writer path when rowWriter is true, the record writers otherwise.
+        withWriteLayout(Forced("k string")) {
+          runClustering(tableName, rowWriter)
+        }
+        val clusteringInstant = completedClusteringInstant(tablePath, leg)
+        assertNestedBaseLayout(tablePath, clusteringInstant, shredded = true, leg)
+        checkAnswer(snapshotQuery)(finalRows: _*)
+        checkAnswer(readOptimizedQuery)(finalRows: _*)
+      }
+    }
+  }
+
+  test("CDC images carry a nested-shredded variant") {
+    assume(HoodieSparkUtils.gteqSpark4_1, SPARK_4_1_GATE)
+
+    // The nested twin of TestVariantDataType's CDC test. OP_KEY_ONLY reconstructs both images by
+    // reading the file slices; DATA_BEFORE_AFTER (the default) reads the update images from the cdc
+    // log instead. The insert leg takes BASE_FILE_INSERT in both modes, which reads the new base
+    // file directly rather than through the reader context.
+    Seq("OP_KEY_ONLY", "DATA_BEFORE_AFTER").foreach { loggingMode =>
+      // SPARK pinned: a cdc-enabled table always writes through FileGroupReaderBasedMergeHandle and
+      // the merger's record type picks that handle's reader context; AVRO would route the update's
+      // base-file read through HoodieAvroParquetReader, the separate defect tracked as #19567.
+      withNestedOnlyVariantTable(s"cdc nested $loggingMode", "cow", props = Seq(
+        "'hoodie.table.cdc.enabled' = 'true'",
+        s"'hoodie.table.cdc.supplemental.logging.mode' = '$loggingMode'",
+        "hoodie.index.type = 'INMEMORY'"),
+        recordTypes = Seq(HoodieRecordType.SPARK)) { (tableName, tablePath, leg) =>
+        withWriteLayout(Forced("k string")) {
+          spark.sql(s"insert into $tableName values " +
+            """(1, named_struct('inner', parse_json('{"k":"c1"}')), 1000)""")
+        }
+        assertVariantLayout(tablePath, shredded = true, leg, column = "s.inner")
+
+        withWriteLayout(Forced("k string")) {
+          spark.sql(s"update $tableName set " +
+            """s = named_struct('inner', parse_json('{"k":"c2"}')), ts = 1001 where id = 1""")
+        }
+        // Layout flip: the second update rewrites the file unshredded, so the images below span
+        // both physical slots.
+        withWriteLayout(Unshredded) {
+          spark.sql(s"update $tableName set " +
+            """s = named_struct('inner', parse_json('{"k":"c3"}')), ts = 1002 where id = 1""")
+        }
+        assertNestedBaseLayout(tablePath, latestCompletedInstant(tablePath), shredded = false, leg)
+
+        val cdc = spark.sql(s"select op, get_json_object(before, '$$.s.inner.k') as before_k, " +
+          s"get_json_object(after, '$$.s.inner.k') as after_k " +
+          s"from hudi_table_changes('$tableName', 'cdc', 'earliest')")
+        val insertRows = cdc.where("op = 'i'").collect()
+        assert(insertRows.length == 1, s"[$leg] expected exactly one insert cdc row")
+        assert(insertRows(0).getString(2) == "c1",
+          s"[$leg] insert after-image lost the nested variant payload: ${insertRows(0)}")
+
+        val updateRows = cdc.where("op = 'u'").orderBy("after_k").collect()
+        assert(updateRows.length == 2, s"[$leg] expected two update cdc rows")
+        assert(updateRows(0).getString(1) == "c1" && updateRows(0).getString(2) == "c2",
+          s"[$leg] first update images lost the nested variant payload: ${updateRows(0)}")
+        assert(updateRows(1).getString(1) == "c2" && updateRows(1).getString(2) == "c3",
+          s"[$leg] second (layout-flipped) update images lost the nested variant payload: ${updateRows(1)}")
+      }
+    }
+  }
+
+  test("variant_get projections and filters resolve a nested-shredded variant") {
+    assume(HoodieSparkUtils.gteqSpark4_1, SPARK_4_1_GATE)
+
+    // The nested twin of the mixed-layout variant_get test above; SPARK pinned for the same reason.
+    // pushVariantIntoScan is swept because the two arms reach the file differently: on, Spark
+    // rewrites the s.inner struct path into its own projection struct and pushes it into the scan;
+    // off, the whole variant is read and variant_get evaluates on top of it.
+    def nestedRowsSql(lo: Int, hi: Int): String =
+      s"""select cast(id as int) as id,
+         | named_struct('inner', parse_json(concat('{"k":"x', id, '"}'))) as s,
+         | 1000L as ts from range($lo, $hi, 1, 1)""".stripMargin
+
+    Seq("true", "false").foreach { pushIntoScan =>
+      withSQLConf("spark.sql.variant.pushVariantIntoScan" -> pushIntoScan) {

Review Comment:
   Added `variantProjectionPushedIntoScan(sql)` to `VariantShreddingTestSupport`: it collects the `FileSourceScanExec` off `sparkPlan` (AQE hides it on `executedPlan`) and asks whether its `requiredSchema` carries a projection struct, through the new `SparkAdapter.containsVariantProjection`. The COW loop and both MOR legs assert it equals their `pushVariantIntoScan` arm, and the new widening leg asserts it stays pushed.
   



##########
hudi-spark-datasource/hudi-spark4-common/src/main/scala/org/apache/spark/sql/adapter/BaseSpark4Adapter.scala:
##########
@@ -320,6 +324,89 @@ abstract class BaseSpark4Adapter extends SparkAdapter with Logging {
     if (rewritten) Some(StructType(fields)) else None
   }
 
+  /**
+   * Shared implementation behind [[SparkAdapter#buildVariantProjector]] for the 4.x adapters
+   * whose planner rewrites variants into projection structs (4.1+).
+   *
+   * Recurses into struct members, mirroring PushVariantIntoScan's `VariantInRelation.rewriteType`:
+   * a variant is rewritten at the root of the relation output or below a STRUCT path, while
+   * arrays and maps keep their native VariantType, so nothing under a collection is projected
+   * here either. Before #19775 this walked top-level fields only, and a projection struct sitting
+   * one struct member down was left holding a raw variant that the plan then read as its
+   * projected children.
+   */
+  protected final def buildVariantProjectorForStructPaths(
+      sparkDataSchema: StructType,
+      sparkRequiredSchema: StructType): Option[InternalRow => InternalRow] = {
+    // Quick check: does any required field carry a variant projection struct, at any depth?
+    if (!sparkRequiredSchema.fields.exists(f => containsVariantProjection(f.dataType))) {
+      None
+    } else {
+      // Surface mismatched schemas with both field lists rather than Spark's bare
+      // IllegalArgumentException from fieldIndex. `path` is the dotted field path of `name`.
+      def lookupDataField(dataStruct: StructType, requiredStruct: StructType,
+                          name: String, path: String): (Int, StructField) = {
+        val idx = dataStruct.getFieldIndex(name).getOrElse(
+          throw new IllegalStateException(
+            s"Required field '$path' is absent from sparkDataSchema; " +
+              s"required=${requiredStruct.fieldNames.mkString("[", ",", "]")}, " +
+              s"data=${dataStruct.fieldNames.mkString("[", ",", "]")}"))
+        (idx, dataStruct.fields(idx))
+      }
+
+      // `ref` reads the data-schema value of type `dataType`; the result has type `requiredType`.
+      def projectionExpr(ref: Expression, dataType: DataType, requiredType: DataType,
+                         path: String): Expression = requiredType match {
+        case projectedStruct: StructType if VariantMetadata.isVariantStruct(projectedStruct) =>
+          require(isVariantType(dataType),
+            s"Expected VariantType for field '$path' in data schema, got $dataType")
+          val childExprs: Seq[Expression] = projectedStruct.fields.toSeq.flatMap { child =>
+            val vm = VariantMetadata.fromMetadata(child.metadata)
+            val pathLit = Literal(UTF8String.fromString(vm.path), StringType)
+            val variantGet: Expression =
+              VariantGet(ref, pathLit, child.dataType, vm.failOnError, Option(vm.timeZoneId))
+            Seq(Literal(UTF8String.fromString(child.name), StringType), variantGet)
+          }
+          CreateNamedStruct(childExprs)

Review Comment:
   Fixed: the projection-struct arm carries the same `If(IsNull(ref), null, ...)` guard as the struct rebuild below it. Pinned twice: the projector unit test feeds a null variant inside a live struct and expects a null struct with the sibling intact, and the nested MOR legs now null id 4's variant through the log and check `where s.inner is null` / `is not null`; the avro-block leg with the rule on returned no row for `is null` before the fix.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(variant): project PushVariantIntoScan struct paths in the ... [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19783:
URL: https://github.com/apache/hudi/pull/19783#issuecomment-5491906177

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e268d40c1df977ced91f3197fa97fef1d0473ac0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16667",
       "triggerID" : "e268d40c1df977ced91f3197fa97fef1d0473ac0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "115b202afb5a10df2382ea1644f37a97ff2b933e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16671",
       "triggerID" : "115b202afb5a10df2382ea1644f37a97ff2b933e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a51b75e41a66ba8982da4aff12edd236dbfe692a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16673",
       "triggerID" : "a51b75e41a66ba8982da4aff12edd236dbfe692a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4e1524d2a0e3e7ea61348dc237e336bf9ff7e19e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16714",
       "triggerID" : "4e1524d2a0e3e7ea61348dc237e336bf9ff7e19e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6b4ffe79db9a996c398df1e4888f93a9937cb98e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16717",
       "triggerID" : "6b4ffe79db9a996c398df1e4888f93a9937cb98e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2ea92de1fcf8cd8a83abe3968d79a303d21d3140",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16729",
       "triggerID" : "2ea92de1fcf8cd8a83abe3968d79a303d21d3140",
       "triggerType" : "PUSH"
     }, {
       "hash" : "42b6ff809b4af335e5becdafa97de014912d3fd6",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16735",
       "triggerID" : "42b6ff809b4af335e5becdafa97de014912d3fd6",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 42b6ff809b4af335e5becdafa97de014912d3fd6 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16735) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(variant): project PushVariantIntoScan struct paths in the ... [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19783:
URL: https://github.com/apache/hudi/pull/19783#issuecomment-5453808105

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e268d40c1df977ced91f3197fa97fef1d0473ac0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16667",
       "triggerID" : "e268d40c1df977ced91f3197fa97fef1d0473ac0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "115b202afb5a10df2382ea1644f37a97ff2b933e",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16671",
       "triggerID" : "115b202afb5a10df2382ea1644f37a97ff2b933e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a51b75e41a66ba8982da4aff12edd236dbfe692a",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16673",
       "triggerID" : "a51b75e41a66ba8982da4aff12edd236dbfe692a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 115b202afb5a10df2382ea1644f37a97ff2b933e Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16671) 
   * a51b75e41a66ba8982da4aff12edd236dbfe692a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16673) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(variant): project PushVariantIntoScan struct paths in the ... [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19783:
URL: https://github.com/apache/hudi/pull/19783#discussion_r3893268251


##########
hudi-spark-datasource/hudi-spark4-common/src/main/scala/org/apache/spark/sql/adapter/BaseSpark4Adapter.scala:
##########
@@ -320,6 +324,89 @@ abstract class BaseSpark4Adapter extends SparkAdapter with Logging {
     if (rewritten) Some(StructType(fields)) else None
   }
 
+  /**
+   * Shared implementation behind [[SparkAdapter#buildVariantProjector]] for the 4.x adapters
+   * whose planner rewrites variants into projection structs (4.1+).
+   *
+   * Recurses into struct members, mirroring PushVariantIntoScan's `VariantInRelation.rewriteType`:
+   * a variant is rewritten at the root of the relation output or below a STRUCT path, while
+   * arrays and maps keep their native VariantType, so nothing under a collection is projected
+   * here either. Before #19775 this walked top-level fields only, and a projection struct sitting
+   * one struct member down was left holding a raw variant that the plan then read as its
+   * projected children.
+   */
+  protected final def buildVariantProjectorForStructPaths(
+      sparkDataSchema: StructType,
+      sparkRequiredSchema: StructType): Option[InternalRow => InternalRow] = {
+    // Quick check: does any required field carry a variant projection struct, at any depth?
+    if (!sparkRequiredSchema.fields.exists(f => containsVariantProjection(f.dataType))) {
+      None
+    } else {
+      // Surface mismatched schemas with both field lists rather than Spark's bare
+      // IllegalArgumentException from fieldIndex. `path` is the dotted field path of `name`.
+      def lookupDataField(dataStruct: StructType, requiredStruct: StructType,
+                          name: String, path: String): (Int, StructField) = {
+        val idx = dataStruct.getFieldIndex(name).getOrElse(
+          throw new IllegalStateException(
+            s"Required field '$path' is absent from sparkDataSchema; " +
+              s"required=${requiredStruct.fieldNames.mkString("[", ",", "]")}, " +
+              s"data=${dataStruct.fieldNames.mkString("[", ",", "]")}"))
+        (idx, dataStruct.fields(idx))
+      }
+
+      // `ref` reads the data-schema value of type `dataType`; the result has type `requiredType`.
+      def projectionExpr(ref: Expression, dataType: DataType, requiredType: DataType,
+                         path: String): Expression = requiredType match {
+        case projectedStruct: StructType if VariantMetadata.isVariantStruct(projectedStruct) =>
+          require(isVariantType(dataType),
+            s"Expected VariantType for field '$path' in data schema, got $dataType")
+          val childExprs: Seq[Expression] = projectedStruct.fields.toSeq.flatMap { child =>
+            val vm = VariantMetadata.fromMetadata(child.metadata)
+            val pathLit = Literal(UTF8String.fromString(vm.path), StringType)
+            val variantGet: Expression =
+              VariantGet(ref, pathLit, child.dataType, vm.failOnError, Option(vm.timeZoneId))
+            Seq(Literal(UTF8String.fromString(child.name), StringType), variantGet)
+          }
+          CreateNamedStruct(childExprs)
+        case requiredStruct: StructType =>
+          dataType match {
+            // Rebuild the struct member by member only when something below it is projected;
+            // otherwise the reference is already in the required shape and is cheaper untouched.
+            case dataStruct: StructType if containsVariantProjection(requiredStruct) =>
+              val childExprs: Seq[Expression] = requiredStruct.fields.toSeq.flatMap { rf =>
+                val childPath = s"$path.${rf.name}"
+                val (childIdx, childField) = lookupDataField(dataStruct, requiredStruct, rf.name, childPath)
+                val childRef = GetStructField(ref, childIdx, Some(rf.name))
+                Seq(Literal(UTF8String.fromString(rf.name), StringType),
+                  projectionExpr(childRef, childField.dataType, rf.dataType, childPath))
+              }
+              val rebuilt = CreateNamedStruct(childExprs)
+              // CreateNamedStruct is never null, so a null struct would come back as a struct of
+              // nulls without this guard.
+              If(IsNull(ref), Literal(null, rebuilt.dataType), rebuilt)
+            case _ => ref
+          }
+        case _ => ref
+      }
+
+      val exprs: Array[Expression] = sparkRequiredSchema.fields.map { rf =>
+        val (dataIdx, dataField) = lookupDataField(sparkDataSchema, sparkRequiredSchema, rf.name, rf.name)
+        val ref: Expression = BoundReference(dataIdx, dataField.dataType, dataField.nullable)
+        projectionExpr(ref, dataField.dataType, rf.dataType, rf.name)
+      }
+
+      val projection = UnsafeProjection.create(exprs.toIndexedSeq, DataTypeUtils.toAttributes(sparkDataSchema))
+      Some(row => projection(row))
+    }
+  }
+
+  /** True when `dataType` is a variant projection struct or holds one below a struct path. */
+  private def containsVariantProjection(dataType: DataType): Boolean = dataType match {

Review Comment:
   Moved to `SparkAdapter` as a concrete method over `isVariantProjectionStruct`; both private copies are gone.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(variant): project PushVariantIntoScan struct paths in the ... [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19783:
URL: https://github.com/apache/hudi/pull/19783#discussion_r3893260104


##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/SparkFileFormatInternalRowReaderContext.scala:
##########
@@ -93,28 +93,41 @@ class SparkFileFormatInternalRowReaderContext(baseFileReader: SparkColumnarFileR
   private lazy val allFilters = filters ++ requiredFilters
 
   // For each field of `target`, replace its dataType with the matching field's projected
-  // variant struct from `source` (when present). Non-matching fields pass through. Why a parallel
-  // `sparkRequiredSchema` overlay exists at all is documented on that constructor parameter.
+  // variant struct from `source` (when present), recursing into struct members so a variant
+  // reached through a struct path is overlaid too. Fields are matched by name (findFieldByName);
+  // non-matching fields pass through. The recursion mirrors PushVariantIntoScan's
+  // VariantInRelation.rewriteType, which rewrites variants at the root of the relation output
+  // and below STRUCT paths only, so an array element or a map value is never overlaid here
+  // either (#19775). Why a parallel `sparkRequiredSchema` overlay exists at all is documented on
+  // that constructor parameter.
   private def overlayVariantProjections(target: StructType, source: StructType): StructType = {
     StructType(target.fields.map { f =>
-      SparkFileFormatInternalRowReaderContext.findFieldByName(source, f.name).map(_.dataType) match {
-        case Some(projStruct: StructType) if sparkAdapter.isVariantProjectionStruct(projStruct) =>
+      (f.dataType, SparkFileFormatInternalRowReaderContext.findFieldByName(source, f.name).map(_.dataType)) match {
+        case (_, Some(projStruct: StructType)) if sparkAdapter.isVariantProjectionStruct(projStruct) =>
           f.copy(dataType = projStruct)
+        case (targetStruct: StructType, Some(sourceStruct: StructType)) =>

Review Comment:
   In scope: `addMissingFields` now has the arm for the pair `isDataTypeEqual` already declares equal. A projection struct requested over a file `VariantType` stays the reader type instead of being folded back to `VariantType` -- which parquet then decoded as a raw variant that the type-change `Cast` cast to the projected struct, i.e. nulls for that member. Pinned in `TestBaseSpark4AdapterVariantMethods.testImplicitSchemaChangeKeepsNestedVariantProjection` (file `s.n` int, table `s.n` long, `s.inner` projected: the reconciled reader type keeps the projection struct) and functionally by the new "Implicit widening of a sibling keeps the nested variant projection" leg in `TestVariantShreddingMixedLayouts`.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(variant): project PushVariantIntoScan struct paths in the ... [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19783:
URL: https://github.com/apache/hudi/pull/19783#issuecomment-5476763877

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e268d40c1df977ced91f3197fa97fef1d0473ac0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16667",
       "triggerID" : "e268d40c1df977ced91f3197fa97fef1d0473ac0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "115b202afb5a10df2382ea1644f37a97ff2b933e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16671",
       "triggerID" : "115b202afb5a10df2382ea1644f37a97ff2b933e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a51b75e41a66ba8982da4aff12edd236dbfe692a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16673",
       "triggerID" : "a51b75e41a66ba8982da4aff12edd236dbfe692a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4e1524d2a0e3e7ea61348dc237e336bf9ff7e19e",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16714",
       "triggerID" : "4e1524d2a0e3e7ea61348dc237e336bf9ff7e19e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6b4ffe79db9a996c398df1e4888f93a9937cb98e",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16717",
       "triggerID" : "6b4ffe79db9a996c398df1e4888f93a9937cb98e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 4e1524d2a0e3e7ea61348dc237e336bf9ff7e19e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16714) 
   * 6b4ffe79db9a996c398df1e4888f93a9937cb98e Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16717) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(variant): project PushVariantIntoScan struct paths in the ... [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous merged PR #19783:
URL: https://github.com/apache/hudi/pull/19783


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(variant): project PushVariantIntoScan struct paths in the ... [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19783:
URL: https://github.com/apache/hudi/pull/19783#issuecomment-5476535580

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e268d40c1df977ced91f3197fa97fef1d0473ac0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16667",
       "triggerID" : "e268d40c1df977ced91f3197fa97fef1d0473ac0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "115b202afb5a10df2382ea1644f37a97ff2b933e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16671",
       "triggerID" : "115b202afb5a10df2382ea1644f37a97ff2b933e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a51b75e41a66ba8982da4aff12edd236dbfe692a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16673",
       "triggerID" : "a51b75e41a66ba8982da4aff12edd236dbfe692a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4e1524d2a0e3e7ea61348dc237e336bf9ff7e19e",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16714",
       "triggerID" : "4e1524d2a0e3e7ea61348dc237e336bf9ff7e19e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6b4ffe79db9a996c398df1e4888f93a9937cb98e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "6b4ffe79db9a996c398df1e4888f93a9937cb98e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 4e1524d2a0e3e7ea61348dc237e336bf9ff7e19e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16714) 
   * 6b4ffe79db9a996c398df1e4888f93a9937cb98e UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(variant): project PushVariantIntoScan struct paths in the ... [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19783:
URL: https://github.com/apache/hudi/pull/19783#issuecomment-5488922525

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e268d40c1df977ced91f3197fa97fef1d0473ac0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16667",
       "triggerID" : "e268d40c1df977ced91f3197fa97fef1d0473ac0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "115b202afb5a10df2382ea1644f37a97ff2b933e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16671",
       "triggerID" : "115b202afb5a10df2382ea1644f37a97ff2b933e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a51b75e41a66ba8982da4aff12edd236dbfe692a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16673",
       "triggerID" : "a51b75e41a66ba8982da4aff12edd236dbfe692a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4e1524d2a0e3e7ea61348dc237e336bf9ff7e19e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16714",
       "triggerID" : "4e1524d2a0e3e7ea61348dc237e336bf9ff7e19e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6b4ffe79db9a996c398df1e4888f93a9937cb98e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16717",
       "triggerID" : "6b4ffe79db9a996c398df1e4888f93a9937cb98e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2ea92de1fcf8cd8a83abe3968d79a303d21d3140",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16729",
       "triggerID" : "2ea92de1fcf8cd8a83abe3968d79a303d21d3140",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 2ea92de1fcf8cd8a83abe3968d79a303d21d3140 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16729) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org