You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "linliu-code (via GitHub)" <gi...@apache.org> on 2026/08/21 23:02:17 UTC

[PR] fix(spark-sql): lay meta fields over the partial-update schema in the global-index merge [hudi]

linliu-code opened a new pull request, #19713:
URL: https://github.com/apache/hudi/pull/19713

   ### Describe the issue this Pull Request addresses
   
   Closes #19712
   
   Stacked on #19709 — review that one first. Until it merges this PR's diff shows its commit as well as this one's.
   
   ### Summary and Changelog
   
   Under MOR partial updates the merge in `HoodieIndexUtils` produces a record carrying only the columns
   named in `UPDATE SET`. That is the intended contract: `HoodieAppendHandle` takes the partial schema as
   its writer schema and `BaseWriteHelper` does the same for dedup. But the global-index merge wrapped
   that record against the **full** write schema, and `HoodieAvroIndexedRecord#prependMetaFields` infers
   the meta-field count as `targetSchema.size() - record.size()`. For a two-field record against a
   ten-field target that is 8, so `JoinedGenericRecord` treated eight slots as meta and the data landed
   at indices 8 and 9 rather than 7 and 8.
   
   Wrap the merged record against its own schema instead, taken from the same config the append handle
   and dedup path read, resolved once in the caller rather than per record, falling back to the write
   schema when partial updates are off. That makes `prependMetaFields`' existing inference correct, so
   `HoodieAvroIndexedRecord` and `JoinedGenericRecord` are untouched.
   
   Second change, required with the first: `inferPartitionPath` was deriving a partition from that same
   merged record, and `KeyGenUtils#getPartitionPath` substitutes the default partition for a field it
   cannot find, so the payload already carried `dt=__HIVE_DEFAULT_PARTITION__`. Harmless only because the
   write failed first; fixing the schema alone would turn a loud failure into a silently mis-partitioned
   row. A merge cannot change a field the record does not carry, so the existing record's partition is
   correct by construction. Gated on partial updates so full-schema paths keep resolving through the key
   generator, and the partition-field check strips the mandatory `field:TYPE` spec that the custom key
   generators keep verbatim in `getPartitionPathFields`.
   
   Tests: five parameterized cases over `GLOBAL_BLOOM`, `GLOBAL_SIMPLE`, `RECORD_INDEX` and
   `GLOBAL_RECORD_LEVEL_INDEX`, plus a boundary case pinning the analysis-time rejection when the index
   updates the partition path. They assert row values, `_hoodie_partition_path`, and the log block via
   the existing `validateLogBlock`, which checks `IS_PARTIAL` and that the block schema is the meta
   fields over the assigned columns only. `validateLogBlock` gained a defaulted `partitionPath`
   parameter so it can address a partitioned table.
   
   ### Impact
   
   Fixes `MERGE INTO` with a partial `UPDATE SET` on MOR with a global bloom or simple index. No
   behaviour change when partial updates are off, since both new schemas then resolve to the write schema
   and the partition guard is gated off.
   
   Two points a reviewer may want to push on, stated rather than left to be found:
   
   The partial schema is read from `config.getPartialUpdateSchema()`. `RecordContext#getSchemaFromBufferRecord`
   would give the merged record's schema directly and be correct by construction; it returned null when
   tried, because each `RecordContext` holds its own schema cache and the merger is built from the
   incoming context. The config route mirrors `HoodieAppendHandle` and `BaseWriteHelper`, but the other
   shape may be preferable and is easy to swap if so.
   
   Relatedly, `WRITE_PARTIAL_UPDATE_SCHEMA` is the union across update clauses while each clause projects
   only its own assignments, so config-equals-record holds for the single-clause shape the tests cover.
   A divergent multi-clause statement faults earlier today, inside the payload's serializer, so this
   change does not introduce that gap, but it is not an invariant either.
   
   ### Risk Level
   
   low
   
   Two methods in one file, gated so that only the partial-update path changes. Verified on this branch:
   the two global bloom and simple cases fail on the parent commit with the exception from #19712 and
   pass here; `TestPartialUpdateForMergeInto`, `TestMergeIntoTable` and `TestMergeIntoTable2` pass
   together (67 tests); checkstyle and scalastyle clean.
   
   An earlier revision of the partition guard was not gated and checked the partition fields by raw name.
   Since the custom key generators return `field:TYPE` verbatim, that would have silently disabled
   partition re-keying for every table using them, including full-schema merges unrelated to partial
   updates. Both the gate and the suffix stripping exist to close that.
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(spark-sql): lay meta fields over the partial-update schema in the global-index merge [hudi]

Posted by "voonhous (via GitHub) via commits" <co...@hudi.apache.org>.
voonhous closed pull request #19713: fix(spark-sql): lay meta fields over the partial-update schema in the global-index merge
URL: https://github.com/apache/hudi/pull/19713


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(spark-sql): lay meta fields over the partial-update schema in the global-index merge [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19713:
URL: https://github.com/apache/hudi/pull/19713#issuecomment-5408150271

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b0211c8bfa6efc54521580e31af0a18e9194917f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16474",
       "triggerID" : "b0211c8bfa6efc54521580e31af0a18e9194917f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b186a3a0dcad3d7809de379c47bcd060182aaa1b",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16495",
       "triggerID" : "b186a3a0dcad3d7809de379c47bcd060182aaa1b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "46032578e799e656f9f886e79b2f3252c8eabb56",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16524",
       "triggerID" : "46032578e799e656f9f886e79b2f3252c8eabb56",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b186a3a0dcad3d7809de379c47bcd060182aaa1b Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16495) 
   * 46032578e799e656f9f886e79b2f3252c8eabb56 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16524) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(spark-sql): lay meta fields over the partial-update schema in the global-index merge [hudi]

Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19713:
URL: https://github.com/apache/hudi/pull/19713#discussion_r3834706487


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java:
##########
@@ -503,6 +548,21 @@ public static <R> HoodieData<HoodieRecord<R>> mergeForPartitionUpdatesAndDeletio
     HoodieSchema writerSchemaWithMetaFields = HoodieSchemaUtils.addMetadataFields(writerSchema, updatedConfig.allowOperationMetadataField());
     HoodieSchemaCache.intern(writerSchema);
     HoodieSchemaCache.intern(writerSchemaWithMetaFields);
+    // Under partial updates the merge produces a record carrying only the assigned columns, and the log
+    // block is written with that schema: HoodieAppendHandle and BaseWriteHelper both take it from this
+    // same config. Meta fields therefore have to be prepended onto the partial schema, not the full
+    // write schema, or they occupy the wrong positions. Resolves to the write schema otherwise.
+    HoodieSchema mergedRecordSchema = updatedConfig.shouldWritePartialUpdates()
+        ? HoodieSchemaCache.intern(HoodieSchema.parse(updatedConfig.getPartialUpdateSchema()))
+        : writerSchema;

Review Comment:
   🤖 nit: `mergedRecordSchema` and `mergedRecordSchemaWithMetaFields` are clear, but the corresponding method parameters are named `mergedSchema` / `mergedSchemaWithMetaFields` — could you align the local variable names with the parameter names (or vice versa) to avoid the asymmetry?
   
   <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java:
##########
@@ -450,7 +494,8 @@ private static <R> Option<HoodieRecord<R>> mergeIncomingWithExistingRecord(
         // the record was deleted
         return Option.empty();

Review Comment:
   🤖 nit: the bare `true` here is a bit opaque — could you add an inline comment like `/* partitionResolvableFromRecord */` or extract it as a local `boolean partitionResolvable = true` with a brief note that this branch never runs partial updates? Saves the next reader a trip to the method signature.
   
   <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(spark-sql): lay meta fields over the partial-update schema in the global-index merge [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19713:
URL: https://github.com/apache/hudi/pull/19713#discussion_r3851475719


##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/others/TestPartialUpdateForMergeInto.scala:
##########
@@ -732,12 +732,144 @@ class TestPartialUpdateForMergeInto extends HoodieSparkSqlTestBase {
     }
   }
 
+  // A partial update names only the columns being changed, so the record key is normally absent

Review Comment:
   The existing test at lines 611-681 is the nearest shape to #19712 (partitioned MOR, `GLOBAL_SIMPLE`, partial `UPDATE SET`, mixed `INSERT`, partition change) and has no assertions at all; it also runs with the default `update.partition.path=true`, so partial updates are off in it, which is why it never caught this, and line 634 sets `hoodie.index.global.index.enable`, which is not a config anywhere in the tree. Relatedly, `TestMergeIntoTable.scala:1496-1498` has its MOR rows commented out citing HUDI-8835, fixed 2025-05-15 by #13263; that table is partitioned, partial-update, has no preCombineField and already sets the small-file limit to 0.
   
   Could we give the test here a `checkAnswer` on the rows plus `_hoodie_partition_path`, set `update.partition.path=false` and drop the bogus key, and re-enable those two MOR rows in `TestMergeIntoTable`, so the mixed-clause, compaction and commit-time-ordering shapes are covered without another cell?



##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/others/TestPartialUpdateForMergeInto.scala:
##########
@@ -732,12 +732,144 @@ class TestPartialUpdateForMergeInto extends HoodieSparkSqlTestBase {
     }
   }
 
+  // A partial update names only the columns being changed, so the record key is normally absent

Review Comment:
   Two more on that test while you are there: line 634 sets `hoodie.index.global.index.enable`, which is not a config anywhere in the tree, so it is silently dropped. And `TestMergeIntoTable.scala:1496-1498` has its MOR rows commented out citing HUDI-8835, fixed 2025-05-15 by #13263; that table is partitioned, partial-update, has no preCombineField and already sets the small-file limit to 0.
   
   Could we re-enable those two rows instead of adding a commit-time-ordering cell here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(spark-sql): lay meta fields over the partial-update schema in the global-index merge [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19713:
URL: https://github.com/apache/hudi/pull/19713#issuecomment-5390542184

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b0211c8bfa6efc54521580e31af0a18e9194917f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16474",
       "triggerID" : "b0211c8bfa6efc54521580e31af0a18e9194917f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b186a3a0dcad3d7809de379c47bcd060182aaa1b",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16495",
       "triggerID" : "b186a3a0dcad3d7809de379c47bcd060182aaa1b",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b186a3a0dcad3d7809de379c47bcd060182aaa1b Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16495) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(spark-sql): lay meta fields over the partial-update schema in the global-index merge [hudi]

Posted by "voonhous (via GitHub) via commits" <co...@hudi.apache.org>.
voonhous commented on PR #19713:
URL: https://github.com/apache/hudi/pull/19713#issuecomment-6701327564

   Temporarily closing this PR while we debug a GitHub settings error on master (#20471); it will be reopened shortly, no action needed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(spark-sql): lay meta fields over the partial-update schema in the global-index merge [hudi]

Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #19713:
URL: https://github.com/apache/hudi/pull/19713#discussion_r3840233581


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java:
##########
@@ -450,7 +494,8 @@ private static <R> Option<HoodieRecord<R>> mergeIncomingWithExistingRecord(
         // the record was deleted
         return Option.empty();

Review Comment:
   Done, took the inline-comment form: `existingRecordContext, mergeResult, /* partitionResolvableFromRecord */ true)`. Kept it as a literal rather than a local because that branch merges against the full write schema, so the partition is always resolvable from the record; the comment now says which parameter it is without implying the value is computed.



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java:
##########
@@ -503,6 +548,21 @@ public static <R> HoodieData<HoodieRecord<R>> mergeForPartitionUpdatesAndDeletio
     HoodieSchema writerSchemaWithMetaFields = HoodieSchemaUtils.addMetadataFields(writerSchema, updatedConfig.allowOperationMetadataField());
     HoodieSchemaCache.intern(writerSchema);
     HoodieSchemaCache.intern(writerSchemaWithMetaFields);
+    // Under partial updates the merge produces a record carrying only the assigned columns, and the log
+    // block is written with that schema: HoodieAppendHandle and BaseWriteHelper both take it from this
+    // same config. Meta fields therefore have to be prepended onto the partial schema, not the full
+    // write schema, or they occupy the wrong positions. Resolves to the write schema otherwise.
+    HoodieSchema mergedRecordSchema = updatedConfig.shouldWritePartialUpdates()
+        ? HoodieSchemaCache.intern(HoodieSchema.parse(updatedConfig.getPartialUpdateSchema()))
+        : writerSchema;

Review Comment:
   Done, aligned the locals to the parameter names: `mergedSchema` / `mergedSchemaWithMetaFields`. That direction rather than renaming the parameters, since the parameter names are the ones already in use at the other call site.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(spark-sql): lay meta fields over the partial-update schema in the global-index merge [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19713:
URL: https://github.com/apache/hudi/pull/19713#issuecomment-5389882141

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b0211c8bfa6efc54521580e31af0a18e9194917f",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16474",
       "triggerID" : "b0211c8bfa6efc54521580e31af0a18e9194917f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b186a3a0dcad3d7809de379c47bcd060182aaa1b",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b186a3a0dcad3d7809de379c47bcd060182aaa1b",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b0211c8bfa6efc54521580e31af0a18e9194917f Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16474) 
   * b186a3a0dcad3d7809de379c47bcd060182aaa1b UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(spark-sql): lay meta fields over the partial-update schema in the global-index merge [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19713:
URL: https://github.com/apache/hudi/pull/19713#issuecomment-5389916374

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b0211c8bfa6efc54521580e31af0a18e9194917f",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16474",
       "triggerID" : "b0211c8bfa6efc54521580e31af0a18e9194917f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b186a3a0dcad3d7809de379c47bcd060182aaa1b",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16495",
       "triggerID" : "b186a3a0dcad3d7809de379c47bcd060182aaa1b",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b0211c8bfa6efc54521580e31af0a18e9194917f Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16474) 
   * b186a3a0dcad3d7809de379c47bcd060182aaa1b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16495) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(spark-sql): lay meta fields over the partial-update schema in the global-index merge [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19713:
URL: https://github.com/apache/hudi/pull/19713#issuecomment-5376327311

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b0211c8bfa6efc54521580e31af0a18e9194917f",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b0211c8bfa6efc54521580e31af0a18e9194917f",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b0211c8bfa6efc54521580e31af0a18e9194917f UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(spark-sql): lay meta fields over the partial-update schema in the global-index merge [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19713:
URL: https://github.com/apache/hudi/pull/19713#discussion_r3851475707


##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/others/TestPartialUpdateForMergeInto.scala:
##########
@@ -732,12 +732,144 @@ class TestPartialUpdateForMergeInto extends HoodieSparkSqlTestBase {
     }
   }
 
+  // A partial update names only the columns being changed, so the record key is normally absent
+  // from the assignments. On MOR the global-index tagging stage merges the incoming record with its
+  // existing version and then asks the key generator for the merged record's partition path - and
+  // that merged record is materialised against WRITE_PARTIAL_UPDATE_SCHEMA, which carries only the
+  // assigned columns. Resolving the partition path must therefore not also require the record key.
+  //
+  // The index types split on whether that tagging stage runs at all: GLOBAL_BLOOM and GLOBAL_SIMPLE
+  // set mayContainDuplicateLookup on MOR and so reach it, while the record-index spellings pass
+  // false and short-circuit. Both cells are covered, so the fix is pinned where it applies and the
+  // already-working path is guarded against regression. The source projects the partition column in
+  // every case, to keep this independent of partition-column resolution (ENG-46864).
+  Seq(
+    ("GLOBAL_BLOOM re-keying", Map(
+      "hoodie.index.type" -> "GLOBAL_BLOOM",
+      "hoodie.bloom.index.update.partition.path" -> "false")),
+    ("GLOBAL_SIMPLE re-keying", Map(
+      "hoodie.index.type" -> "GLOBAL_SIMPLE",
+      "hoodie.simple.index.update.partition.path" -> "false")),
+    ("RECORD_INDEX", Map(
+      "hoodie.index.type" -> "RECORD_INDEX",
+      "hoodie.record.index.update.partition.path" -> "false",
+      "hoodie.metadata.enable" -> "true",
+      "hoodie.metadata.record.index.enable" -> "true")),
+    ("GLOBAL_RECORD_LEVEL_INDEX", Map(
+      "hoodie.index.type" -> "GLOBAL_RECORD_LEVEL_INDEX",
+      "hoodie.record.index.update.partition.path" -> "false",
+      "hoodie.metadata.enable" -> "true",
+      "hoodie.metadata.record.index.enable" -> "true"))

Review Comment:
   This cell is a runtime duplicate of `RECORD_INDEX`: `SparkHoodieIndexFactory.java:65-67` maps both spellings to `SparkMetadataTableGlobalRecordLevelIndex`, reading the same `hoodie.record.index.update.partition.path`. The one cell where the two spellings diverge, and where a wrong `__HIVE_DEFAULT_PARTITION__` would surface as a delete against the real partition plus a partial record inserted into the default one, is this spelling with `update.partition.path=true`, which the NOTE below names and skips; #19610 is the open PR adding the enum mapping. The flag is not even needed to reach that arm: while the RLI partition is not yet built, `SparkMetadataTableGlobalRecordLevelIndex.java:79-88` falls back to `GLOBAL_SIMPLE`, whose flag defaults to `true`.
   
   Could we make this cell the `update.partition.path=true` variant with a source `dt` that differs, asserting the record stays in its partition, and cite #19610 so the cell is retired when the mapping lands?



##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/others/TestPartialUpdateForMergeInto.scala:
##########
@@ -732,12 +732,144 @@ class TestPartialUpdateForMergeInto extends HoodieSparkSqlTestBase {
     }
   }
 
+  // A partial update names only the columns being changed, so the record key is normally absent
+  // from the assignments. On MOR the global-index tagging stage merges the incoming record with its
+  // existing version and then asks the key generator for the merged record's partition path - and
+  // that merged record is materialised against WRITE_PARTIAL_UPDATE_SCHEMA, which carries only the
+  // assigned columns. Resolving the partition path must therefore not also require the record key.
+  //
+  // The index types split on whether that tagging stage runs at all: GLOBAL_BLOOM and GLOBAL_SIMPLE
+  // set mayContainDuplicateLookup on MOR and so reach it, while the record-index spellings pass
+  // false and short-circuit. Both cells are covered, so the fix is pinned where it applies and the
+  // already-working path is guarded against regression. The source projects the partition column in
+  // every case, to keep this independent of partition-column resolution (ENG-46864).
+  Seq(
+    ("GLOBAL_BLOOM re-keying", Map(
+      "hoodie.index.type" -> "GLOBAL_BLOOM",
+      "hoodie.bloom.index.update.partition.path" -> "false")),
+    ("GLOBAL_SIMPLE re-keying", Map(
+      "hoodie.index.type" -> "GLOBAL_SIMPLE",
+      "hoodie.simple.index.update.partition.path" -> "false")),
+    ("RECORD_INDEX", Map(
+      "hoodie.index.type" -> "RECORD_INDEX",
+      "hoodie.record.index.update.partition.path" -> "false",
+      "hoodie.metadata.enable" -> "true",
+      "hoodie.metadata.record.index.enable" -> "true")),
+    ("GLOBAL_RECORD_LEVEL_INDEX", Map(
+      "hoodie.index.type" -> "GLOBAL_RECORD_LEVEL_INDEX",
+      "hoodie.record.index.update.partition.path" -> "false",
+      "hoodie.metadata.enable" -> "true",
+      "hoodie.metadata.record.index.enable" -> "true"))

Review Comment:
   **Correction** to my second paragraph: the matrix does observe `partitionResolvableFromRecord`. With the arm reverted the untagged record counts as an insert, `UpsertPartitioner.java:189-216` opens a small-file UPDATE bucket at the existing fileId, and `BaseSparkDeltaCommitActionExecutor.java:76-79` diverts it to the COW merge handle: zero log files, so `validateLogBlock(..., 1, ...)` fails at `:877`. The duplicate-cell point and the `update.partition.path=true` cell stand. That flag is not even needed to reach the delete+insert arm: while the RLI partition is not yet built, `SparkMetadataTableGlobalRecordLevelIndex.java:79-88` falls back to `GLOBAL_SIMPLE`, whose flag defaults to `true`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(spark-sql): lay meta fields over the partial-update schema in the global-index merge [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19713:
URL: https://github.com/apache/hudi/pull/19713#discussion_r3851475707


##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/others/TestPartialUpdateForMergeInto.scala:
##########
@@ -732,12 +732,144 @@ class TestPartialUpdateForMergeInto extends HoodieSparkSqlTestBase {
     }
   }
 
+  // A partial update names only the columns being changed, so the record key is normally absent
+  // from the assignments. On MOR the global-index tagging stage merges the incoming record with its
+  // existing version and then asks the key generator for the merged record's partition path - and
+  // that merged record is materialised against WRITE_PARTIAL_UPDATE_SCHEMA, which carries only the
+  // assigned columns. Resolving the partition path must therefore not also require the record key.
+  //
+  // The index types split on whether that tagging stage runs at all: GLOBAL_BLOOM and GLOBAL_SIMPLE
+  // set mayContainDuplicateLookup on MOR and so reach it, while the record-index spellings pass
+  // false and short-circuit. Both cells are covered, so the fix is pinned where it applies and the
+  // already-working path is guarded against regression. The source projects the partition column in
+  // every case, to keep this independent of partition-column resolution (ENG-46864).
+  Seq(
+    ("GLOBAL_BLOOM re-keying", Map(
+      "hoodie.index.type" -> "GLOBAL_BLOOM",
+      "hoodie.bloom.index.update.partition.path" -> "false")),
+    ("GLOBAL_SIMPLE re-keying", Map(
+      "hoodie.index.type" -> "GLOBAL_SIMPLE",
+      "hoodie.simple.index.update.partition.path" -> "false")),
+    ("RECORD_INDEX", Map(
+      "hoodie.index.type" -> "RECORD_INDEX",
+      "hoodie.record.index.update.partition.path" -> "false",
+      "hoodie.metadata.enable" -> "true",
+      "hoodie.metadata.record.index.enable" -> "true")),
+    ("GLOBAL_RECORD_LEVEL_INDEX", Map(
+      "hoodie.index.type" -> "GLOBAL_RECORD_LEVEL_INDEX",
+      "hoodie.record.index.update.partition.path" -> "false",
+      "hoodie.metadata.enable" -> "true",
+      "hoodie.metadata.record.index.enable" -> "true"))

Review Comment:
   This cell is a runtime duplicate of `RECORD_INDEX`: `SparkHoodieIndexFactory.java:65-67` maps both spellings to `SparkMetadataTableGlobalRecordLevelIndex`, reading the same `hoodie.record.index.update.partition.path`. Meanwhile nothing in the matrix observes `partitionResolvableFromRecord`: with `update.partition.path=false` the merged key is reset to the existing key whatever the keygen returned. The one cell where a wrong `__HIVE_DEFAULT_PARTITION__` is visible (delete against the real partition, partial record inserted into the default one) is this spelling with `update.partition.path=true`, which the NOTE below names and skips; #19610 is the open PR adding the enum mapping.
   
   Could we make this cell the `update.partition.path=true` variant with a source `dt` that differs, asserting the record stays in its partition, and cite #19610 so the cell is retired when the mapping lands?



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java:
##########
@@ -531,7 +591,8 @@ public static <R> HoodieData<HoodieRecord<R>> mergeForPartitionUpdatesAndDeletio
           HoodieRecord<R> existing = existingOpt.get();
 
           Option<HoodieRecord<R>> mergedOpt = mergeIncomingWithExistingRecord(
-              incoming, existing, writerSchema, writerSchemaWithMetaFields, updatedConfig,
+              incoming, existing, writerSchema, writerSchemaWithMetaFields,
+              mergedSchema, mergedSchemaWithMetaFields, partitionResolvableFromRecord, updatedConfig,

Review Comment:
   The arm this feeds at line 616 (`merged.newInstance(existing.getKey())`, changed partition with `update.partition.path=false`) returns an untagged record: `HoodieAvroRecord.newInstance(HoodieKey)` sets the location to `Option.empty()`, unlike the sibling arm at 608 that calls `tagRecord`. Untagged means it is written as an insert, so the key lands in a second file group unless small-file routing happens to pick the same slice. Pre-existing from #13830 and untested (`TestGlobalIndexEnableUpdatePartitions` only runs with the flag true), but with #19709 merged a partial `update set t.dt = s.dt, t.amount = s.amount` now reaches it.
   
   Could we `tagRecord(merged.newInstance(existing.getKey()), existing.getCurrentLocation())` there, matching the arm's own comment, and add a matrix row that assigns `dt` to a new value asserting a single file slice?



##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/others/TestPartialUpdateForMergeInto.scala:
##########
@@ -732,12 +732,144 @@ class TestPartialUpdateForMergeInto extends HoodieSparkSqlTestBase {
     }
   }
 
+  // A partial update names only the columns being changed, so the record key is normally absent

Review Comment:
   The existing test at lines 611-681 is the nearest shape to #19712 (partitioned MOR, `GLOBAL_SIMPLE`, partial `UPDATE SET`, mixed `INSERT`, partition change) and has no assertions at all; it also runs with the default `update.partition.path=true`, so partial updates are off in it, which is why it never caught this.
   
   Could we give it a `checkAnswer` on the rows plus `_hoodie_partition_path` and set `update.partition.path=false` there, so the mixed-clause and compaction shapes this matrix skips are covered without another cell?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(spark-sql): lay meta fields over the partial-update schema in the global-index merge [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19713:
URL: https://github.com/apache/hudi/pull/19713#discussion_r3852169992


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java:
##########
@@ -377,6 +378,9 @@ private static <R> Option<HoodieRecord<R>> mergeIncomingWithExistingRecordWithEx
       HoodieRecord<R> existing,
       HoodieSchema writeSchema,

Review Comment:
   **nit, feel free to ignore** -- `writeSchema` is now unused in this method: both former uses (`prependMetaFields` and `Option.of(writeSchema)`) became `mergedSchema`. Could we drop the parameter and the argument at line 482?



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java:
##########
@@ -398,27 +402,62 @@ private static <R> Option<HoodieRecord<R>> mergeIncomingWithExistingRecordWithEx
     }
 
     //record is inserted or updated
-    String partitionPath = inferPartitionPath(incoming, existing, writeSchemaWithMetaFields, keyGenerator, existingRecordContext, mergeResult);
+    String partitionPath = inferPartitionPath(incoming, existing, mergedSchemaWithMetaFields, keyGenerator,
+        existingRecordContext, mergeResult, partitionResolvableFromRecord);
     HoodieRecord<R> result = existingRecordContext.constructHoodieRecord(mergeResult, partitionPath);
-    HoodieRecord<R> withMeta = result.prependMetaFields(writeSchema, writeSchemaWithMetaFields,
+    HoodieRecord<R> withMeta = result.prependMetaFields(mergedSchema, mergedSchemaWithMetaFields,

Review Comment:
   **major** -- generic guard behind this fix. `HoodieAvroIndexedRecord.prependMetaFields:206` infers `metaFieldSize = target.size - record.size` (positional since #13860, in 1.1.0 through 1.2.0) and nothing bounds it; #19712's shape gives 8 and `JoinedGenericRecord` writes the shifted slots silently. Seven of the eight callers are pinned to 0/5/6; `HoodieBootstrapRecordIterator.java:67` is the exception and is already malformed when a bootstrap writer schema gained more than one column.
   
   Could we add `checkArgument(0 <= metaFieldSize && metaFieldSize <= HOODIE_META_COLUMNS.size() + 1)` there, so the next mismatched caller fails loudly instead of writing shifted values?



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java:
##########
@@ -398,27 +402,62 @@ private static <R> Option<HoodieRecord<R>> mergeIncomingWithExistingRecordWithEx
     }
 
     //record is inserted or updated
-    String partitionPath = inferPartitionPath(incoming, existing, writeSchemaWithMetaFields, keyGenerator, existingRecordContext, mergeResult);
+    String partitionPath = inferPartitionPath(incoming, existing, mergedSchemaWithMetaFields, keyGenerator,
+        existingRecordContext, mergeResult, partitionResolvableFromRecord);
     HoodieRecord<R> result = existingRecordContext.constructHoodieRecord(mergeResult, partitionPath);
-    HoodieRecord<R> withMeta = result.prependMetaFields(writeSchema, writeSchemaWithMetaFields,
+    HoodieRecord<R> withMeta = result.prependMetaFields(mergedSchema, mergedSchemaWithMetaFields,
         new MetadataValues().setRecordKey(incoming.getRecordKey()).setPartitionPath(partitionPath), properties);
-    return Option.of(withMeta.wrapIntoHoodieRecordPayloadWithParams(writeSchemaWithMetaFields, properties, Option.empty(),
-        config.allowOperationMetadataField(), Option.empty(), false, Option.of(writeSchema)));
+    return Option.of(withMeta.wrapIntoHoodieRecordPayloadWithParams(mergedSchemaWithMetaFields, properties, Option.empty(),
+        config.allowOperationMetadataField(), Option.empty(), false, Option.of(mergedSchema)));
   }
 
   private static <R> String inferPartitionPath(HoodieRecord<R> incoming, HoodieRecord<R> existing, HoodieSchema recordSchema, BaseKeyGenerator keyGenerator,
-                                               RecordContext<R> recordContext, BufferedRecord<R> resultingBufferedRecord) {
+                                               RecordContext<R> recordContext, BufferedRecord<R> resultingBufferedRecord,
+                                               boolean partitionResolvableFromRecord) {
     R record = resultingBufferedRecord.getRecord();
     if (record == incoming.getData()) {
       return incoming.getPartitionPath();
     } else if (record == existing.getData()) {
       return existing.getPartitionPath();
+    } else if (!partitionResolvableFromRecord) {
+      // A partial update carries only the fields named in the assignments, so the partition fields are
+      // absent. The merge cannot have changed a field the record does not carry, so the partition is the
+      // existing record's. Deriving it from the record instead resolves to the default partition,
+      // because KeyGenUtils#getPartitionPath substitutes it for a value it cannot find.
+      return existing.getPartitionPath();
     } else {
       // the merged record is not the same as either incoming or existing, so we need to compute the partition path
       return keyGenerator.getPartitionPath(recordContext.convertToAvroRecord(record, recordSchema));
     }
   }
 
+  /**
+   * Whether the merged record can yield a partition path on its own. False only for a partial update whose
+   * schema omits a partition field; a full merge result always can, so those paths keep resolving through
+   * the key generator. Evaluated once per stage, not per record.
+   */
+  private static boolean isPartitionResolvableFromRecord(HoodieSchema recordSchema, BaseKeyGenerator keyGenerator) {
+    List<String> partitionPathFields = keyGenerator.getPartitionPathFields();
+    if (partitionPathFields == null || partitionPathFields.isEmpty()) {
+      // A non-partitioned table has nothing to resolve, so let the key generator answer as before.
+      return true;
+    }
+    return partitionPathFields.stream()
+        .map(HoodieIndexUtils::partitionFieldRootName)
+        .allMatch(rootField -> rootField.isEmpty() || recordSchema.getField(rootField).isPresent());
+  }
+
+  /**
+   * The schema field name a partition-path config entry refers to. The custom key generators keep the
+   * mandatory "field:TYPE" spec verbatim in getPartitionPathFields, so the type suffix is stripped before
+   * the nested path is reduced to its root.
+   */
+  private static String partitionFieldRootName(String partitionPathField) {

Review Comment:
   **nit, feel free to ignore** -- `CustomAvroKeyGenerator.getPartitionFieldAndKeyType(field).getLeft()` already strips the `:TYPE` suffix, and the root split is `split("\\.")[0]`, which would avoid re-adding the `HoodieAvroUtils` import that #17599 removed from this file. Separately, `inferPartitionPath` (405) and this guard (565) are handed `mergedSchemaWithMetaFields` while the record carries `mergedSchema`; inert on the Avro path, but could we pass `mergedSchema` so the argument matches the record?



##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/others/TestPartialUpdateForMergeInto.scala:
##########
@@ -732,12 +732,144 @@ class TestPartialUpdateForMergeInto extends HoodieSparkSqlTestBase {
     }
   }
 
+  // A partial update names only the columns being changed, so the record key is normally absent
+  // from the assignments. On MOR the global-index tagging stage merges the incoming record with its
+  // existing version and then asks the key generator for the merged record's partition path - and
+  // that merged record is materialised against WRITE_PARTIAL_UPDATE_SCHEMA, which carries only the
+  // assigned columns. Resolving the partition path must therefore not also require the record key.
+  //
+  // The index types split on whether that tagging stage runs at all: GLOBAL_BLOOM and GLOBAL_SIMPLE
+  // set mayContainDuplicateLookup on MOR and so reach it, while the record-index spellings pass
+  // false and short-circuit. Both cells are covered, so the fix is pinned where it applies and the
+  // already-working path is guarded against regression. The source projects the partition column in
+  // every case, to keep this independent of partition-column resolution (ENG-46864).
+  Seq(
+    ("GLOBAL_BLOOM re-keying", Map(
+      "hoodie.index.type" -> "GLOBAL_BLOOM",
+      "hoodie.bloom.index.update.partition.path" -> "false")),
+    ("GLOBAL_SIMPLE re-keying", Map(
+      "hoodie.index.type" -> "GLOBAL_SIMPLE",
+      "hoodie.simple.index.update.partition.path" -> "false")),
+    ("RECORD_INDEX", Map(
+      "hoodie.index.type" -> "RECORD_INDEX",
+      "hoodie.record.index.update.partition.path" -> "false",
+      "hoodie.metadata.enable" -> "true",
+      "hoodie.metadata.record.index.enable" -> "true")),
+    ("GLOBAL_RECORD_LEVEL_INDEX", Map(
+      "hoodie.index.type" -> "GLOBAL_RECORD_LEVEL_INDEX",
+      "hoodie.record.index.update.partition.path" -> "false",
+      "hoodie.metadata.enable" -> "true",
+      "hoodie.metadata.record.index.enable" -> "true"))
+  ).foreach { case (label, indexConfs) =>
+    test(s"Test MOR partial update on a global index without assigning the record key ($label)") {
+      withTempDir { tmp =>
+        withSQLConf(indexConfs.toSeq: _*) {
+          val tableName = generateTableName
+          val basePath = s"${tmp.getCanonicalPath}/$tableName"
+          spark.sql(
+            s"""
+               | create table $tableName (
+               |   id bigint,
+               |   name string,
+               |   amount double,
+               |   ts bigint,
+               |   dt string
+               | ) using hudi
+               | partitioned by (dt)
+               | location '$basePath'
+               | tblproperties (
+               |   type = 'mor',
+               |   primaryKey = 'id',
+               |   preCombineField = 'ts')
+             """.stripMargin)
+          spark.sql(s"insert into $tableName values (1, 'a', 10.0, 1, '2026-08-11')")
+
+          // `id` is deliberately absent from the assignments - that is what makes it partial.
+          spark.sql(
+            s"""
+               | merge into $tableName as t
+               | using (
+               |   select 1L as id, 15.0 as amount, 200L as ts, '2026-08-11' as dt
+               | ) as s
+               | on t.id = s.id
+               | when matched then update set t.amount = s.amount, t.ts = s.ts
+             """.stripMargin)
+
+          // The assigned columns change, the unassigned ones keep their existing values, and the
+          // record stays in its own partition - asserted rather than merely checking row count,
+          // since a dropped record leaves a table that also "looks unchanged".
+          checkAnswer(s"select id, name, amount, ts, dt, _hoodie_partition_path from $tableName")(
+            Seq(1L, "a", 15.0, 200L, "2026-08-11", "dt=2026-08-11")
+          )
+
+          // The rows alone cannot distinguish a correct partial log block from a full-schema one, and
+          // this fix is precisely about which schema the block's meta fields are laid out against. So
+          // assert the block: IS_PARTIAL set, and the schema equal to the meta fields prepended onto
+          // the assigned columns only. Without the fix the meta count is inferred as
+          // targetSchema.size - record.size and the values land in the wrong slots.
+          validateLogBlock(basePath, 1, Seq(Seq("amount", "ts")), isPartial = true, "dt=2026-08-11")
+        }
+      }
+    }
+  }
+
+
+  // update.partition.path enabled is deliberately NOT in the matrix above: for GLOBAL_BLOOM and
+  // GLOBAL_SIMPLE it makes useGlobalIndex true, which disables MOR partial updates altogether
+  // (isPartialUpdateActionForMOR requires !useGlobalIndex), and MOR then requires the record key to be
+  // assigned. So the statement is rejected at analysis time rather than reaching the writer. Pinned
+  // here so the boundary is explicit and nobody mistakes this rejection for the partial-update defect.
+  //
+  // NOTE the exemption is not universal: isGlobalIndexEnabled maps only GLOBAL_SIMPLE, GLOBAL_BLOOM
+  // and RECORD_INDEX, so GLOBAL_RECORD_LEVEL_INDEX returns false there regardless of its
+  // update.partition.path value, while SparkMetadataTableGlobalRecordLevelIndex still honours that
+  // config. That combination therefore reaches the writer WITH partial updates enabled. The
+  // partition-resolution guard in HoodieIndexUtils covers it, and the missing enum mapping is being
+  // added separately, so this test asserts the rejection only for the spellings that actually reject.
+  test("Test MOR merge without assigning the record key is rejected when the global index updates the partition path") {

Review Comment:
   **minor** -- this is a near-clone of the test at line 686: same table shape, same statement, same `checkExceptionContain` on the message from `MergeIntoHoodieTableCommand.scala:1051`, differing only in which `isPartialUpdateActionForMOR` conjunct fails (custom merge mode there, `useGlobalIndex` here). The `insert into` is dead, since `validate` runs at the top of `run()` before any read, and the rejection is in `run()`, not analysis.
   
   Could we fold this into 686 as a two-element `Seq(...).foreach`, or drop it?



##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/others/TestPartialUpdateForMergeInto.scala:
##########
@@ -732,12 +732,144 @@ class TestPartialUpdateForMergeInto extends HoodieSparkSqlTestBase {
     }
   }
 
+  // A partial update names only the columns being changed, so the record key is normally absent
+  // from the assignments. On MOR the global-index tagging stage merges the incoming record with its
+  // existing version and then asks the key generator for the merged record's partition path - and
+  // that merged record is materialised against WRITE_PARTIAL_UPDATE_SCHEMA, which carries only the
+  // assigned columns. Resolving the partition path must therefore not also require the record key.
+  //
+  // The index types split on whether that tagging stage runs at all: GLOBAL_BLOOM and GLOBAL_SIMPLE
+  // set mayContainDuplicateLookup on MOR and so reach it, while the record-index spellings pass
+  // false and short-circuit. Both cells are covered, so the fix is pinned where it applies and the
+  // already-working path is guarded against regression. The source projects the partition column in
+  // every case, to keep this independent of partition-column resolution (ENG-46864).
+  Seq(
+    ("GLOBAL_BLOOM re-keying", Map(

Review Comment:
   **minor** -- the four cells are two behaviours. `GLOBAL_BLOOM` and `GLOBAL_SIMPLE` call `tagGlobalLocationBackToRecords` with identical flags (`HoodieGlobalBloomIndex.java:106-109`, `HoodieGlobalSimpleIndex.java:77-80`), and both record-index spellings map to `SparkMetadataTableGlobalRecordLevelIndex` (`SparkHoodieIndexFactory.java:65-67`). Also `hoodie.metadata.record.index.enable` is a deprecated alias of `hoodie.metadata.global.record.level.index.enable`; if it stops resolving, the index falls back to `GLOBAL_SIMPLE` on a warning and the cell stays green.
   
   Could we collapse to two cells and use the current key (or assert the `record_index` partition exists)?



##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/others/TestPartialUpdateForMergeInto.scala:
##########
@@ -732,12 +732,144 @@ class TestPartialUpdateForMergeInto extends HoodieSparkSqlTestBase {
     }
   }
 
+  // A partial update names only the columns being changed, so the record key is normally absent
+  // from the assignments. On MOR the global-index tagging stage merges the incoming record with its
+  // existing version and then asks the key generator for the merged record's partition path - and
+  // that merged record is materialised against WRITE_PARTIAL_UPDATE_SCHEMA, which carries only the
+  // assigned columns. Resolving the partition path must therefore not also require the record key.
+  //
+  // The index types split on whether that tagging stage runs at all: GLOBAL_BLOOM and GLOBAL_SIMPLE
+  // set mayContainDuplicateLookup on MOR and so reach it, while the record-index spellings pass
+  // false and short-circuit. Both cells are covered, so the fix is pinned where it applies and the
+  // already-working path is guarded against regression. The source projects the partition column in
+  // every case, to keep this independent of partition-column resolution (ENG-46864).

Review Comment:
   **nit, feel free to ignore** -- comment accuracy: this paragraph describes the #19708 key-generator problem rather than the meta-field layout this PR fixes; `ENG-46864` is an internal tracker id; the `validateLogBlock` note at 805-809 overclaims, since it reads only the block header whose schema comes from config on both sides of the fix (the discriminator is the merge no longer throwing); the "re-keying" labels say the opposite of `update.partition.path=false`; "short-circuit" should read "skip the merged lookup" (those cells do enter the stage); double blank line at 815; commit message and body say five cases for four.
   
   Could we tidy these in one pass?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(spark-sql): lay meta fields over the partial-update schema in the global-index merge [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19713:
URL: https://github.com/apache/hudi/pull/19713#discussion_r3851475695


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java:
##########
@@ -531,7 +591,8 @@ public static <R> HoodieData<HoodieRecord<R>> mergeForPartitionUpdatesAndDeletio
           HoodieRecord<R> existing = existingOpt.get();
 
           Option<HoodieRecord<R>> mergedOpt = mergeIncomingWithExistingRecord(
-              incoming, existing, writerSchema, writerSchemaWithMetaFields, updatedConfig,
+              incoming, existing, writerSchema, writerSchemaWithMetaFields,
+              mergedSchema, mergedSchemaWithMetaFields, partitionResolvableFromRecord, updatedConfig,

Review Comment:
   The arm this feeds at line 616 (`merged.newInstance(existing.getKey())`, changed partition with `update.partition.path=false`) returns an untagged record: `HoodieAvroRecord.newInstance(HoodieKey)` sets the location to `Option.empty()`, unlike the sibling arm at 608 that calls `tagRecord`. Untagged means it is written as an insert: the key lands in a second file group unless the slice is still a small-file candidate (no log file, base under `parquet.small.file.limit`, `SparkUpsertDeltaCommitPartitioner.java:107-116`), where `BaseSparkDeltaCommitActionExecutor.java:76-79` merges it through the COW handle and dedups by key, which is why `TestMergeIntoTable.scala:393` passes. Pre-existing from #13830 and shipped (`git tag --contains 7943e1e18849`: release-1.1.0, 1.1.1, 1.2.0); with #19709 merged, a partial `update set t.dt = s.dt, t.amount = s.amount` reaches it too.
   
   Given the release exposure, could `tagRecord(merged.newInstance(existing.getKey()), existing.getCurrentLocation())` be its own one-line PR, backportable on its own, with a test that sets `hoodie.merge.small.file.group.candidates.limit=0` and asserts a single file slice after the partition change?



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java:
##########
@@ -531,7 +591,8 @@ public static <R> HoodieData<HoodieRecord<R>> mergeForPartitionUpdatesAndDeletio
           HoodieRecord<R> existing = existingOpt.get();
 
           Option<HoodieRecord<R>> mergedOpt = mergeIncomingWithExistingRecord(
-              incoming, existing, writerSchema, writerSchemaWithMetaFields, updatedConfig,
+              incoming, existing, writerSchema, writerSchemaWithMetaFields,
+              mergedSchema, mergedSchemaWithMetaFields, partitionResolvableFromRecord, updatedConfig,

Review Comment:
   Two additions. This shipped: `git tag --contains 7943e1e18849` lists release-1.1.0, 1.1.1 and 1.2.0, and #13830 own index tests are COW, where the arm is unreachable. And the masking is exact: `SparkUpsertDeltaCommitPartitioner.java:107-116` only packs the untagged record into a slice with no log file and a base under `parquet.small.file.limit`, after which `BaseSparkDeltaCommitActionExecutor.java:76-79` merges it through the COW handle and dedups by key; `TestMergeIntoTable.scala:393` passes because six small inserts leave exactly that state.
   
   Given the release exposure, could this be its own one-line PR, backportable on its own, rather than part of this one?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(spark-sql): lay meta fields over the partial-update schema in the global-index merge [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19713:
URL: https://github.com/apache/hudi/pull/19713#issuecomment-5376885222

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b0211c8bfa6efc54521580e31af0a18e9194917f",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16474",
       "triggerID" : "b0211c8bfa6efc54521580e31af0a18e9194917f",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b0211c8bfa6efc54521580e31af0a18e9194917f Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16474) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(spark-sql): lay meta fields over the partial-update schema in the global-index merge [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19713:
URL: https://github.com/apache/hudi/pull/19713#issuecomment-5408077472

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b0211c8bfa6efc54521580e31af0a18e9194917f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16474",
       "triggerID" : "b0211c8bfa6efc54521580e31af0a18e9194917f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b186a3a0dcad3d7809de379c47bcd060182aaa1b",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16495",
       "triggerID" : "b186a3a0dcad3d7809de379c47bcd060182aaa1b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "46032578e799e656f9f886e79b2f3252c8eabb56",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "46032578e799e656f9f886e79b2f3252c8eabb56",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b186a3a0dcad3d7809de379c47bcd060182aaa1b Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16495) 
   * 46032578e799e656f9f886e79b2f3252c8eabb56 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(spark-sql): lay meta fields over the partial-update schema in the global-index merge [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19713:
URL: https://github.com/apache/hudi/pull/19713#discussion_r3852165815


##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/others/TestPartialUpdateForMergeInto.scala:
##########
@@ -732,12 +732,144 @@ class TestPartialUpdateForMergeInto extends HoodieSparkSqlTestBase {
     }
   }
 
+  // A partial update names only the columns being changed, so the record key is normally absent

Review Comment:
   Two more on that test while you are there: line 634 sets `hoodie.index.global.index.enable`, which is not a config anywhere in the tree, so it is silently dropped. And `TestMergeIntoTable.scala:1496-1498` has its MOR rows commented out citing HUDI-8835, fixed 2025-05-15 by #13263; that table is partitioned, partial-update, has no preCombineField and already sets the small-file limit to 0.
   
   Could we re-enable those two rows instead of adding a commit-time-ordering cell here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(spark-sql): lay meta fields over the partial-update schema in the global-index merge [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19713:
URL: https://github.com/apache/hudi/pull/19713#issuecomment-5409275890

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b0211c8bfa6efc54521580e31af0a18e9194917f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16474",
       "triggerID" : "b0211c8bfa6efc54521580e31af0a18e9194917f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b186a3a0dcad3d7809de379c47bcd060182aaa1b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16495",
       "triggerID" : "b186a3a0dcad3d7809de379c47bcd060182aaa1b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "46032578e799e656f9f886e79b2f3252c8eabb56",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16524",
       "triggerID" : "46032578e799e656f9f886e79b2f3252c8eabb56",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 46032578e799e656f9f886e79b2f3252c8eabb56 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16524) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(spark-sql): lay meta fields over the partial-update schema in the global-index merge [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19713:
URL: https://github.com/apache/hudi/pull/19713#discussion_r3852162966


##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/others/TestPartialUpdateForMergeInto.scala:
##########
@@ -732,12 +732,144 @@ class TestPartialUpdateForMergeInto extends HoodieSparkSqlTestBase {
     }
   }
 
+  // A partial update names only the columns being changed, so the record key is normally absent
+  // from the assignments. On MOR the global-index tagging stage merges the incoming record with its
+  // existing version and then asks the key generator for the merged record's partition path - and
+  // that merged record is materialised against WRITE_PARTIAL_UPDATE_SCHEMA, which carries only the
+  // assigned columns. Resolving the partition path must therefore not also require the record key.
+  //
+  // The index types split on whether that tagging stage runs at all: GLOBAL_BLOOM and GLOBAL_SIMPLE
+  // set mayContainDuplicateLookup on MOR and so reach it, while the record-index spellings pass
+  // false and short-circuit. Both cells are covered, so the fix is pinned where it applies and the
+  // already-working path is guarded against regression. The source projects the partition column in
+  // every case, to keep this independent of partition-column resolution (ENG-46864).
+  Seq(
+    ("GLOBAL_BLOOM re-keying", Map(
+      "hoodie.index.type" -> "GLOBAL_BLOOM",
+      "hoodie.bloom.index.update.partition.path" -> "false")),
+    ("GLOBAL_SIMPLE re-keying", Map(
+      "hoodie.index.type" -> "GLOBAL_SIMPLE",
+      "hoodie.simple.index.update.partition.path" -> "false")),
+    ("RECORD_INDEX", Map(
+      "hoodie.index.type" -> "RECORD_INDEX",
+      "hoodie.record.index.update.partition.path" -> "false",
+      "hoodie.metadata.enable" -> "true",
+      "hoodie.metadata.record.index.enable" -> "true")),
+    ("GLOBAL_RECORD_LEVEL_INDEX", Map(
+      "hoodie.index.type" -> "GLOBAL_RECORD_LEVEL_INDEX",
+      "hoodie.record.index.update.partition.path" -> "false",
+      "hoodie.metadata.enable" -> "true",
+      "hoodie.metadata.record.index.enable" -> "true"))

Review Comment:
   **Correction** to my second paragraph: the matrix does observe `partitionResolvableFromRecord`. With the arm reverted the untagged record counts as an insert, `UpsertPartitioner.java:189-216` opens a small-file UPDATE bucket at the existing fileId, and `BaseSparkDeltaCommitActionExecutor.java:76-79` diverts it to the COW merge handle: zero log files, so `validateLogBlock(..., 1, ...)` fails at `:877`. The duplicate-cell point and the `update.partition.path=true` cell stand. That flag is not even needed to reach the delete+insert arm: while the RLI partition is not yet built, `SparkMetadataTableGlobalRecordLevelIndex.java:79-88` falls back to `GLOBAL_SIMPLE`, whose flag defaults to `true`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(spark-sql): lay meta fields over the partial-update schema in the global-index merge [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19713:
URL: https://github.com/apache/hudi/pull/19713#discussion_r3852164381


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java:
##########
@@ -531,7 +591,8 @@ public static <R> HoodieData<HoodieRecord<R>> mergeForPartitionUpdatesAndDeletio
           HoodieRecord<R> existing = existingOpt.get();
 
           Option<HoodieRecord<R>> mergedOpt = mergeIncomingWithExistingRecord(
-              incoming, existing, writerSchema, writerSchemaWithMetaFields, updatedConfig,
+              incoming, existing, writerSchema, writerSchemaWithMetaFields,
+              mergedSchema, mergedSchemaWithMetaFields, partitionResolvableFromRecord, updatedConfig,

Review Comment:
   Two additions. This shipped: `git tag --contains 7943e1e18849` lists release-1.1.0, 1.1.1 and 1.2.0, and #13830 own index tests are COW, where the arm is unreachable. And the masking is exact: `SparkUpsertDeltaCommitPartitioner.java:107-116` only packs the untagged record into a slice with no log file and a base under `parquet.small.file.limit`, after which `BaseSparkDeltaCommitActionExecutor.java:76-79` merges it through the COW handle and dedups by key; `TestMergeIntoTable.scala:393` passes because six small inserts leave exactly that state.
   
   Given the release exposure, could this be its own one-line PR, backportable on its own, rather than part of this one?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(spark-sql): lay meta fields over the partial-update schema in the global-index merge [hudi]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19713:
URL: https://github.com/apache/hudi/pull/19713#issuecomment-5376381409

   ## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19713?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :x: Patch coverage is `0%` with `27 lines` in your changes missing coverage. Please review.
   :white_check_mark: Project coverage is 77.83%. Comparing base ([`2c14ff2`](https://app.codecov.io/gh/apache/hudi/commit/2c14ff2779ddd59e58bf67219a965ccc71ad4b7a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`b0211c8`](https://app.codecov.io/gh/apache/hudi/commit/b0211c8bfa6efc54521580e31af0a18e9194917f?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 1 commits behind head on master.
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19713?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [...n/java/org/apache/hudi/index/HoodieIndexUtils.java](https://app.codecov.io/gh/apache/hudi/pull/19713?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Findex%2FHoodieIndexUtils.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2luZGV4L0hvb2RpZUluZGV4VXRpbHMuamF2YQ==) | 0.00% | [25 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19713?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...pache/spark/sql/hudi/command/SqlKeyGenerator.scala](https://app.codecov.io/gh/apache/hudi/pull/19713?src=pr&el=tree&filepath=hudi-spark-datasource%2Fhudi-spark-common%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fspark%2Fsql%2Fhudi%2Fcommand%2FSqlKeyGenerator.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3BhcmstY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvc3Bhcmsvc3FsL2h1ZGkvY29tbWFuZC9TcWxLZXlHZW5lcmF0b3Iuc2NhbGE=) | 0.00% | [1 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19713?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@             Coverage Diff              @@
   ##             master   #19713      +/-   ##
   ============================================
   - Coverage     77.85%   77.83%   -0.03%     
   - Complexity    33189    33207      +18     
   ============================================
     Files          2532     2532              
     Lines        140050   140163     +113     
     Branches      16861    16885      +24     
   ============================================
   + Hits         109040   109094      +54     
   - Misses        23386    23440      +54     
   - Partials       7624     7629       +5     
   ```
   
   | [Components](https://app.codecov.io/gh/apache/hudi/pull/19713/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [hudi-common](https://app.codecov.io/gh/apache/hudi/pull/19713/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.36% <ø> (+<0.01%)` | :arrow_up: |
   | [hudi-client](https://app.codecov.io/gh/apache/hudi/pull/19713/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `82.79% <0.00%> (-0.11%)` | :arrow_down: |
   | [hudi-flink](https://app.codecov.io/gh/apache/hudi/pull/19713/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `85.75% <ø> (+<0.01%)` | :arrow_up: |
   | [hudi-spark-datasource](https://app.codecov.io/gh/apache/hudi/pull/19713/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `72.29% <0.00%> (+<0.01%)` | :arrow_up: |
   | [hudi-utilities](https://app.codecov.io/gh/apache/hudi/pull/19713/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `74.09% <ø> (+0.04%)` | :arrow_up: |
   | [hudi-cli](https://app.codecov.io/gh/apache/hudi/pull/19713/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `15.06% <ø> (ø)` | |
   | [hudi-hadoop](https://app.codecov.io/gh/apache/hudi/pull/19713/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `69.08% <ø> (ø)` | |
   | [hudi-sync](https://app.codecov.io/gh/apache/hudi/pull/19713/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `75.62% <ø> (+0.07%)` | :arrow_up: |
   | [hudi-io](https://app.codecov.io/gh/apache/hudi/pull/19713/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `79.02% <ø> (-0.83%)` | :arrow_down: |
   | [hudi-timeline-service](https://app.codecov.io/gh/apache/hudi/pull/19713/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.44% <ø> (ø)` | |
   | [hudi-cloud](https://app.codecov.io/gh/apache/hudi/pull/19713/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `64.33% <ø> (ø)` | |
   | [hudi-kafka-connect](https://app.codecov.io/gh/apache/hudi/pull/19713/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `53.20% <ø> (ø)` | |
   
   | [Flag](https://app.codecov.io/gh/apache/hudi/pull/19713/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19713/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `43.89% <0.00%> (-0.01%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19713?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [...pache/spark/sql/hudi/command/SqlKeyGenerator.scala](https://app.codecov.io/gh/apache/hudi/pull/19713?src=pr&el=tree&filepath=hudi-spark-datasource%2Fhudi-spark-common%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fspark%2Fsql%2Fhudi%2Fcommand%2FSqlKeyGenerator.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3BhcmstY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvc3Bhcmsvc3FsL2h1ZGkvY29tbWFuZC9TcWxLZXlHZW5lcmF0b3Iuc2NhbGE=) | `71.60% <0.00%> (-2.15%)` | :arrow_down: |
   | [...n/java/org/apache/hudi/index/HoodieIndexUtils.java](https://app.codecov.io/gh/apache/hudi/pull/19713?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Findex%2FHoodieIndexUtils.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2luZGV4L0hvb2RpZUluZGV4VXRpbHMuamF2YQ==) | `84.79% <0.00%> (-7.96%)` | :arrow_down: |
   
   ... and [17 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/19713/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   </details>
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(spark-sql): lay meta fields over the partial-update schema in the global-index merge [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19713:
URL: https://github.com/apache/hudi/pull/19713#issuecomment-5376362009

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b0211c8bfa6efc54521580e31af0a18e9194917f",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16474",
       "triggerID" : "b0211c8bfa6efc54521580e31af0a18e9194917f",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b0211c8bfa6efc54521580e31af0a18e9194917f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16474) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org