You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "lokeshj1703 (via GitHub)" <gi...@apache.org> on 2026/08/07 18:23:16 UTC
[PR] fix(metadata): fix ArrayIndexOutOfBoundsException when 1.x reader merges old MDT records [hudi]
lokeshj1703 opened a new pull request, #19560:
URL: https://github.com/apache/hudi/pull/19560
### Describe the issue this Pull Request addresses
Closes #19559
When a Hudi 1.x reader reads a Metadata Table (MDT) that was written by an older Hudi release, merging log-file delta records against base-file records throws an `ArrayIndexOutOfBoundsException`:
```
Caused by: java.lang.ArrayIndexOutOfBoundsException: 6
at org.apache.avro.generic.GenericData$Record.put(GenericData.java:273)
at org.apache.hudi.metadata.HoodieMetadataPayload.getInsertValue(HoodieMetadataPayload.java:422)
at org.apache.hudi.metadata.HoodieMetadataPayload.combineAndGetUpdateValue(HoodieMetadataPayload.java:399)
at org.apache.hudi.common.model.HoodieAvroRecordMerger.merge(HoodieAvroRecordMerger.java:67)
at org.apache.hudi.common.table.read.BufferedRecordMergerFactory$CustomPayloadRecordMerger.deltaMergeRecords(BufferedRecordMergerFactory.java:404)
at org.apache.hudi.common.table.read.BufferedRecordMergerFactory$BaseCustomMerger.deltaMerge(BufferedRecordMergerFactory.java:439)
at org.apache.hudi.common.table.read.buffer.KeyBasedFileGroupRecordBuffer.processNextDataRecord(...)
```
### Summary and Changelog
**Root cause:** `SecondaryIndexMetadata` was added to `HoodieMetadataRecord` in a later Hudi release, growing the schema from 6 to 7 fields. MDT log blocks written by older releases carry the 6-field writer schema in their block header. When merging, `HoodieAvroRecordMerger` retrieves the writer schema from the buffered record and passes it to `HoodieMetadataPayload.getInsertValue(schema)`.
Inside `getInsertValue`, the fast path uses an object-identity check:
```java
if (schema == null || schema == HOODIE_METADATA_AVRO_SCHEMA) {
// safe path
} else {
// assumes HOODIE_META_COLUMNS are prepended; uses TYPE_FIELD_OFFSET=6
record.put(TYPE_FIELD_OFFSET, type); // index 6 on a 6-field schema → ArrayIndexOutOfBoundsException
}
```
The old 6-field schema is a different `Schema` object from `HOODIE_METADATA_AVRO_SCHEMA` (7 fields), so it fails the identity check and falls into the `else` branch designed for schemas with `HOODIE_META_COLUMNS` prepended. Writing to index 6 on a 6-field schema throws `ArrayIndexOutOfBoundsException`.
**Fix:** Add `isHoodieMetadataRecordSchema(Schema)` that recognises any version of `HoodieMetadataRecord` by Avro record name and namespace. Route these older-version schemas to the same fast path as the current schema, returning a `HoodieMetadataRecord` with the current schema (`SecondaryIndexMetadata` defaults to `null`).
**Changes:**
- `HoodieMetadataPayload.java`: add `isHoodieMetadataRecordSchema()` helper; extend the fast-path guard in `getInsertValue()`
- `TestHoodieMetadataPayload.java`: add `testGetInsertValueWithOldMetadataSchema` regression test that constructs the old 6-field schema and verifies no exception is thrown
### Impact
No user-facing API or config change. Tables written by older Hudi releases can be read safely by a 1.x reader without requiring a table upgrade.
### Risk Level
Low. The change is confined to `getInsertValue()` in `HoodieMetadataPayload`. The added condition only triggers when the schema is a `HoodieMetadataRecord` schema that is not object-identical to `HOODIE_METADATA_AVRO_SCHEMA` (i.e. an older-version schema read from a log block); all other callers are unaffected.
### Documentation Update
None.
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(metadata): fix ArrayIndexOutOfBoundsException when 1.x reader merges old MDT records [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19560:
URL: https://github.com/apache/hudi/pull/19560#discussion_r3737940329
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java:
##########
@@ -426,8 +432,11 @@ public Option<IndexedRecord> getInsertValue(Schema schema, Properties properties
// TODO: feature(schema): Swap this over to HOODIE_METADATA_SCHEMA after HoodieRecordPayload implementations are using HoodieSchema
// Uses cached Avro schema reference for O(1) equality check.
- if (schema == null || schema == HOODIE_METADATA_AVRO_SCHEMA) {
- // If the schema is same or none is provided, we can return the record directly
+ if (schema == null || schema == HOODIE_METADATA_AVRO_SCHEMA || isHoodieMetadataRecordSchema(schema)) {
Review Comment:
🤖 `isHoodieMetadataRecordSchema` only checks the record name/namespace, so it matches *any* schema named `HoodieMetadataRecord` — including one with the 5 Hudi meta columns prepended (`addMetadataFields` preserves the name) or a projected subset. Those cases previously fell into the else-branch. Now they take the fast path and get back the full 7-field record regardless of the requested layout. Is that safe for every caller here (e.g. a `populateMetaFields=true` MDT or a projected read), or could it hand back a record that doesn't match the schema the caller asked for?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java:
##########
@@ -390,6 +390,12 @@ public HoodieMetadataPayload preCombine(HoodieMetadataPayload previousRecord) {
return MetadataPartitionType.get(type).combineMetadataPayloads(previousRecord, this);
}
+ private static boolean isHoodieMetadataRecordSchema(Schema schema) {
Review Comment:
🤖 nit: `isHoodieMetadataRecordSchema` describes the structural check but not the intent — could you rename it to something like `isCompatibleMetadataRecordSchema` or `isOlderVersionMetadataRecordSchema` to signal that this exists specifically to handle older schemas missing newer fields?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(metadata): fix ArrayIndexOutOfBoundsException when 1.x reader merges old MDT records [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19560:
URL: https://github.com/apache/hudi/pull/19560#issuecomment-5220772386
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26b2385d74c86bc0265e669786f4e1befe4adcad",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "26b2385d74c86bc0265e669786f4e1befe4adcad",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 26b2385d74c86bc0265e669786f4e1befe4adcad UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(metadata): fix ArrayIndexOutOfBoundsException when 1.x reader merges old MDT records [hudi]
Posted by "voonhous (via GitHub) via commits" <co...@hudi.apache.org>.
voonhous commented on PR #19560:
URL: https://github.com/apache/hudi/pull/19560#issuecomment-6701269207
Temporarily closing this PR while we debug a GitHub settings error on master (#20471); it will be reopened shortly, no action needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(metadata): fix ArrayIndexOutOfBoundsException when 1.x reader merges old MDT records [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19560:
URL: https://github.com/apache/hudi/pull/19560#issuecomment-5220823699
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26b2385d74c86bc0265e669786f4e1befe4adcad",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16184",
"triggerID" : "26b2385d74c86bc0265e669786f4e1befe4adcad",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 26b2385d74c86bc0265e669786f4e1befe4adcad Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16184)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(metadata): fix ArrayIndexOutOfBoundsException when 1.x reader merges old MDT records [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19560:
URL: https://github.com/apache/hudi/pull/19560#issuecomment-5221026345
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26b2385d74c86bc0265e669786f4e1befe4adcad",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16184",
"triggerID" : "26b2385d74c86bc0265e669786f4e1befe4adcad",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 26b2385d74c86bc0265e669786f4e1befe4adcad Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16184)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(metadata): fix ArrayIndexOutOfBoundsException when 1.x reader merges old MDT records [hudi]
Posted by "voonhous (via GitHub) via commits" <co...@hudi.apache.org>.
voonhous closed pull request #19560: fix(metadata): fix ArrayIndexOutOfBoundsException when 1.x reader merges old MDT records
URL: https://github.com/apache/hudi/pull/19560
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org