You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "deepakpanda93 (via GitHub)" <gi...@apache.org> on 2026/08/25 12:14:58 UTC

[PR] docs(tech-specs): update the delete block format for Avro serde [hudi]

deepakpanda93 opened a new pull request, #19730:
URL: https://github.com/apache/hudi/pull/19730

   ### Describe the issue this Pull Request addresses
   
   Addresses #16138 (JIRA [HUDI-6616](https://issues.apache.org/jira/browse/HUDI-6616)).
   
   HUDI-5760 replaced Kryo with Avro as the serde for delete log blocks, but the tech spec was never updated. The **Delete
   Block (Id: 1)** section still told readers:
   
   > Tombstone of the record to encode a delete. **The following 3 fields are serialized using the KryoSerializer.**
   
   That has not been true since HUDI-5760 landed.
   
   The issue also asks (@vinothchandar's comment) for positional headers, block uuid headers and "other changes". Those have
   since been documented independently: `RECORD_POSITIONS` (5), `BLOCK_IDENTIFIER` (6), `IS_PARTIAL` (7),
   `COMPACTED_BLOCK_TIMES` (4) and `BASE_FILE_INSTANT_TIME_OF_RECORD_POSITIONS` (8) are all present in the **Headers** table
   today. I checked each rather than assuming, so this PR is scoped to what actually remains: the block content tables.
   
   ### Summary and Changelog
   
   One file, `website/learn/tech-specs.md` (+32/−5). `learn/` is an unversioned Docusaurus plugin, so there is a single copy
   and no versioned duplicates.
   
   **1. Delete Block: the payload encoding is versioned, not simply Kryo.** `HoodieDeleteBlock` dispatches on the log block
   version, and older blocks stay readable, so the section now documents all three encodings rather than only the current
   one:
   
   | Block version | Encoding | Tombstone contents |
   |---|---|---|
   | 1 | Kryo-serialized `HoodieKey[]` | Record key and partition path only. **No ordering value.** |
   | 2 | Kryo-serialized `DeleteRecord[]` | Record key, partition path, ordering value |
   | 3 | Avro, binary-encoded `HoodieDeleteRecordList` | Record key, partition path, **typed** ordering value |
   
   Version 3 is what current writers emit, since `HoodieLogBlock.version` is `3`. The v1 row matters in practice: a reader
   encountering a v1 block gets no ordering value at all, which the old text gave no way to anticipate.
   
   **2. The version 3 record fields**, from `HoodieDeleteRecordList.avsc`: `recordKey` and `partitionPath` as nullable
   strings, and `orderingVal` as a union of typed wrappers (`BooleanWrapper`, `IntWrapper`, `LongWrapper`, `FloatWrapper`,
   `DoubleWrapper`, `BytesWrapper`, `StringWrapper`, `DateWrapper`, `DecimalWrapper`, `TimeMicrosWrapper`,
   `TimestampMicrosWrapper`, `ArrayWrapper`). That typed ordering value is the substantive gain over Kryo, alongside the
   block no longer needing a JVM with matching Kryo registrations to read. The schema's own doc string confirms the pairing:
   *"A list of delete records stored in the delete block in log block version 3"*.
   
   <!-- SNAPSHOT 1 HERE: rendered Delete Block (Id: 1) section -->
   
   **3. Byte widths corrected in two tables.** Both the Delete Block `length` field and the Avro Block `record length` field
   were documented as **8** bytes. `HoodieDeleteBlock#getContentBytes` and `HoodieAvroDataBlock#serializeRecords` both write
   them with `output.writeInt`, so both are **4**:
   
   ```java
   // HoodieDeleteBlock#getContentBytes
   output.writeInt(version);
   byte[] bytesToWrite = (version <= 2) ? serializeV2() : serializeV3();
   output.writeInt(bytesToWrite.length);
   output.write(bytesToWrite);
   ```
   
   Worth being explicit that this is not a confusion with the outer log block, which genuinely *does* have an 8-byte
   `content length` field. These tables describe the bytes **inside** the block content. I also checked that
   `HoodieAvroDataBlock` has two serialization paths and that the 8-byte reading does not come from the other one: the
   `@Deprecated getBytes(Schema)` method is a legacy writer that compresses and writes the schema, and is not what current
   writers use.
   
   **4. `format version` renamed to `block version`** in both tables. The value written is `HoodieLogBlock.version`,
   currently `3`, which is the log **block** version. The log **file** format version is a separate concept that the
   Versioning section of this same page documents as currently `1`, so labelling this row "version of the log file format"
   conflated the two. After this change no block table row says "format version"; the remaining occurrences on the page are
   unrelated prose about `hoodie.table.version` and the Versioning section.
   
   <!-- SNAPSHOT 2 HERE: rendered Avro Block (Id: 3) section -->
   
   ### Scope note
   
   The Avro Block correction is slightly wider than issue #16138, which is about the delete block. I included it for two
   reasons: it is the same class of error verified the same way, and renaming the Delete Block's `format version` row would
   otherwise have left the two adjacent tables describing the identical field under different names. Happy to split it out
   if reviewers would rather keep this strictly to the delete block.
   
   ### Verification
   
   Every claim was read out of `release-1.2.0` source rather than inferred: `HoodieDeleteBlock`,
   `HoodieDeleteRecordList.avsc`, `HoodieLogBlock` (for `version = 3`), `SerializationUtils` (confirming v1/v2 are
   Kryo-backed via `KryoSerializerInstance`), `DeleteRecord`, and `HoodieAvroDataBlock` (both the current
   `serializeRecords` and the deprecated `getBytes`).
   
   `npm run build` passes with the warning block **byte-identical** to a baseline built from the same base commit
   (`5971a1ac3ba3`), 13,265 lines each. The branch is rebased on that head, so the workflow's changed-file check sees only
   the one `website/` file. Rendering confirmed under `npm run serve`: both tables render with the corrected widths, the
   Kryo sentence is gone, all twelve wrapper types appear, and the new `RECORD_POSITIONS` cross-reference resolves to
   `#headers`.
   
   One limitation worth stating: this is verified by reading the format code, not by decoding a delete block off disk at each
   version. A committer who can confirm the v1/v2/v3 dispatch against real log files would strengthen it, particularly the
   claim that v1 blocks carry no ordering value.
   
   ### Impact
   
   Documentation only. No code, config, or behaviour change. It does correct statements that are currently wrong about both
   the delete block encoding and two field widths.
   
   ### Risk Level
   
   none
   
   ### Documentation Update
   
   This PR is the documentation update — the tech spec, https://hudi.apache.org/learn/tech-specs.
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   
   cc @vinothchandar (who raised the original list on the issue), @yihua
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] docs(tech-specs): update the delete block format for Avro serde [hudi]

Posted by "deepakpanda93 (via GitHub)" <gi...@apache.org>.
deepakpanda93 commented on code in PR #19730:
URL: https://github.com/apache/hudi/pull/19730#discussion_r3853109634


##########
website/learn/tech-specs.md:
##########
@@ -295,11 +295,38 @@ This denotes that the previous action that wrote the log block was unsuccessful.
 
 ### Delete Block (Id: 1)
 
+A delete block carries the tombstones for records deleted by a commit. Within a batch it is always written after the
+data (Avro/HFile/Parquet) block, so that the deletes are applied after the inserts and updates they accompany.
+
 | Section | #bytes | Description |
 | ---| ---| --- |
-| format version | 4 | version of the log file format |
-| length | 8 | length of the deleted keys section to follow |
-| deleted keys | variable | Tombstone of the record to encode a delete. The following 3 fields are serialized using the KryoSerializer. **Record Key** - Unique record key within the partition to deleted **Partition Path** - Partition path of the record deleted **Ordering Value** - In a particular batch of updates, the delete block is always written after the data (Avro/HFile/Parquet) block. This field would preserve the ordering of deletes and inserts within the same batch. |
+| block version | 4 | The log block version the writer emitted. The reader selects the payload encoding below from this value. |
+| length | 4 | Length in bytes of the payload to follow |
+| payload | variable | The serialized tombstones, in the encoding selected by the block version |
+
+The payload encoding has changed twice, and readers dispatch on the block version so blocks written by older writers
+remain readable:
+
+| Block version | Encoding | Tombstone contents |
+|---------------|----------|--------------------|
+| 1             | Kryo-serialized `HoodieKey[]` | Record key and partition path only. These blocks carry no ordering value. |
+| 2             | Kryo-serialized `DeleteRecord[]` | Record key, partition path and ordering value. |
+| 3             | Avro, binary-encoded `HoodieDeleteRecordList` | Record key, partition path and a typed ordering value. |
+
+Version 3 is what current writers produce. Encoding the payload with Avro rather than Kryo makes a delete block readable
+by any Avro implementation instead of only a JVM with matching Kryo registrations, and it gives the ordering value a
+declared type rather than leaving it an opaque serialized object.
+
+Each element of the version 3 `deleteRecordList` array is a `HoodieDeleteRecord`:
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `recordKey` | nullable `string` | Unique record key within the partition being deleted. |
+| `partitionPath` | nullable `string` | Partition path of the record being deleted. |
+| `orderingVal` | nullable union of typed wrappers | Ordering value used to resolve merge order against other writes to the same key, encoded with the wrapper matching the value's own type: `BooleanWrapper`, `IntWrapper`, `LongWrapper`, `FloatWrapper`, `DoubleWrapper`, `BytesWrapper`, `StringWrapper`, `DateWrapper`, `DecimalWrapper`, `TimeMicrosWrapper`, `TimestampMicrosWrapper` or `ArrayWrapper`. |
+
+A delete block may also carry the [`RECORD_POSITIONS`](#headers) header. When it does, the reader can apply the deletes
+positionally against the base file named by `BASE_FILE_INSTANT_TIME_OF_RECORD_POSITIONS` instead of resolving each key.

Review Comment:
   Good catch, that reads wrong. Fixed in 62a96e91.
   
   You are right that the header holds an instant time. `HoodieAppendHandle` writes `baseInstantTimeForPositions` into it as a `String`:
   
   ```java
   if (baseInstantTimeForPositions.isPresent()) {
     updatedHeader.put(
         HeaderMetadataType.BASE_FILE_INSTANT_TIME_OF_RECORD_POSITIONS,
         baseInstantTimeForPositions.get());
   }
   ```
   
   and the reader gets it back through `HoodieLogBlock#getBaseFileInstantTimeOfPositions`, whose javadoc is explicit that it returns the "base file instant time of the record positions". Nothing in the block carries a path, so "named by" was misleading about what the reader actually has to work with.
   
   What tipped it from a wording nit to something worth fixing is that my sentence was also inconsistent with the rest of the same page. The Headers table already describes this header as the *"Begin (requested) instant time of the base file that the `RECORD_POSITIONS` bitmap is relative to"*, and the log-merge section already says *"the base file identified by `BASE_FILE_INSTANT_TIME_OF_RECORD_POSITIONS`"*. I had introduced a third phrasing for the same thing, in a spec where consistency is most of the value.
   
   It now reads:
   
   > A delete block may also carry the `RECORD_POSITIONS` header. When it does, the reader can apply the deletes positionally instead of resolving each key, against the base file identified by the instant time in `BASE_FILE_INSTANT_TIME_OF_RECORD_POSITIONS`. The reader matches that instant time against the base file of the file slice, which is also how it validates that the positions still refer to the base file the bitmap was built against.
   
   I added that last clause because it answers the question your comment implicitly raises: why carry an instant time rather than something more direct. The Headers table already gives the reason ("Used by the reader to validate that positions still refer to the same base file"), so this connects the delete block note to it instead of leaving the reader to infer the purpose.
   
   Build passes with the warning block still byte-identical to a baseline built at the same base commit, and the rendered page no longer contains "named by".
   
   Thanks also for verifying the byte widths, the version dispatch, the schema field names, the full wrapper list and the current block version against `hudi-common`. Agreed on routing to a committer. @yihua, the two things I would most value a committer's eye on are the v1 claim that those blocks carry no ordering value at all, and whether the "block version" relabel is the terminology the project wants in the spec, given the log *file* format version is separately documented as 1 on the same page.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] docs(tech-specs): update the delete block format for Avro serde [hudi]

Posted by "voonhous (via GitHub) via commits" <co...@hudi.apache.org>.
voonhous commented on PR #19730:
URL: https://github.com/apache/hudi/pull/19730#issuecomment-6701343877

   Temporarily closing this PR while we debug a GitHub settings error on master (#20471); it will be reopened shortly, no action needed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] docs(tech-specs): update the delete block format for Avro serde [hudi]

Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19730:
URL: https://github.com/apache/hudi/pull/19730#discussion_r3852942992


##########
website/learn/tech-specs.md:
##########
@@ -295,11 +295,38 @@ This denotes that the previous action that wrote the log block was unsuccessful.
 
 ### Delete Block (Id: 1)
 
+A delete block carries the tombstones for records deleted by a commit. Within a batch it is always written after the
+data (Avro/HFile/Parquet) block, so that the deletes are applied after the inserts and updates they accompany.
+
 | Section | #bytes | Description |
 | ---| ---| --- |
-| format version | 4 | version of the log file format |
-| length | 8 | length of the deleted keys section to follow |
-| deleted keys | variable | Tombstone of the record to encode a delete. The following 3 fields are serialized using the KryoSerializer. **Record Key** - Unique record key within the partition to deleted **Partition Path** - Partition path of the record deleted **Ordering Value** - In a particular batch of updates, the delete block is always written after the data (Avro/HFile/Parquet) block. This field would preserve the ordering of deletes and inserts within the same batch. |
+| block version | 4 | The log block version the writer emitted. The reader selects the payload encoding below from this value. |
+| length | 4 | Length in bytes of the payload to follow |
+| payload | variable | The serialized tombstones, in the encoding selected by the block version |
+
+The payload encoding has changed twice, and readers dispatch on the block version so blocks written by older writers
+remain readable:
+
+| Block version | Encoding | Tombstone contents |
+|---------------|----------|--------------------|
+| 1             | Kryo-serialized `HoodieKey[]` | Record key and partition path only. These blocks carry no ordering value. |
+| 2             | Kryo-serialized `DeleteRecord[]` | Record key, partition path and ordering value. |
+| 3             | Avro, binary-encoded `HoodieDeleteRecordList` | Record key, partition path and a typed ordering value. |
+
+Version 3 is what current writers produce. Encoding the payload with Avro rather than Kryo makes a delete block readable
+by any Avro implementation instead of only a JVM with matching Kryo registrations, and it gives the ordering value a
+declared type rather than leaving it an opaque serialized object.
+
+Each element of the version 3 `deleteRecordList` array is a `HoodieDeleteRecord`:
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `recordKey` | nullable `string` | Unique record key within the partition being deleted. |
+| `partitionPath` | nullable `string` | Partition path of the record being deleted. |
+| `orderingVal` | nullable union of typed wrappers | Ordering value used to resolve merge order against other writes to the same key, encoded with the wrapper matching the value's own type: `BooleanWrapper`, `IntWrapper`, `LongWrapper`, `FloatWrapper`, `DoubleWrapper`, `BytesWrapper`, `StringWrapper`, `DateWrapper`, `DecimalWrapper`, `TimeMicrosWrapper`, `TimestampMicrosWrapper` or `ArrayWrapper`. |
+
+A delete block may also carry the [`RECORD_POSITIONS`](#headers) header. When it does, the reader can apply the deletes
+positionally against the base file named by `BASE_FILE_INSTANT_TIME_OF_RECORD_POSITIONS` instead of resolving each key.

Review Comment:
   🤖 This positional-delete note is accurate and a nice addition. One small clarification: `BASE_FILE_INSTANT_TIME_OF_RECORD_POSITIONS` holds an *instant time*, not a base-file path, so "the base file named by" reads a bit loosely — the reader resolves the base file within the file slice by matching that instant time. Rewording to something like "the base file identified by the instant time in `BASE_FILE_INSTANT_TIME_OF_RECORD_POSITIONS`" might help readers.
   
   <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] docs(tech-specs): update the delete block format for Avro serde [hudi]

Posted by "voonhous (via GitHub) via commits" <co...@hudi.apache.org>.
voonhous closed pull request #19730: docs(tech-specs): update the delete block format for Avro serde
URL: https://github.com/apache/hudi/pull/19730


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org