You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "deepakpanda93 (via GitHub)" <gi...@apache.org> on 2026/08/10 13:55:31 UTC

[PR] docs(streamer): document checkpoint v2 reset for the Hudi incremental source [hudi]

deepakpanda93 opened a new pull request, #19573:
URL: https://github.com/apache/hudi/pull/19573

   ### Describe the issue this Pull Request addresses
   
   Closes #16813. (JIRA: HUDI-8927.)
   
   #12718 changed how `--checkpoint` is interpreted for the Hudi incremental source once the target table is at table
   version 8 or higher: the value must now be prefixed to say whether it is a requested instant time or a completion time.
   The **Checkpointing** section of the Hudi Streamer page still described only the Kafka and DFS formats, so a user
   resetting a checkpoint on such a table hits
   
   ```
   Illegal checkpoint key override `20250110120000000`. Valid format is either
   `resumeFromInstantRequestTime:<checkpoint value>` or `resumeFromInstantCompletionTime:<checkpoint value>`.
   ```
   
   with no documentation to consult. The PR that introduced this asked for the docs follow-up in its own Documentation
   Update section, which is what this addresses.
   
   ### Summary and Changelog
   
   Added a `#### Resetting the checkpoint for the Hudi incremental source` subsection under **Checkpointing** covering the
   two prefixed forms, the error you get without a prefix, the S3/GCS exclusion, the version floor, and the upgrade
   restriction.
   
   Also corrected an existing inaccuracy in the same section, described below.
   
   ### Reproduction — all six 1.x releases
   
   Ran real `HoodieStreamer` jobs on Spark 3.5.7, each release using its own `hudi-spark3.5-bundle` and
   `hudi-utilities-slim-bundle`, with a source table built by that same release's writer. The reset boundary is source
   instant #2; the bare case was given that instant's requested time, and the prefixed cases named the same instant by
   requested time and by completion time.
   
   | Version | Table ver | bare value | `resumeFromInstantRequestTime:` | `resumeFromInstantCompletionTime:` | Rows ingested (prefixed) |
   |---|---|---|---|---|---|
   | **1.0.0** | 8 | :white_check_mark: accepted | :x: **fails** | :x: **fails** | — |
   | 1.0.1 | 8 | :x: rejected | :white_check_mark: | :white_check_mark: | `31,32,41,42` (identical) |
   | 1.0.2 | 8 | :x: rejected | :white_check_mark: | :white_check_mark: | :white_check_mark: |
   | 1.1.0 | 9 | :x: rejected | :white_check_mark: | :white_check_mark: | :white_check_mark: |
   | 1.1.1 | 9 | :x: rejected | :white_check_mark: | :white_check_mark: | :white_check_mark: |
   | 1.2.0 | 9 | :x: rejected | :white_check_mark: | :white_check_mark: | `31,32,41,42` (identical) |
   
   Every rejection is the `Illegal checkpoint key override` message above. Every success wrote
   `streamer.checkpoint.key.v2` plus `streamer.checkpoint.reset.key.v2`, the latter holding the prefixed value verbatim.
   
   Three things this settled that source reading alone would not have:
   
   1. **The behaviour is completion time based today.** #12718's own note said *"the hoodie incremental sources are still
      using request time based handling in 1.x"*. That is no longer true: a run with no override stored
      `streamer.checkpoint.key.v2 = 20260810122008215`, the completion time of the last source commit. Documenting from the
      PR description alone would have got this backwards.
   2. **Both prefixes name the same boundary.** They produced byte-identical row sets on the earliest and the latest
      affected release, confirming the requested time form is translated to that instant's completion time rather than
      being a separate mode.
   3. **1.0.0 is not simply missing the feature, it behaves oppositely.** The prefixed form dies with
      `DateTimeParseException: Text 'resumeFromInstant' could not be parsed at index 0`, while a bare value is accepted and
      read as a *completion* time — so passing a requested time there silently resumes from the wrong place (six records
      ingested where 1.0.1+ ingests four). That ambiguity is exactly what #12718 removed, and it is why the new text states
      a 1.0.1 floor.
   
   Also verified the `S3EventsHoodieIncrSource` exclusion: with a bare `--checkpoint` the prefix parser never engages
   (zero rejections) and the job proceeds to fail later on unrelated S3 event schema, matching
   `CheckpointUtils.DATASOURCES_NOT_SUPPORTED_WITH_CKPT_V2`.
   
   ### Correcting the checkpoint metadata key
   
   The section said checkpoints are saved as `streamer.checkpoint.key`, which is not a key Hudi writes. The actual keys are
   
   | Checkpoint | Key | Defined at |
   |---|---|---|
   | request time | `deltastreamer.checkpoint.key`, `deltastreamer.checkpoint.reset_key` | `StreamerCheckpointV1.java:29-30`, written at `:55`/`:60` |
   | completion time | `streamer.checkpoint.key.v2`, `streamer.checkpoint.reset.key.v2` | `StreamerCheckpointV2.java:32-33`, written at `:61`/`:66` |
   
   selected by `CheckpointUtils.shouldTargetCheckpointV2` on the write table version. Confirmed by running the streamer
   both ways against the same source: the default version-9 target commit carries `streamer.checkpoint.key.v2`, and a
   target forced to `hoodie.write.table.version=6` carries
   `deltastreamer.checkpoint.key" : "20260810134321000`.
   
   `StreamerCheckpointV2` also has an `addV1Props()` method that would write the v1 keys alongside the v2 ones, so I
   checked before wording this as an either/or: it has no callers anywhere in main or test at `release-1.2.0`, and no
   commit produced during testing carried both sets.
   
   One simplification worth flagging: the corrected sentence ties the split to table version, which is what
   `shouldTargetCheckpointV2` does, but that method also excludes the S3 and GCS incremental sources, so those stay on the
   v1 key even at version 8 and above. That carve-out is stated in the note a few lines below, so the two read together
   correctly. Happy to make the intro sentence carry the exception explicitly if reviewers prefer.
   
   ### Version scope
   
   Applied to `next`, 1.2.0, 1.1.1, 1.0.2 and 1.0.1 — every release where the prefixed form works, verified by execution
   rather than inference, and covering both table version 8 and 9.
   
   **Not 1.0.0**, where the prefixed form fails outright. `HoodieIncrSourceCheckpointValUtils` does not exist at
   `release-1.0.0`, and the test above shows what a user would actually hit. There is no `version-1.1.0` docs folder, so
   although 1.1.0 was tested and behaves like the rest there is nothing to change for it. 0.14.x and 0.15.x predate
   checkpoint v2 entirely.
   
   ### Site verification
   
   `npm run build` passes with the warning set byte-identical to a baseline build of the same base commit — no new warnings
   and no new broken anchors. Rendering checked under `npm run serve` on `/docs`, `/docs/next`, `/docs/1.1.1`,
   `/docs/1.0.2` and `/docs/1.0.1`: the new heading, anchor, TOC entry, both admonitions, all three code fences, the
   corrected key sentence and the 1.0.1 floor are all present. `/docs/1.0.0` correctly shows none of it.
   
   ### Impact
   
   Documentation only. No code, config, or behaviour change.
   
   ### Risk Level
   
   none
   
   ### Documentation Update
   
   This PR is the documentation update — the Hudi Streamer page, `/docs/hoodie_streaming_ingestion` and
   `/docs/next/hoodie_streaming_ingestion`.
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] docs(streamer): document checkpoint v2 reset for the Hudi incremental source [hudi]

Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19573:
URL: https://github.com/apache/hudi/pull/19573#discussion_r3753941406


##########
website/docs/hoodie_streaming_ingestion.md:
##########
@@ -312,14 +312,54 @@ Read more in depth about concurrency control in the [concurrency control concept
 Hudi Streamer uses checkpoints to keep track of what data has been read already so it can resume without needing to reprocess all data.
 When using a Kafka source, the checkpoint is the [Kafka Offset](https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management) 
 When using a DFS source, the checkpoint is the 'last modified' timestamp of the latest file read.
-Checkpoints are saved in the .hoodie commit file as `streamer.checkpoint.key`.
+Checkpoints are saved in the .hoodie commit file, under `streamer.checkpoint.key.v2` for tables at table
+version 8 or higher and `deltastreamer.checkpoint.key` below that.

Review Comment:
   🤖 This line reads as though every source's checkpoint moves to `streamer.checkpoint.key.v2` at table version 8+, but that isn't what the code does — and it sits directly after the Kafka and DFS descriptions, which is exactly where a reader will apply it. In `StreamerCheckpointUtils.shouldTargetCheckpointV2()` the V2 key is used only for the `HoodieIncrSource` family (and excludes `S3EventsHoodieIncrSource`/`GcsEventsHoodieIncrSource`); `StreamSync.getCheckpointCommitMetadata()` otherwise falls back to `createCheckpoint()`, which always produces a `StreamerCheckpointV1` writing under `deltastreamer.checkpoint.key`. So a Kafka or DFS checkpoint stays under `deltastreamer.checkpoint.key` even on a table-version-8+ table. It would help to scope this sentence to the Hudi incremental source rather than to table version alone, so Kafka/DFS users don't go looking for a `.v2` key that never gets written. @yihua could you confirm the intended key semantics here — the checkpoint class
 es carry your TODOs, so you'd know best whether V2 keys are ever written for non-incremental sources at v8+? (Same wording appears in the version-1.0.1/1.0.2/1.1.1/1.2.0 copies.)
   
   <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>



##########
website/docs/hoodie_streaming_ingestion.md:
##########
@@ -312,14 +312,54 @@ Read more in depth about concurrency control in the [concurrency control concept
 Hudi Streamer uses checkpoints to keep track of what data has been read already so it can resume without needing to reprocess all data.
 When using a Kafka source, the checkpoint is the [Kafka Offset](https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management) 
 When using a DFS source, the checkpoint is the 'last modified' timestamp of the latest file read.
-Checkpoints are saved in the .hoodie commit file as `streamer.checkpoint.key`.
+Checkpoints are saved in the .hoodie commit file, under `streamer.checkpoint.key.v2` for tables at table
+version 8 or higher and `deltastreamer.checkpoint.key` below that.
 
 If you need to change the checkpoints for reprocessing or replaying data you can use the following options:
 
-- `--checkpoint` will set `streamer.checkpoint.reset_key` in the commit file to overwrite the current checkpoint. Format of checkpoint depends on [KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype). By default (for type `string`), checkpoint should be provided as: `topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint should be provided as long value of desired timestamp. For type `single_offset`, we assume that topic consists of a single partition, so checkpoint should be provided as long value of desired offset.
+- `--checkpoint` will set the checkpoint reset key in the commit file to overwrite the current checkpoint — `streamer.checkpoint.reset.key.v2` at table version 8 or higher, `deltastreamer.checkpoint.reset_key` below that. Format of checkpoint depends on [KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype). By default (for type `string`), checkpoint should be provided as: `topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint should be provided as long value of desired timestamp. For type `single_offset`, we assume that topic consists of a single partition, so checkpoint should be provided as long value of desired offset.
 - `--source-limit` will set a maximum amount of data to read from the source. For DFS sources, this is max # of bytes read.
 For Kafka, this is the max # of events to read.

Review Comment:
   🤖 Same root cause as the intro line: this bullet now says `--checkpoint` sets `streamer.checkpoint.reset.key.v2` at table version 8+, but `buildCheckpointFromConfigOverride()` only returns the V2-capable (`UnresolvedStreamerCheckpointBasedOnCfg`) checkpoint for the `HoodieIncrSource` family — for Kafka/DFS sources it returns `StreamerCheckpointV1`, whose `getCheckpointCommitMetadata()` writes the reset value under `deltastreamer.checkpoint.reset_key` regardless of table version. Since this bullet is describing the Kafka-format `--checkpoint` value, the reset key it will actually see is `deltastreamer.checkpoint.reset_key`, not the `.v2` one. Consider limiting the `.v2` reset-key mention to the Hudi incremental source (where the new subsection already covers it) and keeping the V1 key for the Kafka/DFS bullet. This is a different point from the earlier name-reconciliation thread — the concern here is that the `.v2` name is wrong for the non-incremental sources this bullet do
 cuments.
   
   <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] docs(streamer): document checkpoint v2 reset for the Hudi incremental source [hudi]

Posted by "deepakpanda93 (via GitHub)" <gi...@apache.org>.
deepakpanda93 commented on code in PR #19573:
URL: https://github.com/apache/hudi/pull/19573#discussion_r3750547072


##########
website/docs/hoodie_streaming_ingestion.md:
##########
@@ -312,14 +312,54 @@ Read more in depth about concurrency control in the [concurrency control concept
 Hudi Streamer uses checkpoints to keep track of what data has been read already so it can resume without needing to reprocess all data.
 When using a Kafka source, the checkpoint is the [Kafka Offset](https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management) 
 When using a DFS source, the checkpoint is the 'last modified' timestamp of the latest file read.
-Checkpoints are saved in the .hoodie commit file as `streamer.checkpoint.key`.
+Checkpoints are saved in the .hoodie commit file, under `streamer.checkpoint.key.v2` for tables at table
+version 8 or higher and `deltastreamer.checkpoint.key` below that.
 
 If you need to change the checkpoints for reprocessing or replaying data you can use the following options:
 
 - `--checkpoint` will set `streamer.checkpoint.reset_key` in the commit file to overwrite the current checkpoint. Format of checkpoint depends on [KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype). By default (for type `string`), checkpoint should be provided as: `topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint should be provided as long value of desired timestamp. For type `single_offset`, we assume that topic consists of a single partition, so checkpoint should be provided as long value of desired offset.
 - `--source-limit` will set a maximum amount of data to read from the source. For DFS sources, this is max # of bytes read.
 For Kafka, this is the max # of events to read.
 
+#### Resetting the checkpoint for the Hudi incremental source
+
+When Hudi Streamer writes a target table at table version 8 or higher using `HoodieIncrSource`, it tracks progress by
+**completion time** instead of requested instant time, and records it in the commit metadata under
+`streamer.checkpoint.key.v2`. A bare timestamp would be ambiguous between the two, so `--checkpoint` has to state which
+one it is. This form is required from Hudi 1.0.1 onward; on 1.0.0 the prefixes are not recognised and `--checkpoint`
+takes a bare completion time.
+
+```shell

Review Comment:
   Good challenge — I re-tested rather than argue from the earlier run, and the doc statement holds: **1.0.0 reads a bare `--checkpoint` as a completion time.**
   
   The inference in the comment is that because the bare value *was accepted* on 1.0.0, and that value happened to be a requested instant time, 1.0.0 must interpret bare values as requested times. Acceptance does not imply interpretation, though — 1.0.0 does no validation of the value at all, so it accepts any timestamp-shaped string regardless of which clock it came from. What the value *means* only shows up in which records get ingested.
   
   So I ran the discriminating pair on 1.0.0. Source instant #2: requested `20260810143709423`, completed `20260810143710155`.
   
   | bare `--checkpoint` | rows ingested | instant #2 |
   |---|---|---|
   | `20260810143709423` (its **requested** time) | `21,22,31,32,41,42` | **included** |
   | `20260810143710155` (its **completion** time) | `31,32,41,42` | **excluded** |
   
   If a bare value were a requested time, passing instant #2's requested time would have resumed *after* instant #2 and excluded its records. It did not. Passing instant #2's **completion** time is what lands the checkpoint exactly after it. That is completion time semantics.
   
   This also explains the original matrix row that prompted the question. The bare value I fed every version was instant #2's requested time, which is *earlier* than its completion time, so read as a completion time it sits mid-way and instant #2 is still ahead of the checkpoint — hence six records on 1.0.0 where 1.0.1+ ingests four.
   
   The source agrees. In `release-1.0.0`, `StreamerCheckpointUtils` L67-68:
   
   ```java
   checkpoint = Option.of(CheckpointUtils.targetCheckpointV2(writeTableVersion)
       ? new StreamerCheckpointV2(streamerConfig.checkpoint) : new StreamerCheckpointV1(streamerConfig.checkpoint));
   ```
   
   At table version 8 `targetCheckpointV2` is true, so the raw string is wrapped in `StreamerCheckpointV2` — a completion time — with no translation and no parsing. `HoodieIncrSourceCheckpointValUtils`, which is what introduces the prefixes and their validation, does not exist in 1.0.0 (404 at that tag).
   
   On #12718's note that 1.x was "still using request time based handling": that was written in January 2025 and is now stale, which the PR body covers. The 1.0.0 run with no override stores `streamer.checkpoint.key.v2` holding a completion time, so even 1.0.0 was already completion time based in storage — it just had no way to disambiguate an override, which is the gap #12718 closed.
   
   No change made for this one. @yihua a confirmation of the pre-1.0.1 semantics would still be welcome.



##########
website/docs/hoodie_streaming_ingestion.md:
##########
@@ -312,14 +312,54 @@ Read more in depth about concurrency control in the [concurrency control concept
 Hudi Streamer uses checkpoints to keep track of what data has been read already so it can resume without needing to reprocess all data.
 When using a Kafka source, the checkpoint is the [Kafka Offset](https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management) 
 When using a DFS source, the checkpoint is the 'last modified' timestamp of the latest file read.
-Checkpoints are saved in the .hoodie commit file as `streamer.checkpoint.key`.
+Checkpoints are saved in the .hoodie commit file, under `streamer.checkpoint.key.v2` for tables at table
+version 8 or higher and `deltastreamer.checkpoint.key` below that.
 
 If you need to change the checkpoints for reprocessing or replaying data you can use the following options:
 
 - `--checkpoint` will set `streamer.checkpoint.reset_key` in the commit file to overwrite the current checkpoint. Format of checkpoint depends on [KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype). By default (for type `string`), checkpoint should be provided as: `topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint should be provided as long value of desired timestamp. For type `single_offset`, we assume that topic consists of a single partition, so checkpoint should be provided as long value of desired offset.
 - `--source-limit` will set a maximum amount of data to read from the source. For DFS sources, this is max # of bytes read.
 For Kafka, this is the max # of events to read.

Review Comment:
   Right that these do not reconcile — and the reason is that `streamer.checkpoint.reset_key` is not a real key either. Fixed in 05e5b6443aa2.
   
   It has the same missing-prefix defect as the checkpoint key line I corrected in d92a88e38e9c. The actual reset keys, at `release-1.2.0`:
   
   | Checkpoint | Reset key | Defined / written |
   |---|---|---|
   | request time | `deltastreamer.checkpoint.reset_key` | `StreamerCheckpointV1.java:30`, written at `:60` |
   | completion time | `streamer.checkpoint.reset.key.v2` | `StreamerCheckpointV2.java:33`, written at `:66` |
   
   Same `CheckpointUtils.shouldTargetCheckpointV2` test on the write table version that selects between `deltastreamer.checkpoint.key` and `streamer.checkpoint.key.v2`. So the bullet was not describing an older key superseded by the new one — it was naming a key Hudi never writes, which is exactly why a reader inspecting commit metadata would not find it.
   
   The bullet now reads:
   
   > `--checkpoint` will set the checkpoint reset key in the commit file to overwrite the current checkpoint — `streamer.checkpoint.reset.key.v2` at table version 8 or higher, `deltastreamer.checkpoint.reset_key` below that.
   
   which mirrors the phrasing used for the checkpoint key a few lines above, so the two lines now agree with each other and with the new subsection. Empirically confirmed earlier in this PR: the version-9 target commit carries `streamer.checkpoint.reset.key.v2` with the prefixed override verbatim, and a target forced to `hoodie.write.table.version=6` carries the `deltastreamer.*` pair.
   
   Build is clean with the warning set byte-identical to the baseline.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] docs(streamer): document checkpoint v2 reset for the Hudi incremental source [hudi]

Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19573:
URL: https://github.com/apache/hudi/pull/19573#discussion_r3750266182


##########
website/docs/hoodie_streaming_ingestion.md:
##########
@@ -312,14 +312,54 @@ Read more in depth about concurrency control in the [concurrency control concept
 Hudi Streamer uses checkpoints to keep track of what data has been read already so it can resume without needing to reprocess all data.
 When using a Kafka source, the checkpoint is the [Kafka Offset](https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management) 
 When using a DFS source, the checkpoint is the 'last modified' timestamp of the latest file read.
-Checkpoints are saved in the .hoodie commit file as `streamer.checkpoint.key`.
+Checkpoints are saved in the .hoodie commit file, under `streamer.checkpoint.key.v2` for tables at table
+version 8 or higher and `deltastreamer.checkpoint.key` below that.
 
 If you need to change the checkpoints for reprocessing or replaying data you can use the following options:
 
 - `--checkpoint` will set `streamer.checkpoint.reset_key` in the commit file to overwrite the current checkpoint. Format of checkpoint depends on [KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype). By default (for type `string`), checkpoint should be provided as: `topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint should be provided as long value of desired timestamp. For type `single_offset`, we assume that topic consists of a single partition, so checkpoint should be provided as long value of desired offset.
 - `--source-limit` will set a maximum amount of data to read from the source. For DFS sources, this is max # of bytes read.
 For Kafka, this is the max # of events to read.
 
+#### Resetting the checkpoint for the Hudi incremental source
+
+When Hudi Streamer writes a target table at table version 8 or higher using `HoodieIncrSource`, it tracks progress by
+**completion time** instead of requested instant time, and records it in the commit metadata under
+`streamer.checkpoint.key.v2`. A bare timestamp would be ambiguous between the two, so `--checkpoint` has to state which
+one it is. This form is required from Hudi 1.0.1 onward; on 1.0.0 the prefixes are not recognised and `--checkpoint`
+takes a bare completion time.
+
+```shell

Review Comment:
   🤖 This section could clarify the 1.0.0 behavior: it says on 1.0.0 `--checkpoint` "takes a bare completion time," but the PR's own reproduction states the bare value fed to every version was the instant's *requested* time, and that value was accepted on 1.0.0. That suggests 1.0.0 interprets a bare `--checkpoint` as a requested instant time (consistent with #12718's note that 1.x incremental sources were still request-time based), not a completion time. Could you double-check whether 1.0.0 treats the bare value as requested or completion time? @yihua could you confirm the pre-1.0.1 checkpoint semantics here?
   
   <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>



##########
website/docs/hoodie_streaming_ingestion.md:
##########
@@ -312,14 +312,54 @@ Read more in depth about concurrency control in the [concurrency control concept
 Hudi Streamer uses checkpoints to keep track of what data has been read already so it can resume without needing to reprocess all data.
 When using a Kafka source, the checkpoint is the [Kafka Offset](https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management) 
 When using a DFS source, the checkpoint is the 'last modified' timestamp of the latest file read.
-Checkpoints are saved in the .hoodie commit file as `streamer.checkpoint.key`.
+Checkpoints are saved in the .hoodie commit file, under `streamer.checkpoint.key.v2` for tables at table
+version 8 or higher and `deltastreamer.checkpoint.key` below that.
 
 If you need to change the checkpoints for reprocessing or replaying data you can use the following options:
 
 - `--checkpoint` will set `streamer.checkpoint.reset_key` in the commit file to overwrite the current checkpoint. Format of checkpoint depends on [KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype). By default (for type `string`), checkpoint should be provided as: `topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint should be provided as long value of desired timestamp. For type `single_offset`, we assume that topic consists of a single partition, so checkpoint should be provided as long value of desired offset.
 - `--source-limit` will set a maximum amount of data to read from the source. For DFS sources, this is max # of bytes read.
 For Kafka, this is the max # of events to read.

Review Comment:
   🤖 It might help to reconcile the key names here. The existing bullet says `--checkpoint` sets `streamer.checkpoint.reset_key`, while the new subsection says the value is recorded under `streamer.checkpoint.reset.key.v2` for table version 8+. A reader inspecting commit metadata may be unsure which key applies to their table. Consider noting that `reset.key.v2` is used for table version 8+ and the older `reset_key` for versions below that, mirroring the `streamer.checkpoint.key.v2` vs `deltastreamer.checkpoint.key` distinction you already added just above.
   
   <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] docs(streamer): document checkpoint v2 reset for the Hudi incremental source [hudi]

Posted by "deepakpanda93 (via GitHub)" <gi...@apache.org>.
deepakpanda93 commented on code in PR #19573:
URL: https://github.com/apache/hudi/pull/19573#discussion_r3755257653


##########
website/docs/hoodie_streaming_ingestion.md:
##########
@@ -312,14 +312,54 @@ Read more in depth about concurrency control in the [concurrency control concept
 Hudi Streamer uses checkpoints to keep track of what data has been read already so it can resume without needing to reprocess all data.
 When using a Kafka source, the checkpoint is the [Kafka Offset](https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management) 
 When using a DFS source, the checkpoint is the 'last modified' timestamp of the latest file read.
-Checkpoints are saved in the .hoodie commit file as `streamer.checkpoint.key`.
+Checkpoints are saved in the .hoodie commit file, under `streamer.checkpoint.key.v2` for tables at table
+version 8 or higher and `deltastreamer.checkpoint.key` below that.

Review Comment:
   You're right that these two lines were wrong, and thanks for pushing on it — but the measurement lands somewhere neither of us had it. Fixed in cc3483885c7c.
   
   **Where the review's mechanism doesn't match `release-1.2.0`.** There is no `createCheckpoint()` in `StreamSync` at this tag. The write path is `StreamSync.extractCheckpointMetadata` (L911-927):
   
   ```java
   if (inputBatch.getCheckpointForNextBatch() != null) {
     return inputBatch.getCheckpointForNextBatch().getCheckpointCommitMetadata(cfg.checkpoint, cfg.ignoreCheckpoint);
   }
   Checkpoint checkpoint = buildCheckpointFromGeneralSource(cfg.sourceClassName, versionCode, null);
   return checkpoint.getCheckpointCommitMetadata(cfg.checkpoint, cfg.ignoreCheckpoint);
   ```
   
   The `buildCheckpointFromGeneralSource` branch — the one carrying the source-class exclusion — is only the *fallback*, used when the source supplied no checkpoint. In the normal case the key comes from whatever `Checkpoint` object the source returned.
   
   **What actually decides it.** `InputBatch(Option<T> batch, String checkpointForNextBatch)` delegates to `this(batch, new StreamerCheckpointV2(checkpointForNextBatch), ...)` (`InputBatch.java:43-44`) — wrapping in **V2 unconditionally**. Sources that hand back a plain string, Kafka and DFS among them, therefore always produce a V2 checkpoint.
   
   So Kafka/DFS do **not** stay on `deltastreamer.checkpoint.key`. I checked rather than reason about it, running `ParquetDFSSource` twice over the same input:
   
   | Target table version | Checkpoint key written |
   |---|---|
   | 9 (default) | `streamer.checkpoint.key.v2` |
   | 6 (`hoodie.write.table.version=6`) | `streamer.checkpoint.key.v2` |
   
   It writes the v2 key at **both**, which also makes my "version 8 or higher" rule wrong — in the opposite direction from the one suggested.
   
   `HoodieIncrSource` behaves differently because it returns a real `Checkpoint` object whose type *is* version-gated, which is why my earlier version-6 run of that source produced `deltastreamer.checkpoint.key`.
   
   **The fix.** The split is by source, and only *within* the Hudi incremental source by table version. Both lines now name the two keys and say what the choice depends on, without asserting a version rule that does not hold for Kafka or DFS:
   
   > Checkpoints are saved in the .hoodie commit file. Completion time checkpoints are stored under `streamer.checkpoint.key.v2` and request time checkpoints under `deltastreamer.checkpoint.key`. Which one applies depends on the source, and for the Hudi incremental source also on the table version — see below.
   
   > `--checkpoint` will set the matching reset key in the commit file to overwrite the current checkpoint, either `streamer.checkpoint.reset.key.v2` or `deltastreamer.checkpoint.reset_key`.
   
   The version-specific detail stays in the incremental source subsection, which is the part backed by the six-release matrix. Note the original text named `streamer.checkpoint.key` and `streamer.checkpoint.reset_key`, neither of which Hudi writes, so reverting was not an option.
   
   Build clean, warnings byte-identical to baseline. @yihua a committer's eye on the Kafka/DFS key behaviour would still be welcome — the `InputBatch` string constructor forcing V2 regardless of table version looks deliberate but is worth confirming.



##########
website/docs/hoodie_streaming_ingestion.md:
##########
@@ -312,14 +312,54 @@ Read more in depth about concurrency control in the [concurrency control concept
 Hudi Streamer uses checkpoints to keep track of what data has been read already so it can resume without needing to reprocess all data.
 When using a Kafka source, the checkpoint is the [Kafka Offset](https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management) 
 When using a DFS source, the checkpoint is the 'last modified' timestamp of the latest file read.
-Checkpoints are saved in the .hoodie commit file as `streamer.checkpoint.key`.
+Checkpoints are saved in the .hoodie commit file, under `streamer.checkpoint.key.v2` for tables at table
+version 8 or higher and `deltastreamer.checkpoint.key` below that.
 
 If you need to change the checkpoints for reprocessing or replaying data you can use the following options:
 
-- `--checkpoint` will set `streamer.checkpoint.reset_key` in the commit file to overwrite the current checkpoint. Format of checkpoint depends on [KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype). By default (for type `string`), checkpoint should be provided as: `topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint should be provided as long value of desired timestamp. For type `single_offset`, we assume that topic consists of a single partition, so checkpoint should be provided as long value of desired offset.
+- `--checkpoint` will set the checkpoint reset key in the commit file to overwrite the current checkpoint — `streamer.checkpoint.reset.key.v2` at table version 8 or higher, `deltastreamer.checkpoint.reset_key` below that. Format of checkpoint depends on [KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype). By default (for type `string`), checkpoint should be provided as: `topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint should be provided as long value of desired timestamp. For type `single_offset`, we assume that topic consists of a single partition, so checkpoint should be provided as long value of desired offset.
 - `--source-limit` will set a maximum amount of data to read from the source. For DFS sources, this is max # of bytes read.
 For Kafka, this is the max # of events to read.

Review Comment:
   Same root cause, same fix — see the reply on the intro line for the detail and the measurements. Corrected in cc3483885c7c.
   
   Short version: `buildCheckpointFromConfigOverride` is not what decides this in the common path. `StreamSync.extractCheckpointMetadata` uses the `Checkpoint` object the source returned, and `InputBatch(batch, String)` wraps a plain string in `StreamerCheckpointV2` unconditionally (`InputBatch.java:43-44`). Running `ParquetDFSSource` at table version 9 and at table version 6 produced `streamer.checkpoint.key.v2` in **both** cases, so a DFS checkpoint does not stay on the `deltastreamer.*` key.
   
   The bullet no longer claims a table-version rule:
   
   > `--checkpoint` will set the matching reset key in the commit file to overwrite the current checkpoint, either `streamer.checkpoint.reset.key.v2` or `deltastreamer.checkpoint.reset_key`.
   
   Worth keeping in mind that the pre-existing text said `streamer.checkpoint.reset_key`, which is not a key Hudi writes at all — so this bullet needed correcting regardless of which rule governs the choice.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] docs(streamer): document checkpoint v2 reset for the Hudi incremental source [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous merged PR #19573:
URL: https://github.com/apache/hudi/pull/19573


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org