You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "shangxinli (via GitHub)" <gi...@apache.org> on 2026/02/07 23:00:09 UTC
[PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
shangxinli opened a new pull request, #18125:
URL: https://github.com/apache/hudi/pull/18125
This is to port internal Hudi changes to OSS to prepare for Hudi version upgrade.
This commit adds utility functions to StreamerUtil for extracting and parsing Kafka offset metadata from Hudi commits, enabling analysis of message processing progress between commits.
Changes:
- Add extractKafkaOffsetMetadata() to extract Kafka offset metadata from commit
- Add parseKafkaOffsets() to parse URL-encoded Kafka offset strings into partition->offset map
- Add calculateKafkaOffsetDifference() to compute total offset delta between two commits
- Add comprehensive unit tests in TestStreamerUtil covering normal cases and edge cases
- Add constants: HOODIE_METADATA_KEY, URL_ENCODED_COLON, PARTITION_SEPARATOR
Use Cases:
- Data loss detection by comparing expected vs actual record counts
- Monitoring Kafka consumption progress across Hudi commits
- Debugging offset tracking issues in Flink-Kafka-Hudi pipelines
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18125:
URL: https://github.com/apache/hudi/pull/18125#issuecomment-3876394977
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0296f2f24aca47aa8d38c23e1620702b45d51163",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11865",
"triggerID" : "0296f2f24aca47aa8d38c23e1620702b45d51163",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0296f2f24aca47aa8d38c23e1620702b45d51163 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11865)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #18125:
URL: https://github.com/apache/hudi/pull/18125#discussion_r2893738277
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java:
##########
@@ -111,6 +111,7 @@ public class StreamerUtil {
public static final String FLINK_CHECKPOINT_ID = "flink_checkpoint_id";
public static final String EMPTY_PARTITION_PATH = "";
+ public static final String HOODIE_METADATA_KEY = "HoodieMetadataKey";
Review Comment:
already got one in `KafkaOffsetParseUtils` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18125:
URL: https://github.com/apache/hudi/pull/18125#issuecomment-3994129520
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0296f2f24aca47aa8d38c23e1620702b45d51163",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11865",
"triggerID" : "0296f2f24aca47aa8d38c23e1620702b45d51163",
"triggerType" : "PUSH"
}, {
"hash" : "86bf764280f3963bacf5a91c9f18827fd361b7ef",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12400",
"triggerID" : "86bf764280f3963bacf5a91c9f18827fd361b7ef",
"triggerType" : "PUSH"
}, {
"hash" : "bbcf710de4aa74feff668c4fb4190dc5780fd6dc",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12402",
"triggerID" : "bbcf710de4aa74feff668c4fb4190dc5780fd6dc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0296f2f24aca47aa8d38c23e1620702b45d51163 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11865)
* 86bf764280f3963bacf5a91c9f18827fd361b7ef Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12400)
* bbcf710de4aa74feff668c4fb4190dc5780fd6dc Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12402)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18125:
URL: https://github.com/apache/hudi/pull/18125#issuecomment-3993898294
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0296f2f24aca47aa8d38c23e1620702b45d51163",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11865",
"triggerID" : "0296f2f24aca47aa8d38c23e1620702b45d51163",
"triggerType" : "PUSH"
}, {
"hash" : "86bf764280f3963bacf5a91c9f18827fd361b7ef",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12400",
"triggerID" : "86bf764280f3963bacf5a91c9f18827fd361b7ef",
"triggerType" : "PUSH"
}, {
"hash" : "bbcf710de4aa74feff668c4fb4190dc5780fd6dc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bbcf710de4aa74feff668c4fb4190dc5780fd6dc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0296f2f24aca47aa8d38c23e1620702b45d51163 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11865)
* 86bf764280f3963bacf5a91c9f18827fd361b7ef Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12400)
* bbcf710de4aa74feff668c4fb4190dc5780fd6dc UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18125:
URL: https://github.com/apache/hudi/pull/18125#issuecomment-3994199264
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0296f2f24aca47aa8d38c23e1620702b45d51163",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11865",
"triggerID" : "0296f2f24aca47aa8d38c23e1620702b45d51163",
"triggerType" : "PUSH"
}, {
"hash" : "86bf764280f3963bacf5a91c9f18827fd361b7ef",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12400",
"triggerID" : "86bf764280f3963bacf5a91c9f18827fd361b7ef",
"triggerType" : "PUSH"
}, {
"hash" : "bbcf710de4aa74feff668c4fb4190dc5780fd6dc",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12402",
"triggerID" : "bbcf710de4aa74feff668c4fb4190dc5780fd6dc",
"triggerType" : "PUSH"
}, {
"hash" : "8fddf1084fe0125a5e383f7cf2ac237da4b0f657",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8fddf1084fe0125a5e383f7cf2ac237da4b0f657",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 86bf764280f3963bacf5a91c9f18827fd361b7ef Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12400)
* bbcf710de4aa74feff668c4fb4190dc5780fd6dc Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12402)
* 8fddf1084fe0125a5e383f7cf2ac237da4b0f657 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18125:
URL: https://github.com/apache/hudi/pull/18125#issuecomment-3993869180
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0296f2f24aca47aa8d38c23e1620702b45d51163",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11865",
"triggerID" : "0296f2f24aca47aa8d38c23e1620702b45d51163",
"triggerType" : "PUSH"
}, {
"hash" : "86bf764280f3963bacf5a91c9f18827fd361b7ef",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12400",
"triggerID" : "86bf764280f3963bacf5a91c9f18827fd361b7ef",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0296f2f24aca47aa8d38c23e1620702b45d51163 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11865)
* 86bf764280f3963bacf5a91c9f18827fd361b7ef Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12400)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18125:
URL: https://github.com/apache/hudi/pull/18125#issuecomment-4016705255
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0296f2f24aca47aa8d38c23e1620702b45d51163",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11865",
"triggerID" : "0296f2f24aca47aa8d38c23e1620702b45d51163",
"triggerType" : "PUSH"
}, {
"hash" : "86bf764280f3963bacf5a91c9f18827fd361b7ef",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12400",
"triggerID" : "86bf764280f3963bacf5a91c9f18827fd361b7ef",
"triggerType" : "PUSH"
}, {
"hash" : "bbcf710de4aa74feff668c4fb4190dc5780fd6dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12402",
"triggerID" : "bbcf710de4aa74feff668c4fb4190dc5780fd6dc",
"triggerType" : "PUSH"
}, {
"hash" : "8fddf1084fe0125a5e383f7cf2ac237da4b0f657",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12404",
"triggerID" : "8fddf1084fe0125a5e383f7cf2ac237da4b0f657",
"triggerType" : "PUSH"
}, {
"hash" : "d5adeaa1818c6c315437ffe2589c2ea661f913e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12419",
"triggerID" : "d5adeaa1818c6c315437ffe2589c2ea661f913e4",
"triggerType" : "PUSH"
}, {
"hash" : "b931ee659898114083659324895f5aac346a4e47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12453",
"triggerID" : "b931ee659898114083659324895f5aac346a4e47",
"triggerType" : "PUSH"
}, {
"hash" : "71655a3f73a1472292fd7011b6e6e164056295cf",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12491",
"triggerID" : "71655a3f73a1472292fd7011b6e6e164056295cf",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 71655a3f73a1472292fd7011b6e6e164056295cf Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12491)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18125:
URL: https://github.com/apache/hudi/pull/18125#issuecomment-3865712430
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b21e77f61e70826afe74e66188afb16a1b1bd74",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4b21e77f61e70826afe74e66188afb16a1b1bd74",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4b21e77f61e70826afe74e66188afb16a1b1bd74 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18125:
URL: https://github.com/apache/hudi/pull/18125#issuecomment-3874776705
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0296f2f24aca47aa8d38c23e1620702b45d51163",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11865",
"triggerID" : "0296f2f24aca47aa8d38c23e1620702b45d51163",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0296f2f24aca47aa8d38c23e1620702b45d51163 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11865)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18125:
URL: https://github.com/apache/hudi/pull/18125#issuecomment-3994349257
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0296f2f24aca47aa8d38c23e1620702b45d51163",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11865",
"triggerID" : "0296f2f24aca47aa8d38c23e1620702b45d51163",
"triggerType" : "PUSH"
}, {
"hash" : "86bf764280f3963bacf5a91c9f18827fd361b7ef",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12400",
"triggerID" : "86bf764280f3963bacf5a91c9f18827fd361b7ef",
"triggerType" : "PUSH"
}, {
"hash" : "bbcf710de4aa74feff668c4fb4190dc5780fd6dc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12402",
"triggerID" : "bbcf710de4aa74feff668c4fb4190dc5780fd6dc",
"triggerType" : "PUSH"
}, {
"hash" : "8fddf1084fe0125a5e383f7cf2ac237da4b0f657",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12404",
"triggerID" : "8fddf1084fe0125a5e383f7cf2ac237da4b0f657",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bbcf710de4aa74feff668c4fb4190dc5780fd6dc Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12402)
* 8fddf1084fe0125a5e383f7cf2ac237da4b0f657 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12404)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18125:
URL: https://github.com/apache/hudi/pull/18125#issuecomment-4024372151
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18125?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 43.38%. Comparing base ([`93b8e9f`](https://app.codecov.io/gh/apache/hudi/commit/93b8e9fc9804f75c1c9e21e5ca421df758743a8c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`fab648e`](https://app.codecov.io/gh/apache/hudi/commit/fab648eaa67a2196bacaa984db93e075c649eea7?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
> :exclamation: There is a different number of reports uploaded between BASE (93b8e9f) and HEAD (fab648e). Click for more details.
>
> <details><summary>HEAD has 10 uploads less than BASE</summary>
>
>| Flag | BASE (93b8e9f) | HEAD (fab648e) |
>|------|------|------|
>|spark-scala-tests|6|3|
>|spark-java-tests|9|2|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18125 +/- ##
=============================================
- Coverage 57.30% 43.38% -13.92%
+ Complexity 18626 13871 -4755
=============================================
Files 1955 1955
Lines 106916 106916
Branches 13225 13225
=============================================
- Hits 61267 46387 -14880
- Misses 39858 55190 +15332
+ Partials 5791 5339 -452
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `45.23% <ø> (+0.01%)` | :arrow_up: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `33.55% <ø> (-13.93%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `33.11% <ø> (-12.46%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
[see 787 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18125/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18125:
URL: https://github.com/apache/hudi/pull/18125#issuecomment-4024596054
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18125?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 57.30%. Comparing base ([`93b8e9f`](https://app.codecov.io/gh/apache/hudi/commit/93b8e9fc9804f75c1c9e21e5ca421df758743a8c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`fab648e`](https://app.codecov.io/gh/apache/hudi/commit/fab648eaa67a2196bacaa984db93e075c649eea7?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18125 +/- ##
=========================================
Coverage 57.30% 57.30%
Complexity 18626 18626
=========================================
Files 1955 1955
Lines 106916 106916
Branches 13225 13225
=========================================
+ Hits 61267 61271 +4
+ Misses 39858 39856 -2
+ Partials 5791 5789 -2
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `45.23% <ø> (+0.01%)` | :arrow_up: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `47.48% <ø> (+0.01%)` | :arrow_up: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `45.57% <ø> (+<0.01%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
[see 3 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18125/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18125:
URL: https://github.com/apache/hudi/pull/18125#issuecomment-4024463166
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18125?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 55.54%. Comparing base ([`93b8e9f`](https://app.codecov.io/gh/apache/hudi/commit/93b8e9fc9804f75c1c9e21e5ca421df758743a8c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`fab648e`](https://app.codecov.io/gh/apache/hudi/commit/fab648eaa67a2196bacaa984db93e075c649eea7?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18125 +/- ##
============================================
- Coverage 57.30% 55.54% -1.76%
+ Complexity 18626 18000 -626
============================================
Files 1955 1955
Lines 106916 106916
Branches 13225 13225
============================================
- Hits 61267 59391 -1876
- Misses 39858 41666 +1808
- Partials 5791 5859 +68
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `45.23% <ø> (+0.01%)` | :arrow_up: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `43.51% <ø> (-3.97%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `45.57% <ø> (+<0.01%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
[see 231 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18125/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18125:
URL: https://github.com/apache/hudi/pull/18125#issuecomment-4024370084
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18125?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 43.31%. Comparing base ([`93b8e9f`](https://app.codecov.io/gh/apache/hudi/commit/93b8e9fc9804f75c1c9e21e5ca421df758743a8c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`fab648e`](https://app.codecov.io/gh/apache/hudi/commit/fab648eaa67a2196bacaa984db93e075c649eea7?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
> :exclamation: There is a different number of reports uploaded between BASE (93b8e9f) and HEAD (fab648e). Click for more details.
>
> <details><summary>HEAD has 11 uploads less than BASE</summary>
>
>| Flag | BASE (93b8e9f) | HEAD (fab648e) |
>|------|------|------|
>|spark-scala-tests|6|3|
>|spark-java-tests|9|1|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18125 +/- ##
=============================================
- Coverage 57.30% 43.31% -13.99%
+ Complexity 18626 13848 -4778
=============================================
Files 1955 1955
Lines 106916 106916
Branches 13225 13225
=============================================
- Hits 61267 46311 -14956
- Misses 39858 55256 +15398
+ Partials 5791 5349 -442
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `45.23% <ø> (+0.01%)` | :arrow_up: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `33.94% <ø> (-13.54%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `33.11% <ø> (-12.46%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
[see 788 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18125/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18125:
URL: https://github.com/apache/hudi/pull/18125#issuecomment-4024453869
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18125?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 55.47%. Comparing base ([`93b8e9f`](https://app.codecov.io/gh/apache/hudi/commit/93b8e9fc9804f75c1c9e21e5ca421df758743a8c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`fab648e`](https://app.codecov.io/gh/apache/hudi/commit/fab648eaa67a2196bacaa984db93e075c649eea7?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18125 +/- ##
============================================
- Coverage 57.30% 55.47% -1.83%
+ Complexity 18626 17970 -656
============================================
Files 1955 1955
Lines 106916 106916
Branches 13225 13225
============================================
- Hits 61267 59315 -1952
- Misses 39858 41730 +1872
- Partials 5791 5871 +80
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `45.23% <ø> (+0.01%)` | :arrow_up: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `43.39% <ø> (-4.08%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `45.57% <ø> (+<0.01%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
[see 250 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18125/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18125:
URL: https://github.com/apache/hudi/pull/18125#issuecomment-4007869210
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0296f2f24aca47aa8d38c23e1620702b45d51163",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11865",
"triggerID" : "0296f2f24aca47aa8d38c23e1620702b45d51163",
"triggerType" : "PUSH"
}, {
"hash" : "86bf764280f3963bacf5a91c9f18827fd361b7ef",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12400",
"triggerID" : "86bf764280f3963bacf5a91c9f18827fd361b7ef",
"triggerType" : "PUSH"
}, {
"hash" : "bbcf710de4aa74feff668c4fb4190dc5780fd6dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12402",
"triggerID" : "bbcf710de4aa74feff668c4fb4190dc5780fd6dc",
"triggerType" : "PUSH"
}, {
"hash" : "8fddf1084fe0125a5e383f7cf2ac237da4b0f657",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12404",
"triggerID" : "8fddf1084fe0125a5e383f7cf2ac237da4b0f657",
"triggerType" : "PUSH"
}, {
"hash" : "d5adeaa1818c6c315437ffe2589c2ea661f913e4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12419",
"triggerID" : "d5adeaa1818c6c315437ffe2589c2ea661f913e4",
"triggerType" : "PUSH"
}, {
"hash" : "b931ee659898114083659324895f5aac346a4e47",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12453",
"triggerID" : "b931ee659898114083659324895f5aac346a4e47",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d5adeaa1818c6c315437ffe2589c2ea661f913e4 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12419)
* b931ee659898114083659324895f5aac346a4e47 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12453)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "shangxinli (via GitHub)" <gi...@apache.org>.
shangxinli commented on code in PR #18125:
URL: https://github.com/apache/hudi/pull/18125#discussion_r2880979404
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java:
##########
@@ -729,4 +733,153 @@ public static void validateWriteStatus(
});
}
}
+
+ /**
+ * Extracts Kafka offset metadata from a Hudi commit.
+ *
+ * @param commitInstant The commit instant to extract metadata from
+ * @param timeline The timeline to read commit details from
+ * @return The Kafka offset metadata string, or null if not found
+ * @throws IOException if error reading commit metadata
+ */
+ public static String extractKafkaOffsetMetadata(final HoodieInstant commitInstant,
+ final HoodieTimeline timeline) throws IOException {
+ HoodieCommitMetadata commitMetadata = TimelineUtils.getCommitMetadata(commitInstant, timeline);
+ return commitMetadata.getExtraMetadata().get(HOODIE_METADATA_KEY);
+ }
+
+ /**
+ * Parses Kafka offset string and restores partition offsets as Map.
+ *
+ * @param kafkaOffsetString URL-encoded Kafka offset string in format:
+ * "kafka_metadata%3Atopic%3Apartition:offset;kafka_metadata%3Atopic%3Apartition:offset"
+ * @return Map of partition ID to offset, or empty map if parsing fails
+ */
+ public static Map<Integer, Long> parseKafkaOffsets(final String kafkaOffsetString) {
+ Map<Integer, Long> partitionOffsets = new HashMap<>();
+
+ if (kafkaOffsetString == null || kafkaOffsetString.isEmpty()) {
+ return partitionOffsets;
+ }
+
+ try {
+ // Split by semicolon to get individual partition entries
+ String[] partitionEntries = kafkaOffsetString.split(PARTITION_SEPARATOR);
+
+ for (String entry : partitionEntries) {
+ entry = entry.trim();
+
+ // Skip cluster metadata entries (they don't contain partition:offset format)
+ if (!entry.contains(":") || entry.contains("kafka_cluster")) {
+ continue;
+ }
+
+ // Entry format: kafka_metadata%3Atopic%3Apartition:offset
+ // Find the last colon which separates partition and offset
+ int lastColonIndex = entry.lastIndexOf(':');
+ if (lastColonIndex == -1) {
+ continue;
+ }
+
+ String offsetStr = entry.substring(lastColonIndex + 1);
+ String beforeOffset = entry.substring(0, lastColonIndex);
+
+ // Find the partition number (everything after the last %3A before the colon)
+ int lastEncodedColonIndex = beforeOffset.lastIndexOf(URL_ENCODED_COLON);
+ if (lastEncodedColonIndex == -1) {
+ continue;
+ }
+
+ String partitionStr = beforeOffset.substring(lastEncodedColonIndex + URL_ENCODED_COLON.length());
+
+ try {
+ int partitionId = Integer.parseInt(partitionStr);
+ long offset = Long.parseLong(offsetStr);
+ partitionOffsets.put(partitionId, offset);
+ } catch (NumberFormatException e) {
+ log.warn("Failed to parse partition ID or offset from entry: {}", entry, e);
+ }
+ }
+ } catch (Exception e) {
+ log.error("Failed to parse Kafka offset string: {}", kafkaOffsetString, e);
+ }
+
+ return partitionOffsets;
+ }
+
+ /**
+ * Calculates the total difference count between Kafka offsets of two Hudi commits.
+ * The difference is: current commit's kafka offset minus previous commit's kafka offset
+ * for each partition. All partition differences are summed together.
+ *
+ * @param currentCommitInstant The current (newer) commit instant
+ * @param previousCommitInstant The previous (older) commit instant
+ * @param timeline The timeline to read commit details from
+ * @return The total count of offset differences across all partitions, or 0 if calculation fails
+ */
+ public static long calculateKafkaOffsetDifference(
+ final HoodieInstant currentCommitInstant,
+ final HoodieInstant previousCommitInstant,
+ final HoodieTimeline timeline) {
+
+ if (currentCommitInstant == null || previousCommitInstant == null || timeline == null) {
+ log.warn("Invalid input parameters for calculating Kafka offset difference");
+ return 0L;
+ }
+
+ try {
+ // Extract Kafka offset metadata from both commits
+ String currentOffsetString = extractKafkaOffsetMetadata(currentCommitInstant, timeline);
+ String previousOffsetString = extractKafkaOffsetMetadata(previousCommitInstant, timeline);
+
+ if (currentOffsetString == null || previousOffsetString == null) {
+ log.warn("Kafka offset metadata not found in one or both commits: current={}, previous={}",
+ currentCommitInstant.requestedTime(), previousCommitInstant.requestedTime());
+ return 0L;
+ }
+
+ // Parse offset strings into partition -> offset maps
+ Map<Integer, Long> currentOffsets = parseKafkaOffsets(currentOffsetString);
+ Map<Integer, Long> previousOffsets = parseKafkaOffsets(previousOffsetString);
+
+ if (currentOffsets.isEmpty() || previousOffsets.isEmpty()) {
+ log.warn("Failed to parse Kafka offsets from commits: current={}, previous={}",
+ currentCommitInstant.requestedTime(), previousCommitInstant.requestedTime());
+ return 0L;
+ }
+
+ // Calculate total difference across all partitions
+ long totalDifference = 0L;
+
+ for (Map.Entry<Integer, Long> currentEntry : currentOffsets.entrySet()) {
Review Comment:
A partition in the previous commit but missing from the current one means no records were consumed from it in this batch — so its offset diff contribution is correctly 0, and we don't need to include it in the sum.
However, a disappearing partition could signal topic reconfiguration or a consumer rebalance issue, so I added a LOG.warn to flag it. No change to the diff calculation logic itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18125:
URL: https://github.com/apache/hudi/pull/18125#issuecomment-4024437918
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18125?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 51.86%. Comparing base ([`93b8e9f`](https://app.codecov.io/gh/apache/hudi/commit/93b8e9fc9804f75c1c9e21e5ca421df758743a8c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`fab648e`](https://app.codecov.io/gh/apache/hudi/commit/fab648eaa67a2196bacaa984db93e075c649eea7?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
> :exclamation: There is a different number of reports uploaded between BASE (93b8e9f) and HEAD (fab648e). Click for more details.
>
> <details><summary>HEAD has 7 uploads less than BASE</summary>
>
>| Flag | BASE (93b8e9f) | HEAD (fab648e) |
>|------|------|------|
>|spark-scala-tests|6|5|
>|spark-java-tests|9|3|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18125 +/- ##
============================================
- Coverage 57.30% 51.86% -5.45%
+ Complexity 18626 16650 -1976
============================================
Files 1955 1955
Lines 106916 106916
Branches 13225 13225
============================================
- Hits 61267 55447 -5820
- Misses 39858 45502 +5644
- Partials 5791 5967 +176
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `45.23% <ø> (+0.01%)` | :arrow_up: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `33.18% <ø> (-14.29%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `45.50% <ø> (-0.07%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
[see 508 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18125/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18125:
URL: https://github.com/apache/hudi/pull/18125#issuecomment-4024156646
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18125?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 45.23%. Comparing base ([`93b8e9f`](https://app.codecov.io/gh/apache/hudi/commit/93b8e9fc9804f75c1c9e21e5ca421df758743a8c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`fab648e`](https://app.codecov.io/gh/apache/hudi/commit/fab648eaa67a2196bacaa984db93e075c649eea7?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
> :exclamation: There is a different number of reports uploaded between BASE (93b8e9f) and HEAD (fab648e). Click for more details.
>
> <details><summary>HEAD has 15 uploads less than BASE</summary>
>
>| Flag | BASE (93b8e9f) | HEAD (fab648e) |
>|------|------|------|
>|spark-scala-tests|6|0|
>|spark-java-tests|9|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18125 +/- ##
=============================================
- Coverage 57.30% 45.23% -12.08%
+ Complexity 18626 8414 -10212
=============================================
Files 1955 1186 -769
Lines 106916 60840 -46076
Branches 13225 6488 -6737
=============================================
- Hits 61267 27518 -33749
+ Misses 39858 30340 -9518
+ Partials 5791 2982 -2809
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `45.23% <ø> (+0.01%)` | :arrow_up: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
[see 1385 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18125/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18125:
URL: https://github.com/apache/hudi/pull/18125#issuecomment-4016582470
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0296f2f24aca47aa8d38c23e1620702b45d51163",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11865",
"triggerID" : "0296f2f24aca47aa8d38c23e1620702b45d51163",
"triggerType" : "PUSH"
}, {
"hash" : "86bf764280f3963bacf5a91c9f18827fd361b7ef",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12400",
"triggerID" : "86bf764280f3963bacf5a91c9f18827fd361b7ef",
"triggerType" : "PUSH"
}, {
"hash" : "bbcf710de4aa74feff668c4fb4190dc5780fd6dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12402",
"triggerID" : "bbcf710de4aa74feff668c4fb4190dc5780fd6dc",
"triggerType" : "PUSH"
}, {
"hash" : "8fddf1084fe0125a5e383f7cf2ac237da4b0f657",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12404",
"triggerID" : "8fddf1084fe0125a5e383f7cf2ac237da4b0f657",
"triggerType" : "PUSH"
}, {
"hash" : "d5adeaa1818c6c315437ffe2589c2ea661f913e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12419",
"triggerID" : "d5adeaa1818c6c315437ffe2589c2ea661f913e4",
"triggerType" : "PUSH"
}, {
"hash" : "b931ee659898114083659324895f5aac346a4e47",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12453",
"triggerID" : "b931ee659898114083659324895f5aac346a4e47",
"triggerType" : "PUSH"
}, {
"hash" : "71655a3f73a1472292fd7011b6e6e164056295cf",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12491",
"triggerID" : "71655a3f73a1472292fd7011b6e6e164056295cf",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b931ee659898114083659324895f5aac346a4e47 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12453)
* 71655a3f73a1472292fd7011b6e6e164056295cf Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12491)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "shangxinli (via GitHub)" <gi...@apache.org>.
shangxinli commented on code in PR #18125:
URL: https://github.com/apache/hudi/pull/18125#discussion_r2880716221
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java:
##########
@@ -729,4 +733,153 @@ public static void validateWriteStatus(
});
}
}
+
+ /**
+ * Extracts Kafka offset metadata from a Hudi commit.
+ *
+ * @param commitInstant The commit instant to extract metadata from
+ * @param timeline The timeline to read commit details from
+ * @return The Kafka offset metadata string, or null if not found
+ * @throws IOException if error reading commit metadata
+ */
+ public static String extractKafkaOffsetMetadata(final HoodieInstant commitInstant,
+ final HoodieTimeline timeline) throws IOException {
+ HoodieCommitMetadata commitMetadata = TimelineUtils.getCommitMetadata(commitInstant, timeline);
+ return commitMetadata.getExtraMetadata().get(HOODIE_METADATA_KEY);
+ }
+
+ /**
+ * Parses Kafka offset string and restores partition offsets as Map.
+ *
+ * @param kafkaOffsetString URL-encoded Kafka offset string in format:
+ * "kafka_metadata%3Atopic%3Apartition:offset;kafka_metadata%3Atopic%3Apartition:offset"
+ * @return Map of partition ID to offset, or empty map if parsing fails
+ */
+ public static Map<Integer, Long> parseKafkaOffsets(final String kafkaOffsetString) {
Review Comment:
Good catch — made it package-private. It is only used internally by `calculateKafkaOffsetDifference` within the same class, and the tests are in the same package so they still have access.
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java:
##########
@@ -729,4 +733,153 @@ public static void validateWriteStatus(
});
}
}
+
+ /**
+ * Extracts Kafka offset metadata from a Hudi commit.
+ *
+ * @param commitInstant The commit instant to extract metadata from
+ * @param timeline The timeline to read commit details from
+ * @return The Kafka offset metadata string, or null if not found
+ * @throws IOException if error reading commit metadata
+ */
+ public static String extractKafkaOffsetMetadata(final HoodieInstant commitInstant,
+ final HoodieTimeline timeline) throws IOException {
+ HoodieCommitMetadata commitMetadata = TimelineUtils.getCommitMetadata(commitInstant, timeline);
+ return commitMetadata.getExtraMetadata().get(HOODIE_METADATA_KEY);
+ }
+
+ /**
+ * Parses Kafka offset string and restores partition offsets as Map.
+ *
+ * @param kafkaOffsetString URL-encoded Kafka offset string in format:
+ * "kafka_metadata%3Atopic%3Apartition:offset;kafka_metadata%3Atopic%3Apartition:offset"
+ * @return Map of partition ID to offset, or empty map if parsing fails
+ */
+ public static Map<Integer, Long> parseKafkaOffsets(final String kafkaOffsetString) {
+ Map<Integer, Long> partitionOffsets = new HashMap<>();
+
+ if (kafkaOffsetString == null || kafkaOffsetString.isEmpty()) {
+ return partitionOffsets;
+ }
+
+ try {
+ // Split by semicolon to get individual partition entries
+ String[] partitionEntries = kafkaOffsetString.split(PARTITION_SEPARATOR);
+
+ for (String entry : partitionEntries) {
+ entry = entry.trim();
+
+ // Skip cluster metadata entries (they don't contain partition:offset format)
+ if (!entry.contains(":") || entry.contains("kafka_cluster")) {
+ continue;
+ }
+
+ // Entry format: kafka_metadata%3Atopic%3Apartition:offset
+ // Find the last colon which separates partition and offset
+ int lastColonIndex = entry.lastIndexOf(':');
+ if (lastColonIndex == -1) {
+ continue;
+ }
+
+ String offsetStr = entry.substring(lastColonIndex + 1);
+ String beforeOffset = entry.substring(0, lastColonIndex);
+
+ // Find the partition number (everything after the last %3A before the colon)
+ int lastEncodedColonIndex = beforeOffset.lastIndexOf(URL_ENCODED_COLON);
+ if (lastEncodedColonIndex == -1) {
+ continue;
+ }
+
+ String partitionStr = beforeOffset.substring(lastEncodedColonIndex + URL_ENCODED_COLON.length());
+
+ try {
+ int partitionId = Integer.parseInt(partitionStr);
+ long offset = Long.parseLong(offsetStr);
+ partitionOffsets.put(partitionId, offset);
+ } catch (NumberFormatException e) {
+ log.warn("Failed to parse partition ID or offset from entry: {}", entry, e);
+ }
+ }
+ } catch (Exception e) {
+ log.error("Failed to parse Kafka offset string: {}", kafkaOffsetString, e);
+ }
+
+ return partitionOffsets;
+ }
+
+ /**
+ * Calculates the total difference count between Kafka offsets of two Hudi commits.
+ * The difference is: current commit's kafka offset minus previous commit's kafka offset
+ * for each partition. All partition differences are summed together.
+ *
+ * @param currentCommitInstant The current (newer) commit instant
+ * @param previousCommitInstant The previous (older) commit instant
+ * @param timeline The timeline to read commit details from
+ * @return The total count of offset differences across all partitions, or 0 if calculation fails
+ */
+ public static long calculateKafkaOffsetDifference(
Review Comment:
Yes, that is possible if the commit has been archived. Added a Javadoc note that both commit instants must be present in the provided timeline — if a commit has been archived and is no longer in the active timeline, this method will fail. The caller is responsible for ensuring the instants are accessible.
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java:
##########
@@ -729,4 +733,153 @@ public static void validateWriteStatus(
});
}
}
+
+ /**
+ * Extracts Kafka offset metadata from a Hudi commit.
+ *
+ * @param commitInstant The commit instant to extract metadata from
+ * @param timeline The timeline to read commit details from
+ * @return The Kafka offset metadata string, or null if not found
+ * @throws IOException if error reading commit metadata
+ */
+ public static String extractKafkaOffsetMetadata(final HoodieInstant commitInstant,
+ final HoodieTimeline timeline) throws IOException {
+ HoodieCommitMetadata commitMetadata = TimelineUtils.getCommitMetadata(commitInstant, timeline);
+ return commitMetadata.getExtraMetadata().get(HOODIE_METADATA_KEY);
+ }
+
+ /**
+ * Parses Kafka offset string and restores partition offsets as Map.
+ *
+ * @param kafkaOffsetString URL-encoded Kafka offset string in format:
+ * "kafka_metadata%3Atopic%3Apartition:offset;kafka_metadata%3Atopic%3Apartition:offset"
+ * @return Map of partition ID to offset, or empty map if parsing fails
+ */
+ public static Map<Integer, Long> parseKafkaOffsets(final String kafkaOffsetString) {
+ Map<Integer, Long> partitionOffsets = new HashMap<>();
+
+ if (kafkaOffsetString == null || kafkaOffsetString.isEmpty()) {
+ return partitionOffsets;
+ }
+
+ try {
+ // Split by semicolon to get individual partition entries
+ String[] partitionEntries = kafkaOffsetString.split(PARTITION_SEPARATOR);
+
+ for (String entry : partitionEntries) {
+ entry = entry.trim();
+
+ // Skip cluster metadata entries (they don't contain partition:offset format)
+ if (!entry.contains(":") || entry.contains("kafka_cluster")) {
+ continue;
+ }
+
+ // Entry format: kafka_metadata%3Atopic%3Apartition:offset
+ // Find the last colon which separates partition and offset
+ int lastColonIndex = entry.lastIndexOf(':');
+ if (lastColonIndex == -1) {
+ continue;
+ }
+
+ String offsetStr = entry.substring(lastColonIndex + 1);
+ String beforeOffset = entry.substring(0, lastColonIndex);
+
+ // Find the partition number (everything after the last %3A before the colon)
+ int lastEncodedColonIndex = beforeOffset.lastIndexOf(URL_ENCODED_COLON);
+ if (lastEncodedColonIndex == -1) {
+ continue;
+ }
+
+ String partitionStr = beforeOffset.substring(lastEncodedColonIndex + URL_ENCODED_COLON.length());
+
+ try {
+ int partitionId = Integer.parseInt(partitionStr);
+ long offset = Long.parseLong(offsetStr);
+ partitionOffsets.put(partitionId, offset);
+ } catch (NumberFormatException e) {
+ log.warn("Failed to parse partition ID or offset from entry: {}", entry, e);
+ }
+ }
+ } catch (Exception e) {
+ log.error("Failed to parse Kafka offset string: {}", kafkaOffsetString, e);
Review Comment:
You are right — changed to throw `HoodieException` instead of silently swallowing the error. The individual per-entry `NumberFormatException` still logs a warning and skips (for resilient partial parsing of individual malformed entries), but any unexpected exception during overall parsing now propagates to the caller.
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java:
##########
@@ -729,4 +733,153 @@ public static void validateWriteStatus(
});
}
}
+
+ /**
+ * Extracts Kafka offset metadata from a Hudi commit.
+ *
+ * @param commitInstant The commit instant to extract metadata from
+ * @param timeline The timeline to read commit details from
+ * @return The Kafka offset metadata string, or null if not found
+ * @throws IOException if error reading commit metadata
+ */
+ public static String extractKafkaOffsetMetadata(final HoodieInstant commitInstant,
+ final HoodieTimeline timeline) throws IOException {
+ HoodieCommitMetadata commitMetadata = TimelineUtils.getCommitMetadata(commitInstant, timeline);
+ return commitMetadata.getExtraMetadata().get(HOODIE_METADATA_KEY);
+ }
+
+ /**
+ * Parses Kafka offset string and restores partition offsets as Map.
+ *
+ * @param kafkaOffsetString URL-encoded Kafka offset string in format:
+ * "kafka_metadata%3Atopic%3Apartition:offset;kafka_metadata%3Atopic%3Apartition:offset"
+ * @return Map of partition ID to offset, or empty map if parsing fails
+ */
+ public static Map<Integer, Long> parseKafkaOffsets(final String kafkaOffsetString) {
+ Map<Integer, Long> partitionOffsets = new HashMap<>();
+
+ if (kafkaOffsetString == null || kafkaOffsetString.isEmpty()) {
+ return partitionOffsets;
+ }
+
+ try {
+ // Split by semicolon to get individual partition entries
+ String[] partitionEntries = kafkaOffsetString.split(PARTITION_SEPARATOR);
+
+ for (String entry : partitionEntries) {
+ entry = entry.trim();
+
+ // Skip cluster metadata entries (they don't contain partition:offset format)
+ if (!entry.contains(":") || entry.contains("kafka_cluster")) {
+ continue;
+ }
+
+ // Entry format: kafka_metadata%3Atopic%3Apartition:offset
+ // Find the last colon which separates partition and offset
+ int lastColonIndex = entry.lastIndexOf(':');
+ if (lastColonIndex == -1) {
+ continue;
+ }
+
+ String offsetStr = entry.substring(lastColonIndex + 1);
+ String beforeOffset = entry.substring(0, lastColonIndex);
+
+ // Find the partition number (everything after the last %3A before the colon)
+ int lastEncodedColonIndex = beforeOffset.lastIndexOf(URL_ENCODED_COLON);
+ if (lastEncodedColonIndex == -1) {
+ continue;
+ }
+
+ String partitionStr = beforeOffset.substring(lastEncodedColonIndex + URL_ENCODED_COLON.length());
+
+ try {
+ int partitionId = Integer.parseInt(partitionStr);
+ long offset = Long.parseLong(offsetStr);
+ partitionOffsets.put(partitionId, offset);
+ } catch (NumberFormatException e) {
+ log.warn("Failed to parse partition ID or offset from entry: {}", entry, e);
+ }
+ }
+ } catch (Exception e) {
+ log.error("Failed to parse Kafka offset string: {}", kafkaOffsetString, e);
+ }
+
+ return partitionOffsets;
+ }
+
+ /**
+ * Calculates the total difference count between Kafka offsets of two Hudi commits.
+ * The difference is: current commit's kafka offset minus previous commit's kafka offset
+ * for each partition. All partition differences are summed together.
+ *
+ * @param currentCommitInstant The current (newer) commit instant
+ * @param previousCommitInstant The previous (older) commit instant
+ * @param timeline The timeline to read commit details from
+ * @return The total count of offset differences across all partitions, or 0 if calculation fails
+ */
+ public static long calculateKafkaOffsetDifference(
+ final HoodieInstant currentCommitInstant,
+ final HoodieInstant previousCommitInstant,
+ final HoodieTimeline timeline) {
+
+ if (currentCommitInstant == null || previousCommitInstant == null || timeline == null) {
+ log.warn("Invalid input parameters for calculating Kafka offset difference");
+ return 0L;
+ }
+
+ try {
+ // Extract Kafka offset metadata from both commits
+ String currentOffsetString = extractKafkaOffsetMetadata(currentCommitInstant, timeline);
+ String previousOffsetString = extractKafkaOffsetMetadata(previousCommitInstant, timeline);
+
+ if (currentOffsetString == null || previousOffsetString == null) {
+ log.warn("Kafka offset metadata not found in one or both commits: current={}, previous={}",
+ currentCommitInstant.requestedTime(), previousCommitInstant.requestedTime());
+ return 0L;
+ }
+
+ // Parse offset strings into partition -> offset maps
+ Map<Integer, Long> currentOffsets = parseKafkaOffsets(currentOffsetString);
+ Map<Integer, Long> previousOffsets = parseKafkaOffsets(previousOffsetString);
+
+ if (currentOffsets.isEmpty() || previousOffsets.isEmpty()) {
+ log.warn("Failed to parse Kafka offsets from commits: current={}, previous={}",
+ currentCommitInstant.requestedTime(), previousCommitInstant.requestedTime());
+ return 0L;
+ }
+
+ // Calculate total difference across all partitions
+ long totalDifference = 0L;
+
+ for (Map.Entry<Integer, Long> currentEntry : currentOffsets.entrySet()) {
+ Integer partitionId = currentEntry.getKey();
+ Long currentOffset = currentEntry.getValue();
+ Long previousOffset = previousOffsets.get(partitionId);
+
+ // If partition doesn't exist in previous commit (new partition), use 0 as base
+ if (previousOffset == null) {
+ previousOffset = 0L;
+ log.debug("New partition {} detected, using 0 as previous offset baseline",
+ partitionId);
+ }
+
+ long difference = currentOffset - previousOffset;
+ totalDifference += difference;
+
+ log.debug("Partition {} offset difference: {} - {} = {}",
+ partitionId, currentOffset, previousOffset, difference);
+ }
+
+ log.info("Total Kafka offset difference between commits {} and {}: {}",
+ currentCommitInstant.requestedTime(), previousCommitInstant.requestedTime(),
+ totalDifference);
+
+ return totalDifference;
+
+ } catch (Exception e) {
+ log.error("Failed to calculate Kafka offset difference between commits {} and {}",
+ currentCommitInstant.requestedTime(), previousCommitInstant.requestedTime(), e);
+ return 0L;
Review Comment:
Agreed — returning 0 silently masks real failures. Changed all error paths to throw:
- Null parameters → `IllegalArgumentException`
- Missing metadata → `HoodieException`
- Empty parsed offsets → `HoodieException`
- `IOException` from metadata extraction now propagates directly
Updated tests to assert exceptions for these cases instead of expecting 0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "shangxinli (via GitHub)" <gi...@apache.org>.
shangxinli commented on PR #18125:
URL: https://github.com/apache/hudi/pull/18125#issuecomment-3867369419
@prashantwason Can you have a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #18125:
URL: https://github.com/apache/hudi/pull/18125#discussion_r2881571185
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/util/TestStreamerUtil.java:
##########
@@ -0,0 +1,279 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.util;
+
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineUtils;
+
+import org.junit.jupiter.api.Test;
+import org.mockito.MockedStatic;
+import org.mockito.Mockito;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.hudi.exception.HoodieException;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Tests for StreamerUtil Kafka offset methods.
+ */
+public class TestStreamerUtil {
Review Comment:
we already have a test class `TestStreamerUtil`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18125:
URL: https://github.com/apache/hudi/pull/18125#issuecomment-3999215294
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0296f2f24aca47aa8d38c23e1620702b45d51163",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11865",
"triggerID" : "0296f2f24aca47aa8d38c23e1620702b45d51163",
"triggerType" : "PUSH"
}, {
"hash" : "86bf764280f3963bacf5a91c9f18827fd361b7ef",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12400",
"triggerID" : "86bf764280f3963bacf5a91c9f18827fd361b7ef",
"triggerType" : "PUSH"
}, {
"hash" : "bbcf710de4aa74feff668c4fb4190dc5780fd6dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12402",
"triggerID" : "bbcf710de4aa74feff668c4fb4190dc5780fd6dc",
"triggerType" : "PUSH"
}, {
"hash" : "8fddf1084fe0125a5e383f7cf2ac237da4b0f657",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12404",
"triggerID" : "8fddf1084fe0125a5e383f7cf2ac237da4b0f657",
"triggerType" : "PUSH"
}, {
"hash" : "d5adeaa1818c6c315437ffe2589c2ea661f913e4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12419",
"triggerID" : "d5adeaa1818c6c315437ffe2589c2ea661f913e4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d5adeaa1818c6c315437ffe2589c2ea661f913e4 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12419)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18125:
URL: https://github.com/apache/hudi/pull/18125#issuecomment-4016579136
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0296f2f24aca47aa8d38c23e1620702b45d51163",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11865",
"triggerID" : "0296f2f24aca47aa8d38c23e1620702b45d51163",
"triggerType" : "PUSH"
}, {
"hash" : "86bf764280f3963bacf5a91c9f18827fd361b7ef",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12400",
"triggerID" : "86bf764280f3963bacf5a91c9f18827fd361b7ef",
"triggerType" : "PUSH"
}, {
"hash" : "bbcf710de4aa74feff668c4fb4190dc5780fd6dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12402",
"triggerID" : "bbcf710de4aa74feff668c4fb4190dc5780fd6dc",
"triggerType" : "PUSH"
}, {
"hash" : "8fddf1084fe0125a5e383f7cf2ac237da4b0f657",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12404",
"triggerID" : "8fddf1084fe0125a5e383f7cf2ac237da4b0f657",
"triggerType" : "PUSH"
}, {
"hash" : "d5adeaa1818c6c315437ffe2589c2ea661f913e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12419",
"triggerID" : "d5adeaa1818c6c315437ffe2589c2ea661f913e4",
"triggerType" : "PUSH"
}, {
"hash" : "b931ee659898114083659324895f5aac346a4e47",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12453",
"triggerID" : "b931ee659898114083659324895f5aac346a4e47",
"triggerType" : "PUSH"
}, {
"hash" : "71655a3f73a1472292fd7011b6e6e164056295cf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "71655a3f73a1472292fd7011b6e6e164056295cf",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b931ee659898114083659324895f5aac346a4e47 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12453)
* 71655a3f73a1472292fd7011b6e6e164056295cf UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18125:
URL: https://github.com/apache/hudi/pull/18125#issuecomment-4024444506
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18125?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 55.32%. Comparing base ([`93b8e9f`](https://app.codecov.io/gh/apache/hudi/commit/93b8e9fc9804f75c1c9e21e5ca421df758743a8c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`fab648e`](https://app.codecov.io/gh/apache/hudi/commit/fab648eaa67a2196bacaa984db93e075c649eea7?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18125 +/- ##
============================================
- Coverage 57.30% 55.32% -1.99%
+ Complexity 18626 17937 -689
============================================
Files 1955 1955
Lines 106916 106916
Branches 13225 13225
============================================
- Hits 61267 59146 -2121
- Misses 39858 41920 +2062
- Partials 5791 5850 +59
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `45.23% <ø> (+0.01%)` | :arrow_up: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `43.18% <ø> (-4.29%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `45.57% <ø> (+<0.01%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
[see 266 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18125/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18125:
URL: https://github.com/apache/hudi/pull/18125#issuecomment-4024072908
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0296f2f24aca47aa8d38c23e1620702b45d51163",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11865",
"triggerID" : "0296f2f24aca47aa8d38c23e1620702b45d51163",
"triggerType" : "PUSH"
}, {
"hash" : "86bf764280f3963bacf5a91c9f18827fd361b7ef",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12400",
"triggerID" : "86bf764280f3963bacf5a91c9f18827fd361b7ef",
"triggerType" : "PUSH"
}, {
"hash" : "bbcf710de4aa74feff668c4fb4190dc5780fd6dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12402",
"triggerID" : "bbcf710de4aa74feff668c4fb4190dc5780fd6dc",
"triggerType" : "PUSH"
}, {
"hash" : "8fddf1084fe0125a5e383f7cf2ac237da4b0f657",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12404",
"triggerID" : "8fddf1084fe0125a5e383f7cf2ac237da4b0f657",
"triggerType" : "PUSH"
}, {
"hash" : "d5adeaa1818c6c315437ffe2589c2ea661f913e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12419",
"triggerID" : "d5adeaa1818c6c315437ffe2589c2ea661f913e4",
"triggerType" : "PUSH"
}, {
"hash" : "b931ee659898114083659324895f5aac346a4e47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12453",
"triggerID" : "b931ee659898114083659324895f5aac346a4e47",
"triggerType" : "PUSH"
}, {
"hash" : "71655a3f73a1472292fd7011b6e6e164056295cf",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12491",
"triggerID" : "71655a3f73a1472292fd7011b6e6e164056295cf",
"triggerType" : "PUSH"
}, {
"hash" : "fab648eaa67a2196bacaa984db93e075c649eea7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fab648eaa67a2196bacaa984db93e075c649eea7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 71655a3f73a1472292fd7011b6e6e164056295cf Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12491)
* fab648eaa67a2196bacaa984db93e075c649eea7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "shangxinli (via GitHub)" <gi...@apache.org>.
shangxinli commented on code in PR #18125:
URL: https://github.com/apache/hudi/pull/18125#discussion_r2892433096
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java:
##########
@@ -112,6 +112,7 @@ public class StreamerUtil {
public static final String FLINK_CHECKPOINT_ID = "flink_checkpoint_id";
public static final String EMPTY_PARTITION_PATH = "";
+
Review Comment:
done
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/utils/TestStreamerUtil.java:
##########
@@ -199,5 +211,234 @@ void testTableExist() throws IOException {
assertTrue(StreamerUtil.tableExists(basePath, HadoopConfigurations.getHadoopConf(conf)));
}
}
+
+ // ---- KafkaOffsetParseUtils tests ----
+
+ @Test
+ void testExtractKafkaOffsetMetadata() throws Exception {
Review Comment:
Done. Created a separate TestKafkaOffsetParseUtils test class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18125:
URL: https://github.com/apache/hudi/pull/18125#issuecomment-3994310531
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0296f2f24aca47aa8d38c23e1620702b45d51163",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11865",
"triggerID" : "0296f2f24aca47aa8d38c23e1620702b45d51163",
"triggerType" : "PUSH"
}, {
"hash" : "86bf764280f3963bacf5a91c9f18827fd361b7ef",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12400",
"triggerID" : "86bf764280f3963bacf5a91c9f18827fd361b7ef",
"triggerType" : "PUSH"
}, {
"hash" : "bbcf710de4aa74feff668c4fb4190dc5780fd6dc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12402",
"triggerID" : "bbcf710de4aa74feff668c4fb4190dc5780fd6dc",
"triggerType" : "PUSH"
}, {
"hash" : "8fddf1084fe0125a5e383f7cf2ac237da4b0f657",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8fddf1084fe0125a5e383f7cf2ac237da4b0f657",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bbcf710de4aa74feff668c4fb4190dc5780fd6dc Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12402)
* 8fddf1084fe0125a5e383f7cf2ac237da4b0f657 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "shangxinli (via GitHub)" <gi...@apache.org>.
shangxinli commented on PR #18125:
URL: https://github.com/apache/hudi/pull/18125#issuecomment-3993886401
**Re: partition disappearing between commits in `calculateKafkaOffsetDifference`**
Good question — yes, it's possible a partition exists in the previous commit but not in the current one (e.g., topic reconfiguration, consumer rebalance).
The offset diff contribution for such a partition is correctly 0 — no records were consumed from it in this batch, so there's nothing to count. However, a disappearing partition may indicate an issue worth investigating, so I added a `LOG.warn` that flags each such partition along with its previous offset.
No functional change to the diff calculation itself — we still only iterate over `currentOffsets` since those are the partitions that actually produced records in this batch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #18125:
URL: https://github.com/apache/hudi/pull/18125#discussion_r2887441830
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/utils/TestStreamerUtil.java:
##########
@@ -199,5 +211,234 @@ void testTableExist() throws IOException {
assertTrue(StreamerUtil.tableExists(basePath, HadoopConfigurations.getHadoopConf(conf)));
}
}
+
+ // ---- KafkaOffsetParseUtils tests ----
+
+ @Test
+ void testExtractKafkaOffsetMetadata() throws Exception {
Review Comment:
we can write a separate test class `TestKafkaOffsetParseUtils`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18125:
URL: https://github.com/apache/hudi/pull/18125#issuecomment-4024656565
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0296f2f24aca47aa8d38c23e1620702b45d51163",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11865",
"triggerID" : "0296f2f24aca47aa8d38c23e1620702b45d51163",
"triggerType" : "PUSH"
}, {
"hash" : "86bf764280f3963bacf5a91c9f18827fd361b7ef",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12400",
"triggerID" : "86bf764280f3963bacf5a91c9f18827fd361b7ef",
"triggerType" : "PUSH"
}, {
"hash" : "bbcf710de4aa74feff668c4fb4190dc5780fd6dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12402",
"triggerID" : "bbcf710de4aa74feff668c4fb4190dc5780fd6dc",
"triggerType" : "PUSH"
}, {
"hash" : "8fddf1084fe0125a5e383f7cf2ac237da4b0f657",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12404",
"triggerID" : "8fddf1084fe0125a5e383f7cf2ac237da4b0f657",
"triggerType" : "PUSH"
}, {
"hash" : "d5adeaa1818c6c315437ffe2589c2ea661f913e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12419",
"triggerID" : "d5adeaa1818c6c315437ffe2589c2ea661f913e4",
"triggerType" : "PUSH"
}, {
"hash" : "b931ee659898114083659324895f5aac346a4e47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12453",
"triggerID" : "b931ee659898114083659324895f5aac346a4e47",
"triggerType" : "PUSH"
}, {
"hash" : "71655a3f73a1472292fd7011b6e6e164056295cf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12491",
"triggerID" : "71655a3f73a1472292fd7011b6e6e164056295cf",
"triggerType" : "PUSH"
}, {
"hash" : "fab648eaa67a2196bacaa984db93e075c649eea7",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12502",
"triggerID" : "fab648eaa67a2196bacaa984db93e075c649eea7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fab648eaa67a2196bacaa984db93e075c649eea7 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12502)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #18125:
URL: https://github.com/apache/hudi/pull/18125
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18125:
URL: https://github.com/apache/hudi/pull/18125#issuecomment-4008521058
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0296f2f24aca47aa8d38c23e1620702b45d51163",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11865",
"triggerID" : "0296f2f24aca47aa8d38c23e1620702b45d51163",
"triggerType" : "PUSH"
}, {
"hash" : "86bf764280f3963bacf5a91c9f18827fd361b7ef",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12400",
"triggerID" : "86bf764280f3963bacf5a91c9f18827fd361b7ef",
"triggerType" : "PUSH"
}, {
"hash" : "bbcf710de4aa74feff668c4fb4190dc5780fd6dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12402",
"triggerID" : "bbcf710de4aa74feff668c4fb4190dc5780fd6dc",
"triggerType" : "PUSH"
}, {
"hash" : "8fddf1084fe0125a5e383f7cf2ac237da4b0f657",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12404",
"triggerID" : "8fddf1084fe0125a5e383f7cf2ac237da4b0f657",
"triggerType" : "PUSH"
}, {
"hash" : "d5adeaa1818c6c315437ffe2589c2ea661f913e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12419",
"triggerID" : "d5adeaa1818c6c315437ffe2589c2ea661f913e4",
"triggerType" : "PUSH"
}, {
"hash" : "b931ee659898114083659324895f5aac346a4e47",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12453",
"triggerID" : "b931ee659898114083659324895f5aac346a4e47",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b931ee659898114083659324895f5aac346a4e47 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12453)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "shangxinli (via GitHub)" <gi...@apache.org>.
shangxinli commented on code in PR #18125:
URL: https://github.com/apache/hudi/pull/18125#discussion_r2884965321
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java:
##########
@@ -729,4 +733,163 @@ public static void validateWriteStatus(
});
}
}
+
+ /**
+ * Extracts Kafka offset metadata from a Hudi commit.
+ *
+ * @param commitInstant The commit instant to extract metadata from
+ * @param timeline The timeline to read commit details from
+ * @return The Kafka offset metadata string, or null if not found
+ * @throws IOException if error reading commit metadata
+ */
+ public static String extractKafkaOffsetMetadata(final HoodieInstant commitInstant,
Review Comment:
Done. Moved `extractKafkaOffsetMetadata`, `parseKafkaOffsets`, and `calculateKafkaOffsetDifference` into a new `KafkaOffsetParseUtils` class along with the related constants.
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/util/TestStreamerUtil.java:
##########
@@ -0,0 +1,279 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.util;
+
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineUtils;
+
+import org.junit.jupiter.api.Test;
+import org.mockito.MockedStatic;
+import org.mockito.Mockito;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.hudi.exception.HoodieException;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Tests for StreamerUtil Kafka offset methods.
+ */
+public class TestStreamerUtil {
Review Comment:
Done. Removed the separate test class and merged all Kafka offset tests into the existing `TestStreamerUtil` in `org.apache.hudi.utils`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18125:
URL: https://github.com/apache/hudi/pull/18125#issuecomment-3994181198
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0296f2f24aca47aa8d38c23e1620702b45d51163",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11865",
"triggerID" : "0296f2f24aca47aa8d38c23e1620702b45d51163",
"triggerType" : "PUSH"
}, {
"hash" : "86bf764280f3963bacf5a91c9f18827fd361b7ef",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12400",
"triggerID" : "86bf764280f3963bacf5a91c9f18827fd361b7ef",
"triggerType" : "PUSH"
}, {
"hash" : "bbcf710de4aa74feff668c4fb4190dc5780fd6dc",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12402",
"triggerID" : "bbcf710de4aa74feff668c4fb4190dc5780fd6dc",
"triggerType" : "PUSH"
}, {
"hash" : "8fddf1084fe0125a5e383f7cf2ac237da4b0f657",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8fddf1084fe0125a5e383f7cf2ac237da4b0f657",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0296f2f24aca47aa8d38c23e1620702b45d51163 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11865)
* 86bf764280f3963bacf5a91c9f18827fd361b7ef Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12400)
* bbcf710de4aa74feff668c4fb4190dc5780fd6dc Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12402)
* 8fddf1084fe0125a5e383f7cf2ac237da4b0f657 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #18125:
URL: https://github.com/apache/hudi/pull/18125#discussion_r2881574245
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java:
##########
@@ -729,4 +733,163 @@ public static void validateWriteStatus(
});
}
}
+
+ /**
+ * Extracts Kafka offset metadata from a Hudi commit.
+ *
+ * @param commitInstant The commit instant to extract metadata from
+ * @param timeline The timeline to read commit details from
+ * @return The Kafka offset metadata string, or null if not found
+ * @throws IOException if error reading commit metadata
+ */
+ public static String extractKafkaOffsetMetadata(final HoodieInstant commitInstant,
Review Comment:
can we move these new utils into a separate class probably named `KafkaOffsetParseUtils`? there is no need to we put every thing into the `StreamerUtil`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18125:
URL: https://github.com/apache/hudi/pull/18125#issuecomment-4007855590
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0296f2f24aca47aa8d38c23e1620702b45d51163",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11865",
"triggerID" : "0296f2f24aca47aa8d38c23e1620702b45d51163",
"triggerType" : "PUSH"
}, {
"hash" : "86bf764280f3963bacf5a91c9f18827fd361b7ef",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12400",
"triggerID" : "86bf764280f3963bacf5a91c9f18827fd361b7ef",
"triggerType" : "PUSH"
}, {
"hash" : "bbcf710de4aa74feff668c4fb4190dc5780fd6dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12402",
"triggerID" : "bbcf710de4aa74feff668c4fb4190dc5780fd6dc",
"triggerType" : "PUSH"
}, {
"hash" : "8fddf1084fe0125a5e383f7cf2ac237da4b0f657",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12404",
"triggerID" : "8fddf1084fe0125a5e383f7cf2ac237da4b0f657",
"triggerType" : "PUSH"
}, {
"hash" : "d5adeaa1818c6c315437ffe2589c2ea661f913e4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12419",
"triggerID" : "d5adeaa1818c6c315437ffe2589c2ea661f913e4",
"triggerType" : "PUSH"
}, {
"hash" : "b931ee659898114083659324895f5aac346a4e47",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b931ee659898114083659324895f5aac346a4e47",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d5adeaa1818c6c315437ffe2589c2ea661f913e4 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12419)
* b931ee659898114083659324895f5aac346a4e47 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18125:
URL: https://github.com/apache/hudi/pull/18125#issuecomment-4024351215
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18125?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 39.63%. Comparing base ([`93b8e9f`](https://app.codecov.io/gh/apache/hudi/commit/93b8e9fc9804f75c1c9e21e5ca421df758743a8c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`fab648e`](https://app.codecov.io/gh/apache/hudi/commit/fab648eaa67a2196bacaa984db93e075c649eea7?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
> :exclamation: There is a different number of reports uploaded between BASE (93b8e9f) and HEAD (fab648e). Click for more details.
>
> <details><summary>HEAD has 12 uploads less than BASE</summary>
>
>| Flag | BASE (93b8e9f) | HEAD (fab648e) |
>|------|------|------|
>|spark-scala-tests|6|3|
>|spark-java-tests|9|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18125 +/- ##
=============================================
- Coverage 57.30% 39.63% -17.68%
+ Complexity 18626 12620 -6006
=============================================
Files 1955 1955
Lines 106916 106916
Branches 13225 13225
=============================================
- Hits 61267 42373 -18894
- Misses 39858 59450 +19592
+ Partials 5791 5093 -698
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `45.23% <ø> (+0.01%)` | :arrow_up: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `33.11% <ø> (-12.46%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
[see 860 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18125/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18125:
URL: https://github.com/apache/hudi/pull/18125#issuecomment-4024093333
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0296f2f24aca47aa8d38c23e1620702b45d51163",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11865",
"triggerID" : "0296f2f24aca47aa8d38c23e1620702b45d51163",
"triggerType" : "PUSH"
}, {
"hash" : "86bf764280f3963bacf5a91c9f18827fd361b7ef",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12400",
"triggerID" : "86bf764280f3963bacf5a91c9f18827fd361b7ef",
"triggerType" : "PUSH"
}, {
"hash" : "bbcf710de4aa74feff668c4fb4190dc5780fd6dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12402",
"triggerID" : "bbcf710de4aa74feff668c4fb4190dc5780fd6dc",
"triggerType" : "PUSH"
}, {
"hash" : "8fddf1084fe0125a5e383f7cf2ac237da4b0f657",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12404",
"triggerID" : "8fddf1084fe0125a5e383f7cf2ac237da4b0f657",
"triggerType" : "PUSH"
}, {
"hash" : "d5adeaa1818c6c315437ffe2589c2ea661f913e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12419",
"triggerID" : "d5adeaa1818c6c315437ffe2589c2ea661f913e4",
"triggerType" : "PUSH"
}, {
"hash" : "b931ee659898114083659324895f5aac346a4e47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12453",
"triggerID" : "b931ee659898114083659324895f5aac346a4e47",
"triggerType" : "PUSH"
}, {
"hash" : "71655a3f73a1472292fd7011b6e6e164056295cf",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12491",
"triggerID" : "71655a3f73a1472292fd7011b6e6e164056295cf",
"triggerType" : "PUSH"
}, {
"hash" : "fab648eaa67a2196bacaa984db93e075c649eea7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12502",
"triggerID" : "fab648eaa67a2196bacaa984db93e075c649eea7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 71655a3f73a1472292fd7011b6e6e164056295cf Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12491)
* fab648eaa67a2196bacaa984db93e075c649eea7 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12502)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18125:
URL: https://github.com/apache/hudi/pull/18125#issuecomment-3865716363
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b21e77f61e70826afe74e66188afb16a1b1bd74",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4b21e77f61e70826afe74e66188afb16a1b1bd74",
"triggerType" : "PUSH"
}, {
"hash" : "14b3884aa832d437ff362c482bc2d9d02fc27d57",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "14b3884aa832d437ff362c482bc2d9d02fc27d57",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4b21e77f61e70826afe74e66188afb16a1b1bd74 UNKNOWN
* 14b3884aa832d437ff362c482bc2d9d02fc27d57 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18125:
URL: https://github.com/apache/hudi/pull/18125#issuecomment-4024584613
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18125?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 57.29%. Comparing base ([`93b8e9f`](https://app.codecov.io/gh/apache/hudi/commit/93b8e9fc9804f75c1c9e21e5ca421df758743a8c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`fab648e`](https://app.codecov.io/gh/apache/hudi/commit/fab648eaa67a2196bacaa984db93e075c649eea7?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18125 +/- ##
============================================
- Coverage 57.30% 57.29% -0.01%
Complexity 18626 18626
============================================
Files 1955 1955
Lines 106916 106916
Branches 13225 13225
============================================
- Hits 61267 61261 -6
- Misses 39858 39863 +5
- Partials 5791 5792 +1
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `45.23% <ø> (+0.01%)` | :arrow_up: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `47.44% <ø> (-0.03%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `45.57% <ø> (+<0.01%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
[see 4 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18125/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18125:
URL: https://github.com/apache/hudi/pull/18125#issuecomment-4024352475
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18125?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 39.63%. Comparing base ([`93b8e9f`](https://app.codecov.io/gh/apache/hudi/commit/93b8e9fc9804f75c1c9e21e5ca421df758743a8c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`fab648e`](https://app.codecov.io/gh/apache/hudi/commit/fab648eaa67a2196bacaa984db93e075c649eea7?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
> :exclamation: There is a different number of reports uploaded between BASE (93b8e9f) and HEAD (fab648e). Click for more details.
>
> <details><summary>HEAD has 12 uploads less than BASE</summary>
>
>| Flag | BASE (93b8e9f) | HEAD (fab648e) |
>|------|------|------|
>|spark-scala-tests|6|3|
>|spark-java-tests|9|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18125 +/- ##
=============================================
- Coverage 57.30% 39.63% -17.68%
+ Complexity 18626 12620 -6006
=============================================
Files 1955 1955
Lines 106916 106916
Branches 13225 13225
=============================================
- Hits 61267 42373 -18894
- Misses 39858 59450 +19592
+ Partials 5791 5093 -698
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `45.23% <ø> (+0.01%)` | :arrow_up: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `33.11% <ø> (-12.46%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
[see 860 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18125/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "shangxinli (via GitHub)" <gi...@apache.org>.
shangxinli commented on code in PR #18125:
URL: https://github.com/apache/hudi/pull/18125#discussion_r2899783965
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java:
##########
@@ -111,6 +111,7 @@ public class StreamerUtil {
public static final String FLINK_CHECKPOINT_ID = "flink_checkpoint_id";
public static final String EMPTY_PARTITION_PATH = "";
+ public static final String HOODIE_METADATA_KEY = "HoodieMetadataKey";
Review Comment:
Done. Removed the duplicate constant from StreamerUtil — it's only in KafkaOffsetParseUtils now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18125:
URL: https://github.com/apache/hudi/pull/18125#issuecomment-3993852689
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0296f2f24aca47aa8d38c23e1620702b45d51163",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11865",
"triggerID" : "0296f2f24aca47aa8d38c23e1620702b45d51163",
"triggerType" : "PUSH"
}, {
"hash" : "86bf764280f3963bacf5a91c9f18827fd361b7ef",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "86bf764280f3963bacf5a91c9f18827fd361b7ef",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0296f2f24aca47aa8d38c23e1620702b45d51163 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11865)
* 86bf764280f3963bacf5a91c9f18827fd361b7ef UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18125:
URL: https://github.com/apache/hudi/pull/18125#issuecomment-3998874719
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0296f2f24aca47aa8d38c23e1620702b45d51163",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11865",
"triggerID" : "0296f2f24aca47aa8d38c23e1620702b45d51163",
"triggerType" : "PUSH"
}, {
"hash" : "86bf764280f3963bacf5a91c9f18827fd361b7ef",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12400",
"triggerID" : "86bf764280f3963bacf5a91c9f18827fd361b7ef",
"triggerType" : "PUSH"
}, {
"hash" : "bbcf710de4aa74feff668c4fb4190dc5780fd6dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12402",
"triggerID" : "bbcf710de4aa74feff668c4fb4190dc5780fd6dc",
"triggerType" : "PUSH"
}, {
"hash" : "8fddf1084fe0125a5e383f7cf2ac237da4b0f657",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12404",
"triggerID" : "8fddf1084fe0125a5e383f7cf2ac237da4b0f657",
"triggerType" : "PUSH"
}, {
"hash" : "d5adeaa1818c6c315437ffe2589c2ea661f913e4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d5adeaa1818c6c315437ffe2589c2ea661f913e4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8fddf1084fe0125a5e383f7cf2ac237da4b0f657 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12404)
* d5adeaa1818c6c315437ffe2589c2ea661f913e4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #18125:
URL: https://github.com/apache/hudi/pull/18125#discussion_r2887439385
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java:
##########
@@ -112,6 +112,7 @@ public class StreamerUtil {
public static final String FLINK_CHECKPOINT_ID = "flink_checkpoint_id";
public static final String EMPTY_PARTITION_PATH = "";
+
Review Comment:
revert the changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18125:
URL: https://github.com/apache/hudi/pull/18125#issuecomment-4024545821
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18125?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 57.29%. Comparing base ([`93b8e9f`](https://app.codecov.io/gh/apache/hudi/commit/93b8e9fc9804f75c1c9e21e5ca421df758743a8c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`fab648e`](https://app.codecov.io/gh/apache/hudi/commit/fab648eaa67a2196bacaa984db93e075c649eea7?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18125 +/- ##
============================================
- Coverage 57.30% 57.29% -0.01%
+ Complexity 18626 18625 -1
============================================
Files 1955 1955
Lines 106916 106916
Branches 13225 13225
============================================
- Hits 61267 61261 -6
- Misses 39858 39863 +5
- Partials 5791 5792 +1
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `45.23% <ø> (+0.01%)` | :arrow_up: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `47.41% <ø> (-0.07%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `45.57% <ø> (+<0.01%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
[see 4 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18125/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18125:
URL: https://github.com/apache/hudi/pull/18125#issuecomment-4024392605
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18125?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 43.42%. Comparing base ([`93b8e9f`](https://app.codecov.io/gh/apache/hudi/commit/93b8e9fc9804f75c1c9e21e5ca421df758743a8c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`fab648e`](https://app.codecov.io/gh/apache/hudi/commit/fab648eaa67a2196bacaa984db93e075c649eea7?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
> :exclamation: There is a different number of reports uploaded between BASE (93b8e9f) and HEAD (fab648e). Click for more details.
>
> <details><summary>HEAD has 9 uploads less than BASE</summary>
>
>| Flag | BASE (93b8e9f) | HEAD (fab648e) |
>|------|------|------|
>|spark-scala-tests|6|3|
>|spark-java-tests|9|3|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18125 +/- ##
=============================================
- Coverage 57.30% 43.42% -13.88%
+ Complexity 18626 13881 -4745
=============================================
Files 1955 1955
Lines 106916 106916
Branches 13225 13225
=============================================
- Hits 61267 46432 -14835
- Misses 39858 55158 +15300
+ Partials 5791 5326 -465
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `45.23% <ø> (+0.01%)` | :arrow_up: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `33.18% <ø> (-14.29%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `33.11% <ø> (-12.46%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
[see 786 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18125/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18125:
URL: https://github.com/apache/hudi/pull/18125#issuecomment-3994291858
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18125?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 45.27%. Comparing base ([`ec04479`](https://app.codecov.io/gh/apache/hudi/commit/ec044790fe71f04710472ad63e7859f1b29e3985?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`8fddf10`](https://app.codecov.io/gh/apache/hudi/commit/8fddf1084fe0125a5e383f7cf2ac237da4b0f657?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 1013 commits behind head on master.
> :exclamation: There is a different number of reports uploaded between BASE (ec04479) and HEAD (8fddf10). Click for more details.
>
> <details><summary>HEAD has 1 upload less than BASE</summary>
>
>| Flag | BASE (ec04479) | HEAD (8fddf10) |
>|------|------|------|
>||1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18125 +/- ##
=============================================
- Coverage 61.43% 45.27% -16.17%
+ Complexity 23082 8400 -14682
=============================================
Files 2108 1180 -928
Lines 127636 60661 -66975
Branches 14534 6476 -8058
=============================================
- Hits 78409 27462 -50947
+ Misses 42873 30221 -12652
+ Partials 6354 2978 -3376
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18125/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `45.27% <ø> (?)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
[see 3246 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18125/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18125:
URL: https://github.com/apache/hudi/pull/18125#issuecomment-3994664812
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0296f2f24aca47aa8d38c23e1620702b45d51163",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11865",
"triggerID" : "0296f2f24aca47aa8d38c23e1620702b45d51163",
"triggerType" : "PUSH"
}, {
"hash" : "86bf764280f3963bacf5a91c9f18827fd361b7ef",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12400",
"triggerID" : "86bf764280f3963bacf5a91c9f18827fd361b7ef",
"triggerType" : "PUSH"
}, {
"hash" : "bbcf710de4aa74feff668c4fb4190dc5780fd6dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12402",
"triggerID" : "bbcf710de4aa74feff668c4fb4190dc5780fd6dc",
"triggerType" : "PUSH"
}, {
"hash" : "8fddf1084fe0125a5e383f7cf2ac237da4b0f657",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12404",
"triggerID" : "8fddf1084fe0125a5e383f7cf2ac237da4b0f657",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8fddf1084fe0125a5e383f7cf2ac237da4b0f657 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12404)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18125:
URL: https://github.com/apache/hudi/pull/18125#issuecomment-3998892881
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0296f2f24aca47aa8d38c23e1620702b45d51163",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11865",
"triggerID" : "0296f2f24aca47aa8d38c23e1620702b45d51163",
"triggerType" : "PUSH"
}, {
"hash" : "86bf764280f3963bacf5a91c9f18827fd361b7ef",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12400",
"triggerID" : "86bf764280f3963bacf5a91c9f18827fd361b7ef",
"triggerType" : "PUSH"
}, {
"hash" : "bbcf710de4aa74feff668c4fb4190dc5780fd6dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12402",
"triggerID" : "bbcf710de4aa74feff668c4fb4190dc5780fd6dc",
"triggerType" : "PUSH"
}, {
"hash" : "8fddf1084fe0125a5e383f7cf2ac237da4b0f657",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12404",
"triggerID" : "8fddf1084fe0125a5e383f7cf2ac237da4b0f657",
"triggerType" : "PUSH"
}, {
"hash" : "d5adeaa1818c6c315437ffe2589c2ea661f913e4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12419",
"triggerID" : "d5adeaa1818c6c315437ffe2589c2ea661f913e4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8fddf1084fe0125a5e383f7cf2ac237da4b0f657 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12404)
* d5adeaa1818c6c315437ffe2589c2ea661f913e4 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12419)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18125:
URL: https://github.com/apache/hudi/pull/18125#issuecomment-3874758254
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0296f2f24aca47aa8d38c23e1620702b45d51163",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0296f2f24aca47aa8d38c23e1620702b45d51163",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0296f2f24aca47aa8d38c23e1620702b45d51163 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Add helper functions to parse Kafka offset differences b… [hudi]
Posted by "shangxinli (via GitHub)" <gi...@apache.org>.
shangxinli closed pull request #18125: feat(flink): Add helper functions to parse Kafka offset differences b…
URL: https://github.com/apache/hudi/pull/18125
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org