You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/03/05 19:09:54 UTC
[GitHub] [incubator-hudi] lamber-ken opened a new pull request #1377:
[HUDI-663] Fix HoodieDeltaStreamer offset not handled correctly
lamber-ken opened a new pull request #1377: [HUDI-663] Fix HoodieDeltaStreamer offset not handled correctly
URL: https://github.com/apache/incubator-hudi/pull/1377
## What is the purpose of the pull request
More detail, please go ahead with https://github.com/apache/incubator-hudi/issues/1375
1. the first commit content like following:
```
{
"partitionToWriteStats" : { },
"compacted" : false,
"extraMetadata" : {
"ROLLING_STAT" : "{\n \"partitionToRollingStats\" : { },\n \"actionType\" : \"commit\"\n}",
"schema" : "{\"type\":\"record\",\"name\":\"stock_ticks\",\"fields\":[{\"name\":\"volume\",\"type\":\"long\"},{\"name\":\"ts\",\"type\":\"string\"},{\"name\":\"symbol\",\"type\":\"string\"},{\"name\":\"year\",\"type\":\"int\"},{\"name\":\"month\",\"type\":\"string\"},{\"name\":\"high\",\"type\":\"double\"},{\"name\":\"low\",\"type\":\"double\"},{\"name\":\"key\",\"type\":\"string\"},{\"name\":\"date\",\"type\":\"string\"},{\"name\":\"close\",\"type\":\"double\"},{\"name\":\"open\",\"type\":\"double\"},{\"name\":\"day\",\"type\":\"string\"}]}",
"deltastreamer.checkpoint.key" : ""
},
"operationType" : "UPSERT",
"fileIdAndRelativePaths" : { },
"totalRecordsDeleted" : 0,
"totalLogRecordsCompacted" : 0,
"totalScanTime" : 0,
"totalCreateTime" : 0,
"totalUpsertTime" : 0,
"totalCompactedRecordsUpdated" : 0,
"totalLogFilesCompacted" : 0,
"totalLogFilesSize" : 0
}
```
2. the second commit will use the last checkpoint `{}`, which means the fromoffset is `0`.
but the previous messages may be removed because of kafka retention mechanism.
so the low_watermark is greater than 0.
## Brief change log
- double check the lastCheckpointStr
## Verify this pull request
Follow steps: https://github.com/apache/incubator-hudi/issues/1375
## Committer checklist
- [X] Has a corresponding JIRA in PR title & commit
- [X] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on issue #1377: [HUDI-663]
Fix HoodieDeltaStreamer offset not handled correctly
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on issue #1377: [HUDI-663] Fix HoodieDeltaStreamer offset not handled correctly
URL: https://github.com/apache/incubator-hudi/pull/1377#issuecomment-595907417
@garyli1019 can you drive this review? :). I can merge once I have a π from you
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] bvaradar commented on a change in pull request
#1377: [HUDI-663] Fix HoodieDeltaStreamer offset not handled correctly
Posted by GitBox <gi...@apache.org>.
bvaradar commented on a change in pull request #1377: [HUDI-663] Fix HoodieDeltaStreamer offset not handled correctly
URL: https://github.com/apache/incubator-hudi/pull/1377#discussion_r390466238
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -180,7 +180,7 @@ public KafkaOffsetGen(TypedProperties props) {
.map(x -> new TopicPartition(x.topic(), x.partition())).collect(Collectors.toSet());
// Determine the offset ranges to read from
- if (lastCheckpointStr.isPresent()) {
+ if (lastCheckpointStr.isPresent() && !lastCheckpointStr.get().isEmpty()) {
Review comment:
@lamber-ken : Also, Instead of handling the empty checkpoints only for kafka, can we handle it generically in DeltaSync (https://github.com/apache/incubator-hudi/blob/77d5b92d88d6583bdfc09e4c10ecfe7ddbb04806/hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java#L262) so that we can have an uniform handling of checkpoints across sources ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] garyli1019 commented on a change in pull request
#1377: [HUDI-663] Fix HoodieDeltaStreamer offset not handled correctly
Posted by GitBox <gi...@apache.org>.
garyli1019 commented on a change in pull request #1377: [HUDI-663] Fix HoodieDeltaStreamer offset not handled correctly
URL: https://github.com/apache/incubator-hudi/pull/1377#discussion_r391344867
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -180,7 +180,7 @@ public KafkaOffsetGen(TypedProperties props) {
.map(x -> new TopicPartition(x.topic(), x.partition())).collect(Collectors.toSet());
// Determine the offset ranges to read from
- if (lastCheckpointStr.isPresent()) {
+ if (lastCheckpointStr.isPresent() && !lastCheckpointStr.get().isEmpty()) {
Review comment:
I agree with @bvaradar here.
I tried
```java
else if (commitMetadata.getMetadata(CHECKPOINT_KEY) != null && !commitMetadata.getMetadata(CHECKPOINT_KEY).isEmpty()) {
resumeCheckpointStr = Option.of(commitMetadata.getMetadata(CHECKPOINT_KEY));
}
```
The unit tests were passed.
If you mean when the checkpoint was empty, the application was throwing an Exception. I think it should the desired behavior, WDYT?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] garyli1019 commented on issue #1377: [HUDI-663]
Fix HoodieDeltaStreamer offset not handled correctly
Posted by GitBox <gi...@apache.org>.
garyli1019 commented on issue #1377: [HUDI-663] Fix HoodieDeltaStreamer offset not handled correctly
URL: https://github.com/apache/incubator-hudi/pull/1377#issuecomment-601412133
@vinothchandar I thought the empty checkpoint was created by a bug before, but if the empty checkpoint is intended then we need to somehow handle it like this PR
If resetting the checkpoint when an empty checkpoint present makes sense to you guys then I think this PR should be good to go.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] garyli1019 commented on issue #1377: [HUDI-663]
Fix HoodieDeltaStreamer offset not handled correctly
Posted by GitBox <gi...@apache.org>.
garyli1019 commented on issue #1377: [HUDI-663] Fix HoodieDeltaStreamer offset not handled correctly
URL: https://github.com/apache/incubator-hudi/pull/1377#issuecomment-595592624
Thanks @lamber-ken for digging into this issue.
I am curious about how did you store the empty checkpoint in the metadata? Is that produced by the delta streamer?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] lamber-ken commented on issue #1377: [HUDI-663]
Fix HoodieDeltaStreamer offset not handled correctly
Posted by GitBox <gi...@apache.org>.
lamber-ken commented on issue #1377: [HUDI-663] Fix HoodieDeltaStreamer offset not handled correctly
URL: https://github.com/apache/incubator-hudi/pull/1377#issuecomment-595787065
> Thanks @lamber-ken for digging into this issue.
> I am curious about how did you store the empty checkpoint in the metadata? Is that produced by the delta streamer?
π thanks, you're welcome. There are two different cases will store the empty checkpoint.
1. Create a kafka topic(don't produce any data to kafka) which means a topic with no messages, then start the delta streamer, hudi will store the empty checkpoint.
2. Set `auto.offset.reset=latest` and use a new target-base-path. like #1375 says.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] garyli1019 commented on a change in pull request
#1377: [HUDI-663] Fix HoodieDeltaStreamer offset not handled correctly
Posted by GitBox <gi...@apache.org>.
garyli1019 commented on a change in pull request #1377: [HUDI-663] Fix HoodieDeltaStreamer offset not handled correctly
URL: https://github.com/apache/incubator-hudi/pull/1377#discussion_r391350401
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -180,7 +180,7 @@ public KafkaOffsetGen(TypedProperties props) {
.map(x -> new TopicPartition(x.topic(), x.partition())).collect(Collectors.toSet());
// Determine the offset ranges to read from
- if (lastCheckpointStr.isPresent()) {
+ if (lastCheckpointStr.isPresent() && !lastCheckpointStr.get().isEmpty()) {
Review comment:
In this case I think the user could use `cfg.checkpoint = xxx` to reset the checkpoint. It would be concerning for me if the deltastreamer automatically reset the checkpoint for me and I didn't aware of it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] lamber-ken commented on a change in pull request
#1377: [HUDI-663] Fix HoodieDeltaStreamer offset not handled correctly
Posted by GitBox <gi...@apache.org>.
lamber-ken commented on a change in pull request #1377: [HUDI-663] Fix HoodieDeltaStreamer offset not handled correctly
URL: https://github.com/apache/incubator-hudi/pull/1377#discussion_r391352601
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -180,7 +180,7 @@ public KafkaOffsetGen(TypedProperties props) {
.map(x -> new TopicPartition(x.topic(), x.partition())).collect(Collectors.toSet());
// Determine the offset ranges to read from
- if (lastCheckpointStr.isPresent()) {
+ if (lastCheckpointStr.isPresent() && !lastCheckpointStr.get().isEmpty()) {
Review comment:
Thanks, I need to think about it, wait a moment : )
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] lamber-ken commented on a change in pull request
#1377: [HUDI-663] Fix HoodieDeltaStreamer offset not handled correctly
Posted by GitBox <gi...@apache.org>.
lamber-ken commented on a change in pull request #1377: [HUDI-663] Fix HoodieDeltaStreamer offset not handled correctly
URL: https://github.com/apache/incubator-hudi/pull/1377#discussion_r391348111
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -180,7 +180,7 @@ public KafkaOffsetGen(TypedProperties props) {
.map(x -> new TopicPartition(x.topic(), x.partition())).collect(Collectors.toSet());
// Determine the offset ranges to read from
- if (lastCheckpointStr.isPresent()) {
+ if (lastCheckpointStr.isPresent() && !lastCheckpointStr.get().isEmpty()) {
Review comment:
hi @garyli1019, let's imagine a scenario that a topic with no data, the first commit will save empty checkpoint, then the second commit will always throw exception(even if we send msg to kafka). In that case, `EARLIEST` or `LATEST` will no longer has any effect.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar merged pull request #1377:
[HUDI-663] Fix HoodieDeltaStreamer offset not handled correctly
Posted by GitBox <gi...@apache.org>.
vinothchandar merged pull request #1377: [HUDI-663] Fix HoodieDeltaStreamer offset not handled correctly
URL: https://github.com/apache/incubator-hudi/pull/1377
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on issue #1377: [HUDI-663]
Fix HoodieDeltaStreamer offset not handled correctly
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on issue #1377: [HUDI-663] Fix HoodieDeltaStreamer offset not handled correctly
URL: https://github.com/apache/incubator-hudi/pull/1377#issuecomment-596358157
>>Create a kafka topic(don't produce any data to kafka) which means a topic with no messages, then start the delta streamer, hudi will store the empty checkpoint.
I assume there will be partitions created already.. So instead of special casing the empty checkpoint for a specific source (which seems higher maintenance), can we make the code store an actual checkpoint.. Our contract can be that if a checkpoint is stored, then it is indeed non empty..
I am with @garyli1019 that erroring out on empty checkpoints is better.. Not sure if we expect an empty checkpoint in other sources as well
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] garyli1019 commented on a change in pull request
#1377: [HUDI-663] Fix HoodieDeltaStreamer offset not handled correctly
Posted by GitBox <gi...@apache.org>.
garyli1019 commented on a change in pull request #1377: [HUDI-663] Fix HoodieDeltaStreamer offset not handled correctly
URL: https://github.com/apache/incubator-hudi/pull/1377#discussion_r388705608
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -180,7 +180,7 @@ public KafkaOffsetGen(TypedProperties props) {
.map(x -> new TopicPartition(x.topic(), x.partition())).collect(Collectors.toSet());
// Determine the offset ranges to read from
- if (lastCheckpointStr.isPresent()) {
+ if (lastCheckpointStr.isPresent() && !lastCheckpointStr.get().isEmpty()) {
Review comment:
I think this may potentially hide some concerning errors.
e.g. The delta streamer is consuming Kafka source, but a hidden bug happens and stored an empty checkpoint. The next run will just ignore the empty checkpoint and reset to the `LATEST`. Then there will be data loss
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] lamber-ken edited a comment on issue #1377:
[HUDI-663] Fix HoodieDeltaStreamer offset not handled correctly
Posted by GitBox <gi...@apache.org>.
lamber-ken edited a comment on issue #1377: [HUDI-663] Fix HoodieDeltaStreamer offset not handled correctly
URL: https://github.com/apache/incubator-hudi/pull/1377#issuecomment-597431353
Sorry, I revert it, hudi already handled the empty checkpoint by `KafkaOffsetGen.KafkaResetOffsetStrategies`.
User can decide `EARLIEST` or `LATEST` by using `auto.offset.reset` property.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] lamber-ken commented on a change in pull request
#1377: [HUDI-663] Fix HoodieDeltaStreamer offset not handled correctly
Posted by GitBox <gi...@apache.org>.
lamber-ken commented on a change in pull request #1377: [HUDI-663] Fix HoodieDeltaStreamer offset not handled correctly
URL: https://github.com/apache/incubator-hudi/pull/1377#discussion_r391348916
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -180,7 +180,7 @@ public KafkaOffsetGen(TypedProperties props) {
.map(x -> new TopicPartition(x.topic(), x.partition())).collect(Collectors.toSet());
// Determine the offset ranges to read from
- if (lastCheckpointStr.isPresent()) {
+ if (lastCheckpointStr.isPresent() && !lastCheckpointStr.get().isEmpty()) {
Review comment:
Prefer to pushdown the control bebavior to datasource(e.g kafka / pulsar)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] lamber-ken commented on issue #1377: [HUDI-663]
Fix HoodieDeltaStreamer offset not handled correctly
Posted by GitBox <gi...@apache.org>.
lamber-ken commented on issue #1377: [HUDI-663] Fix HoodieDeltaStreamer offset not handled correctly
URL: https://github.com/apache/incubator-hudi/pull/1377#issuecomment-597428982
Thanks for reviewing @garyli1019 @vinothchandar. Had updated the pr by double check empty checkpoint in `KafkaOffsetGen#checkupValidOffsets`.
> I'd appreciate it if we took into consideration how checkpoint is handled in a general source agnostic way and also fix this issue..
This is good idea as bvaradar suggested, but it seems impossible, because different data streams handle empty checkpoint in different way.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on issue #1377: [HUDI-663]
Fix HoodieDeltaStreamer offset not handled correctly
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on issue #1377: [HUDI-663] Fix HoodieDeltaStreamer offset not handled correctly
URL: https://github.com/apache/incubator-hudi/pull/1377#issuecomment-599717588
I am bit confused at this point for the PR.. Can you summarize where we are at?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] lamber-ken commented on issue #1377: [HUDI-663]
Fix HoodieDeltaStreamer offset not handled correctly
Posted by GitBox <gi...@apache.org>.
lamber-ken commented on issue #1377: [HUDI-663] Fix HoodieDeltaStreamer offset not handled correctly
URL: https://github.com/apache/incubator-hudi/pull/1377#issuecomment-597431353
Sorry, I revert it, already handled the empty checkpoint by `KafkaOffsetGen.KafkaResetOffsetStrategies`

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] lamber-ken commented on issue #1377: [HUDI-663]
Fix HoodieDeltaStreamer offset not handled correctly
Posted by GitBox <gi...@apache.org>.
lamber-ken commented on issue #1377: [HUDI-663] Fix HoodieDeltaStreamer offset not handled correctly
URL: https://github.com/apache/incubator-hudi/pull/1377#issuecomment-601486061
@garyli1019 thanks very much for your detail comment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on issue #1377: [HUDI-663]
Fix HoodieDeltaStreamer offset not handled correctly
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on issue #1377: [HUDI-663] Fix HoodieDeltaStreamer offset not handled correctly
URL: https://github.com/apache/incubator-hudi/pull/1377#issuecomment-601303031
>https://github.com/apache/incubator-hudi/blob/master/hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java#L331
This is handling the case that the transformer filtered everything out, but we want to do an empty commit to record that the offsets moved.
The resets happening around Kafka, should be left alone IMO.. It can happen due to variety of reasons like topic retention kicking in etc..
For this PR, I am not sure if we want to special case an empty string.. ? Can we formulate the problem we are trying to solve again..
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] lamber-ken commented on issue #1377: [HUDI-663]
Fix HoodieDeltaStreamer offset not handled correctly
Posted by GitBox <gi...@apache.org>.
lamber-ken commented on issue #1377: [HUDI-663] Fix HoodieDeltaStreamer offset not handled correctly
URL: https://github.com/apache/incubator-hudi/pull/1377#issuecomment-596299950
Will come back, after #1373, it may needs more time to debuging.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] lamber-ken commented on a change in pull request
#1377: [HUDI-663] Fix HoodieDeltaStreamer offset not handled correctly
Posted by GitBox <gi...@apache.org>.
lamber-ken commented on a change in pull request #1377: [HUDI-663] Fix HoodieDeltaStreamer offset not handled correctly
URL: https://github.com/apache/incubator-hudi/pull/1377#discussion_r390695516
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -180,7 +180,7 @@ public KafkaOffsetGen(TypedProperties props) {
.map(x -> new TopicPartition(x.topic(), x.partition())).collect(Collectors.toSet());
// Determine the offset ranges to read from
- if (lastCheckpointStr.isPresent()) {
+ if (lastCheckpointStr.isPresent() && !lastCheckpointStr.get().isEmpty()) {
Review comment:
hi @bvaradar, add `!commitMetadata.getMetadata(CHECKPOINT_KEY).isEmpty()`.
but if we do that, the application will always throw `HoodieDeltaStreamerException`

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] codecov-io commented on issue #1377: [HUDI-663]
Fix HoodieDeltaStreamer offset not handled correctly
Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #1377: [HUDI-663] Fix HoodieDeltaStreamer offset not handled correctly
URL: https://github.com/apache/incubator-hudi/pull/1377#issuecomment-597426610
# [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1377?src=pr&el=h1) Report
> Merging [#1377](https://codecov.io/gh/apache/incubator-hudi/pull/1377?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-hudi/commit/5f85c267040fd51c186794fdae900162ab176b14?src=pr&el=desc) will **decrease** coverage by `66.32%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/incubator-hudi/pull/1377?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #1377 +/- ##
============================================
- Coverage 66.96% 0.64% -66.33%
+ Complexity 223 2 -221
============================================
Files 334 289 -45
Lines 16276 14375 -1901
Branches 1661 1467 -194
============================================
- Hits 10900 92 -10808
- Misses 4639 14280 +9641
+ Partials 737 3 -734
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-hudi/pull/1377?src=pr&el=tree) | Coverage Ξ | Complexity Ξ | |
|---|---|---|---|
| [...apache/hudi/common/model/HoodieDeltaWriteStat.java](https://codecov.io/gh/apache/incubator-hudi/pull/1377/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL0hvb2RpZURlbHRhV3JpdGVTdGF0LmphdmE=) | `0% <0%> (-100%)` | `0% <0%> (ΓΈ)` | |
| [...org/apache/hudi/common/model/HoodieFileFormat.java](https://codecov.io/gh/apache/incubator-hudi/pull/1377/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL0hvb2RpZUZpbGVGb3JtYXQuamF2YQ==) | `0% <0%> (-100%)` | `0% <0%> (ΓΈ)` | |
| [...g/apache/hudi/execution/BulkInsertMapFunction.java](https://codecov.io/gh/apache/incubator-hudi/pull/1377/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhlY3V0aW9uL0J1bGtJbnNlcnRNYXBGdW5jdGlvbi5qYXZh) | `0% <0%> (-100%)` | `0% <0%> (ΓΈ)` | |
| [.../common/util/queue/IteratorBasedQueueProducer.java](https://codecov.io/gh/apache/incubator-hudi/pull/1377/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3V0aWwvcXVldWUvSXRlcmF0b3JCYXNlZFF1ZXVlUHJvZHVjZXIuamF2YQ==) | `0% <0%> (-100%)` | `0% <0%> (ΓΈ)` | |
| [...rg/apache/hudi/index/bloom/KeyRangeLookupTree.java](https://codecov.io/gh/apache/incubator-hudi/pull/1377/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW5kZXgvYmxvb20vS2V5UmFuZ2VMb29rdXBUcmVlLmphdmE=) | `0% <0%> (-100%)` | `0% <0%> (ΓΈ)` | |
| [...e/hudi/common/table/timeline/dto/FileGroupDTO.java](https://codecov.io/gh/apache/incubator-hudi/pull/1377/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3RpbWVsaW5lL2R0by9GaWxlR3JvdXBEVE8uamF2YQ==) | `0% <0%> (-100%)` | `0% <0%> (ΓΈ)` | |
| [...apache/hudi/timeline/service/handlers/Handler.java](https://codecov.io/gh/apache/incubator-hudi/pull/1377/diff?src=pr&el=tree#diff-aHVkaS10aW1lbGluZS1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL3RpbWVsaW5lL3NlcnZpY2UvaGFuZGxlcnMvSGFuZGxlci5qYXZh) | `0% <0%> (-100%)` | `0% <0%> (ΓΈ)` | |
| [...che/hudi/common/table/timeline/dto/LogFileDTO.java](https://codecov.io/gh/apache/incubator-hudi/pull/1377/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3RpbWVsaW5lL2R0by9Mb2dGaWxlRFRPLmphdmE=) | `0% <0%> (-100%)` | `0% <0%> (ΓΈ)` | |
| [.../common/util/queue/FunctionBasedQueueProducer.java](https://codecov.io/gh/apache/incubator-hudi/pull/1377/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3V0aWwvcXVldWUvRnVuY3Rpb25CYXNlZFF1ZXVlUHJvZHVjZXIuamF2YQ==) | `0% <0%> (-100%)` | `0% <0%> (ΓΈ)` | |
| [...che/hudi/index/bloom/ListBasedIndexFileFilter.java](https://codecov.io/gh/apache/incubator-hudi/pull/1377/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW5kZXgvYmxvb20vTGlzdEJhc2VkSW5kZXhGaWxlRmlsdGVyLmphdmE=) | `0% <0%> (-100%)` | `0% <0%> (ΓΈ)` | |
| ... and [299 more](https://codecov.io/gh/apache/incubator-hudi/pull/1377/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1377?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Ξ = absolute <relative> (impact)`, `ΓΈ = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1377?src=pr&el=footer). Last update [5f85c26...0197788](https://codecov.io/gh/apache/incubator-hudi/pull/1377?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar edited a comment on issue #1377:
[HUDI-663] Fix HoodieDeltaStreamer offset not handled correctly
Posted by GitBox <gi...@apache.org>.
vinothchandar edited a comment on issue #1377: [HUDI-663] Fix HoodieDeltaStreamer offset not handled correctly
URL: https://github.com/apache/incubator-hudi/pull/1377#issuecomment-596357138
>>then start the delta streamer, hudi will store the empty checkpoint.
Re-reading this again.. Is this the right behavior? I think there are a few cases now handled in delta-streamer that has made life a bit complicated..
Reason for writing such empty checkpoint could be that - we want to write checkpoints even for empty commits, since it could have read data but the transformer could have filtered all of that out..
I think the right fix could be to checkpoint the actual fromOffsets instead of empty checkpoint..
>>the second commit will use the last checkpoint {}, which means the fromoffset is 0.
but the previous messages may be removed because of kafka retention mechanism.
And this is because we enter `checkupValidOffsets` right?
I'd appreciate it if we took into consideration how checkpoint is handled in a general source agnostic way and also fix this issue..
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on issue #1377: [HUDI-663]
Fix HoodieDeltaStreamer offset not handled correctly
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on issue #1377: [HUDI-663] Fix HoodieDeltaStreamer offset not handled correctly
URL: https://github.com/apache/incubator-hudi/pull/1377#issuecomment-596357138
>>then start the delta streamer, hudi will store the empty checkpoint.
Re-reading this again.. Is this the right behavior? I think there are a few cases now handled in delta-streamer that has made life a bit complicated..
Reason for writing such empty checkpoint could be that - we want to write checkpoints even for empty commits, since it could have read data but the transformer could have filtered all of that out..
I think the right fix could be to checkpoint the actual fromOffsets instead of empty checkpoint..
>>the second commit will use the last checkpoint {}, which means the fromoffset is 0.
but the previous messages may be removed because of kafka retention mechanism.
I'd appreciate it if we took into consideration how checkpoint is handled in a general source agnostic way and also fix this issue..
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] lamber-ken commented on a change in pull request
#1377: [HUDI-663] Fix HoodieDeltaStreamer offset not handled correctly
Posted by GitBox <gi...@apache.org>.
lamber-ken commented on a change in pull request #1377: [HUDI-663] Fix HoodieDeltaStreamer offset not handled correctly
URL: https://github.com/apache/incubator-hudi/pull/1377#discussion_r388930286
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -180,7 +180,7 @@ public KafkaOffsetGen(TypedProperties props) {
.map(x -> new TopicPartition(x.topic(), x.partition())).collect(Collectors.toSet());
// Determine the offset ranges to read from
- if (lastCheckpointStr.isPresent()) {
+ if (lastCheckpointStr.isPresent() && !lastCheckpointStr.get().isEmpty()) {
Review comment:
hi, if a hidden bug happens, the `*.commit` file will not be created, the next run will still read from the last successful commit, so no data will be lossed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] lamber-ken edited a comment on issue #1377:
[HUDI-663] Fix HoodieDeltaStreamer offset not handled correctly
Posted by GitBox <gi...@apache.org>.
lamber-ken edited a comment on issue #1377: [HUDI-663] Fix HoodieDeltaStreamer offset not handled correctly
URL: https://github.com/apache/incubator-hudi/pull/1377#issuecomment-597431353
Sorry, I revert it, hudi already handled the empty checkpoint by `KafkaOffsetGen.KafkaResetOffsetStrategies`
So, IMO, we

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] lamber-ken commented on issue #1377: [HUDI-663]
Fix HoodieDeltaStreamer offset not handled correctly
Posted by GitBox <gi...@apache.org>.
lamber-ken commented on issue #1377: [HUDI-663] Fix HoodieDeltaStreamer offset not handled correctly
URL: https://github.com/apache/incubator-hudi/pull/1377#issuecomment-600217114
> I am bit confused at this point for the PR.. Can you summarize where we are at?
hi @garyli1019 as we talked:
Both `cfg.checkpoint` and `deltastreamer.checkpoint.reset_key` are same thing.
User can use `auto.offset.reset` to control bebavior to datasource. do you have any other ideas?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] garyli1019 commented on issue #1377: [HUDI-663]
Fix HoodieDeltaStreamer offset not handled correctly
Posted by GitBox <gi...@apache.org>.
garyli1019 commented on issue #1377: [HUDI-663] Fix HoodieDeltaStreamer offset not handled correctly
URL: https://github.com/apache/incubator-hudi/pull/1377#issuecomment-600344505
discussed with @lamber-ken offline, summary:
- We found the empty checkpoint is actually not a bug. https://github.com/apache/incubator-hudi/blob/master/hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java#L331
We need to think about how to handle it. Should we handle the empty checkpoint or we set a special marker to identify this case? This question should be addressed in this PR.
- There are three places that will possibly reset the checkpoint
https://github.com/apache/incubator-hudi/blob/master/hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java#L260
https://github.com/apache/incubator-hudi/blob/master/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java#L186
https://github.com/apache/incubator-hudi/blob/master/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java#L222
We need somehow simplify this. This could be a separate task from this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] garyli1019 commented on a change in pull request
#1377: [HUDI-663] Fix HoodieDeltaStreamer offset not handled correctly
Posted by GitBox <gi...@apache.org>.
garyli1019 commented on a change in pull request #1377: [HUDI-663] Fix HoodieDeltaStreamer offset not handled correctly
URL: https://github.com/apache/incubator-hudi/pull/1377#discussion_r389089300
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -180,7 +180,7 @@ public KafkaOffsetGen(TypedProperties props) {
.map(x -> new TopicPartition(x.topic(), x.partition())).collect(Collectors.toSet());
// Determine the offset ranges to read from
- if (lastCheckpointStr.isPresent()) {
+ if (lastCheckpointStr.isPresent() && !lastCheckpointStr.get().isEmpty()) {
Review comment:
Right. As you mentioned it is still possible that some wrong user behaviors might lead to an empty checkpoint. From a user perspective, I'd say if there is an empty checkpoint in the last commit, I will prefer to let the job fail other than automatically reset the checkpoint. Throw an exception if the checkpoint is empty would make more sense to me and let the user decide whether they wanna reset or not. WDYT?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services