You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/06/27 05:15:54 UTC
[GitHub] [hudi] veenaypatil opened a new pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
veenaypatil opened a new pull request #3092:
URL: https://github.com/apache/hudi/pull/3092
## What is the purpose of the pull request
This change commits the offset to Kafka after a successful Hudi commit, this will help in monitoring the consumer lag
## Brief change log
- Added commitOffsetToKafka function in KafkaOffsetGen
- Added config `KafkaOffsetGen.Config.ENABLE_KAFKA_COMMIT_OFFSET` to be set in properties file
## Verify this pull request
This change added tests and can be verified as follows:
- Added `testCommitOffsetToKafka` test in TestKafkaSource
## Testing on Actual Environment
- Tested this change by building from this branch and deployed the job on YARN in continuous mode
- Added the following config in kafka-source.properties file for delta streamer
```
hoodie.deltastreamer.source.enable.kafka.commit.offset=true
```
- Pushed 9M records to Kafka topic, in this case offset will be committed in two different batches.
<img width="1674" alt="Screenshot 2021-06-20 at 9 48 25 PM" src="https://user-images.githubusercontent.com/52563354/122681486-23650780-d212-11eb-8687-c287b490893f.png">
## Committer checklist
- [X] Has a corresponding JIRA in PR title & commit
- [X] Commit message is descriptive of the change
- [X] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3092: [HUDI-1910] [WIP] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-862136673
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=219",
"triggerID" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"triggerType" : "PUSH"
}, {
"hash" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=290",
"triggerID" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cd25e8a35b0dba2de3479d3a5b58f60d7b758e02 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=219)
* 7dc7b1f9434ea3e602e2886e357bc467ab56aaf3 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=290)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-864517429
# [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3092](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f3cc39f) into [master](https://codecov.io/gh/apache/hudi/commit/f73bedd374952580c4d6bc9b2e30607c4efdd538?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f73bedd) will **decrease** coverage by `43.29%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3092 +/- ##
============================================
- Coverage 46.36% 3.07% -43.30%
+ Complexity 5390 82 -5308
============================================
Files 920 280 -640
Lines 39939 10875 -29064
Branches 4309 1106 -3203
============================================
- Hits 18517 334 -18183
+ Misses 19543 10515 -9028
+ Partials 1879 26 -1853
```
| Flag | Coverage Δ | |
|---|---|---|
| hudicli | `?` | |
| hudiclient | `0.00% <ø> (-30.46%)` | :arrow_down: |
| hudicommon | `?` | |
| hudiflink | `?` | |
| hudihadoopmr | `?` | |
| hudisparkdatasource | `?` | |
| hudisync | `5.38% <ø> (-48.67%)` | :arrow_down: |
| huditimelineservice | `?` | |
| hudiutilities | `9.46% <0.00%> (-49.63%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [.../hudi/utilities/callback/SourceCommitCallback.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NhbGxiYWNrL1NvdXJjZUNvbW1pdENhbGxiYWNrLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `0.00% <0.00%> (-71.29%)` | :arrow_down: |
| [...i/utilities/deltastreamer/SourceFormatAdapter.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvU291cmNlRm9ybWF0QWRhcHRlci5qYXZh) | `0.00% <0.00%> (-86.85%)` | :arrow_down: |
| [...apache/hudi/utilities/sources/AvroKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb0thZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...java/org/apache/hudi/utilities/sources/Source.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvU291cmNlLmphdmE=) | `0.00% <ø> (-90.00%)` | :arrow_down: |
| [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `0.00% <0.00%> (-90.06%)` | :arrow_down: |
| [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [765 more](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [f73bedd...f3cc39f](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] veenaypatil commented on a change in pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
veenaypatil commented on a change in pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#discussion_r655498636
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -297,7 +295,37 @@ public String getTopicName() {
return topicName;
}
- public HashMap<String, Object> getKafkaParams() {
+ public Map<String, Object> getKafkaParams() {
+ return kafkaParams;
+ }
+
+ private static Map<String, Object> excludeHoodieConfigs(TypedProperties props) {
+ Map<String, Object> kafkaParams = new HashMap<>();
+ props.keySet().stream().filter(prop -> {
+ // In order to prevent printing unnecessary warn logs, here filter out the hoodie
+ // configuration items before passing to kafkaParams
+ return !prop.toString().startsWith("hoodie.");
+ }).forEach(prop -> {
+ kafkaParams.put(prop.toString(), props.get(prop.toString()));
+ });
return kafkaParams;
}
+
+ /**
+ * Commit offsets to Kafka only after hoodie commit is successful.
+ * @param checkpointStr checkpoint string containing offsets.
+ * @param props properties for Kafka consumer.
+ */
+ public static void commitOffsetToKafka(String checkpointStr, TypedProperties props) {
+ DataSourceUtils.checkRequiredProperties(props, Collections.singletonList(ConsumerConfig.GROUP_ID_CONFIG));
+ Map<TopicPartition, Long> offsetMap = KafkaOffsetGen.CheckpointUtils.strToOffsets(checkpointStr);
+ Map<String, Object> kafkaParams = KafkaOffsetGen.excludeHoodieConfigs(props);
+ Map<TopicPartition, OffsetAndMetadata> offsetAndMetadataMap = new HashMap<>(offsetMap.size());
+ try (KafkaConsumer consumer = new KafkaConsumer(kafkaParams)) {
+ offsetMap.forEach((topicPartition, offset) -> offsetAndMetadataMap.put(topicPartition, new OffsetAndMetadata(offset)));
+ consumer.commitSync(offsetAndMetadataMap);
+ } catch (CommitFailedException | TimeoutException e) {
+ LOG.warn("Committing offsets to Kafka failed, this does not impact processing of records", e);
Review comment:
committing offset to Kafka is optional for the user, failure of single commit should not fail the complete job, hence logged this exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] veenaypatil commented on a change in pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
veenaypatil commented on a change in pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#discussion_r655512304
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -297,7 +295,37 @@ public String getTopicName() {
return topicName;
}
- public HashMap<String, Object> getKafkaParams() {
+ public Map<String, Object> getKafkaParams() {
+ return kafkaParams;
+ }
+
+ private static Map<String, Object> excludeHoodieConfigs(TypedProperties props) {
Review comment:
Are you proposing to create `KafkaOffsetGen` object in `DeltaSync` as well to call `commitOffsetToKafka` method ? I wanted to avoid doing it as we are already doing so in respective Source classes. Also, I saw `excludeHoodieConfigs ` as helper method, hence created static
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-862136673
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-864517429
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-862136673
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=219",
"triggerID" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"triggerType" : "PUSH"
}, {
"hash" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=290",
"triggerID" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"triggerType" : "PUSH"
}, {
"hash" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=311",
"triggerID" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"triggerType" : "PUSH"
}, {
"hash" : "5a2beb75f789afe8f5874e27b4d7b95427a1ea7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=315",
"triggerID" : "5a2beb75f789afe8f5874e27b4d7b95427a1ea7f",
"triggerType" : "PUSH"
}, {
"hash" : "2cb652554386030acaf569a73f9e074cbc72ffa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=324",
"triggerID" : "2cb652554386030acaf569a73f9e074cbc72ffa3",
"triggerType" : "PUSH"
}, {
"hash" : "1ae031303ebfa92fd153b8b6c0a404d1aac39a0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=393",
"triggerID" : "1ae031303ebfa92fd153b8b6c0a404d1aac39a0a",
"triggerType" : "PUSH"
}, {
"hash" : "e90cbcdc4a4926230d2440b486c52e39850cdb1a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=442",
"triggerID" : "e90cbcdc4a4926230d2440b486c52e39850cdb1a",
"triggerType" : "PUSH"
}, {
"hash" : "2d5c2588f9088daef2800600558ebaa7d6d05212",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=443",
"triggerID" : "2d5c2588f9088daef2800600558ebaa7d6d05212",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e90cbcdc4a4926230d2440b486c52e39850cdb1a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=442)
* 2d5c2588f9088daef2800600558ebaa7d6d05212 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=443)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-862136673
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=219",
"triggerID" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"triggerType" : "PUSH"
}, {
"hash" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=290",
"triggerID" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"triggerType" : "PUSH"
}, {
"hash" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=311",
"triggerID" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"triggerType" : "PUSH"
}, {
"hash" : "5a2beb75f789afe8f5874e27b4d7b95427a1ea7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=315",
"triggerID" : "5a2beb75f789afe8f5874e27b4d7b95427a1ea7f",
"triggerType" : "PUSH"
}, {
"hash" : "2cb652554386030acaf569a73f9e074cbc72ffa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=324",
"triggerID" : "2cb652554386030acaf569a73f9e074cbc72ffa3",
"triggerType" : "PUSH"
}, {
"hash" : "1ae031303ebfa92fd153b8b6c0a404d1aac39a0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=393",
"triggerID" : "1ae031303ebfa92fd153b8b6c0a404d1aac39a0a",
"triggerType" : "PUSH"
}, {
"hash" : "e90cbcdc4a4926230d2440b486c52e39850cdb1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=442",
"triggerID" : "e90cbcdc4a4926230d2440b486c52e39850cdb1a",
"triggerType" : "PUSH"
}, {
"hash" : "2d5c2588f9088daef2800600558ebaa7d6d05212",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=443",
"triggerID" : "2d5c2588f9088daef2800600558ebaa7d6d05212",
"triggerType" : "PUSH"
}, {
"hash" : "3b02a03a355c85723b69c9d8b4557c5983c8914f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=461",
"triggerID" : "3b02a03a355c85723b69c9d8b4557c5983c8914f",
"triggerType" : "PUSH"
}, {
"hash" : "3b02a03a355c85723b69c9d8b4557c5983c8914f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=461",
"triggerID" : "869102385",
"triggerType" : "MANUAL"
}, {
"hash" : "aa5414320f403ffa2224798e89cd7aa1752fdd12",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aa5414320f403ffa2224798e89cd7aa1752fdd12",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3b02a03a355c85723b69c9d8b4557c5983c8914f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=461)
* aa5414320f403ffa2224798e89cd7aa1752fdd12 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] sbernauer commented on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
sbernauer commented on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-870727246
@veenaypatil no problem ;)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] veenaypatil commented on a change in pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
veenaypatil commented on a change in pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#discussion_r659271355
##########
File path: hudi-examples/src/main/java/org/apache/hudi/examples/common/RandomJsonSource.java
##########
@@ -45,4 +45,8 @@ public RandomJsonSource(TypedProperties props, JavaSparkContext sparkContext, Sp
return new InputBatch<>(Option.of(sparkContext.parallelize(inserts, 1)), commitTime);
}
+
+ @Override
+ public void onCommit(String lastCkptStr) {
Review comment:
Yes, I thought about it first but Source being an abstract class dropped that idea, should we add a new interface like `SourceCallback` which will contain this default method ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-864517429
# [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3092](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (1ae0313) into [master](https://codecov.io/gh/apache/hudi/commit/e039e0ff6d04b9b0684dbf29452af8a8dae94b09?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e039e0f) will **decrease** coverage by `29.51%`.
> The diff coverage is `81.81%`.
[](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3092 +/- ##
=============================================
- Coverage 46.13% 16.62% -29.52%
+ Complexity 5325 471 -4854
=============================================
Files 915 274 -641
Lines 39622 10665 -28957
Branches 4272 1089 -3183
=============================================
- Hits 18281 1773 -16508
+ Misses 19470 8733 -10737
+ Partials 1871 159 -1712
```
| Flag | Coverage Δ | |
|---|---|---|
| hudicli | `?` | |
| hudiclient | `0.00% <ø> (-30.46%)` | :arrow_down: |
| hudicommon | `?` | |
| hudiflink | `?` | |
| hudihadoopmr | `?` | |
| hudisparkdatasource | `?` | |
| hudisync | `6.72% <ø> (-45.01%)` | :arrow_down: |
| huditimelineservice | `?` | |
| hudiutilities | `58.50% <81.81%> (+0.12%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `70.70% <0.00%> (-0.15%)` | :arrow_down: |
| [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `87.09% <90.00%> (-0.18%)` | :arrow_down: |
| [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...rg/apache/hudi/client/bootstrap/BootstrapMode.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9ib290c3RyYXAvQm9vdHN0cmFwTW9kZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...he/hudi/hive/HiveStylePartitionValueExtractor.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN0eWxlUGFydGl0aW9uVmFsdWVFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/hudi/keygen/constant/KeyGeneratorType.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2tleWdlbi9jb25zdGFudC9LZXlHZW5lcmF0b3JUeXBlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...pache/hudi/client/utils/ConcatenatingIterator.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC91dGlscy9Db25jYXRlbmF0aW5nSXRlcmF0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [706 more](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e039e0f...1ae0313](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] veenaypatil commented on a change in pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
veenaypatil commented on a change in pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#discussion_r658715036
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java
##########
@@ -473,7 +476,9 @@ public void refreshTimeline() throws IOException {
boolean success = writeClient.commit(instantTime, writeStatusRDD, Option.of(checkpointCommitMetadata));
if (success) {
LOG.info("Commit " + instantTime + " successful!");
-
+ if (this.props.getBoolean(ENABLE_KAFKA_COMMIT_OFFSET, DEFAULT_ENABLE_KAFKA_COMMIT_OFFSET)) {
Review comment:
@n3nash @vinothchandar added `onCommit` method to Source, let me know if this approach works
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-862136673
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=219",
"triggerID" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"triggerType" : "PUSH"
}, {
"hash" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=290",
"triggerID" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"triggerType" : "PUSH"
}, {
"hash" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=311",
"triggerID" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"triggerType" : "PUSH"
}, {
"hash" : "5a2beb75f789afe8f5874e27b4d7b95427a1ea7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=315",
"triggerID" : "5a2beb75f789afe8f5874e27b4d7b95427a1ea7f",
"triggerType" : "PUSH"
}, {
"hash" : "2cb652554386030acaf569a73f9e074cbc72ffa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=324",
"triggerID" : "2cb652554386030acaf569a73f9e074cbc72ffa3",
"triggerType" : "PUSH"
}, {
"hash" : "1ae031303ebfa92fd153b8b6c0a404d1aac39a0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=393",
"triggerID" : "1ae031303ebfa92fd153b8b6c0a404d1aac39a0a",
"triggerType" : "PUSH"
}, {
"hash" : "e90cbcdc4a4926230d2440b486c52e39850cdb1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=442",
"triggerID" : "e90cbcdc4a4926230d2440b486c52e39850cdb1a",
"triggerType" : "PUSH"
}, {
"hash" : "2d5c2588f9088daef2800600558ebaa7d6d05212",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=443",
"triggerID" : "2d5c2588f9088daef2800600558ebaa7d6d05212",
"triggerType" : "PUSH"
}, {
"hash" : "3b02a03a355c85723b69c9d8b4557c5983c8914f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=461",
"triggerID" : "3b02a03a355c85723b69c9d8b4557c5983c8914f",
"triggerType" : "PUSH"
}, {
"hash" : "3b02a03a355c85723b69c9d8b4557c5983c8914f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=461",
"triggerID" : "869102385",
"triggerType" : "MANUAL"
}, {
"hash" : "aa5414320f403ffa2224798e89cd7aa1752fdd12",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aa5414320f403ffa2224798e89cd7aa1752fdd12",
"triggerType" : "PUSH"
}, {
"hash" : "f3cc39fc2b35b1b935e24f120b14db515748a6b9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=491",
"triggerID" : "f3cc39fc2b35b1b935e24f120b14db515748a6b9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aa5414320f403ffa2224798e89cd7aa1752fdd12 UNKNOWN
* f3cc39fc2b35b1b935e24f120b14db515748a6b9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=491)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-864517429
# [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3092](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5a2beb7) into [master](https://codecov.io/gh/apache/hudi/commit/61efc6af79c389ef0a77cda75e4f562ed59ef86b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (61efc6a) will **decrease** coverage by `16.87%`.
> The diff coverage is `81.81%`.
[](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3092 +/- ##
=============================================
- Coverage 45.75% 28.88% -16.88%
+ Complexity 5262 1257 -4005
=============================================
Files 909 370 -539
Lines 39356 14112 -25244
Branches 4239 1441 -2798
=============================================
- Hits 18008 4076 -13932
+ Misses 19504 9740 -9764
+ Partials 1844 296 -1548
```
| Flag | Coverage Δ | |
|---|---|---|
| hudicli | `?` | |
| hudiclient | `22.27% <ø> (-8.12%)` | :arrow_down: |
| hudicommon | `?` | |
| hudiflink | `?` | |
| hudihadoopmr | `?` | |
| hudisparkdatasource | `?` | |
| hudisync | `6.72% <ø> (-44.74%)` | :arrow_down: |
| huditimelineservice | `?` | |
| hudiutilities | `58.46% <81.81%> (+1.82%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `70.37% <0.00%> (-0.48%)` | :arrow_down: |
| [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `87.09% <90.00%> (-0.18%)` | :arrow_down: |
| [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...rg/apache/hudi/client/bootstrap/BootstrapMode.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9ib290c3RyYXAvQm9vdHN0cmFwTW9kZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...he/hudi/hive/HiveStylePartitionValueExtractor.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN0eWxlUGFydGl0aW9uVmFsdWVFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/hudi/keygen/constant/KeyGeneratorType.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2tleWdlbi9jb25zdGFudC9LZXlHZW5lcmF0b3JUeXBlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...pache/hudi/client/utils/ConcatenatingIterator.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC91dGlscy9Db25jYXRlbmF0aW5nSXRlcmF0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [613 more](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [61efc6a...5a2beb7](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-864517429
# [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3092](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3b02a03) into [master](https://codecov.io/gh/apache/hudi/commit/f73bedd374952580c4d6bc9b2e30607c4efdd538?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f73bedd) will **increase** coverage by `14.78%`.
> The diff coverage is `75.00%`.
[](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3092 +/- ##
=============================================
+ Coverage 44.31% 59.09% +14.78%
+ Complexity 4574 1192 -3382
=============================================
Files 824 162 -662
Lines 36375 6398 -29977
Branches 3937 670 -3267
=============================================
- Hits 16120 3781 -12339
+ Misses 18511 2345 -16166
+ Partials 1744 272 -1472
```
| Flag | Coverage Δ | |
|---|---|---|
| hudicli | `?` | |
| hudiclient | `66.85% <ø> (+50.39%)` | :arrow_up: |
| hudicommon | `?` | |
| hudiflink | `?` | |
| hudihadoopmr | `?` | |
| hudisparkdatasource | `?` | |
| hudisync | `?` | |
| huditimelineservice | `?` | |
| hudiutilities | `50.01% <75.00%> (-8.28%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...g/apache/hudi/utilities/sources/AvroDFSSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb0RGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...apache/hudi/utilities/sources/AvroKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb0thZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...che/hudi/utilities/sources/HiveIncrPullSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSGl2ZUluY3JQdWxsU291cmNlLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/JsonDFSSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkRGU1NvdXJjZS5qYXZh) | `90.00% <0.00%> (-10.00%)` | :arrow_down: |
| [...java/org/apache/hudi/utilities/sources/Source.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvU291cmNlLmphdmE=) | `87.50% <ø> (ø)` | |
| [...a/org/apache/hudi/utilities/sources/SqlSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvU3FsU291cmNlLmphdmE=) | `61.11% <0.00%> (-3.60%)` | :arrow_down: |
| [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `87.09% <90.00%> (+0.61%)` | :arrow_up: |
| [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `71.28% <100.00%> (+0.09%)` | :arrow_up: |
| [...i/utilities/deltastreamer/SourceFormatAdapter.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvU291cmNlRm9ybWF0QWRhcHRlci5qYXZh) | `86.84% <100.00%> (+0.35%)` | :arrow_up: |
| [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQ3N2REZTU291cmNlLmphdmE=) | `100.00% <100.00%> (ø)` | |
| ... and [865 more](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [f73bedd...3b02a03](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#discussion_r655478803
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java
##########
@@ -473,7 +474,9 @@ public void refreshTimeline() throws IOException {
boolean success = writeClient.commit(instantTime, writeStatusRDD, Option.of(checkpointCommitMetadata));
if (success) {
LOG.info("Commit " + instantTime + " successful!");
-
+ if (this.props.getBoolean(KafkaOffsetGen.Config.ENABLE_KAFKA_COMMIT_OFFSET,KafkaOffsetGen.Config.DEFAULT_ENABLE_KAFKA_COMMIT_OFFSET)) {
Review comment:
code format `KafkaOffsetGen.Config.ENABLE_KAFKA_COMMIT_OFFSET, KafkaOffsetGen.Config.DEFAULT_ENABLE_KAFKA_COMMIT_OFFSET`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-864517429
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#discussion_r655485904
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -297,7 +295,37 @@ public String getTopicName() {
return topicName;
}
- public HashMap<String, Object> getKafkaParams() {
+ public Map<String, Object> getKafkaParams() {
+ return kafkaParams;
+ }
+
+ private static Map<String, Object> excludeHoodieConfigs(TypedProperties props) {
+ Map<String, Object> kafkaParams = new HashMap<>();
+ props.keySet().stream().filter(prop -> {
+ // In order to prevent printing unnecessary warn logs, here filter out the hoodie
+ // configuration items before passing to kafkaParams
+ return !prop.toString().startsWith("hoodie.");
+ }).forEach(prop -> {
+ kafkaParams.put(prop.toString(), props.get(prop.toString()));
+ });
return kafkaParams;
}
+
+ /**
+ * Commit offsets to Kafka only after hoodie commit is successful.
+ * @param checkpointStr checkpoint string containing offsets.
+ * @param props properties for Kafka consumer.
+ */
+ public static void commitOffsetToKafka(String checkpointStr, TypedProperties props) {
+ DataSourceUtils.checkRequiredProperties(props, Collections.singletonList(ConsumerConfig.GROUP_ID_CONFIG));
+ Map<TopicPartition, Long> offsetMap = KafkaOffsetGen.CheckpointUtils.strToOffsets(checkpointStr);
+ Map<String, Object> kafkaParams = KafkaOffsetGen.excludeHoodieConfigs(props);
+ Map<TopicPartition, OffsetAndMetadata> offsetAndMetadataMap = new HashMap<>(offsetMap.size());
+ try (KafkaConsumer consumer = new KafkaConsumer(kafkaParams)) {
+ offsetMap.forEach((topicPartition, offset) -> offsetAndMetadataMap.put(topicPartition, new OffsetAndMetadata(offset)));
+ consumer.commitSync(offsetAndMetadataMap);
+ } catch (CommitFailedException | TimeoutException e) {
+ LOG.warn("Committing offsets to Kafka failed, this does not impact processing of records", e);
Review comment:
should we throw exception here instead of just logging ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3092: [HUDI-1910] [WIP] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-864517429
# [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3092](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7dc7b1f) into [master](https://codecov.io/gh/apache/hudi/commit/61efc6af79c389ef0a77cda75e4f562ed59ef86b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (61efc6a) will **decrease** coverage by `29.14%`.
> The diff coverage is `89.47%`.
[](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3092 +/- ##
=============================================
- Coverage 45.75% 16.60% -29.15%
+ Complexity 5262 470 -4792
=============================================
Files 909 274 -635
Lines 39356 10663 -28693
Branches 4239 1089 -3150
=============================================
- Hits 18008 1771 -16237
+ Misses 19504 8732 -10772
+ Partials 1844 160 -1684
```
| Flag | Coverage Δ | |
|---|---|---|
| hudicli | `?` | |
| hudiclient | `0.00% <ø> (-30.40%)` | :arrow_down: |
| hudicommon | `?` | |
| hudiflink | `?` | |
| hudihadoopmr | `?` | |
| hudisparkdatasource | `?` | |
| hudisync | `6.72% <ø> (-44.74%)` | :arrow_down: |
| huditimelineservice | `?` | |
| hudiutilities | `58.49% <89.47%> (+1.85%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `70.37% <0.00%> (-0.48%)` | :arrow_down: |
| [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `88.42% <100.00%> (+1.15%)` | :arrow_up: |
| [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...rg/apache/hudi/client/bootstrap/BootstrapMode.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9ib290c3RyYXAvQm9vdHN0cmFwTW9kZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...he/hudi/hive/HiveStylePartitionValueExtractor.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN0eWxlUGFydGl0aW9uVmFsdWVFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/hudi/keygen/constant/KeyGeneratorType.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2tleWdlbi9jb25zdGFudC9LZXlHZW5lcmF0b3JUeXBlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...pache/hudi/client/utils/ConcatenatingIterator.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC91dGlscy9Db25jYXRlbmF0aW5nSXRlcmF0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [709 more](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [61efc6a...7dc7b1f](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on a change in pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#discussion_r655327890
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java
##########
@@ -506,6 +509,8 @@ public void refreshTimeline() throws IOException {
return Pair.of(scheduledCompactionInstant, writeStatusRDD);
}
+
+
Review comment:
unnecessary empty lines
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] veenaypatil commented on a change in pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
veenaypatil commented on a change in pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#discussion_r655526600
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java
##########
@@ -473,7 +474,9 @@ public void refreshTimeline() throws IOException {
boolean success = writeClient.commit(instantTime, writeStatusRDD, Option.of(checkpointCommitMetadata));
if (success) {
LOG.info("Commit " + instantTime + " successful!");
-
+ if (this.props.getBoolean(KafkaOffsetGen.Config.ENABLE_KAFKA_COMMIT_OFFSET,KafkaOffsetGen.Config.DEFAULT_ENABLE_KAFKA_COMMIT_OFFSET)) {
+ KafkaOffsetGen.commitOffsetToKafka(checkpointStr, this.props);
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-864517429
# [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3092](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (4cf4e2e) into [master](https://codecov.io/gh/apache/hudi/commit/61efc6af79c389ef0a77cda75e4f562ed59ef86b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (61efc6a) will **decrease** coverage by `42.62%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3092 +/- ##
============================================
- Coverage 45.75% 3.13% -42.63%
+ Complexity 5262 82 -5180
============================================
Files 909 274 -635
Lines 39356 10666 -28690
Branches 4239 1089 -3150
============================================
- Hits 18008 334 -17674
+ Misses 19504 10306 -9198
+ Partials 1844 26 -1818
```
| Flag | Coverage Δ | |
|---|---|---|
| hudicli | `?` | |
| hudiclient | `0.00% <ø> (-30.40%)` | :arrow_down: |
| hudicommon | `?` | |
| hudiflink | `?` | |
| hudihadoopmr | `?` | |
| hudisparkdatasource | `?` | |
| hudisync | `6.72% <ø> (-44.74%)` | :arrow_down: |
| huditimelineservice | `?` | |
| hudiutilities | `9.47% <0.00%> (-47.17%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `0.00% <0.00%> (-70.85%)` | :arrow_down: |
| [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `0.00% <0.00%> (-87.28%)` | :arrow_down: |
| [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQ3N2REZTU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...g/apache/hudi/utilities/sources/JsonDFSSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkRGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [750 more](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [61efc6a...4cf4e2e](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] veenaypatil commented on a change in pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
veenaypatil commented on a change in pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#discussion_r655334095
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java
##########
@@ -506,6 +509,8 @@ public void refreshTimeline() throws IOException {
return Pair.of(scheduledCompactionInstant, writeStatusRDD);
}
+
+
Review comment:
removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] sbernauer commented on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
sbernauer commented on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-870673044
Hi together, this PR broke/overwrote the new change introduced in https://github.com/apache/hudi/pull/3111
```
return !prop.toString().startsWith("hoodie.")
// We need to pass some properties to kafka client so that KafkaAvroSchemaDeserializer can use it
|| prop.toString().startsWith("hoodie.deltastreamer.source.kafka.value.deserializer.");
```
We need to pass the `hoodie.deltastreamer.source.kafka.value.deserializer.*` configs to the kafka client.
Happily i noticed it ;) I opened https://github.com/apache/hudi/pull/3172 that will revert your change to include the needed properties again, so no further action needed besides looking at my PR ;)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#discussion_r659269069
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -297,7 +295,37 @@ public String getTopicName() {
return topicName;
}
- public HashMap<String, Object> getKafkaParams() {
+ public Map<String, Object> getKafkaParams() {
+ return kafkaParams;
+ }
+
+ private static Map<String, Object> excludeHoodieConfigs(TypedProperties props) {
+ Map<String, Object> kafkaParams = new HashMap<>();
+ props.keySet().stream().filter(prop -> {
+ // In order to prevent printing unnecessary warn logs, here filter out the hoodie
+ // configuration items before passing to kafkaParams
+ return !prop.toString().startsWith("hoodie.");
+ }).forEach(prop -> {
+ kafkaParams.put(prop.toString(), props.get(prop.toString()));
+ });
return kafkaParams;
}
+
+ /**
+ * Commit offsets to Kafka only after hoodie commit is successful.
+ * @param checkpointStr checkpoint string containing offsets.
+ * @param props properties for Kafka consumer.
+ */
+ public static void commitOffsetToKafka(String checkpointStr, TypedProperties props) {
+ DataSourceUtils.checkRequiredProperties(props, Collections.singletonList(ConsumerConfig.GROUP_ID_CONFIG));
+ Map<TopicPartition, Long> offsetMap = KafkaOffsetGen.CheckpointUtils.strToOffsets(checkpointStr);
+ Map<String, Object> kafkaParams = KafkaOffsetGen.excludeHoodieConfigs(props);
+ Map<TopicPartition, OffsetAndMetadata> offsetAndMetadataMap = new HashMap<>(offsetMap.size());
+ try (KafkaConsumer consumer = new KafkaConsumer(kafkaParams)) {
+ offsetMap.forEach((topicPartition, offset) -> offsetAndMetadataMap.put(topicPartition, new OffsetAndMetadata(offset)));
+ consumer.commitSync(offsetAndMetadataMap);
+ } catch (CommitFailedException | TimeoutException e) {
+ LOG.warn("Committing offsets to Kafka failed, this does not impact processing of records", e);
Review comment:
after restarting will data duplicated?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-862136673
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=219",
"triggerID" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"triggerType" : "PUSH"
}, {
"hash" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=290",
"triggerID" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"triggerType" : "PUSH"
}, {
"hash" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=311",
"triggerID" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"triggerType" : "PUSH"
}, {
"hash" : "5a2beb75f789afe8f5874e27b4d7b95427a1ea7f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=315",
"triggerID" : "5a2beb75f789afe8f5874e27b4d7b95427a1ea7f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5a2beb75f789afe8f5874e27b4d7b95427a1ea7f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=315)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf merged pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
leesf merged pull request #3092:
URL: https://github.com/apache/hudi/pull/3092
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-862136673
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=219",
"triggerID" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"triggerType" : "PUSH"
}, {
"hash" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=290",
"triggerID" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"triggerType" : "PUSH"
}, {
"hash" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=311",
"triggerID" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"triggerType" : "PUSH"
}, {
"hash" : "5a2beb75f789afe8f5874e27b4d7b95427a1ea7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=315",
"triggerID" : "5a2beb75f789afe8f5874e27b4d7b95427a1ea7f",
"triggerType" : "PUSH"
}, {
"hash" : "2cb652554386030acaf569a73f9e074cbc72ffa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=324",
"triggerID" : "2cb652554386030acaf569a73f9e074cbc72ffa3",
"triggerType" : "PUSH"
}, {
"hash" : "1ae031303ebfa92fd153b8b6c0a404d1aac39a0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=393",
"triggerID" : "1ae031303ebfa92fd153b8b6c0a404d1aac39a0a",
"triggerType" : "PUSH"
}, {
"hash" : "e90cbcdc4a4926230d2440b486c52e39850cdb1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=442",
"triggerID" : "e90cbcdc4a4926230d2440b486c52e39850cdb1a",
"triggerType" : "PUSH"
}, {
"hash" : "2d5c2588f9088daef2800600558ebaa7d6d05212",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=443",
"triggerID" : "2d5c2588f9088daef2800600558ebaa7d6d05212",
"triggerType" : "PUSH"
}, {
"hash" : "3b02a03a355c85723b69c9d8b4557c5983c8914f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b02a03a355c85723b69c9d8b4557c5983c8914f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2d5c2588f9088daef2800600558ebaa7d6d05212 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=443)
* 3b02a03a355c85723b69c9d8b4557c5983c8914f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3092: [HUDI-1910] [WIP] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-862136673
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=219",
"triggerID" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cd25e8a35b0dba2de3479d3a5b58f60d7b758e02 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=219)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-862136673
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=219",
"triggerID" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"triggerType" : "PUSH"
}, {
"hash" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=290",
"triggerID" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"triggerType" : "PUSH"
}, {
"hash" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=311",
"triggerID" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"triggerType" : "PUSH"
}, {
"hash" : "5a2beb75f789afe8f5874e27b4d7b95427a1ea7f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=315",
"triggerID" : "5a2beb75f789afe8f5874e27b4d7b95427a1ea7f",
"triggerType" : "PUSH"
}, {
"hash" : "2cb652554386030acaf569a73f9e074cbc72ffa3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2cb652554386030acaf569a73f9e074cbc72ffa3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5a2beb75f789afe8f5874e27b4d7b95427a1ea7f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=315)
* 2cb652554386030acaf569a73f9e074cbc72ffa3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#discussion_r659272256
##########
File path: hudi-examples/src/main/java/org/apache/hudi/examples/common/RandomJsonSource.java
##########
@@ -45,4 +45,8 @@ public RandomJsonSource(TypedProperties props, JavaSparkContext sparkContext, Sp
return new InputBatch<>(Option.of(sparkContext.parallelize(inserts, 1)), commitTime);
}
+
+ @Override
+ public void onCommit(String lastCkptStr) {
Review comment:
add a new interface sounds good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3092: [HUDI-1910] [WIP] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-864517429
# [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3092](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7dc7b1f) into [master](https://codecov.io/gh/apache/hudi/commit/61efc6af79c389ef0a77cda75e4f562ed59ef86b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (61efc6a) will **decrease** coverage by `1.71%`.
> The diff coverage is `89.47%`.
[](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3092 +/- ##
============================================
- Coverage 45.75% 44.03% -1.72%
+ Complexity 5262 4522 -740
============================================
Files 909 815 -94
Lines 39356 36043 -3313
Branches 4239 3903 -336
============================================
- Hits 18008 15873 -2135
+ Misses 19504 18451 -1053
+ Partials 1844 1719 -125
```
| Flag | Coverage Δ | |
|---|---|---|
| hudicli | `39.95% <ø> (ø)` | |
| hudiclient | `16.44% <ø> (-13.95%)` | :arrow_down: |
| hudicommon | `47.56% <ø> (+<0.01%)` | :arrow_up: |
| hudiflink | `61.33% <ø> (+0.59%)` | :arrow_up: |
| hudihadoopmr | `51.29% <ø> (ø)` | |
| hudisparkdatasource | `66.52% <ø> (+0.04%)` | :arrow_up: |
| hudisync | `51.73% <ø> (+0.27%)` | :arrow_up: |
| huditimelineservice | `64.36% <ø> (ø)` | |
| hudiutilities | `58.49% <89.47%> (+1.85%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `70.37% <0.00%> (-0.48%)` | :arrow_down: |
| [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `88.42% <100.00%> (+1.15%)` | :arrow_up: |
| [.../org/apache/hudi/sink/compact/CompactFunction.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL2NvbXBhY3QvQ29tcGFjdEZ1bmN0aW9uLmphdmE=) | `86.66% <0.00%> (-13.34%)` | :arrow_down: |
| [...e/hudi/sink/partitioner/profile/WriteProfiles.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL3BhcnRpdGlvbmVyL3Byb2ZpbGUvV3JpdGVQcm9maWxlcy5qYXZh) | `52.50% <0.00%> (-3.39%)` | :arrow_down: |
| [...c/main/java/org/apache/hudi/util/StreamerUtil.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS91dGlsL1N0cmVhbWVyVXRpbC5qYXZh) | `55.46% <0.00%> (-0.95%)` | :arrow_down: |
| [...he/hudi/sink/partitioner/profile/WriteProfile.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL3BhcnRpdGlvbmVyL3Byb2ZpbGUvV3JpdGVQcm9maWxlLmphdmE=) | `87.50% <0.00%> (-0.50%)` | :arrow_down: |
| [...java/org/apache/hudi/sink/StreamWriteFunction.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL1N0cmVhbVdyaXRlRnVuY3Rpb24uamF2YQ==) | `84.84% <0.00%> (-0.16%)` | :arrow_down: |
| [...va/org/apache/hudi/configuration/FlinkOptions.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9jb25maWd1cmF0aW9uL0ZsaW5rT3B0aW9ucy5qYXZh) | `96.37% <0.00%> (-0.05%)` | :arrow_down: |
| [...main/java/org/apache/hudi/util/CompactionUtil.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS91dGlsL0NvbXBhY3Rpb25VdGlsLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...in/java/org/apache/hudi/table/HoodieTableSink.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS90YWJsZS9Ib29kaWVUYWJsZVNpbmsuamF2YQ==) | `10.52% <0.00%> (ø)` | |
| ... and [119 more](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [61efc6a...7dc7b1f](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-862136673
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=219",
"triggerID" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"triggerType" : "PUSH"
}, {
"hash" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=290",
"triggerID" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"triggerType" : "PUSH"
}, {
"hash" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=311",
"triggerID" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"triggerType" : "PUSH"
}, {
"hash" : "5a2beb75f789afe8f5874e27b4d7b95427a1ea7f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=315",
"triggerID" : "5a2beb75f789afe8f5874e27b4d7b95427a1ea7f",
"triggerType" : "PUSH"
}, {
"hash" : "2cb652554386030acaf569a73f9e074cbc72ffa3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=324",
"triggerID" : "2cb652554386030acaf569a73f9e074cbc72ffa3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5a2beb75f789afe8f5874e27b4d7b95427a1ea7f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=315)
* 2cb652554386030acaf569a73f9e074cbc72ffa3 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=324)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-864517429
# [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3092](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (4cf4e2e) into [master](https://codecov.io/gh/apache/hudi/commit/61efc6af79c389ef0a77cda75e4f562ed59ef86b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (61efc6a) will **increase** coverage by `0.28%`.
> The diff coverage is `81.81%`.
[](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3092 +/- ##
============================================
+ Coverage 45.75% 46.03% +0.28%
- Complexity 5262 5311 +49
============================================
Files 909 911 +2
Lines 39356 39492 +136
Branches 4239 4255 +16
============================================
+ Hits 18008 18181 +173
+ Misses 19504 19457 -47
- Partials 1844 1854 +10
```
| Flag | Coverage Δ | |
|---|---|---|
| hudicli | `39.95% <ø> (ø)` | |
| hudiclient | `30.44% <ø> (+0.05%)` | :arrow_up: |
| hudicommon | `47.58% <ø> (+0.01%)` | :arrow_up: |
| hudiflink | `61.33% <ø> (+0.59%)` | :arrow_up: |
| hudihadoopmr | `51.29% <ø> (ø)` | |
| hudisparkdatasource | `66.52% <ø> (+0.04%)` | :arrow_up: |
| hudisync | `51.73% <ø> (+0.27%)` | :arrow_up: |
| huditimelineservice | `64.36% <ø> (ø)` | |
| hudiutilities | `58.50% <81.81%> (+1.86%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `70.70% <0.00%> (-0.15%)` | :arrow_down: |
| [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `87.09% <90.00%> (-0.18%)` | :arrow_down: |
| [.../org/apache/hudi/sink/compact/CompactFunction.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL2NvbXBhY3QvQ29tcGFjdEZ1bmN0aW9uLmphdmE=) | `86.66% <0.00%> (-13.34%)` | :arrow_down: |
| [...e/hudi/sink/partitioner/profile/WriteProfiles.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL3BhcnRpdGlvbmVyL3Byb2ZpbGUvV3JpdGVQcm9maWxlcy5qYXZh) | `52.50% <0.00%> (-3.39%)` | :arrow_down: |
| [...c/main/java/org/apache/hudi/util/StreamerUtil.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS91dGlsL1N0cmVhbWVyVXRpbC5qYXZh) | `55.46% <0.00%> (-0.95%)` | :arrow_down: |
| [...he/hudi/sink/partitioner/profile/WriteProfile.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL3BhcnRpdGlvbmVyL3Byb2ZpbGUvV3JpdGVQcm9maWxlLmphdmE=) | `87.50% <0.00%> (-0.50%)` | :arrow_down: |
| [...java/org/apache/hudi/sink/StreamWriteFunction.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL1N0cmVhbVdyaXRlRnVuY3Rpb24uamF2YQ==) | `84.84% <0.00%> (-0.16%)` | :arrow_down: |
| [...va/org/apache/hudi/configuration/FlinkOptions.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9jb25maWd1cmF0aW9uL0ZsaW5rT3B0aW9ucy5qYXZh) | `96.37% <0.00%> (-0.05%)` | :arrow_down: |
| [...main/java/org/apache/hudi/util/CompactionUtil.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS91dGlsL0NvbXBhY3Rpb25VdGlsLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...in/java/org/apache/hudi/table/HoodieTableSink.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS90YWJsZS9Ib29kaWVUYWJsZVNpbmsuamF2YQ==) | `10.52% <0.00%> (ø)` | |
| ... and [25 more](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [61efc6a...4cf4e2e](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3092: [HUDI-1910] [WIP] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-864517429
# [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3092](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7dc7b1f) into [master](https://codecov.io/gh/apache/hudi/commit/61efc6af79c389ef0a77cda75e4f562ed59ef86b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (61efc6a) will **increase** coverage by `0.27%`.
> The diff coverage is `89.47%`.
[](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3092 +/- ##
============================================
+ Coverage 45.75% 46.03% +0.27%
- Complexity 5262 5309 +47
============================================
Files 909 911 +2
Lines 39356 39489 +133
Branches 4239 4255 +16
============================================
+ Hits 18008 18177 +169
+ Misses 19504 19457 -47
- Partials 1844 1855 +11
```
| Flag | Coverage Δ | |
|---|---|---|
| hudicli | `39.95% <ø> (ø)` | |
| hudiclient | `30.44% <ø> (+0.05%)` | :arrow_up: |
| hudicommon | `47.56% <ø> (+<0.01%)` | :arrow_up: |
| hudiflink | `61.33% <ø> (+0.59%)` | :arrow_up: |
| hudihadoopmr | `51.29% <ø> (ø)` | |
| hudisparkdatasource | `66.52% <ø> (+0.04%)` | :arrow_up: |
| hudisync | `51.73% <ø> (+0.27%)` | :arrow_up: |
| huditimelineservice | `64.36% <ø> (ø)` | |
| hudiutilities | `58.49% <89.47%> (+1.85%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `70.37% <0.00%> (-0.48%)` | :arrow_down: |
| [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `88.42% <100.00%> (+1.15%)` | :arrow_up: |
| [.../org/apache/hudi/sink/compact/CompactFunction.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL2NvbXBhY3QvQ29tcGFjdEZ1bmN0aW9uLmphdmE=) | `86.66% <0.00%> (-13.34%)` | :arrow_down: |
| [...e/hudi/sink/partitioner/profile/WriteProfiles.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL3BhcnRpdGlvbmVyL3Byb2ZpbGUvV3JpdGVQcm9maWxlcy5qYXZh) | `52.50% <0.00%> (-3.39%)` | :arrow_down: |
| [...c/main/java/org/apache/hudi/util/StreamerUtil.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS91dGlsL1N0cmVhbWVyVXRpbC5qYXZh) | `55.46% <0.00%> (-0.95%)` | :arrow_down: |
| [...he/hudi/sink/partitioner/profile/WriteProfile.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL3BhcnRpdGlvbmVyL3Byb2ZpbGUvV3JpdGVQcm9maWxlLmphdmE=) | `87.50% <0.00%> (-0.50%)` | :arrow_down: |
| [...java/org/apache/hudi/sink/StreamWriteFunction.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL1N0cmVhbVdyaXRlRnVuY3Rpb24uamF2YQ==) | `84.84% <0.00%> (-0.16%)` | :arrow_down: |
| [...va/org/apache/hudi/configuration/FlinkOptions.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9jb25maWd1cmF0aW9uL0ZsaW5rT3B0aW9ucy5qYXZh) | `96.37% <0.00%> (-0.05%)` | :arrow_down: |
| [...main/java/org/apache/hudi/util/CompactionUtil.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS91dGlsL0NvbXBhY3Rpb25VdGlsLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...in/java/org/apache/hudi/table/HoodieTableSink.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS90YWJsZS9Ib29kaWVUYWJsZVNpbmsuamF2YQ==) | `10.52% <0.00%> (ø)` | |
| ... and [23 more](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [61efc6a...7dc7b1f](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] veenaypatil commented on a change in pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
veenaypatil commented on a change in pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#discussion_r655525696
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java
##########
@@ -473,7 +474,9 @@ public void refreshTimeline() throws IOException {
boolean success = writeClient.commit(instantTime, writeStatusRDD, Option.of(checkpointCommitMetadata));
if (success) {
LOG.info("Commit " + instantTime + " successful!");
-
+ if (this.props.getBoolean(KafkaOffsetGen.Config.ENABLE_KAFKA_COMMIT_OFFSET,KafkaOffsetGen.Config.DEFAULT_ENABLE_KAFKA_COMMIT_OFFSET)) {
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] veenaypatil commented on a change in pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
veenaypatil commented on a change in pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#discussion_r658703745
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -297,7 +295,37 @@ public String getTopicName() {
return topicName;
}
- public HashMap<String, Object> getKafkaParams() {
+ public Map<String, Object> getKafkaParams() {
+ return kafkaParams;
+ }
+
+ private static Map<String, Object> excludeHoodieConfigs(TypedProperties props) {
Review comment:
@leesf done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] veenaypatil commented on a change in pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
veenaypatil commented on a change in pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#discussion_r659277370
##########
File path: hudi-examples/src/main/java/org/apache/hudi/examples/common/RandomJsonSource.java
##########
@@ -45,4 +45,8 @@ public RandomJsonSource(TypedProperties props, JavaSparkContext sparkContext, Sp
return new InputBatch<>(Option.of(sparkContext.parallelize(inserts, 1)), commitTime);
}
+
+ @Override
+ public void onCommit(String lastCkptStr) {
Review comment:
done, as discussed added `SourceCommitCallback` interface with default method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] veenaypatil commented on a change in pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
veenaypatil commented on a change in pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#discussion_r656759751
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java
##########
@@ -473,7 +476,9 @@ public void refreshTimeline() throws IOException {
boolean success = writeClient.commit(instantTime, writeStatusRDD, Option.of(checkpointCommitMetadata));
if (success) {
LOG.info("Commit " + instantTime + " successful!");
-
+ if (this.props.getBoolean(ENABLE_KAFKA_COMMIT_OFFSET, DEFAULT_ENABLE_KAFKA_COMMIT_OFFSET)) {
Review comment:
Agree, @n3nash and me discussed on various approaches here https://issues.apache.org/jira/browse/HUDI-1910 and picked this one , hence did not add the config to DeltaSync.
Should we implement callback approach instead ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-864517429
# [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3092](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (aa54143) into [master](https://codecov.io/gh/apache/hudi/commit/f73bedd374952580c4d6bc9b2e30607c4efdd538?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f73bedd) will **decrease** coverage by `17.89%`.
> The diff coverage is `82.75%`.
[](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3092 +/- ##
=============================================
- Coverage 46.36% 28.47% -17.90%
+ Complexity 5390 1261 -4129
=============================================
Files 920 376 -544
Lines 39939 14326 -25613
Branches 4309 1458 -2851
=============================================
- Hits 18517 4079 -14438
+ Misses 19543 9951 -9592
+ Partials 1879 296 -1583
```
| Flag | Coverage Δ | |
|---|---|---|
| hudicli | `?` | |
| hudiclient | `22.29% <ø> (-8.17%)` | :arrow_down: |
| hudicommon | `?` | |
| hudiflink | `?` | |
| hudihadoopmr | `?` | |
| hudisparkdatasource | `?` | |
| hudisync | `5.38% <ø> (-48.67%)` | :arrow_down: |
| huditimelineservice | `?` | |
| hudiutilities | `58.40% <82.75%> (-0.69%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...apache/hudi/utilities/sources/AvroKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb0thZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...java/org/apache/hudi/utilities/sources/Source.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvU291cmNlLmphdmE=) | `87.50% <ø> (-2.50%)` | :arrow_down: |
| [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `87.09% <90.00%> (-2.96%)` | :arrow_down: |
| [.../hudi/utilities/callback/SourceCommitCallback.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NhbGxiYWNrL1NvdXJjZUNvbW1pdENhbGxiYWNrLmphdmE=) | `100.00% <100.00%> (ø)` | |
| [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `70.94% <100.00%> (-0.34%)` | :arrow_down: |
| [...i/utilities/deltastreamer/SourceFormatAdapter.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvU291cmNlRm9ybWF0QWRhcHRlci5qYXZh) | `86.84% <100.00%> (ø)` | |
| [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `100.00% <100.00%> (ø)` | |
| [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [634 more](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [f73bedd...aa54143](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-864517429
# [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3092](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5a2beb7) into [master](https://codecov.io/gh/apache/hudi/commit/61efc6af79c389ef0a77cda75e4f562ed59ef86b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (61efc6a) will **increase** coverage by `0.04%`.
> The diff coverage is `81.81%`.
[](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3092 +/- ##
============================================
+ Coverage 45.75% 45.80% +0.04%
+ Complexity 5262 5247 -15
============================================
Files 909 905 -4
Lines 39356 39055 -301
Branches 4239 4232 -7
============================================
- Hits 18008 17888 -120
+ Misses 19504 19330 -174
+ Partials 1844 1837 -7
```
| Flag | Coverage Δ | |
|---|---|---|
| hudicli | `39.95% <ø> (ø)` | |
| hudiclient | `30.44% <ø> (+0.05%)` | :arrow_up: |
| hudicommon | `47.56% <ø> (+<0.01%)` | :arrow_up: |
| hudiflink | `61.12% <ø> (+0.38%)` | :arrow_up: |
| hudihadoopmr | `51.29% <ø> (ø)` | |
| hudisparkdatasource | `66.52% <ø> (+0.04%)` | :arrow_up: |
| hudisync | `51.73% <ø> (+0.27%)` | :arrow_up: |
| huditimelineservice | `?` | |
| hudiutilities | `58.46% <81.81%> (+1.82%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `70.37% <0.00%> (-0.48%)` | :arrow_down: |
| [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `87.09% <90.00%> (-0.18%)` | :arrow_down: |
| [.../org/apache/hudi/sink/compact/CompactFunction.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL2NvbXBhY3QvQ29tcGFjdEZ1bmN0aW9uLmphdmE=) | `86.66% <0.00%> (-13.34%)` | :arrow_down: |
| [...e/hudi/sink/partitioner/profile/WriteProfiles.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL3BhcnRpdGlvbmVyL3Byb2ZpbGUvV3JpdGVQcm9maWxlcy5qYXZh) | `52.50% <0.00%> (-3.39%)` | :arrow_down: |
| [...c/main/java/org/apache/hudi/util/StreamerUtil.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS91dGlsL1N0cmVhbWVyVXRpbC5qYXZh) | `55.00% <0.00%> (-1.42%)` | :arrow_down: |
| [...java/org/apache/hudi/sink/StreamWriteFunction.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL1N0cmVhbVdyaXRlRnVuY3Rpb24uamF2YQ==) | `84.34% <0.00%> (-0.66%)` | :arrow_down: |
| [...he/hudi/sink/partitioner/profile/WriteProfile.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL3BhcnRpdGlvbmVyL3Byb2ZpbGUvV3JpdGVQcm9maWxlLmphdmE=) | `87.50% <0.00%> (-0.50%)` | :arrow_down: |
| [...va/org/apache/hudi/configuration/FlinkOptions.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9jb25maWd1cmF0aW9uL0ZsaW5rT3B0aW9ucy5qYXZh) | `96.37% <0.00%> (-0.05%)` | :arrow_down: |
| [...main/java/org/apache/hudi/util/CompactionUtil.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS91dGlsL0NvbXBhY3Rpb25VdGlsLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...in/java/org/apache/hudi/table/HoodieTableSink.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS90YWJsZS9Ib29kaWVUYWJsZVNpbmsuamF2YQ==) | `10.52% <0.00%> (ø)` | |
| ... and [29 more](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [61efc6a...5a2beb7](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#discussion_r655485179
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java
##########
@@ -473,7 +474,9 @@ public void refreshTimeline() throws IOException {
boolean success = writeClient.commit(instantTime, writeStatusRDD, Option.of(checkpointCommitMetadata));
if (success) {
LOG.info("Commit " + instantTime + " successful!");
-
+ if (this.props.getBoolean(KafkaOffsetGen.Config.ENABLE_KAFKA_COMMIT_OFFSET,KafkaOffsetGen.Config.DEFAULT_ENABLE_KAFKA_COMMIT_OFFSET)) {
+ KafkaOffsetGen.commitOffsetToKafka(checkpointStr, this.props);
Review comment:
remove `KafkaOffsetGen.` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-862136673
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=219",
"triggerID" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"triggerType" : "PUSH"
}, {
"hash" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=290",
"triggerID" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"triggerType" : "PUSH"
}, {
"hash" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=311",
"triggerID" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"triggerType" : "PUSH"
}, {
"hash" : "5a2beb75f789afe8f5874e27b4d7b95427a1ea7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=315",
"triggerID" : "5a2beb75f789afe8f5874e27b4d7b95427a1ea7f",
"triggerType" : "PUSH"
}, {
"hash" : "2cb652554386030acaf569a73f9e074cbc72ffa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=324",
"triggerID" : "2cb652554386030acaf569a73f9e074cbc72ffa3",
"triggerType" : "PUSH"
}, {
"hash" : "1ae031303ebfa92fd153b8b6c0a404d1aac39a0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=393",
"triggerID" : "1ae031303ebfa92fd153b8b6c0a404d1aac39a0a",
"triggerType" : "PUSH"
}, {
"hash" : "e90cbcdc4a4926230d2440b486c52e39850cdb1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=442",
"triggerID" : "e90cbcdc4a4926230d2440b486c52e39850cdb1a",
"triggerType" : "PUSH"
}, {
"hash" : "2d5c2588f9088daef2800600558ebaa7d6d05212",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=443",
"triggerID" : "2d5c2588f9088daef2800600558ebaa7d6d05212",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2d5c2588f9088daef2800600558ebaa7d6d05212 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=443)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-862136673
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=219",
"triggerID" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"triggerType" : "PUSH"
}, {
"hash" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=290",
"triggerID" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"triggerType" : "PUSH"
}, {
"hash" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=311",
"triggerID" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"triggerType" : "PUSH"
}, {
"hash" : "5a2beb75f789afe8f5874e27b4d7b95427a1ea7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=315",
"triggerID" : "5a2beb75f789afe8f5874e27b4d7b95427a1ea7f",
"triggerType" : "PUSH"
}, {
"hash" : "2cb652554386030acaf569a73f9e074cbc72ffa3",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=324",
"triggerID" : "2cb652554386030acaf569a73f9e074cbc72ffa3",
"triggerType" : "PUSH"
}, {
"hash" : "1ae031303ebfa92fd153b8b6c0a404d1aac39a0a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=393",
"triggerID" : "1ae031303ebfa92fd153b8b6c0a404d1aac39a0a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2cb652554386030acaf569a73f9e074cbc72ffa3 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=324)
* 1ae031303ebfa92fd153b8b6c0a404d1aac39a0a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=393)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-862136673
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=219",
"triggerID" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"triggerType" : "PUSH"
}, {
"hash" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=290",
"triggerID" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"triggerType" : "PUSH"
}, {
"hash" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=311",
"triggerID" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=311)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] veenaypatil commented on a change in pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
veenaypatil commented on a change in pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#discussion_r655334095
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java
##########
@@ -506,6 +509,8 @@ public void refreshTimeline() throws IOException {
return Pair.of(scheduledCompactionInstant, writeStatusRDD);
}
+
+
Review comment:
removed
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -297,7 +295,37 @@ public String getTopicName() {
return topicName;
}
- public HashMap<String, Object> getKafkaParams() {
+ public Map<String, Object> getKafkaParams() {
+ return kafkaParams;
+ }
+
+ private static Map<String, Object> excludeHoodieConfigs(TypedProperties props) {
+ Map<String, Object> kafkaParams = new HashMap<>();
+ props.keySet().stream().filter(prop -> {
+ // In order to prevent printing unnecessary warn logs, here filter out the hoodie
+ // configuration items before passing to kafkaParams
+ return !prop.toString().startsWith("hoodie.");
+ }).forEach(prop -> {
+ kafkaParams.put(prop.toString(), props.get(prop.toString()));
+ });
return kafkaParams;
}
+
+ /**
+ * Commit offsets to Kafka only after hoodie commit is successful.
+ * @param checkpointStr checkpoint string containing offsets.
+ * @param props properties for Kafka consumer.
+ */
+ public static void commitOffsetToKafka(String checkpointStr, TypedProperties props) {
+ DataSourceUtils.checkRequiredProperties(props, Collections.singletonList(ConsumerConfig.GROUP_ID_CONFIG));
+ Map<TopicPartition, Long> offsetMap = KafkaOffsetGen.CheckpointUtils.strToOffsets(checkpointStr);
+ Map<String, Object> kafkaParams = KafkaOffsetGen.excludeHoodieConfigs(props);
+ Map<TopicPartition, OffsetAndMetadata> offsetAndMetadataMap = new HashMap<>(offsetMap.size());
+ try (KafkaConsumer consumer = new KafkaConsumer(kafkaParams)) {
+ offsetMap.forEach((topicPartition, offset) -> offsetAndMetadataMap.put(topicPartition, new OffsetAndMetadata(offset)));
+ consumer.commitSync(offsetAndMetadataMap);
+ } catch (CommitFailedException | TimeoutException e) {
+ LOG.warn("Committing offsets to Kafka failed, this does not impact processing of records", e);
Review comment:
committing offset to Kafka is optional for the user, failure of single commit should not fail the complete job, hence logged this exception.
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -297,7 +295,37 @@ public String getTopicName() {
return topicName;
}
- public HashMap<String, Object> getKafkaParams() {
+ public Map<String, Object> getKafkaParams() {
+ return kafkaParams;
+ }
+
+ private static Map<String, Object> excludeHoodieConfigs(TypedProperties props) {
Review comment:
Are you proposing to create `KafkaOffsetGen` object in `DeltaSync` as well to call `commitOffsetToKafka` method ? I wanted to avoid doing it as we are already doing so in respective Source classes. Also, I saw `excludeHoodieConfigs ` as helper method, hence created static
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java
##########
@@ -473,7 +474,9 @@ public void refreshTimeline() throws IOException {
boolean success = writeClient.commit(instantTime, writeStatusRDD, Option.of(checkpointCommitMetadata));
if (success) {
LOG.info("Commit " + instantTime + " successful!");
-
+ if (this.props.getBoolean(KafkaOffsetGen.Config.ENABLE_KAFKA_COMMIT_OFFSET,KafkaOffsetGen.Config.DEFAULT_ENABLE_KAFKA_COMMIT_OFFSET)) {
Review comment:
done
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -157,29 +161,23 @@ public static long totalNewMessages(OffsetRange[] ranges) {
private static final String KAFKA_TOPIC_NAME = "hoodie.deltastreamer.source.kafka.topic";
private static final String MAX_EVENTS_FROM_KAFKA_SOURCE_PROP = "hoodie.deltastreamer.kafka.source.maxEvents";
+ public static final String ENABLE_KAFKA_COMMIT_OFFSET = "hoodie.deltastreamer.source.enable.kafka.commit.offset";
Review comment:
make sense, updated code, did not rename `hoodie.deltastreamer.kafka.source.maxEvents` in this PR, will discuss more on compatibility
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -157,29 +161,23 @@ public static long totalNewMessages(OffsetRange[] ranges) {
private static final String KAFKA_TOPIC_NAME = "hoodie.deltastreamer.source.kafka.topic";
private static final String MAX_EVENTS_FROM_KAFKA_SOURCE_PROP = "hoodie.deltastreamer.kafka.source.maxEvents";
+ public static final String ENABLE_KAFKA_COMMIT_OFFSET = "hoodie.deltastreamer.source.enable.kafka.commit.offset";
// "auto.offset.reset" is kafka native config param. Do not change the config param name.
public static final String KAFKA_AUTO_OFFSET_RESET = "auto.offset.reset";
private static final KafkaResetOffsetStrategies DEFAULT_KAFKA_AUTO_OFFSET_RESET = KafkaResetOffsetStrategies.LATEST;
public static final long DEFAULT_MAX_EVENTS_FROM_KAFKA_SOURCE = 5000000;
public static long maxEventsFromKafkaSource = DEFAULT_MAX_EVENTS_FROM_KAFKA_SOURCE;
+ public static final Boolean DEFAULT_ENABLE_KAFKA_COMMIT_OFFSET = false;
}
- private final HashMap<String, Object> kafkaParams;
+ private final Map<String, Object> kafkaParams;
private final TypedProperties props;
protected final String topicName;
private KafkaResetOffsetStrategies autoResetValue;
public KafkaOffsetGen(TypedProperties props) {
this.props = props;
-
- kafkaParams = new HashMap<>();
- props.keySet().stream().filter(prop -> {
- // In order to prevent printing unnecessary warn logs, here filter out the hoodie
- // configuration items before passing to kafkaParams
- return !prop.toString().startsWith("hoodie.");
- }).forEach(prop -> {
- kafkaParams.put(prop.toString(), props.get(prop.toString()));
- });
+ kafkaParams = KafkaOffsetGen.excludeHoodieConfigs(props);
Review comment:
done
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java
##########
@@ -473,7 +474,9 @@ public void refreshTimeline() throws IOException {
boolean success = writeClient.commit(instantTime, writeStatusRDD, Option.of(checkpointCommitMetadata));
if (success) {
LOG.info("Commit " + instantTime + " successful!");
-
+ if (this.props.getBoolean(KafkaOffsetGen.Config.ENABLE_KAFKA_COMMIT_OFFSET,KafkaOffsetGen.Config.DEFAULT_ENABLE_KAFKA_COMMIT_OFFSET)) {
+ KafkaOffsetGen.commitOffsetToKafka(checkpointStr, this.props);
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] veenaypatil commented on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
veenaypatil commented on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-869102385
@hudi-bot run travis
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] veenaypatil commented on a change in pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
veenaypatil commented on a change in pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#discussion_r656757106
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -157,29 +161,23 @@ public static long totalNewMessages(OffsetRange[] ranges) {
private static final String KAFKA_TOPIC_NAME = "hoodie.deltastreamer.source.kafka.topic";
private static final String MAX_EVENTS_FROM_KAFKA_SOURCE_PROP = "hoodie.deltastreamer.kafka.source.maxEvents";
+ public static final String ENABLE_KAFKA_COMMIT_OFFSET = "hoodie.deltastreamer.source.kafka.enable.commit.offset";
Review comment:
that is `auto.commit.offset` config of Kafka, this will be a custom optional config
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] veenaypatil commented on a change in pull request #3092: [HUDI-1910] [WIP] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
veenaypatil commented on a change in pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#discussion_r654938113
##########
File path: hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestKafkaSource.java
##########
@@ -58,7 +67,7 @@
@BeforeAll
public static void initClass() throws Exception {
- UtilitiesTestBase.initClass();
+ UtilitiesTestBase.initClass(false);
Review comment:
Starting hive is not required for this test, this helps to reduce overall time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-862136673
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=219",
"triggerID" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"triggerType" : "PUSH"
}, {
"hash" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=290",
"triggerID" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"triggerType" : "PUSH"
}, {
"hash" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=311",
"triggerID" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"triggerType" : "PUSH"
}, {
"hash" : "5a2beb75f789afe8f5874e27b4d7b95427a1ea7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=315",
"triggerID" : "5a2beb75f789afe8f5874e27b4d7b95427a1ea7f",
"triggerType" : "PUSH"
}, {
"hash" : "2cb652554386030acaf569a73f9e074cbc72ffa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=324",
"triggerID" : "2cb652554386030acaf569a73f9e074cbc72ffa3",
"triggerType" : "PUSH"
}, {
"hash" : "1ae031303ebfa92fd153b8b6c0a404d1aac39a0a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=393",
"triggerID" : "1ae031303ebfa92fd153b8b6c0a404d1aac39a0a",
"triggerType" : "PUSH"
}, {
"hash" : "e90cbcdc4a4926230d2440b486c52e39850cdb1a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e90cbcdc4a4926230d2440b486c52e39850cdb1a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1ae031303ebfa92fd153b8b6c0a404d1aac39a0a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=393)
* e90cbcdc4a4926230d2440b486c52e39850cdb1a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-864517429
# [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3092](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2d5c258) into [master](https://codecov.io/gh/apache/hudi/commit/23dbc09a0d5d677ed3728dbd5357320590ca2833?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (23dbc09) will **decrease** coverage by `43.17%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3092 +/- ##
============================================
- Coverage 46.24% 3.06% -43.18%
+ Complexity 5362 82 -5280
============================================
Files 920 279 -641
Lines 39842 10884 -28958
Branches 4291 1106 -3185
============================================
- Hits 18425 334 -18091
+ Misses 19535 10524 -9011
+ Partials 1882 26 -1856
```
| Flag | Coverage Δ | |
|---|---|---|
| hudicli | `?` | |
| hudiclient | `0.00% <ø> (-30.46%)` | :arrow_down: |
| hudicommon | `?` | |
| hudiflink | `?` | |
| hudihadoopmr | `?` | |
| hudisparkdatasource | `?` | |
| hudisync | `5.38% <ø> (-48.67%)` | :arrow_down: |
| huditimelineservice | `?` | |
| hudiutilities | `9.43% <0.00%> (-48.87%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `0.00% <0.00%> (-71.19%)` | :arrow_down: |
| [...i/utilities/deltastreamer/SourceFormatAdapter.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvU291cmNlRm9ybWF0QWRhcHRlci5qYXZh) | `0.00% <0.00%> (-86.49%)` | :arrow_down: |
| [...g/apache/hudi/utilities/sources/AvroDFSSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb0RGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...apache/hudi/utilities/sources/AvroKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb0thZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQ3N2REZTU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...che/hudi/utilities/sources/HiveIncrPullSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSGl2ZUluY3JQdWxsU291cmNlLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...pache/hudi/utilities/sources/HoodieIncrSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSG9vZGllSW5jclNvdXJjZS5qYXZh) | `0.00% <0.00%> (-92.60%)` | :arrow_down: |
| [.../org/apache/hudi/utilities/sources/JdbcSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSmRiY1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-92.00%)` | :arrow_down: |
| [...g/apache/hudi/utilities/sources/JsonDFSSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkRGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [768 more](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [23dbc09...2d5c258](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-862136673
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=219",
"triggerID" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"triggerType" : "PUSH"
}, {
"hash" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=290",
"triggerID" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"triggerType" : "PUSH"
}, {
"hash" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=311",
"triggerID" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"triggerType" : "PUSH"
}, {
"hash" : "5a2beb75f789afe8f5874e27b4d7b95427a1ea7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=315",
"triggerID" : "5a2beb75f789afe8f5874e27b4d7b95427a1ea7f",
"triggerType" : "PUSH"
}, {
"hash" : "2cb652554386030acaf569a73f9e074cbc72ffa3",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=324",
"triggerID" : "2cb652554386030acaf569a73f9e074cbc72ffa3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2cb652554386030acaf569a73f9e074cbc72ffa3 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=324)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-864517429
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-864517429
# [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3092](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5a2beb7) into [master](https://codecov.io/gh/apache/hudi/commit/61efc6af79c389ef0a77cda75e4f562ed59ef86b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (61efc6a) will **increase** coverage by `0.25%`.
> The diff coverage is `81.81%`.
[](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3092 +/- ##
============================================
+ Coverage 45.75% 46.01% +0.25%
- Complexity 5262 5309 +47
============================================
Files 909 911 +2
Lines 39356 39504 +148
Branches 4239 4256 +17
============================================
+ Hits 18008 18177 +169
+ Misses 19504 19471 -33
- Partials 1844 1856 +12
```
| Flag | Coverage Δ | |
|---|---|---|
| hudicli | `39.95% <ø> (ø)` | |
| hudiclient | `30.44% <ø> (+0.05%)` | :arrow_up: |
| hudicommon | `47.56% <ø> (+<0.01%)` | :arrow_up: |
| hudiflink | `61.12% <ø> (+0.38%)` | :arrow_up: |
| hudihadoopmr | `51.29% <ø> (ø)` | |
| hudisparkdatasource | `66.52% <ø> (+0.04%)` | :arrow_up: |
| hudisync | `51.73% <ø> (+0.27%)` | :arrow_up: |
| huditimelineservice | `64.36% <ø> (ø)` | |
| hudiutilities | `58.46% <81.81%> (+1.82%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `70.37% <0.00%> (-0.48%)` | :arrow_down: |
| [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `87.09% <90.00%> (-0.18%)` | :arrow_down: |
| [.../org/apache/hudi/sink/compact/CompactFunction.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL2NvbXBhY3QvQ29tcGFjdEZ1bmN0aW9uLmphdmE=) | `86.66% <0.00%> (-13.34%)` | :arrow_down: |
| [...e/hudi/sink/partitioner/profile/WriteProfiles.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL3BhcnRpdGlvbmVyL3Byb2ZpbGUvV3JpdGVQcm9maWxlcy5qYXZh) | `52.50% <0.00%> (-3.39%)` | :arrow_down: |
| [...c/main/java/org/apache/hudi/util/StreamerUtil.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS91dGlsL1N0cmVhbWVyVXRpbC5qYXZh) | `55.00% <0.00%> (-1.42%)` | :arrow_down: |
| [...java/org/apache/hudi/sink/StreamWriteFunction.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL1N0cmVhbVdyaXRlRnVuY3Rpb24uamF2YQ==) | `84.34% <0.00%> (-0.66%)` | :arrow_down: |
| [...he/hudi/sink/partitioner/profile/WriteProfile.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL3BhcnRpdGlvbmVyL3Byb2ZpbGUvV3JpdGVQcm9maWxlLmphdmE=) | `87.50% <0.00%> (-0.50%)` | :arrow_down: |
| [...va/org/apache/hudi/configuration/FlinkOptions.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9jb25maWd1cmF0aW9uL0ZsaW5rT3B0aW9ucy5qYXZh) | `96.37% <0.00%> (-0.05%)` | :arrow_down: |
| [...main/java/org/apache/hudi/util/CompactionUtil.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS91dGlsL0NvbXBhY3Rpb25VdGlsLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...in/java/org/apache/hudi/table/HoodieTableSink.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS90YWJsZS9Ib29kaWVUYWJsZVNpbmsuamF2YQ==) | `10.52% <0.00%> (ø)` | |
| ... and [23 more](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [61efc6a...5a2beb7](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-864517429
# [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3092](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (4cf4e2e) into [master](https://codecov.io/gh/apache/hudi/commit/61efc6af79c389ef0a77cda75e4f562ed59ef86b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (61efc6a) will **decrease** coverage by `29.13%`.
> The diff coverage is `81.81%`.
[](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3092 +/- ##
=============================================
- Coverage 45.75% 16.62% -29.14%
+ Complexity 5262 471 -4791
=============================================
Files 909 274 -635
Lines 39356 10666 -28690
Branches 4239 1089 -3150
=============================================
- Hits 18008 1773 -16235
+ Misses 19504 8734 -10770
+ Partials 1844 159 -1685
```
| Flag | Coverage Δ | |
|---|---|---|
| hudicli | `?` | |
| hudiclient | `0.00% <ø> (-30.40%)` | :arrow_down: |
| hudicommon | `?` | |
| hudiflink | `?` | |
| hudihadoopmr | `?` | |
| hudisparkdatasource | `?` | |
| hudisync | `6.72% <ø> (-44.74%)` | :arrow_down: |
| huditimelineservice | `?` | |
| hudiutilities | `58.50% <81.81%> (+1.86%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `70.70% <0.00%> (-0.15%)` | :arrow_down: |
| [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `87.09% <90.00%> (-0.18%)` | :arrow_down: |
| [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...rg/apache/hudi/client/bootstrap/BootstrapMode.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9ib290c3RyYXAvQm9vdHN0cmFwTW9kZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...he/hudi/hive/HiveStylePartitionValueExtractor.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN0eWxlUGFydGl0aW9uVmFsdWVFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/hudi/keygen/constant/KeyGeneratorType.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2tleWdlbi9jb25zdGFudC9LZXlHZW5lcmF0b3JUeXBlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...pache/hudi/client/utils/ConcatenatingIterator.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC91dGlscy9Db25jYXRlbmF0aW5nSXRlcmF0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [710 more](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [61efc6a...4cf4e2e](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on a change in pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#discussion_r656753261
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -157,29 +161,23 @@ public static long totalNewMessages(OffsetRange[] ranges) {
private static final String KAFKA_TOPIC_NAME = "hoodie.deltastreamer.source.kafka.topic";
private static final String MAX_EVENTS_FROM_KAFKA_SOURCE_PROP = "hoodie.deltastreamer.kafka.source.maxEvents";
+ public static final String ENABLE_KAFKA_COMMIT_OFFSET = "hoodie.deltastreamer.source.kafka.enable.commit.offset";
Review comment:
IIRC all these are just kafka configs right?
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java
##########
@@ -473,7 +476,9 @@ public void refreshTimeline() throws IOException {
boolean success = writeClient.commit(instantTime, writeStatusRDD, Option.of(checkpointCommitMetadata));
if (success) {
LOG.info("Commit " + instantTime + " successful!");
-
+ if (this.props.getBoolean(ENABLE_KAFKA_COMMIT_OFFSET, DEFAULT_ENABLE_KAFKA_COMMIT_OFFSET)) {
Review comment:
I am trying to see how we can move such source specific logic back to the source. Ideally `DeltaSync` should be agnostic of Kafka etc. Can we add an `onCommit()` or some callback on the Source?
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -157,29 +161,23 @@ public static long totalNewMessages(OffsetRange[] ranges) {
private static final String KAFKA_TOPIC_NAME = "hoodie.deltastreamer.source.kafka.topic";
private static final String MAX_EVENTS_FROM_KAFKA_SOURCE_PROP = "hoodie.deltastreamer.kafka.source.maxEvents";
+ public static final String ENABLE_KAFKA_COMMIT_OFFSET = "hoodie.deltastreamer.source.kafka.enable.commit.offset";
// "auto.offset.reset" is kafka native config param. Do not change the config param name.
public static final String KAFKA_AUTO_OFFSET_RESET = "auto.offset.reset";
private static final KafkaResetOffsetStrategies DEFAULT_KAFKA_AUTO_OFFSET_RESET = KafkaResetOffsetStrategies.LATEST;
public static final long DEFAULT_MAX_EVENTS_FROM_KAFKA_SOURCE = 5000000;
public static long maxEventsFromKafkaSource = DEFAULT_MAX_EVENTS_FROM_KAFKA_SOURCE;
+ public static final Boolean DEFAULT_ENABLE_KAFKA_COMMIT_OFFSET = false;
Review comment:
can we move this closer to the prop name?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-862136673
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=219",
"triggerID" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"triggerType" : "PUSH"
}, {
"hash" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=290",
"triggerID" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"triggerType" : "PUSH"
}, {
"hash" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=311",
"triggerID" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"triggerType" : "PUSH"
}, {
"hash" : "5a2beb75f789afe8f5874e27b4d7b95427a1ea7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=315",
"triggerID" : "5a2beb75f789afe8f5874e27b4d7b95427a1ea7f",
"triggerType" : "PUSH"
}, {
"hash" : "2cb652554386030acaf569a73f9e074cbc72ffa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=324",
"triggerID" : "2cb652554386030acaf569a73f9e074cbc72ffa3",
"triggerType" : "PUSH"
}, {
"hash" : "1ae031303ebfa92fd153b8b6c0a404d1aac39a0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=393",
"triggerID" : "1ae031303ebfa92fd153b8b6c0a404d1aac39a0a",
"triggerType" : "PUSH"
}, {
"hash" : "e90cbcdc4a4926230d2440b486c52e39850cdb1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=442",
"triggerID" : "e90cbcdc4a4926230d2440b486c52e39850cdb1a",
"triggerType" : "PUSH"
}, {
"hash" : "2d5c2588f9088daef2800600558ebaa7d6d05212",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=443",
"triggerID" : "2d5c2588f9088daef2800600558ebaa7d6d05212",
"triggerType" : "PUSH"
}, {
"hash" : "3b02a03a355c85723b69c9d8b4557c5983c8914f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=461",
"triggerID" : "3b02a03a355c85723b69c9d8b4557c5983c8914f",
"triggerType" : "PUSH"
}, {
"hash" : "3b02a03a355c85723b69c9d8b4557c5983c8914f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=461",
"triggerID" : "869102385",
"triggerType" : "MANUAL"
}, {
"hash" : "aa5414320f403ffa2224798e89cd7aa1752fdd12",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aa5414320f403ffa2224798e89cd7aa1752fdd12",
"triggerType" : "PUSH"
}, {
"hash" : "f3cc39fc2b35b1b935e24f120b14db515748a6b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f3cc39fc2b35b1b935e24f120b14db515748a6b9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3b02a03a355c85723b69c9d8b4557c5983c8914f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=461)
* aa5414320f403ffa2224798e89cd7aa1752fdd12 UNKNOWN
* f3cc39fc2b35b1b935e24f120b14db515748a6b9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-862136673
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=219",
"triggerID" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"triggerType" : "PUSH"
}, {
"hash" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=290",
"triggerID" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"triggerType" : "PUSH"
}, {
"hash" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=311",
"triggerID" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"triggerType" : "PUSH"
}, {
"hash" : "5a2beb75f789afe8f5874e27b4d7b95427a1ea7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=315",
"triggerID" : "5a2beb75f789afe8f5874e27b4d7b95427a1ea7f",
"triggerType" : "PUSH"
}, {
"hash" : "2cb652554386030acaf569a73f9e074cbc72ffa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=324",
"triggerID" : "2cb652554386030acaf569a73f9e074cbc72ffa3",
"triggerType" : "PUSH"
}, {
"hash" : "1ae031303ebfa92fd153b8b6c0a404d1aac39a0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=393",
"triggerID" : "1ae031303ebfa92fd153b8b6c0a404d1aac39a0a",
"triggerType" : "PUSH"
}, {
"hash" : "e90cbcdc4a4926230d2440b486c52e39850cdb1a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=442",
"triggerID" : "e90cbcdc4a4926230d2440b486c52e39850cdb1a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e90cbcdc4a4926230d2440b486c52e39850cdb1a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=442)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#discussion_r659268783
##########
File path: hudi-examples/src/main/java/org/apache/hudi/examples/common/RandomJsonSource.java
##########
@@ -45,4 +45,8 @@ public RandomJsonSource(TypedProperties props, JavaSparkContext sparkContext, Sp
return new InputBatch<>(Option.of(sparkContext.parallelize(inserts, 1)), commitTime);
}
+
+ @Override
+ public void onCommit(String lastCkptStr) {
Review comment:
for these empty `onCommit` method, would we define a default method `onCommit(String lastCkpStr)`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-864517429
# [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3092](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3b02a03) into [master](https://codecov.io/gh/apache/hudi/commit/f73bedd374952580c4d6bc9b2e30607c4efdd538?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f73bedd) will **increase** coverage by `5.70%`.
> The diff coverage is `75.00%`.
[](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3092 +/- ##
============================================
+ Coverage 44.31% 50.01% +5.70%
+ Complexity 4574 404 -4170
============================================
Files 824 66 -758
Lines 36375 2947 -33428
Branches 3937 318 -3619
============================================
- Hits 16120 1474 -14646
+ Misses 18511 1338 -17173
+ Partials 1744 135 -1609
```
| Flag | Coverage Δ | |
|---|---|---|
| hudicli | `?` | |
| hudiclient | `?` | |
| hudicommon | `?` | |
| hudiflink | `?` | |
| hudihadoopmr | `?` | |
| hudisparkdatasource | `?` | |
| hudisync | `?` | |
| huditimelineservice | `?` | |
| hudiutilities | `50.01% <75.00%> (-8.28%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...g/apache/hudi/utilities/sources/AvroDFSSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb0RGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...apache/hudi/utilities/sources/AvroKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb0thZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...che/hudi/utilities/sources/HiveIncrPullSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSGl2ZUluY3JQdWxsU291cmNlLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/JsonDFSSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkRGU1NvdXJjZS5qYXZh) | `90.00% <0.00%> (-10.00%)` | :arrow_down: |
| [...java/org/apache/hudi/utilities/sources/Source.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvU291cmNlLmphdmE=) | `87.50% <ø> (ø)` | |
| [...a/org/apache/hudi/utilities/sources/SqlSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvU3FsU291cmNlLmphdmE=) | `61.11% <0.00%> (-3.60%)` | :arrow_down: |
| [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `87.09% <90.00%> (+0.61%)` | :arrow_up: |
| [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `71.28% <100.00%> (+0.09%)` | :arrow_up: |
| [...i/utilities/deltastreamer/SourceFormatAdapter.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvU291cmNlRm9ybWF0QWRhcHRlci5qYXZh) | `86.84% <100.00%> (+0.35%)` | :arrow_up: |
| [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQ3N2REZTU291cmNlLmphdmE=) | `100.00% <100.00%> (ø)` | |
| ... and [769 more](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [f73bedd...3b02a03](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-commenter commented on pull request #3092: [HUDI-1910] [WIP] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-864517429
# [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3092](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7dc7b1f) into [master](https://codecov.io/gh/apache/hudi/commit/61efc6af79c389ef0a77cda75e4f562ed59ef86b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (61efc6a) will **decrease** coverage by `42.62%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3092 +/- ##
============================================
- Coverage 45.75% 3.13% -42.63%
+ Complexity 5262 82 -5180
============================================
Files 909 274 -635
Lines 39356 10663 -28693
Branches 4239 1089 -3150
============================================
- Hits 18008 334 -17674
+ Misses 19504 10303 -9201
+ Partials 1844 26 -1818
```
| Flag | Coverage Δ | |
|---|---|---|
| hudicli | `?` | |
| hudiclient | `0.00% <ø> (-30.40%)` | :arrow_down: |
| hudicommon | `?` | |
| hudiflink | `?` | |
| hudihadoopmr | `?` | |
| hudisparkdatasource | `?` | |
| hudisync | `6.72% <ø> (-44.74%)` | :arrow_down: |
| huditimelineservice | `?` | |
| hudiutilities | `9.48% <0.00%> (-47.16%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `0.00% <0.00%> (-70.85%)` | :arrow_down: |
| [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `0.00% <0.00%> (-87.28%)` | :arrow_down: |
| [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQ3N2REZTU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...g/apache/hudi/utilities/sources/JsonDFSSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkRGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [750 more](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [61efc6a...7dc7b1f](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-862136673
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=219",
"triggerID" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"triggerType" : "PUSH"
}, {
"hash" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=290",
"triggerID" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"triggerType" : "PUSH"
}, {
"hash" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=311",
"triggerID" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"triggerType" : "PUSH"
}, {
"hash" : "5a2beb75f789afe8f5874e27b4d7b95427a1ea7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=315",
"triggerID" : "5a2beb75f789afe8f5874e27b4d7b95427a1ea7f",
"triggerType" : "PUSH"
}, {
"hash" : "2cb652554386030acaf569a73f9e074cbc72ffa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=324",
"triggerID" : "2cb652554386030acaf569a73f9e074cbc72ffa3",
"triggerType" : "PUSH"
}, {
"hash" : "1ae031303ebfa92fd153b8b6c0a404d1aac39a0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=393",
"triggerID" : "1ae031303ebfa92fd153b8b6c0a404d1aac39a0a",
"triggerType" : "PUSH"
}, {
"hash" : "e90cbcdc4a4926230d2440b486c52e39850cdb1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=442",
"triggerID" : "e90cbcdc4a4926230d2440b486c52e39850cdb1a",
"triggerType" : "PUSH"
}, {
"hash" : "2d5c2588f9088daef2800600558ebaa7d6d05212",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=443",
"triggerID" : "2d5c2588f9088daef2800600558ebaa7d6d05212",
"triggerType" : "PUSH"
}, {
"hash" : "3b02a03a355c85723b69c9d8b4557c5983c8914f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=461",
"triggerID" : "3b02a03a355c85723b69c9d8b4557c5983c8914f",
"triggerType" : "PUSH"
}, {
"hash" : "3b02a03a355c85723b69c9d8b4557c5983c8914f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "869102385",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 3b02a03a355c85723b69c9d8b4557c5983c8914f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=461)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#discussion_r655483483
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -157,29 +161,23 @@ public static long totalNewMessages(OffsetRange[] ranges) {
private static final String KAFKA_TOPIC_NAME = "hoodie.deltastreamer.source.kafka.topic";
private static final String MAX_EVENTS_FROM_KAFKA_SOURCE_PROP = "hoodie.deltastreamer.kafka.source.maxEvents";
+ public static final String ENABLE_KAFKA_COMMIT_OFFSET = "hoodie.deltastreamer.source.enable.kafka.commit.offset";
// "auto.offset.reset" is kafka native config param. Do not change the config param name.
public static final String KAFKA_AUTO_OFFSET_RESET = "auto.offset.reset";
private static final KafkaResetOffsetStrategies DEFAULT_KAFKA_AUTO_OFFSET_RESET = KafkaResetOffsetStrategies.LATEST;
public static final long DEFAULT_MAX_EVENTS_FROM_KAFKA_SOURCE = 5000000;
public static long maxEventsFromKafkaSource = DEFAULT_MAX_EVENTS_FROM_KAFKA_SOURCE;
+ public static final Boolean DEFAULT_ENABLE_KAFKA_COMMIT_OFFSET = false;
}
- private final HashMap<String, Object> kafkaParams;
+ private final Map<String, Object> kafkaParams;
private final TypedProperties props;
protected final String topicName;
private KafkaResetOffsetStrategies autoResetValue;
public KafkaOffsetGen(TypedProperties props) {
this.props = props;
-
- kafkaParams = new HashMap<>();
- props.keySet().stream().filter(prop -> {
- // In order to prevent printing unnecessary warn logs, here filter out the hoodie
- // configuration items before passing to kafkaParams
- return !prop.toString().startsWith("hoodie.");
- }).forEach(prop -> {
- kafkaParams.put(prop.toString(), props.get(prop.toString()));
- });
+ kafkaParams = KafkaOffsetGen.excludeHoodieConfigs(props);
Review comment:
remove `KafkaOffsetGen.`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-864517429
# [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3092](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (1ae0313) into [master](https://codecov.io/gh/apache/hudi/commit/e039e0ff6d04b9b0684dbf29452af8a8dae94b09?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e039e0f) will **decrease** coverage by `43.00%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3092 +/- ##
============================================
- Coverage 46.13% 3.13% -43.01%
+ Complexity 5325 82 -5243
============================================
Files 915 274 -641
Lines 39622 10665 -28957
Branches 4272 1089 -3183
============================================
- Hits 18281 334 -17947
+ Misses 19470 10305 -9165
+ Partials 1871 26 -1845
```
| Flag | Coverage Δ | |
|---|---|---|
| hudicli | `?` | |
| hudiclient | `0.00% <ø> (-30.46%)` | :arrow_down: |
| hudicommon | `?` | |
| hudiflink | `?` | |
| hudihadoopmr | `?` | |
| hudisparkdatasource | `?` | |
| hudisync | `6.72% <ø> (-45.01%)` | :arrow_down: |
| huditimelineservice | `?` | |
| hudiutilities | `9.47% <0.00%> (-48.91%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `0.00% <0.00%> (-70.85%)` | :arrow_down: |
| [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `0.00% <0.00%> (-87.28%)` | :arrow_down: |
| [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQ3N2REZTU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...g/apache/hudi/utilities/sources/JsonDFSSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkRGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [751 more](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e039e0f...1ae0313](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#discussion_r655483820
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -297,7 +295,37 @@ public String getTopicName() {
return topicName;
}
- public HashMap<String, Object> getKafkaParams() {
+ public Map<String, Object> getKafkaParams() {
+ return kafkaParams;
+ }
+
+ private static Map<String, Object> excludeHoodieConfigs(TypedProperties props) {
Review comment:
remove static ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-864517429
# [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3092](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2cb6525) into [master](https://codecov.io/gh/apache/hudi/commit/61efc6af79c389ef0a77cda75e4f562ed59ef86b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (61efc6a) will **decrease** coverage by `42.62%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3092 +/- ##
============================================
- Coverage 45.75% 3.13% -42.63%
+ Complexity 5262 82 -5180
============================================
Files 909 274 -635
Lines 39356 10666 -28690
Branches 4239 1089 -3150
============================================
- Hits 18008 334 -17674
+ Misses 19504 10306 -9198
+ Partials 1844 26 -1818
```
| Flag | Coverage Δ | |
|---|---|---|
| hudicli | `?` | |
| hudiclient | `0.00% <ø> (-30.40%)` | :arrow_down: |
| hudicommon | `?` | |
| hudiflink | `?` | |
| hudihadoopmr | `?` | |
| hudisparkdatasource | `?` | |
| hudisync | `6.72% <ø> (-44.74%)` | :arrow_down: |
| huditimelineservice | `?` | |
| hudiutilities | `9.47% <0.00%> (-47.17%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `0.00% <0.00%> (-70.85%)` | :arrow_down: |
| [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `0.00% <0.00%> (-87.28%)` | :arrow_down: |
| [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQ3N2REZTU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...g/apache/hudi/utilities/sources/JsonDFSSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkRGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [750 more](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [61efc6a...2cb6525](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] veenaypatil closed pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
veenaypatil closed pull request #3092:
URL: https://github.com/apache/hudi/pull/3092
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] veenaypatil commented on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
veenaypatil commented on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-870697459
@sbernauer ohh, my bad 😞
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-862136673
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=219",
"triggerID" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"triggerType" : "PUSH"
}, {
"hash" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=290",
"triggerID" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"triggerType" : "PUSH"
}, {
"hash" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=311",
"triggerID" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"triggerType" : "PUSH"
}, {
"hash" : "5a2beb75f789afe8f5874e27b4d7b95427a1ea7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=315",
"triggerID" : "5a2beb75f789afe8f5874e27b4d7b95427a1ea7f",
"triggerType" : "PUSH"
}, {
"hash" : "2cb652554386030acaf569a73f9e074cbc72ffa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=324",
"triggerID" : "2cb652554386030acaf569a73f9e074cbc72ffa3",
"triggerType" : "PUSH"
}, {
"hash" : "1ae031303ebfa92fd153b8b6c0a404d1aac39a0a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=393",
"triggerID" : "1ae031303ebfa92fd153b8b6c0a404d1aac39a0a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1ae031303ebfa92fd153b8b6c0a404d1aac39a0a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=393)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-862136673
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=219",
"triggerID" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"triggerType" : "PUSH"
}, {
"hash" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=290",
"triggerID" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"triggerType" : "PUSH"
}, {
"hash" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=311",
"triggerID" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"triggerType" : "PUSH"
}, {
"hash" : "5a2beb75f789afe8f5874e27b4d7b95427a1ea7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=315",
"triggerID" : "5a2beb75f789afe8f5874e27b4d7b95427a1ea7f",
"triggerType" : "PUSH"
}, {
"hash" : "2cb652554386030acaf569a73f9e074cbc72ffa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=324",
"triggerID" : "2cb652554386030acaf569a73f9e074cbc72ffa3",
"triggerType" : "PUSH"
}, {
"hash" : "1ae031303ebfa92fd153b8b6c0a404d1aac39a0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=393",
"triggerID" : "1ae031303ebfa92fd153b8b6c0a404d1aac39a0a",
"triggerType" : "PUSH"
}, {
"hash" : "e90cbcdc4a4926230d2440b486c52e39850cdb1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=442",
"triggerID" : "e90cbcdc4a4926230d2440b486c52e39850cdb1a",
"triggerType" : "PUSH"
}, {
"hash" : "2d5c2588f9088daef2800600558ebaa7d6d05212",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=443",
"triggerID" : "2d5c2588f9088daef2800600558ebaa7d6d05212",
"triggerType" : "PUSH"
}, {
"hash" : "3b02a03a355c85723b69c9d8b4557c5983c8914f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=461",
"triggerID" : "3b02a03a355c85723b69c9d8b4557c5983c8914f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2d5c2588f9088daef2800600558ebaa7d6d05212 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=443)
* 3b02a03a355c85723b69c9d8b4557c5983c8914f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=461)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-864517429
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-862136673
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=219",
"triggerID" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"triggerType" : "PUSH"
}, {
"hash" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=290",
"triggerID" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"triggerType" : "PUSH"
}, {
"hash" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=311",
"triggerID" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"triggerType" : "PUSH"
}, {
"hash" : "5a2beb75f789afe8f5874e27b4d7b95427a1ea7f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=315",
"triggerID" : "5a2beb75f789afe8f5874e27b4d7b95427a1ea7f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=311)
* 5a2beb75f789afe8f5874e27b4d7b95427a1ea7f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=315)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-864517429
# [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3092](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5a2beb7) into [master](https://codecov.io/gh/apache/hudi/commit/61efc6af79c389ef0a77cda75e4f562ed59ef86b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (61efc6a) will **decrease** coverage by `29.14%`.
> The diff coverage is `81.81%`.
[](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3092 +/- ##
=============================================
- Coverage 45.75% 16.61% -29.15%
+ Complexity 5262 470 -4792
=============================================
Files 909 274 -635
Lines 39356 10666 -28690
Branches 4239 1089 -3150
=============================================
- Hits 18008 1772 -16236
+ Misses 19504 8734 -10770
+ Partials 1844 160 -1684
```
| Flag | Coverage Δ | |
|---|---|---|
| hudicli | `?` | |
| hudiclient | `0.00% <ø> (-30.40%)` | :arrow_down: |
| hudicommon | `?` | |
| hudiflink | `?` | |
| hudihadoopmr | `?` | |
| hudisparkdatasource | `?` | |
| hudisync | `6.72% <ø> (-44.74%)` | :arrow_down: |
| huditimelineservice | `?` | |
| hudiutilities | `58.46% <81.81%> (+1.82%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `70.37% <0.00%> (-0.48%)` | :arrow_down: |
| [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `87.09% <90.00%> (-0.18%)` | :arrow_down: |
| [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...rg/apache/hudi/client/bootstrap/BootstrapMode.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9ib290c3RyYXAvQm9vdHN0cmFwTW9kZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...he/hudi/hive/HiveStylePartitionValueExtractor.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN0eWxlUGFydGl0aW9uVmFsdWVFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/hudi/keygen/constant/KeyGeneratorType.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2tleWdlbi9jb25zdGFudC9LZXlHZW5lcmF0b3JUeXBlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...pache/hudi/client/utils/ConcatenatingIterator.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC91dGlscy9Db25jYXRlbmF0aW5nSXRlcmF0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [709 more](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [61efc6a...5a2beb7](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] veenaypatil commented on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
veenaypatil commented on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-864579008
@n3nash @vinothchandar all the changes are done and tested, can you please review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] veenaypatil commented on pull request #3092: [HUDI-1910] [WIP] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
veenaypatil commented on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-862135078
@n3nash can you pls review and provide initial thoughts on the approach. I am yet to add test cases, facing few issues in testing this out locally after running `TestKafkaSource`
```
12571 [pool-18-thread-3] ERROR hive.log - Got exception: org.apache.hadoop.security.AccessControlException Permission denied: user=anonymous, access=WRITE, inode="/var/folders/pd/pnhdkpt97c58vvm2y7_x73dh0000gp/T/1623828588133-3088184389848693840/testdb1.db":vinaypatil:supergroup:drwxr-xr-x
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] veenaypatil commented on a change in pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
veenaypatil commented on a change in pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#discussion_r655526318
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -157,29 +161,23 @@ public static long totalNewMessages(OffsetRange[] ranges) {
private static final String KAFKA_TOPIC_NAME = "hoodie.deltastreamer.source.kafka.topic";
private static final String MAX_EVENTS_FROM_KAFKA_SOURCE_PROP = "hoodie.deltastreamer.kafka.source.maxEvents";
+ public static final String ENABLE_KAFKA_COMMIT_OFFSET = "hoodie.deltastreamer.source.enable.kafka.commit.offset";
Review comment:
make sense, updated code, did not rename `hoodie.deltastreamer.kafka.source.maxEvents` in this PR, will discuss more on compatibility
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -157,29 +161,23 @@ public static long totalNewMessages(OffsetRange[] ranges) {
private static final String KAFKA_TOPIC_NAME = "hoodie.deltastreamer.source.kafka.topic";
private static final String MAX_EVENTS_FROM_KAFKA_SOURCE_PROP = "hoodie.deltastreamer.kafka.source.maxEvents";
+ public static final String ENABLE_KAFKA_COMMIT_OFFSET = "hoodie.deltastreamer.source.enable.kafka.commit.offset";
// "auto.offset.reset" is kafka native config param. Do not change the config param name.
public static final String KAFKA_AUTO_OFFSET_RESET = "auto.offset.reset";
private static final KafkaResetOffsetStrategies DEFAULT_KAFKA_AUTO_OFFSET_RESET = KafkaResetOffsetStrategies.LATEST;
public static final long DEFAULT_MAX_EVENTS_FROM_KAFKA_SOURCE = 5000000;
public static long maxEventsFromKafkaSource = DEFAULT_MAX_EVENTS_FROM_KAFKA_SOURCE;
+ public static final Boolean DEFAULT_ENABLE_KAFKA_COMMIT_OFFSET = false;
}
- private final HashMap<String, Object> kafkaParams;
+ private final Map<String, Object> kafkaParams;
private final TypedProperties props;
protected final String topicName;
private KafkaResetOffsetStrategies autoResetValue;
public KafkaOffsetGen(TypedProperties props) {
this.props = props;
-
- kafkaParams = new HashMap<>();
- props.keySet().stream().filter(prop -> {
- // In order to prevent printing unnecessary warn logs, here filter out the hoodie
- // configuration items before passing to kafkaParams
- return !prop.toString().startsWith("hoodie.");
- }).forEach(prop -> {
- kafkaParams.put(prop.toString(), props.get(prop.toString()));
- });
+ kafkaParams = KafkaOffsetGen.excludeHoodieConfigs(props);
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] veenaypatil commented on a change in pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
veenaypatil commented on a change in pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#discussion_r659271949
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -297,7 +295,37 @@ public String getTopicName() {
return topicName;
}
- public HashMap<String, Object> getKafkaParams() {
+ public Map<String, Object> getKafkaParams() {
+ return kafkaParams;
+ }
+
+ private static Map<String, Object> excludeHoodieConfigs(TypedProperties props) {
+ Map<String, Object> kafkaParams = new HashMap<>();
+ props.keySet().stream().filter(prop -> {
+ // In order to prevent printing unnecessary warn logs, here filter out the hoodie
+ // configuration items before passing to kafkaParams
+ return !prop.toString().startsWith("hoodie.");
+ }).forEach(prop -> {
+ kafkaParams.put(prop.toString(), props.get(prop.toString()));
+ });
return kafkaParams;
}
+
+ /**
+ * Commit offsets to Kafka only after hoodie commit is successful.
+ * @param checkpointStr checkpoint string containing offsets.
+ * @param props properties for Kafka consumer.
+ */
+ public static void commitOffsetToKafka(String checkpointStr, TypedProperties props) {
+ DataSourceUtils.checkRequiredProperties(props, Collections.singletonList(ConsumerConfig.GROUP_ID_CONFIG));
+ Map<TopicPartition, Long> offsetMap = KafkaOffsetGen.CheckpointUtils.strToOffsets(checkpointStr);
+ Map<String, Object> kafkaParams = KafkaOffsetGen.excludeHoodieConfigs(props);
+ Map<TopicPartition, OffsetAndMetadata> offsetAndMetadataMap = new HashMap<>(offsetMap.size());
+ try (KafkaConsumer consumer = new KafkaConsumer(kafkaParams)) {
+ offsetMap.forEach((topicPartition, offset) -> offsetAndMetadataMap.put(topicPartition, new OffsetAndMetadata(offset)));
+ consumer.commitSync(offsetAndMetadataMap);
+ } catch (CommitFailedException | TimeoutException e) {
+ LOG.warn("Committing offsets to Kafka failed, this does not impact processing of records", e);
Review comment:
This change will not have any effect on job restarts. Hudi will not rely on the offsets committed to Kafka after restarts. This is similar to how Flink behaves which only commits offsets to Kafka (configurable) but relies on it's own checkpointing if the job restarts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-864517429
# [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3092](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2d5c258) into [master](https://codecov.io/gh/apache/hudi/commit/23dbc09a0d5d677ed3728dbd5357320590ca2833?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (23dbc09) will **increase** coverage by `0.01%`.
> The diff coverage is `69.44%`.
[](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3092 +/- ##
============================================
+ Coverage 46.24% 46.26% +0.01%
- Complexity 5362 5371 +9
============================================
Files 920 920
Lines 39842 39871 +29
Branches 4291 4290 -1
============================================
+ Hits 18425 18446 +21
- Misses 19535 19544 +9
+ Partials 1882 1881 -1
```
| Flag | Coverage Δ | |
|---|---|---|
| hudicli | `39.95% <ø> (ø)` | |
| hudiclient | `30.45% <ø> (ø)` | |
| hudicommon | `47.58% <ø> (+0.01%)` | :arrow_up: |
| hudiflink | `61.25% <ø> (ø)` | |
| hudihadoopmr | `51.29% <ø> (ø)` | |
| hudisparkdatasource | `67.06% <ø> (ø)` | |
| hudisync | `54.05% <ø> (ø)` | |
| huditimelineservice | `64.36% <ø> (ø)` | |
| hudiutilities | `58.36% <69.44%> (+0.07%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...g/apache/hudi/utilities/sources/AvroDFSSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb0RGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...apache/hudi/utilities/sources/AvroKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb0thZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...che/hudi/utilities/sources/HiveIncrPullSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSGl2ZUluY3JQdWxsU291cmNlLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/JsonDFSSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkRGU1NvdXJjZS5qYXZh) | `90.00% <0.00%> (-10.00%)` | :arrow_down: |
| [...java/org/apache/hudi/utilities/sources/Source.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvU291cmNlLmphdmE=) | `87.50% <ø> (ø)` | |
| [...a/org/apache/hudi/utilities/sources/SqlSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvU3FsU291cmNlLmphdmE=) | `61.11% <0.00%> (-3.60%)` | :arrow_down: |
| [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `89.47% <33.33%> (-10.53%)` | :arrow_down: |
| [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `87.09% <90.00%> (+0.61%)` | :arrow_up: |
| [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `71.28% <100.00%> (+0.09%)` | :arrow_up: |
| [...i/utilities/deltastreamer/SourceFormatAdapter.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvU291cmNlRm9ybWF0QWRhcHRlci5qYXZh) | `86.84% <100.00%> (+0.35%)` | :arrow_up: |
| ... and [5 more](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [23dbc09...2d5c258](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-864517429
# [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3092](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2d5c258) into [master](https://codecov.io/gh/apache/hudi/commit/23dbc09a0d5d677ed3728dbd5357320590ca2833?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (23dbc09) will **decrease** coverage by `17.76%`.
> The diff coverage is `69.44%`.
[](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3092 +/- ##
=============================================
- Coverage 46.24% 28.48% -17.77%
+ Complexity 5362 1264 -4098
=============================================
Files 920 375 -545
Lines 39842 14335 -25507
Branches 4291 1458 -2833
=============================================
- Hits 18425 4083 -14342
+ Misses 19535 9956 -9579
+ Partials 1882 296 -1586
```
| Flag | Coverage Δ | |
|---|---|---|
| hudicli | `?` | |
| hudiclient | `22.29% <ø> (-8.17%)` | :arrow_down: |
| hudicommon | `?` | |
| hudiflink | `?` | |
| hudihadoopmr | `?` | |
| hudisparkdatasource | `?` | |
| hudisync | `5.38% <ø> (-48.67%)` | :arrow_down: |
| huditimelineservice | `?` | |
| hudiutilities | `58.36% <69.44%> (+0.07%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...g/apache/hudi/utilities/sources/AvroDFSSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb0RGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...apache/hudi/utilities/sources/AvroKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb0thZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...che/hudi/utilities/sources/HiveIncrPullSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSGl2ZUluY3JQdWxsU291cmNlLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/JsonDFSSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkRGU1NvdXJjZS5qYXZh) | `90.00% <0.00%> (-10.00%)` | :arrow_down: |
| [...java/org/apache/hudi/utilities/sources/Source.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvU291cmNlLmphdmE=) | `87.50% <ø> (ø)` | |
| [...a/org/apache/hudi/utilities/sources/SqlSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvU3FsU291cmNlLmphdmE=) | `61.11% <0.00%> (-3.60%)` | :arrow_down: |
| [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `89.47% <33.33%> (-10.53%)` | :arrow_down: |
| [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `87.09% <90.00%> (+0.61%)` | :arrow_up: |
| [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `71.28% <100.00%> (+0.09%)` | :arrow_up: |
| [...i/utilities/deltastreamer/SourceFormatAdapter.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvU291cmNlRm9ybWF0QWRhcHRlci5qYXZh) | `86.84% <100.00%> (+0.35%)` | :arrow_up: |
| ... and [626 more](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [23dbc09...2d5c258](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on a change in pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#discussion_r655327890
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java
##########
@@ -506,6 +509,8 @@ public void refreshTimeline() throws IOException {
return Pair.of(scheduledCompactionInstant, writeStatusRDD);
}
+
+
Review comment:
unnecessary empty lines
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-864517429
# [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3092](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (aa54143) into [master](https://codecov.io/gh/apache/hudi/commit/f73bedd374952580c4d6bc9b2e30607c4efdd538?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f73bedd) will **decrease** coverage by `43.29%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3092 +/- ##
============================================
- Coverage 46.36% 3.07% -43.30%
+ Complexity 5390 82 -5308
============================================
Files 920 280 -640
Lines 39939 10875 -29064
Branches 4309 1106 -3203
============================================
- Hits 18517 334 -18183
+ Misses 19543 10515 -9028
+ Partials 1879 26 -1853
```
| Flag | Coverage Δ | |
|---|---|---|
| hudicli | `?` | |
| hudiclient | `0.00% <ø> (-30.46%)` | :arrow_down: |
| hudicommon | `?` | |
| hudiflink | `?` | |
| hudihadoopmr | `?` | |
| hudisparkdatasource | `?` | |
| hudisync | `5.38% <ø> (-48.67%)` | :arrow_down: |
| huditimelineservice | `?` | |
| hudiutilities | `9.46% <0.00%> (-49.63%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [.../hudi/utilities/callback/SourceCommitCallback.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NhbGxiYWNrL1NvdXJjZUNvbW1pdENhbGxiYWNrLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `0.00% <0.00%> (-71.29%)` | :arrow_down: |
| [...i/utilities/deltastreamer/SourceFormatAdapter.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvU291cmNlRm9ybWF0QWRhcHRlci5qYXZh) | `0.00% <0.00%> (-86.85%)` | :arrow_down: |
| [...apache/hudi/utilities/sources/AvroKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb0thZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...java/org/apache/hudi/utilities/sources/Source.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvU291cmNlLmphdmE=) | `0.00% <ø> (-90.00%)` | :arrow_down: |
| [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `0.00% <0.00%> (-90.06%)` | :arrow_down: |
| [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [765 more](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [f73bedd...aa54143](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-864517429
# [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3092](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (aa54143) into [master](https://codecov.io/gh/apache/hudi/commit/f73bedd374952580c4d6bc9b2e30607c4efdd538?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f73bedd) will **decrease** coverage by `30.06%`.
> The diff coverage is `82.75%`.
[](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3092 +/- ##
=============================================
- Coverage 46.36% 16.29% -30.07%
+ Complexity 5390 473 -4917
=============================================
Files 920 280 -640
Lines 39939 10875 -29064
Branches 4309 1106 -3203
=============================================
- Hits 18517 1772 -16745
+ Misses 19543 8944 -10599
+ Partials 1879 159 -1720
```
| Flag | Coverage Δ | |
|---|---|---|
| hudicli | `?` | |
| hudiclient | `0.00% <ø> (-30.46%)` | :arrow_down: |
| hudicommon | `?` | |
| hudiflink | `?` | |
| hudihadoopmr | `?` | |
| hudisparkdatasource | `?` | |
| hudisync | `5.38% <ø> (-48.67%)` | :arrow_down: |
| huditimelineservice | `?` | |
| hudiutilities | `58.40% <82.75%> (-0.69%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...apache/hudi/utilities/sources/AvroKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb0thZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...java/org/apache/hudi/utilities/sources/Source.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvU291cmNlLmphdmE=) | `87.50% <ø> (-2.50%)` | :arrow_down: |
| [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `87.09% <90.00%> (-2.96%)` | :arrow_down: |
| [.../hudi/utilities/callback/SourceCommitCallback.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NhbGxiYWNrL1NvdXJjZUNvbW1pdENhbGxiYWNrLmphdmE=) | `100.00% <100.00%> (ø)` | |
| [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `70.94% <100.00%> (-0.34%)` | :arrow_down: |
| [...i/utilities/deltastreamer/SourceFormatAdapter.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvU291cmNlRm9ybWF0QWRhcHRlci5qYXZh) | `86.84% <100.00%> (ø)` | |
| [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `100.00% <100.00%> (ø)` | |
| [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [730 more](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [f73bedd...aa54143](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-862136673
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=219",
"triggerID" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"triggerType" : "PUSH"
}, {
"hash" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=290",
"triggerID" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"triggerType" : "PUSH"
}, {
"hash" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=311",
"triggerID" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"triggerType" : "PUSH"
}, {
"hash" : "5a2beb75f789afe8f5874e27b4d7b95427a1ea7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=315",
"triggerID" : "5a2beb75f789afe8f5874e27b4d7b95427a1ea7f",
"triggerType" : "PUSH"
}, {
"hash" : "2cb652554386030acaf569a73f9e074cbc72ffa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=324",
"triggerID" : "2cb652554386030acaf569a73f9e074cbc72ffa3",
"triggerType" : "PUSH"
}, {
"hash" : "1ae031303ebfa92fd153b8b6c0a404d1aac39a0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=393",
"triggerID" : "1ae031303ebfa92fd153b8b6c0a404d1aac39a0a",
"triggerType" : "PUSH"
}, {
"hash" : "e90cbcdc4a4926230d2440b486c52e39850cdb1a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=442",
"triggerID" : "e90cbcdc4a4926230d2440b486c52e39850cdb1a",
"triggerType" : "PUSH"
}, {
"hash" : "2d5c2588f9088daef2800600558ebaa7d6d05212",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2d5c2588f9088daef2800600558ebaa7d6d05212",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e90cbcdc4a4926230d2440b486c52e39850cdb1a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=442)
* 2d5c2588f9088daef2800600558ebaa7d6d05212 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-862136673
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=219",
"triggerID" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"triggerType" : "PUSH"
}, {
"hash" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=290",
"triggerID" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"triggerType" : "PUSH"
}, {
"hash" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=311",
"triggerID" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"triggerType" : "PUSH"
}, {
"hash" : "5a2beb75f789afe8f5874e27b4d7b95427a1ea7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=315",
"triggerID" : "5a2beb75f789afe8f5874e27b4d7b95427a1ea7f",
"triggerType" : "PUSH"
}, {
"hash" : "2cb652554386030acaf569a73f9e074cbc72ffa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=324",
"triggerID" : "2cb652554386030acaf569a73f9e074cbc72ffa3",
"triggerType" : "PUSH"
}, {
"hash" : "1ae031303ebfa92fd153b8b6c0a404d1aac39a0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=393",
"triggerID" : "1ae031303ebfa92fd153b8b6c0a404d1aac39a0a",
"triggerType" : "PUSH"
}, {
"hash" : "e90cbcdc4a4926230d2440b486c52e39850cdb1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=442",
"triggerID" : "e90cbcdc4a4926230d2440b486c52e39850cdb1a",
"triggerType" : "PUSH"
}, {
"hash" : "2d5c2588f9088daef2800600558ebaa7d6d05212",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=443",
"triggerID" : "2d5c2588f9088daef2800600558ebaa7d6d05212",
"triggerType" : "PUSH"
}, {
"hash" : "3b02a03a355c85723b69c9d8b4557c5983c8914f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=461",
"triggerID" : "3b02a03a355c85723b69c9d8b4557c5983c8914f",
"triggerType" : "PUSH"
}, {
"hash" : "3b02a03a355c85723b69c9d8b4557c5983c8914f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=461",
"triggerID" : "869102385",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 3b02a03a355c85723b69c9d8b4557c5983c8914f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=461)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-864517429
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#discussion_r655478803
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java
##########
@@ -473,7 +474,9 @@ public void refreshTimeline() throws IOException {
boolean success = writeClient.commit(instantTime, writeStatusRDD, Option.of(checkpointCommitMetadata));
if (success) {
LOG.info("Commit " + instantTime + " successful!");
-
+ if (this.props.getBoolean(KafkaOffsetGen.Config.ENABLE_KAFKA_COMMIT_OFFSET,KafkaOffsetGen.Config.DEFAULT_ENABLE_KAFKA_COMMIT_OFFSET)) {
Review comment:
code format `KafkaOffsetGen.Config.ENABLE_KAFKA_COMMIT_OFFSET, KafkaOffsetGen.Config.DEFAULT_ENABLE_KAFKA_COMMIT_OFFSET`
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -157,29 +161,23 @@ public static long totalNewMessages(OffsetRange[] ranges) {
private static final String KAFKA_TOPIC_NAME = "hoodie.deltastreamer.source.kafka.topic";
private static final String MAX_EVENTS_FROM_KAFKA_SOURCE_PROP = "hoodie.deltastreamer.kafka.source.maxEvents";
+ public static final String ENABLE_KAFKA_COMMIT_OFFSET = "hoodie.deltastreamer.source.enable.kafka.commit.offset";
Review comment:
can we rename to `hoodie.deltastreamer.source.kafka.enable.commit.offset` to align with `hoodie.deltastreamer.source.kafka.topic` and would possibly rename `hoodie.deltastreamer.kafka.source.maxEvents` to `hoodie.deltastreamer.source.kafka.maxEvents`(should consider the compatibility)
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -157,29 +161,23 @@ public static long totalNewMessages(OffsetRange[] ranges) {
private static final String KAFKA_TOPIC_NAME = "hoodie.deltastreamer.source.kafka.topic";
private static final String MAX_EVENTS_FROM_KAFKA_SOURCE_PROP = "hoodie.deltastreamer.kafka.source.maxEvents";
+ public static final String ENABLE_KAFKA_COMMIT_OFFSET = "hoodie.deltastreamer.source.enable.kafka.commit.offset";
// "auto.offset.reset" is kafka native config param. Do not change the config param name.
public static final String KAFKA_AUTO_OFFSET_RESET = "auto.offset.reset";
private static final KafkaResetOffsetStrategies DEFAULT_KAFKA_AUTO_OFFSET_RESET = KafkaResetOffsetStrategies.LATEST;
public static final long DEFAULT_MAX_EVENTS_FROM_KAFKA_SOURCE = 5000000;
public static long maxEventsFromKafkaSource = DEFAULT_MAX_EVENTS_FROM_KAFKA_SOURCE;
+ public static final Boolean DEFAULT_ENABLE_KAFKA_COMMIT_OFFSET = false;
}
- private final HashMap<String, Object> kafkaParams;
+ private final Map<String, Object> kafkaParams;
private final TypedProperties props;
protected final String topicName;
private KafkaResetOffsetStrategies autoResetValue;
public KafkaOffsetGen(TypedProperties props) {
this.props = props;
-
- kafkaParams = new HashMap<>();
- props.keySet().stream().filter(prop -> {
- // In order to prevent printing unnecessary warn logs, here filter out the hoodie
- // configuration items before passing to kafkaParams
- return !prop.toString().startsWith("hoodie.");
- }).forEach(prop -> {
- kafkaParams.put(prop.toString(), props.get(prop.toString()));
- });
+ kafkaParams = KafkaOffsetGen.excludeHoodieConfigs(props);
Review comment:
remove `KafkaOffsetGen.`?
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -297,7 +295,37 @@ public String getTopicName() {
return topicName;
}
- public HashMap<String, Object> getKafkaParams() {
+ public Map<String, Object> getKafkaParams() {
+ return kafkaParams;
+ }
+
+ private static Map<String, Object> excludeHoodieConfigs(TypedProperties props) {
Review comment:
remove static ?
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java
##########
@@ -473,7 +474,9 @@ public void refreshTimeline() throws IOException {
boolean success = writeClient.commit(instantTime, writeStatusRDD, Option.of(checkpointCommitMetadata));
if (success) {
LOG.info("Commit " + instantTime + " successful!");
-
+ if (this.props.getBoolean(KafkaOffsetGen.Config.ENABLE_KAFKA_COMMIT_OFFSET,KafkaOffsetGen.Config.DEFAULT_ENABLE_KAFKA_COMMIT_OFFSET)) {
+ KafkaOffsetGen.commitOffsetToKafka(checkpointStr, this.props);
Review comment:
remove `KafkaOffsetGen.` ?
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -297,7 +295,37 @@ public String getTopicName() {
return topicName;
}
- public HashMap<String, Object> getKafkaParams() {
+ public Map<String, Object> getKafkaParams() {
+ return kafkaParams;
+ }
+
+ private static Map<String, Object> excludeHoodieConfigs(TypedProperties props) {
+ Map<String, Object> kafkaParams = new HashMap<>();
+ props.keySet().stream().filter(prop -> {
+ // In order to prevent printing unnecessary warn logs, here filter out the hoodie
+ // configuration items before passing to kafkaParams
+ return !prop.toString().startsWith("hoodie.");
+ }).forEach(prop -> {
+ kafkaParams.put(prop.toString(), props.get(prop.toString()));
+ });
return kafkaParams;
}
+
+ /**
+ * Commit offsets to Kafka only after hoodie commit is successful.
+ * @param checkpointStr checkpoint string containing offsets.
+ * @param props properties for Kafka consumer.
+ */
+ public static void commitOffsetToKafka(String checkpointStr, TypedProperties props) {
+ DataSourceUtils.checkRequiredProperties(props, Collections.singletonList(ConsumerConfig.GROUP_ID_CONFIG));
+ Map<TopicPartition, Long> offsetMap = KafkaOffsetGen.CheckpointUtils.strToOffsets(checkpointStr);
+ Map<String, Object> kafkaParams = KafkaOffsetGen.excludeHoodieConfigs(props);
+ Map<TopicPartition, OffsetAndMetadata> offsetAndMetadataMap = new HashMap<>(offsetMap.size());
+ try (KafkaConsumer consumer = new KafkaConsumer(kafkaParams)) {
+ offsetMap.forEach((topicPartition, offset) -> offsetAndMetadataMap.put(topicPartition, new OffsetAndMetadata(offset)));
+ consumer.commitSync(offsetAndMetadataMap);
+ } catch (CommitFailedException | TimeoutException e) {
+ LOG.warn("Committing offsets to Kafka failed, this does not impact processing of records", e);
Review comment:
should we throw exception here instead of just logging ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-862136673
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=219",
"triggerID" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"triggerType" : "PUSH"
}, {
"hash" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=290",
"triggerID" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"triggerType" : "PUSH"
}, {
"hash" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=311",
"triggerID" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"triggerType" : "PUSH"
}, {
"hash" : "5a2beb75f789afe8f5874e27b4d7b95427a1ea7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=315",
"triggerID" : "5a2beb75f789afe8f5874e27b4d7b95427a1ea7f",
"triggerType" : "PUSH"
}, {
"hash" : "2cb652554386030acaf569a73f9e074cbc72ffa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=324",
"triggerID" : "2cb652554386030acaf569a73f9e074cbc72ffa3",
"triggerType" : "PUSH"
}, {
"hash" : "1ae031303ebfa92fd153b8b6c0a404d1aac39a0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=393",
"triggerID" : "1ae031303ebfa92fd153b8b6c0a404d1aac39a0a",
"triggerType" : "PUSH"
}, {
"hash" : "e90cbcdc4a4926230d2440b486c52e39850cdb1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=442",
"triggerID" : "e90cbcdc4a4926230d2440b486c52e39850cdb1a",
"triggerType" : "PUSH"
}, {
"hash" : "2d5c2588f9088daef2800600558ebaa7d6d05212",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=443",
"triggerID" : "2d5c2588f9088daef2800600558ebaa7d6d05212",
"triggerType" : "PUSH"
}, {
"hash" : "3b02a03a355c85723b69c9d8b4557c5983c8914f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=461",
"triggerID" : "3b02a03a355c85723b69c9d8b4557c5983c8914f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3b02a03a355c85723b69c9d8b4557c5983c8914f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=461)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3092: [HUDI-1910] [WIP] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-862136673
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=219",
"triggerID" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cd25e8a35b0dba2de3479d3a5b58f60d7b758e02 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=219)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-864517429
# [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3092](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (1ae0313) into [master](https://codecov.io/gh/apache/hudi/commit/e039e0ff6d04b9b0684dbf29452af8a8dae94b09?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e039e0f) will **decrease** coverage by `17.23%`.
> The diff coverage is `81.81%`.
[](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3092 +/- ##
=============================================
- Coverage 46.13% 28.90% -17.24%
+ Complexity 5325 1259 -4066
=============================================
Files 915 370 -545
Lines 39622 14116 -25506
Branches 4272 1441 -2831
=============================================
- Hits 18281 4080 -14201
+ Misses 19470 9740 -9730
+ Partials 1871 296 -1575
```
| Flag | Coverage Δ | |
|---|---|---|
| hudicli | `?` | |
| hudiclient | `22.29% <ø> (-8.17%)` | :arrow_down: |
| hudicommon | `?` | |
| hudiflink | `?` | |
| hudihadoopmr | `?` | |
| hudisparkdatasource | `?` | |
| hudisync | `6.72% <ø> (-45.01%)` | :arrow_down: |
| huditimelineservice | `?` | |
| hudiutilities | `58.50% <81.81%> (+0.12%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `70.70% <0.00%> (-0.15%)` | :arrow_down: |
| [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `87.09% <90.00%> (-0.18%)` | :arrow_down: |
| [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...rg/apache/hudi/client/bootstrap/BootstrapMode.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9ib290c3RyYXAvQm9vdHN0cmFwTW9kZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...he/hudi/hive/HiveStylePartitionValueExtractor.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN0eWxlUGFydGl0aW9uVmFsdWVFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/hudi/keygen/constant/KeyGeneratorType.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2tleWdlbi9jb25zdGFudC9LZXlHZW5lcmF0b3JUeXBlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...pache/hudi/client/utils/ConcatenatingIterator.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC91dGlscy9Db25jYXRlbmF0aW5nSXRlcmF0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [610 more](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e039e0f...1ae0313](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-864517429
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-862136673
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=219",
"triggerID" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"triggerType" : "PUSH"
}, {
"hash" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=290",
"triggerID" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"triggerType" : "PUSH"
}, {
"hash" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=311",
"triggerID" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"triggerType" : "PUSH"
}, {
"hash" : "5a2beb75f789afe8f5874e27b4d7b95427a1ea7f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5a2beb75f789afe8f5874e27b4d7b95427a1ea7f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=311)
* 5a2beb75f789afe8f5874e27b4d7b95427a1ea7f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-864517429
# [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3092](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f3cc39f) into [master](https://codecov.io/gh/apache/hudi/commit/f73bedd374952580c4d6bc9b2e30607c4efdd538?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f73bedd) will **decrease** coverage by `17.89%`.
> The diff coverage is `82.75%`.
[](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3092 +/- ##
=============================================
- Coverage 46.36% 28.47% -17.90%
+ Complexity 5390 1261 -4129
=============================================
Files 920 376 -544
Lines 39939 14326 -25613
Branches 4309 1458 -2851
=============================================
- Hits 18517 4079 -14438
+ Misses 19543 9951 -9592
+ Partials 1879 296 -1583
```
| Flag | Coverage Δ | |
|---|---|---|
| hudicli | `?` | |
| hudiclient | `22.29% <ø> (-8.17%)` | :arrow_down: |
| hudicommon | `?` | |
| hudiflink | `?` | |
| hudihadoopmr | `?` | |
| hudisparkdatasource | `?` | |
| hudisync | `5.38% <ø> (-48.67%)` | :arrow_down: |
| huditimelineservice | `?` | |
| hudiutilities | `58.40% <82.75%> (-0.69%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...apache/hudi/utilities/sources/AvroKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb0thZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...java/org/apache/hudi/utilities/sources/Source.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvU291cmNlLmphdmE=) | `87.50% <ø> (-2.50%)` | :arrow_down: |
| [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `87.09% <90.00%> (-2.96%)` | :arrow_down: |
| [.../hudi/utilities/callback/SourceCommitCallback.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NhbGxiYWNrL1NvdXJjZUNvbW1pdENhbGxiYWNrLmphdmE=) | `100.00% <100.00%> (ø)` | |
| [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `70.94% <100.00%> (-0.34%)` | :arrow_down: |
| [...i/utilities/deltastreamer/SourceFormatAdapter.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvU291cmNlRm9ybWF0QWRhcHRlci5qYXZh) | `86.84% <100.00%> (ø)` | |
| [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `100.00% <100.00%> (ø)` | |
| [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [634 more](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [f73bedd...f3cc39f](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-862136673
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=219",
"triggerID" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"triggerType" : "PUSH"
}, {
"hash" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=290",
"triggerID" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"triggerType" : "PUSH"
}, {
"hash" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=311",
"triggerID" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"triggerType" : "PUSH"
}, {
"hash" : "5a2beb75f789afe8f5874e27b4d7b95427a1ea7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=315",
"triggerID" : "5a2beb75f789afe8f5874e27b4d7b95427a1ea7f",
"triggerType" : "PUSH"
}, {
"hash" : "2cb652554386030acaf569a73f9e074cbc72ffa3",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=324",
"triggerID" : "2cb652554386030acaf569a73f9e074cbc72ffa3",
"triggerType" : "PUSH"
}, {
"hash" : "1ae031303ebfa92fd153b8b6c0a404d1aac39a0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1ae031303ebfa92fd153b8b6c0a404d1aac39a0a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2cb652554386030acaf569a73f9e074cbc72ffa3 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=324)
* 1ae031303ebfa92fd153b8b6c0a404d1aac39a0a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-862136673
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=219",
"triggerID" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"triggerType" : "PUSH"
}, {
"hash" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=290",
"triggerID" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"triggerType" : "PUSH"
}, {
"hash" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7dc7b1f9434ea3e602e2886e357bc467ab56aaf3 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=290)
* 4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf merged pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
leesf merged pull request #3092:
URL: https://github.com/apache/hudi/pull/3092
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash commented on a change in pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
n3nash commented on a change in pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#discussion_r658465855
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java
##########
@@ -473,7 +476,9 @@ public void refreshTimeline() throws IOException {
boolean success = writeClient.commit(instantTime, writeStatusRDD, Option.of(checkpointCommitMetadata));
if (success) {
LOG.info("Commit " + instantTime + " successful!");
-
+ if (this.props.getBoolean(ENABLE_KAFKA_COMMIT_OFFSET, DEFAULT_ENABLE_KAFKA_COMMIT_OFFSET)) {
Review comment:
@veenaypatil I think the callback approach sounds cleaner and reduces complex if-else logic in DeltaSync. Let's do that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-864517429
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] veenaypatil commented on a change in pull request #3092: [HUDI-1910] [WIP] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
veenaypatil commented on a change in pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#discussion_r654937577
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -297,7 +293,35 @@ public String getTopicName() {
return topicName;
}
- public HashMap<String, Object> getKafkaParams() {
+ public Map<String, Object> getKafkaParams() {
+ return kafkaParams;
+ }
+
+ private static Map<String, Object> excludeHoodieConfigs(TypedProperties props) {
+ Map<String, Object> kafkaParams = new HashMap<>();
+ props.keySet().stream().filter(prop -> {
+ // In order to prevent printing unnecessary warn logs, here filter out the hoodie
+ // configuration items before passing to kafkaParams
+ return !prop.toString().startsWith("hoodie.");
+ }).forEach(prop -> {
+ kafkaParams.put(prop.toString(), props.get(prop.toString()));
+ });
return kafkaParams;
}
+
+ /**
+ * Commit offsets to Kafka only after hoodie commit is successful.
+ * @param checkpointStr checkpoint string containing offsets.
+ * @param props properties for Kafka consumer.
+ */
+ public static void commitOffsetToKafka(String checkpointStr, TypedProperties props) {
+ DataSourceUtils.checkRequiredProperties(props, Collections.singletonList(ConsumerConfig.GROUP_ID_CONFIG));
+ Map<TopicPartition, Long> offsetMap = KafkaOffsetGen.CheckpointUtils.strToOffsets(checkpointStr);
+ Map<String, Object> kafkaParams = KafkaOffsetGen.excludeHoodieConfigs(props);
+ Map<TopicPartition, OffsetAndMetadata> offsetAndMetadataMap = new HashMap<>(offsetMap.size());
+ try (KafkaConsumer consumer = new KafkaConsumer(kafkaParams)) {
+ offsetMap.forEach((topicPartition, offset) -> offsetAndMetadataMap.put(topicPartition, new OffsetAndMetadata(offset)));
+ consumer.commitSync(offsetAndMetadataMap);
Review comment:
somehow the commitAsync method is not working with Unit tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-862136673
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=219",
"triggerID" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"triggerType" : "PUSH"
}, {
"hash" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=290",
"triggerID" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"triggerType" : "PUSH"
}, {
"hash" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=311",
"triggerID" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"triggerType" : "PUSH"
}, {
"hash" : "5a2beb75f789afe8f5874e27b4d7b95427a1ea7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=315",
"triggerID" : "5a2beb75f789afe8f5874e27b4d7b95427a1ea7f",
"triggerType" : "PUSH"
}, {
"hash" : "2cb652554386030acaf569a73f9e074cbc72ffa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=324",
"triggerID" : "2cb652554386030acaf569a73f9e074cbc72ffa3",
"triggerType" : "PUSH"
}, {
"hash" : "1ae031303ebfa92fd153b8b6c0a404d1aac39a0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=393",
"triggerID" : "1ae031303ebfa92fd153b8b6c0a404d1aac39a0a",
"triggerType" : "PUSH"
}, {
"hash" : "e90cbcdc4a4926230d2440b486c52e39850cdb1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=442",
"triggerID" : "e90cbcdc4a4926230d2440b486c52e39850cdb1a",
"triggerType" : "PUSH"
}, {
"hash" : "2d5c2588f9088daef2800600558ebaa7d6d05212",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=443",
"triggerID" : "2d5c2588f9088daef2800600558ebaa7d6d05212",
"triggerType" : "PUSH"
}, {
"hash" : "3b02a03a355c85723b69c9d8b4557c5983c8914f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=461",
"triggerID" : "3b02a03a355c85723b69c9d8b4557c5983c8914f",
"triggerType" : "PUSH"
}, {
"hash" : "3b02a03a355c85723b69c9d8b4557c5983c8914f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=461",
"triggerID" : "869102385",
"triggerType" : "MANUAL"
}, {
"hash" : "aa5414320f403ffa2224798e89cd7aa1752fdd12",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aa5414320f403ffa2224798e89cd7aa1752fdd12",
"triggerType" : "PUSH"
}, {
"hash" : "f3cc39fc2b35b1b935e24f120b14db515748a6b9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=491",
"triggerID" : "f3cc39fc2b35b1b935e24f120b14db515748a6b9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3b02a03a355c85723b69c9d8b4557c5983c8914f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=461)
* aa5414320f403ffa2224798e89cd7aa1752fdd12 UNKNOWN
* f3cc39fc2b35b1b935e24f120b14db515748a6b9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=491)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3092: [HUDI-1910] [WIP] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-862136673
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=219",
"triggerID" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"triggerType" : "PUSH"
}, {
"hash" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cd25e8a35b0dba2de3479d3a5b58f60d7b758e02 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=219)
* 7dc7b1f9434ea3e602e2886e357bc467ab56aaf3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-862136673
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=219",
"triggerID" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"triggerType" : "PUSH"
}, {
"hash" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=290",
"triggerID" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"triggerType" : "PUSH"
}, {
"hash" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=311",
"triggerID" : "4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7dc7b1f9434ea3e602e2886e357bc467ab56aaf3 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=290)
* 4cf4e2e60c4341ef95263e3c2a86d7d0b22d07b5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=311)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-864517429
# [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3092](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2cb6525) into [master](https://codecov.io/gh/apache/hudi/commit/61efc6af79c389ef0a77cda75e4f562ed59ef86b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (61efc6a) will **decrease** coverage by `29.13%`.
> The diff coverage is `81.81%`.
[](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3092 +/- ##
=============================================
- Coverage 45.75% 16.62% -29.14%
+ Complexity 5262 471 -4791
=============================================
Files 909 274 -635
Lines 39356 10666 -28690
Branches 4239 1089 -3150
=============================================
- Hits 18008 1773 -16235
+ Misses 19504 8734 -10770
+ Partials 1844 159 -1685
```
| Flag | Coverage Δ | |
|---|---|---|
| hudicli | `?` | |
| hudiclient | `0.00% <ø> (-30.40%)` | :arrow_down: |
| hudicommon | `?` | |
| hudiflink | `?` | |
| hudihadoopmr | `?` | |
| hudisparkdatasource | `?` | |
| hudisync | `6.72% <ø> (-44.74%)` | :arrow_down: |
| huditimelineservice | `?` | |
| hudiutilities | `58.50% <81.81%> (+1.86%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `70.70% <0.00%> (-0.15%)` | :arrow_down: |
| [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `87.09% <90.00%> (-0.18%)` | :arrow_down: |
| [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...rg/apache/hudi/client/bootstrap/BootstrapMode.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9ib290c3RyYXAvQm9vdHN0cmFwTW9kZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...he/hudi/hive/HiveStylePartitionValueExtractor.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN0eWxlUGFydGl0aW9uVmFsdWVFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/hudi/keygen/constant/KeyGeneratorType.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2tleWdlbi9jb25zdGFudC9LZXlHZW5lcmF0b3JUeXBlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...pache/hudi/client/utils/ConcatenatingIterator.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC91dGlscy9Db25jYXRlbmF0aW5nSXRlcmF0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [710 more](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [61efc6a...2cb6525](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#discussion_r655482935
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -157,29 +161,23 @@ public static long totalNewMessages(OffsetRange[] ranges) {
private static final String KAFKA_TOPIC_NAME = "hoodie.deltastreamer.source.kafka.topic";
private static final String MAX_EVENTS_FROM_KAFKA_SOURCE_PROP = "hoodie.deltastreamer.kafka.source.maxEvents";
+ public static final String ENABLE_KAFKA_COMMIT_OFFSET = "hoodie.deltastreamer.source.enable.kafka.commit.offset";
Review comment:
can we rename to `hoodie.deltastreamer.source.kafka.enable.commit.offset` to align with `hoodie.deltastreamer.source.kafka.topic` and would possibly rename `hoodie.deltastreamer.kafka.source.maxEvents` to `hoodie.deltastreamer.source.kafka.maxEvents`(should consider the compatibility)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3092: [HUDI-1910] [WIP] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-864517429
# [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3092](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7dc7b1f) into [master](https://codecov.io/gh/apache/hudi/commit/61efc6af79c389ef0a77cda75e4f562ed59ef86b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (61efc6a) will **decrease** coverage by `3.12%`.
> The diff coverage is `89.47%`.
[](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3092 +/- ##
============================================
- Coverage 45.75% 42.63% -3.13%
+ Complexity 5262 4314 -948
============================================
Files 909 805 -104
Lines 39356 35215 -4141
Branches 4239 3850 -389
============================================
- Hits 18008 15013 -2995
+ Misses 19504 18566 -938
+ Partials 1844 1636 -208
```
| Flag | Coverage Δ | |
|---|---|---|
| hudicli | `39.95% <ø> (ø)` | |
| hudiclient | `16.44% <ø> (-13.95%)` | :arrow_down: |
| hudicommon | `47.56% <ø> (+<0.01%)` | :arrow_up: |
| hudiflink | `61.33% <ø> (+0.59%)` | :arrow_up: |
| hudihadoopmr | `51.29% <ø> (ø)` | |
| hudisparkdatasource | `66.52% <ø> (+0.04%)` | :arrow_up: |
| hudisync | `6.72% <ø> (-44.74%)` | :arrow_down: |
| huditimelineservice | `?` | |
| hudiutilities | `58.49% <89.47%> (+1.85%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `70.37% <0.00%> (-0.48%)` | :arrow_down: |
| [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `88.42% <100.00%> (+1.15%)` | :arrow_up: |
| [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...he/hudi/hive/HiveStylePartitionValueExtractor.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN0eWxlUGFydGl0aW9uVmFsdWVFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...main/java/org/apache/hudi/hive/HiveSyncConfig.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWcuamF2YQ==) | `0.00% <0.00%> (-97.83%)` | :arrow_down: |
| [.../apache/hudi/hive/MultiPartKeysValueExtractor.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTXVsdGlQYXJ0S2V5c1ZhbHVlRXh0cmFjdG9yLmphdmE=) | `0.00% <0.00%> (-90.91%)` | :arrow_down: |
| [...in/java/org/apache/hudi/hive/SchemaDifference.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvU2NoZW1hRGlmZmVyZW5jZS5qYXZh) | `0.00% <0.00%> (-84.85%)` | :arrow_down: |
| [...in/java/org/apache/hudi/hive/util/ConfigUtils.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9Db25maWdVdGlscy5qYXZh) | `0.00% <0.00%> (-73.92%)` | :arrow_down: |
| [...in/java/org/apache/hudi/hive/HoodieHiveClient.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZUNsaWVudC5qYXZh) | `0.00% <0.00%> (-71.61%)` | :arrow_down: |
| [...va/org/apache/hudi/hive/util/ColumnNameXLator.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9Db2x1bW5OYW1lWExhdG9yLmphdmE=) | `0.00% <0.00%> (-70.00%)` | :arrow_down: |
| ... and [140 more](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [61efc6a...7dc7b1f](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3092: [HUDI-1910] [WIP] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-862136673
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=219",
"triggerID" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"triggerType" : "PUSH"
}, {
"hash" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=290",
"triggerID" : "7dc7b1f9434ea3e602e2886e357bc467ab56aaf3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7dc7b1f9434ea3e602e2886e357bc467ab56aaf3 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=290)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3092: [HUDI-1910] [WIP] Commit Offset to Kafka after successful Hudi commit
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#issuecomment-862136673
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cd25e8a35b0dba2de3479d3a5b58f60d7b758e02",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cd25e8a35b0dba2de3479d3a5b58f60d7b758e02 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org