You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/11/03 13:53:23 UTC
[GitHub] [hudi] liujinhui1994 opened a new pull request #2227: [HUDI-1367] Make delastreamer transition from dfsSouce to kafkasouce
liujinhui1994 opened a new pull request #2227:
URL: https://github.com/apache/hudi/pull/2227
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contributing.html before opening a pull request.*
## What is the purpose of the pull request
*(For example: This pull request adds quick-start document.)*
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2227: [HUDI-1367] Make deltaStreamer transition from dfsSouce to kafkasouce
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2227:
URL: https://github.com/apache/hudi/pull/2227#issuecomment-720892015
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2227?src=pr&el=h1) Report
> Merging [#2227](https://codecov.io/gh/apache/hudi/pull/2227?src=pr&el=desc) (f14edeb) into [master](https://codecov.io/gh/apache/hudi/commit/77ba561a6bacbf9e7dc9c1582eb068f7ad800cd9?el=desc) (77ba561) will **increase** coverage by `18.42%`.
> The diff coverage is `69.23%`.
[](https://codecov.io/gh/apache/hudi/pull/2227?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2227 +/- ##
=============================================
+ Coverage 51.17% 69.59% +18.42%
+ Complexity 3226 557 -2669
=============================================
Files 438 89 -349
Lines 20089 3490 -16599
Branches 2068 433 -1635
=============================================
- Hits 10281 2429 -7852
+ Misses 8961 816 -8145
+ Partials 847 245 -602
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| hudicli | `?` | `?` | |
| hudiclient | `?` | `?` | |
| hudicommon | `?` | `?` | |
| hudiflink | `?` | `?` | |
| hudihadoopmr | `?` | `?` | |
| hudisparkdatasource | `69.75% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudisync | `?` | `?` | |
| huditimelineservice | `?` | `?` | |
| hudiutilities | `69.47% <69.23%> (+0.11%)` | `0.00 <4.00> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2227?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `87.61% <69.23%> (-1.16%)` | `19.00 <4.00> (+3.00)` | :arrow_down: |
| [...a/org/apache/hudi/avro/HoodieAvroWriteSupport.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvYXZyby9Ib29kaWVBdnJvV3JpdGVTdXBwb3J0LmphdmE=) | | | |
| [.../hudi/common/config/SerializableConfiguration.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2NvbmZpZy9TZXJpYWxpemFibGVDb25maWd1cmF0aW9uLmphdmE=) | | | |
| [.../java/org/apache/hudi/common/util/StringUtils.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3V0aWwvU3RyaW5nVXRpbHMuamF2YQ==) | | | |
| [...i/common/util/collection/ExternalSpillableMap.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3V0aWwvY29sbGVjdGlvbi9FeHRlcm5hbFNwaWxsYWJsZU1hcC5qYXZh) | | | |
| [.../apache/hudi/common/config/SerializableSchema.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2NvbmZpZy9TZXJpYWxpemFibGVTY2hlbWEuamF2YQ==) | | | |
| [...che/hudi/common/util/collection/ImmutablePair.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3V0aWwvY29sbGVjdGlvbi9JbW11dGFibGVQYWlyLmphdmE=) | | | |
| [...org/apache/hudi/hadoop/HoodieHFileInputFormat.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL0hvb2RpZUhGaWxlSW5wdXRGb3JtYXQuamF2YQ==) | | | |
| [...che/hudi/common/table/timeline/HoodieTimeline.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3RpbWVsaW5lL0hvb2RpZVRpbWVsaW5lLmphdmE=) | | | |
| [.../org/apache/hudi/common/model/HoodieFileGroup.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL0hvb2RpZUZpbGVHcm91cC5qYXZh) | | | |
| ... and [341 more](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree-more) | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2227: [HUDI-1367] Make deltaStreamer transition from dfsSouce to kafkasouce
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2227:
URL: https://github.com/apache/hudi/pull/2227#issuecomment-720892015
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2227?src=pr&el=h1) Report
> Merging [#2227](https://codecov.io/gh/apache/hudi/pull/2227?src=pr&el=desc) (39adecb) into [master](https://codecov.io/gh/apache/hudi/commit/77ba561a6bacbf9e7dc9c1582eb068f7ad800cd9?el=desc) (77ba561) will **decrease** coverage by `41.51%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/hudi/pull/2227?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2227 +/- ##
============================================
- Coverage 51.17% 9.65% -41.52%
+ Complexity 3226 48 -3178
============================================
Files 438 53 -385
Lines 20089 1936 -18153
Branches 2068 232 -1836
============================================
- Hits 10281 187 -10094
+ Misses 8961 1736 -7225
+ Partials 847 13 -834
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| hudicli | `?` | `?` | |
| hudiclient | `?` | `?` | |
| hudicommon | `?` | `?` | |
| hudiflink | `?` | `?` | |
| hudihadoopmr | `?` | `?` | |
| hudisparkdatasource | `?` | `?` | |
| hudisync | `?` | `?` | |
| huditimelineservice | `?` | `?` | |
| hudiutilities | `9.65% <0.00%> (-59.71%)` | `0.00 <0.00> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2227?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `0.00% <0.00%> (-88.78%)` | `0.00 <0.00> (-16.00)` | |
| [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | |
| [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | |
| [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQ3N2REZTU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-10.00%)` | |
| [...g/apache/hudi/utilities/sources/JsonDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkRGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-6.00%)` | |
| [...pache/hudi/utilities/sources/ParquetDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUGFycXVldERGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-5.00%)` | |
| ... and [416 more](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree-more) | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] wangxianghu commented on pull request #2227: [HUDI-1367] Make delastreamer transition from dfsSouce to kafkasouce
Posted by GitBox <gi...@apache.org>.
wangxianghu commented on pull request #2227:
URL: https://github.com/apache/hudi/pull/2227#issuecomment-721490027
Hi @liujinhui1994 , thanks for your contribution, I left a comment you can consider. the rest LGTM
BTW, add a unit test again this change should be better
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io commented on pull request #2227: [HUDI-1367] Make delastreamer transition from dfsSouce to kafkasouce
Posted by GitBox <gi...@apache.org>.
codecov-io commented on pull request #2227:
URL: https://github.com/apache/hudi/pull/2227#issuecomment-720892015
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2227?src=pr&el=h1) Report
> Merging [#2227](https://codecov.io/gh/apache/hudi/pull/2227?src=pr&el=desc) into [master](https://codecov.io/gh/apache/hudi/commit/a205dd10faba0a83dcb39a12abb6f744b5224992?el=desc) will **decrease** coverage by `0.49%`.
> The diff coverage is `46.15%`.
[](https://codecov.io/gh/apache/hudi/pull/2227?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2227 +/- ##
============================================
- Coverage 53.68% 53.18% -0.50%
+ Complexity 2849 2710 -139
============================================
Files 359 346 -13
Lines 16565 15929 -636
Branches 1782 1635 -147
============================================
- Hits 8893 8472 -421
+ Misses 6915 6769 -146
+ Partials 757 688 -69
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| hudicli | `38.37% <ø> (ø)` | `193.00 <ø> (ø)` | |
| hudiclient | `100.00% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudicommon | `54.81% <ø> (+0.11%)` | `1803.00 <ø> (+9.00)` | |
| hudihadoopmr | `33.05% <ø> (ø)` | `181.00 <ø> (ø)` | |
| hudispark | `65.58% <ø> (-0.37%)` | `155.00 <ø> (-149.00)` | |
| huditimelineservice | `62.29% <ø> (ø)` | `50.00 <ø> (ø)` | |
| hudiutilities | `69.93% <46.15%> (-0.17%)` | `328.00 <4.00> (+1.00)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2227?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `84.76% <46.15%> (-4.02%)` | `17.00 <4.00> (+1.00)` | :arrow_down: |
| [...e/hudi/exception/HoodieDeltaStreamerException.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1zcGFyay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9leGNlcHRpb24vSG9vZGllRGVsdGFTdHJlYW1lckV4Y2VwdGlvbi5qYXZh) | `0.00% <0.00%> (-50.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [...src/main/java/org/apache/hudi/DataSourceUtils.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1zcGFyay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9EYXRhU291cmNlVXRpbHMuamF2YQ==) | `40.95% <0.00%> (-3.81%)` | `19.00% <0.00%> (-3.00%)` | |
| [...main/scala/org/apache/hudi/DataSourceOptions.scala](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1zcGFyay9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2h1ZGkvRGF0YVNvdXJjZU9wdGlvbnMuc2NhbGE=) | `94.91% <0.00%> (ø)` | `0.00% <0.00%> (ø%)` | |
| [...n/scala/org/apache/hudi/HoodieMergeOnReadRDD.scala](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1zcGFyay9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2h1ZGkvSG9vZGllTWVyZ2VPblJlYWRSREQuc2NhbGE=) | `82.19% <0.00%> (ø)` | `10.00% <0.00%> (ø%)` | |
| [...apache/hudi/keygen/NonpartitionedKeyGenerator.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1zcGFyay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9rZXlnZW4vTm9ucGFydGl0aW9uZWRLZXlHZW5lcmF0b3IuamF2YQ==) | | | |
| [...va/org/apache/hudi/keygen/BuiltinKeyGenerator.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1zcGFyay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9rZXlnZW4vQnVpbHRpbktleUdlbmVyYXRvci5qYXZh) | | | |
| [.../org/apache/hudi/keygen/RowKeyGeneratorHelper.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1zcGFyay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9rZXlnZW4vUm93S2V5R2VuZXJhdG9ySGVscGVyLmphdmE=) | | | |
| [...g/apache/hudi/keygen/GlobalDeleteKeyGenerator.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1zcGFyay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9rZXlnZW4vR2xvYmFsRGVsZXRlS2V5R2VuZXJhdG9yLmphdmE=) | | | |
| [...e/hudi/keygen/parser/HoodieDateTimeParserImpl.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1zcGFyay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9rZXlnZW4vcGFyc2VyL0hvb2RpZURhdGVUaW1lUGFyc2VySW1wbC5qYXZh) | | | |
| ... and [10 more](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree-more) | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2227: [HUDI-1367] Make deltaStreamer transition from dfsSouce to kafkasouce
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2227:
URL: https://github.com/apache/hudi/pull/2227#issuecomment-720892015
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2227?src=pr&el=h1) Report
> Merging [#2227](https://codecov.io/gh/apache/hudi/pull/2227?src=pr&el=desc) (43d4f7f) into [master](https://codecov.io/gh/apache/hudi/commit/b0010bf3b449a9d2e01955b0746b795e22e577db?el=desc) (b0010bf) will **decrease** coverage by `0.01%`.
> The diff coverage is `46.15%`.
[](https://codecov.io/gh/apache/hudi/pull/2227?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2227 +/- ##
============================================
- Coverage 51.15% 51.14% -0.02%
- Complexity 3212 3216 +4
============================================
Files 436 438 +2
Lines 19987 20050 +63
Branches 2057 2067 +10
============================================
+ Hits 10224 10254 +30
- Misses 8922 8951 +29
- Partials 841 845 +4
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| hudicli | `36.87% <ø> (-0.04%)` | `0.00 <ø> (ø)` | |
| hudiclient | `100.00% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudicommon | `51.39% <ø> (+<0.01%)` | `0.00 <ø> (ø)` | |
| hudiflink | `45.44% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudihadoopmr | `33.16% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudisparkdatasource | `69.75% <ø> (+0.01%)` | `0.00 <ø> (ø)` | |
| hudisync | `48.61% <ø> (ø)` | `0.00 <ø> (ø)` | |
| huditimelineservice | `66.49% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudiutilities | `69.21% <46.15%> (-0.18%)` | `0.00 <4.00> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2227?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `84.76% <46.15%> (-4.02%)` | `17.00 <4.00> (+1.00)` | :arrow_down: |
| [...java/org/apache/hudi/common/util/CleanerUtils.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3V0aWwvQ2xlYW5lclV0aWxzLmphdmE=) | `47.72% <0.00%> (-15.91%)` | `6.00% <0.00%> (ø%)` | |
| [...rg/apache/hudi/cli/commands/SavepointsCommand.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1jbGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpL2NvbW1hbmRzL1NhdmVwb2ludHNDb21tYW5kLmphdmE=) | `13.84% <0.00%> (-0.44%)` | `3.00% <0.00%> (ø%)` | |
| [...i/utilities/deltastreamer/HoodieDeltaStreamer.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvSG9vZGllRGVsdGFTdHJlYW1lci5qYXZh) | `68.72% <0.00%> (-0.26%)` | `18.00% <0.00%> (ø%)` | |
| [...pache/hudi/cli/commands/FileSystemViewCommand.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1jbGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpL2NvbW1hbmRzL0ZpbGVTeXN0ZW1WaWV3Q29tbWFuZC5qYXZh) | `82.45% <0.00%> (-0.16%)` | `19.00% <0.00%> (ø%)` | |
| [...che/hudi/common/table/timeline/HoodieTimeline.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3RpbWVsaW5lL0hvb2RpZVRpbWVsaW5lLmphdmE=) | `91.30% <0.00%> (ø)` | `44.00% <0.00%> (ø%)` | |
| [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `70.00% <0.00%> (ø)` | `50.00% <0.00%> (ø%)` | |
| [...di/common/table/timeline/HoodieActiveTimeline.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3RpbWVsaW5lL0hvb2RpZUFjdGl2ZVRpbWVsaW5lLmphdmE=) | `70.77% <0.00%> (ø)` | `43.00% <0.00%> (ø%)` | |
| [...pache/hudi/exception/HoodieHeartbeatException.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhjZXB0aW9uL0hvb2RpZUhlYXJ0YmVhdEV4Y2VwdGlvbi5qYXZh) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (?%)` | |
| [...common/model/HoodieFailedWritesCleaningPolicy.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL0hvb2RpZUZhaWxlZFdyaXRlc0NsZWFuaW5nUG9saWN5LmphdmE=) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (?%)` | |
| ... and [5 more](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree-more) | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #2227: [HUDI-1367] Make deltaStreamer transition from dfsSouce to kafkasouce
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #2227:
URL: https://github.com/apache/hudi/pull/2227#issuecomment-782770366
@liujinhui1994 : I pushed a commit to this PR with tests to delta streamer to test transition from parquet dfs to kaka dfs :) Hope that's fine with you. If you plan to make more changes, feel free to pull in latest commits and add more commits to it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2227: [HUDI-1367] Make deltaStreamer transition from dfsSouce to kafkasouce
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2227:
URL: https://github.com/apache/hudi/pull/2227#issuecomment-720892015
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2227?src=pr&el=h1) Report
> Merging [#2227](https://codecov.io/gh/apache/hudi/pull/2227?src=pr&el=desc) (43d4f7f) into [master](https://codecov.io/gh/apache/hudi/commit/b0010bf3b449a9d2e01955b0746b795e22e577db?el=desc) (b0010bf) will **decrease** coverage by `0.18%`.
> The diff coverage is `46.15%`.
[](https://codecov.io/gh/apache/hudi/pull/2227?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2227 +/- ##
============================================
- Coverage 51.15% 50.96% -0.19%
+ Complexity 3212 3033 -179
============================================
Files 436 417 -19
Lines 19987 18616 -1371
Branches 2057 1933 -124
============================================
- Hits 10224 9488 -736
+ Misses 8922 8349 -573
+ Partials 841 779 -62
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| hudicli | `36.87% <ø> (-0.04%)` | `0.00 <ø> (ø)` | |
| hudiclient | `100.00% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudicommon | `51.39% <ø> (+<0.01%)` | `0.00 <ø> (ø)` | |
| hudiflink | `45.44% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudihadoopmr | `33.16% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudisparkdatasource | `69.75% <ø> (+0.01%)` | `0.00 <ø> (ø)` | |
| hudisync | `?` | `?` | |
| huditimelineservice | `?` | `?` | |
| hudiutilities | `69.21% <46.15%> (-0.18%)` | `0.00 <4.00> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2227?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `84.76% <46.15%> (-4.02%)` | `17.00 <4.00> (+1.00)` | :arrow_down: |
| [...java/org/apache/hudi/common/util/CleanerUtils.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3V0aWwvQ2xlYW5lclV0aWxzLmphdmE=) | `47.72% <0.00%> (-15.91%)` | `6.00% <0.00%> (ø%)` | |
| [...rg/apache/hudi/cli/commands/SavepointsCommand.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1jbGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpL2NvbW1hbmRzL1NhdmVwb2ludHNDb21tYW5kLmphdmE=) | `13.84% <0.00%> (-0.44%)` | `3.00% <0.00%> (ø%)` | |
| [...i/utilities/deltastreamer/HoodieDeltaStreamer.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvSG9vZGllRGVsdGFTdHJlYW1lci5qYXZh) | `68.72% <0.00%> (-0.26%)` | `18.00% <0.00%> (ø%)` | |
| [...pache/hudi/cli/commands/FileSystemViewCommand.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1jbGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpL2NvbW1hbmRzL0ZpbGVTeXN0ZW1WaWV3Q29tbWFuZC5qYXZh) | `82.45% <0.00%> (-0.16%)` | `19.00% <0.00%> (ø%)` | |
| [...che/hudi/common/table/timeline/HoodieTimeline.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3RpbWVsaW5lL0hvb2RpZVRpbWVsaW5lLmphdmE=) | `91.30% <0.00%> (ø)` | `44.00% <0.00%> (ø%)` | |
| [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `70.00% <0.00%> (ø)` | `50.00% <0.00%> (ø%)` | |
| [...di/common/table/timeline/HoodieActiveTimeline.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3RpbWVsaW5lL0hvb2RpZUFjdGl2ZVRpbWVsaW5lLmphdmE=) | `70.77% <0.00%> (ø)` | `43.00% <0.00%> (ø%)` | |
| [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | | | |
| [.../hive/SlashEncodedHourPartitionValueExtractor.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvU2xhc2hFbmNvZGVkSG91clBhcnRpdGlvblZhbHVlRXh0cmFjdG9yLmphdmE=) | | | |
| ... and [26 more](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree-more) | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on pull request #2227: [HUDI-1367] Make deltaStreamer transition from dfsSouce to kafkasouce
Posted by GitBox <gi...@apache.org>.
yanghua commented on pull request #2227:
URL: https://github.com/apache/hudi/pull/2227#issuecomment-782611052
@nsivabalan Let you take over this PR. Please go ahead.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] wangxianghu edited a comment on pull request #2227: [HUDI-1367] Make delastreamer transition from dfsSouce to kafkasouce
Posted by GitBox <gi...@apache.org>.
wangxianghu edited a comment on pull request #2227:
URL: https://github.com/apache/hudi/pull/2227#issuecomment-721490027
Hi @liujinhui1994 , thanks for your contribution, I left a comment you can consider. the rest LGTM
BTW, add a unit test against this change should be better
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #2227: [HUDI-1367] Make deltaStreamer transition from dfsSouce to kafkasouce
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #2227:
URL: https://github.com/apache/hudi/pull/2227#discussion_r579684280
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -156,15 +163,18 @@ public static long totalNewMessages(OffsetRange[] ranges) {
private final HashMap<String, Object> kafkaParams;
private final TypedProperties props;
protected final String topicName;
+ private final KafkaResetOffsetStrategies autoResetValue;
public KafkaOffsetGen(TypedProperties props) {
this.props = props;
+
kafkaParams = new HashMap<>();
for (Object prop : props.keySet()) {
kafkaParams.put(prop.toString(), props.get(prop.toString()));
}
DataSourceUtils.checkRequiredProperties(props, Collections.singletonList(Config.KAFKA_TOPIC_NAME));
topicName = props.getString(Config.KAFKA_TOPIC_NAME);
+ autoResetValue = KafkaResetOffsetStrategies.valueOf(props.getString("auto.offset.reset", Config.DEFAULT_AUTO_RESET_OFFSET.toString()).toUpperCase());
Review comment:
lets add this to Config (in line 154 in this file)
Also, lets name this our usual way.
"hoodie.deltastreamer.source.kafka.auto.offset.reset"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] liujinhui1994 commented on a change in pull request #2227: [HUDI-1367] Make deltaStreamer transition from dfsSouce to kafkasouce
Posted by GitBox <gi...@apache.org>.
liujinhui1994 commented on a change in pull request #2227:
URL: https://github.com/apache/hudi/pull/2227#discussion_r579630158
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -227,12 +232,23 @@ public KafkaOffsetGen(TypedProperties props) {
// else return earliest offsets
private Map<TopicPartition, Long> checkupValidOffsets(KafkaConsumer consumer,
Option<String> lastCheckpointStr, Set<TopicPartition> topicPartitions) {
- Map<TopicPartition, Long> checkpointOffsets = CheckpointUtils.strToOffsets(lastCheckpointStr.get());
Map<TopicPartition, Long> earliestOffsets = consumer.beginningOffsets(topicPartitions);
+ if (checkTopicCheckPoint(lastCheckpointStr)) {
+ Map<TopicPartition, Long> checkpointOffsets = CheckpointUtils.strToOffsets(lastCheckpointStr.get());
+ boolean checkpointOffsetReseter = checkpointOffsets.entrySet().stream()
+ .anyMatch(offset -> offset.getValue() < earliestOffsets.get(offset.getKey()));
+ return checkpointOffsetReseter ? earliestOffsets : checkpointOffsets;
+ }
+
+ switch (autoResetValue) {
+ case EARLIEST:
+ return earliestOffsets;
+ case LATEST:
+ return consumer.endOffsets(topicPartitions);
+ default:
Review comment:
I also don’t think it’s necessary to add NONE
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -257,6 +273,11 @@ public boolean checkTopicExists(KafkaConsumer consumer) {
return result.containsKey(topicName);
}
+ public boolean checkTopicCheckPoint(Option<String> lastCheckpointStr) {
Review comment:
thanks for pointing out
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] liujinhui1994 commented on pull request #2227: [HUDI-1367] Make delastreamer transition from dfsSouce to kafkasouce
Posted by GitBox <gi...@apache.org>.
liujinhui1994 commented on pull request #2227:
URL: https://github.com/apache/hudi/pull/2227#issuecomment-745731566
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #2227: [HUDI-1367] Make deltaStreamer transition from dfsSouce to kafkasouce
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #2227:
URL: https://github.com/apache/hudi/pull/2227#issuecomment-782770387
> @nsivabalan Let you take over this PR. Please go ahead.
Sure.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2227: [HUDI-1367] Make deltaStreamer transition from dfsSouce to kafkasouce
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2227:
URL: https://github.com/apache/hudi/pull/2227#issuecomment-720892015
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2227?src=pr&el=h1) Report
> Merging [#2227](https://codecov.io/gh/apache/hudi/pull/2227?src=pr&el=desc) (f14edeb) into [master](https://codecov.io/gh/apache/hudi/commit/77ba561a6bacbf9e7dc9c1582eb068f7ad800cd9?el=desc) (77ba561) will **increase** coverage by `0.01%`.
> The diff coverage is `69.23%`.
[](https://codecov.io/gh/apache/hudi/pull/2227?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2227 +/- ##
============================================
+ Coverage 51.17% 51.19% +0.01%
- Complexity 3226 3232 +6
============================================
Files 438 438
Lines 20089 20096 +7
Branches 2068 2070 +2
============================================
+ Hits 10281 10288 +7
- Misses 8961 8962 +1
+ Partials 847 846 -1
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| hudicli | `36.87% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudiclient | `100.00% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudicommon | `51.35% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudiflink | `46.38% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudihadoopmr | `33.16% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudisparkdatasource | `69.75% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudisync | `48.61% <ø> (ø)` | `0.00 <ø> (ø)` | |
| huditimelineservice | `66.49% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudiutilities | `69.47% <69.23%> (+0.11%)` | `0.00 <4.00> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2227?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `87.61% <69.23%> (-1.16%)` | `19.00 <4.00> (+3.00)` | :arrow_down: |
| [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `70.71% <0.00%> (+0.71%)` | `52.00% <0.00%> (+2.00%)` | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on pull request #2227: [HUDI-1367] Make delastreamer transition from dfsSouce to kafkasouce
Posted by GitBox <gi...@apache.org>.
yanghua commented on pull request #2227:
URL: https://github.com/apache/hudi/pull/2227#issuecomment-720964360
@liujinhui1994 Please add more description of your motivation and purpose. then @wangxianghu Please help to review this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2227: [HUDI-1367] Make deltaStreamer transition from dfsSouce to kafkasouce
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2227:
URL: https://github.com/apache/hudi/pull/2227#issuecomment-720892015
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2227?src=pr&el=h1) Report
> Merging [#2227](https://codecov.io/gh/apache/hudi/pull/2227?src=pr&el=desc) (8d1257d) into [master](https://codecov.io/gh/apache/hudi/commit/b0010bf3b449a9d2e01955b0746b795e22e577db?el=desc) (b0010bf) will **decrease** coverage by `6.37%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/hudi/pull/2227?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2227 +/- ##
============================================
- Coverage 51.15% 44.77% -6.38%
+ Complexity 3212 2724 -488
============================================
Files 436 417 -19
Lines 19987 18616 -1371
Branches 2057 1933 -124
============================================
- Hits 10224 8335 -1889
- Misses 8922 9623 +701
+ Partials 841 658 -183
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| hudicli | `36.87% <ø> (-0.04%)` | `0.00 <ø> (ø)` | |
| hudiclient | `100.00% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudicommon | `51.39% <ø> (+<0.01%)` | `0.00 <ø> (ø)` | |
| hudiflink | `45.44% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudihadoopmr | `33.16% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudisparkdatasource | `69.75% <ø> (+0.01%)` | `0.00 <ø> (ø)` | |
| hudisync | `?` | `?` | |
| huditimelineservice | `?` | `?` | |
| hudiutilities | `9.65% <0.00%> (-59.74%)` | `0.00 <0.00> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2227?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `0.00% <0.00%> (-88.78%)` | `0.00 <0.00> (-16.00)` | |
| [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | |
| [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | |
| [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQ3N2REZTU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-10.00%)` | |
| [...g/apache/hudi/utilities/sources/JsonDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkRGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-6.00%)` | |
| [...pache/hudi/utilities/sources/ParquetDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUGFycXVldERGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-5.00%)` | |
| ... and [64 more](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree-more) | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan edited a comment on pull request #2227: [HUDI-1367] Make deltaStreamer transition from dfsSouce to kafkasouce
Posted by GitBox <gi...@apache.org>.
nsivabalan edited a comment on pull request #2227:
URL: https://github.com/apache/hudi/pull/2227#issuecomment-785545020
@yanghua : Have addressed all feedback and have added tests. Can you skim through once. Will wait for a day. if not, will go ahead and merge this in.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on pull request #2227: [HUDI-1367] Make deltaStreamer transition from dfsSouce to kafkasouce
Posted by GitBox <gi...@apache.org>.
yanghua commented on pull request #2227:
URL: https://github.com/apache/hudi/pull/2227#issuecomment-774469401
@liujinhui1994 and guys let's make this PR done ASAP?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] liujinhui1994 removed a comment on pull request #2227: [HUDI-1367] Make deltaStreamer transition from dfsSouce to kafkasouce
Posted by GitBox <gi...@apache.org>.
liujinhui1994 removed a comment on pull request #2227:
URL: https://github.com/apache/hudi/pull/2227#issuecomment-784032481
> @ liujinhui1994：我通过对三角流的测试来推动对此PR的提交，以测试从镶木地板dfs到kaka dfs的过渡:)希望您一切都好。如果您打算进行更多更改，请随时获取最新的提交并向其中添加更多的提交。
ok thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on a change in pull request #2227: [HUDI-1367] Make deltaStreamer transition from dfsSouce to kafkasouce
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #2227:
URL: https://github.com/apache/hudi/pull/2227#discussion_r571428718
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -49,6 +51,8 @@
private static final Logger LOG = LogManager.getLogger(KafkaOffsetGen.class);
+ private final Pattern pattern = Pattern.compile(".*,.*:.*");
Review comment:
Please add some comments to describe the pattern? i.e. an example checkpoint string?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2227: [HUDI-1367] Make deltaStreamer transition from dfsSouce to kafkasouce
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2227:
URL: https://github.com/apache/hudi/pull/2227#issuecomment-720892015
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2227?src=pr&el=h1) Report
> Merging [#2227](https://codecov.io/gh/apache/hudi/pull/2227?src=pr&el=desc) (439ea88) into [master](https://codecov.io/gh/apache/hudi/commit/b0010bf3b449a9d2e01955b0746b795e22e577db?el=desc) (b0010bf) will **decrease** coverage by `41.49%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/hudi/pull/2227?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2227 +/- ##
============================================
- Coverage 51.15% 9.65% -41.50%
+ Complexity 3212 48 -3164
============================================
Files 436 53 -383
Lines 19987 1936 -18051
Branches 2057 232 -1825
============================================
- Hits 10224 187 -10037
+ Misses 8922 1736 -7186
+ Partials 841 13 -828
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| hudicli | `?` | `?` | |
| hudiclient | `?` | `?` | |
| hudicommon | `?` | `?` | |
| hudiflink | `?` | `?` | |
| hudihadoopmr | `?` | `?` | |
| hudisparkdatasource | `?` | `?` | |
| hudisync | `?` | `?` | |
| huditimelineservice | `?` | `?` | |
| hudiutilities | `9.65% <0.00%> (-59.74%)` | `0.00 <0.00> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2227?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `0.00% <0.00%> (-88.78%)` | `0.00 <0.00> (-16.00)` | |
| [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | |
| [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | |
| [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQ3N2REZTU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-10.00%)` | |
| [...g/apache/hudi/utilities/sources/JsonDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkRGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-6.00%)` | |
| [...pache/hudi/utilities/sources/ParquetDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUGFycXVldERGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-5.00%)` | |
| ... and [414 more](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree-more) | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan merged pull request #2227: [HUDI-1367] Make deltaStreamer transition from dfsSouce to kafkasouce
Posted by GitBox <gi...@apache.org>.
nsivabalan merged pull request #2227:
URL: https://github.com/apache/hudi/pull/2227
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] liujinhui1994 commented on a change in pull request #2227: [HUDI-1367] Make deltaStreamer transition from dfsSouce to kafkasouce
Posted by GitBox <gi...@apache.org>.
liujinhui1994 commented on a change in pull request #2227:
URL: https://github.com/apache/hudi/pull/2227#discussion_r580881266
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -156,15 +163,18 @@ public static long totalNewMessages(OffsetRange[] ranges) {
private final HashMap<String, Object> kafkaParams;
private final TypedProperties props;
protected final String topicName;
+ private final KafkaResetOffsetStrategies autoResetValue;
public KafkaOffsetGen(TypedProperties props) {
this.props = props;
+
kafkaParams = new HashMap<>();
for (Object prop : props.keySet()) {
kafkaParams.put(prop.toString(), props.get(prop.toString()));
}
DataSourceUtils.checkRequiredProperties(props, Collections.singletonList(Config.KAFKA_TOPIC_NAME));
topicName = props.getString(Config.KAFKA_TOPIC_NAME);
+ autoResetValue = KafkaResetOffsetStrategies.valueOf(props.getString("auto.offset.reset", Config.DEFAULT_AUTO_RESET_OFFSET.toString()).toUpperCase());
Review comment:
ok
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2227: [HUDI-1367] Make delastreamer transition from dfsSouce to kafkasouce
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2227:
URL: https://github.com/apache/hudi/pull/2227#issuecomment-720892015
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2227?src=pr&el=h1) Report
> Merging [#2227](https://codecov.io/gh/apache/hudi/pull/2227?src=pr&el=desc) (d3fe5e0) into [master](https://codecov.io/gh/apache/hudi/commit/a205dd10faba0a83dcb39a12abb6f744b5224992?el=desc) (a205dd1) will **decrease** coverage by `0.14%`.
> The diff coverage is `50.00%`.
[](https://codecov.io/gh/apache/hudi/pull/2227?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2227 +/- ##
============================================
- Coverage 53.68% 53.53% -0.15%
+ Complexity 2849 2771 -78
============================================
Files 359 348 -11
Lines 16565 16116 -449
Branches 1782 1645 -137
============================================
- Hits 8893 8628 -265
+ Misses 6915 6789 -126
+ Partials 757 699 -58
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| hudicli | `38.37% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudiclient | `100.00% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudicommon | `55.33% <ø> (+0.62%)` | `0.00 <ø> (ø)` | |
| hudihadoopmr | `32.94% <ø> (-0.12%)` | `0.00 <ø> (ø)` | |
| hudispark | `65.58% <ø> (-0.37%)` | `0.00 <ø> (ø)` | |
| huditimelineservice | `65.30% <ø> (+3.01%)` | `0.00 <ø> (ø)` | |
| hudiutilities | `69.93% <50.00%> (-0.17%)` | `0.00 <4.00> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2227?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `84.76% <50.00%> (-4.02%)` | `17.00 <4.00> (+1.00)` | :arrow_down: |
| [...e/hudi/exception/HoodieDeltaStreamerException.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1zcGFyay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9leGNlcHRpb24vSG9vZGllRGVsdGFTdHJlYW1lckV4Y2VwdGlvbi5qYXZh) | `0.00% <0.00%> (-50.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [...i/common/table/view/HoodieTableFileSystemView.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3ZpZXcvSG9vZGllVGFibGVGaWxlU3lzdGVtVmlldy5qYXZh) | `66.66% <0.00%> (-8.61%)` | `34.00% <0.00%> (+3.00%)` | :arrow_down: |
| [...src/main/java/org/apache/hudi/DataSourceUtils.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1zcGFyay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9EYXRhU291cmNlVXRpbHMuamF2YQ==) | `40.95% <0.00%> (-3.81%)` | `19.00% <0.00%> (-3.00%)` | |
| [...che/hudi/common/table/log/HoodieLogFileReader.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL2xvZy9Ib29kaWVMb2dGaWxlUmVhZGVyLmphdmE=) | `71.06% <0.00%> (-2.48%)` | `23.00% <0.00%> (-2.00%)` | |
| [.../hadoop/utils/HoodieRealtimeRecordReaderUtils.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3V0aWxzL0hvb2RpZVJlYWx0aW1lUmVjb3JkUmVhZGVyVXRpbHMuamF2YQ==) | `73.04% <0.00%> (-1.52%)` | `30.00% <0.00%> (+1.00%)` | :arrow_down: |
| [...common/table/view/PriorityBasedFileSystemView.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3ZpZXcvUHJpb3JpdHlCYXNlZEZpbGVTeXN0ZW1WaWV3LmphdmE=) | `97.10% <0.00%> (-1.43%)` | `33.00% <0.00%> (ø%)` | |
| [.../common/table/view/RocksDbBasedFileSystemView.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3ZpZXcvUm9ja3NEYkJhc2VkRmlsZVN5c3RlbVZpZXcuamF2YQ==) | `80.40% <0.00%> (-0.06%)` | `72.00% <0.00%> (+10.00%)` | :arrow_down: |
| [...main/scala/org/apache/hudi/DataSourceOptions.scala](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1zcGFyay9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2h1ZGkvRGF0YVNvdXJjZU9wdGlvbnMuc2NhbGE=) | `94.91% <0.00%> (ø)` | `0.00% <0.00%> (ø%)` | |
| [...n/scala/org/apache/hudi/HoodieMergeOnReadRDD.scala](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1zcGFyay9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2h1ZGkvSG9vZGllTWVyZ2VPblJlYWRSREQuc2NhbGE=) | `82.19% <0.00%> (ø)` | `10.00% <0.00%> (ø%)` | |
| ... and [36 more](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree-more) | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] wangxianghu commented on pull request #2227: [HUDI-1367] Make delastreamer transition from dfsSouce to kafkasouce
Posted by GitBox <gi...@apache.org>.
wangxianghu commented on pull request #2227:
URL: https://github.com/apache/hudi/pull/2227#issuecomment-731516353
Hi @liujinhui1994 is this pr ready for review ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on a change in pull request #2227: [HUDI-1367] Make deltaStreamer transition from dfsSouce to kafkasouce
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #2227:
URL: https://github.com/apache/hudi/pull/2227#discussion_r571427186
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -257,6 +273,11 @@ public boolean checkTopicExists(KafkaConsumer consumer) {
return result.containsKey(topicName);
}
+ public boolean checkTopicCheckPoint(Option<String> lastCheckpointStr) {
Review comment:
`checkTopicCheckpoint ` looks better？
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #2227: [HUDI-1367] Make delastreamer transition from dfsSouce to kafkasouce
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #2227:
URL: https://github.com/apache/hudi/pull/2227#issuecomment-751317235
@vinothchandar : do you think we should make this release-blocker for 0.7.0 ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] wangxianghu commented on pull request #2227: [HUDI-1367] Make delastreamer transition from dfsSouce to kafkasouce
Posted by GitBox <gi...@apache.org>.
wangxianghu commented on pull request #2227:
URL: https://github.com/apache/hudi/pull/2227#issuecomment-721031091
> @liujinhui1994 Please add more description of your motivation and purpose. then @wangxianghu Please help to review this PR.
Ack, will do the review when understanding the purpose of this pr
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] liujinhui1994 commented on pull request #2227: [HUDI-1367] Make deltaStreamer transition from dfsSouce to kafkasouce
Posted by GitBox <gi...@apache.org>.
liujinhui1994 commented on pull request #2227:
URL: https://github.com/apache/hudi/pull/2227#issuecomment-784041273
> @liujinhui1994 : I pushed a commit to this PR with tests to delta streamer to test transition from parquet dfs to kaka dfs :) Hope that's fine with you. If you plan to make more changes, feel free to pull in latest commits and add more commits to it.
ok thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] liujinhui1994 commented on pull request #2227: [HUDI-1367] Make deltaStreamer transition from dfsSouce to kafkasouce
Posted by GitBox <gi...@apache.org>.
liujinhui1994 commented on pull request #2227:
URL: https://github.com/apache/hudi/pull/2227#issuecomment-784032481
> @ liujinhui1994：我通过对三角流的测试来推动对此PR的提交，以测试从镶木地板dfs到kaka dfs的过渡:)希望您一切都好。如果您打算进行更多更改，请随时获取最新的提交并向其中添加更多的提交。
ok thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan edited a comment on pull request #2227: [HUDI-1367] Make deltaStreamer transition from dfsSouce to kafkasouce
Posted by GitBox <gi...@apache.org>.
nsivabalan edited a comment on pull request #2227:
URL: https://github.com/apache/hudi/pull/2227#issuecomment-782770387
> @nsivabalan Let you take over this PR. Please go ahead.
Sure. by take over you mean, just reviewing right? or you want me to address all comments by myself and get it to finish line?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #2227: [HUDI-1367] Make deltaStreamer transition from dfsSouce to kafkasouce
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #2227:
URL: https://github.com/apache/hudi/pull/2227#issuecomment-785545020
@yanghua : Have addressed all feedback and have added tests. Can you skim through once.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2227: [HUDI-1367] Make deltaStreamer transition from dfsSouce to kafkasouce
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2227:
URL: https://github.com/apache/hudi/pull/2227#issuecomment-720892015
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2227?src=pr&el=h1) Report
> Merging [#2227](https://codecov.io/gh/apache/hudi/pull/2227?src=pr&el=desc) (3ea040e) into [master](https://codecov.io/gh/apache/hudi/commit/77ba561a6bacbf9e7dc9c1582eb068f7ad800cd9?el=desc) (77ba561) will **increase** coverage by `18.37%`.
> The diff coverage is `68.18%`.
[](https://codecov.io/gh/apache/hudi/pull/2227?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2227 +/- ##
=============================================
+ Coverage 51.17% 69.54% +18.37%
+ Complexity 3226 363 -2863
=============================================
Files 438 53 -385
Lines 20089 1944 -18145
Branches 2068 235 -1833
=============================================
- Hits 10281 1352 -8929
+ Misses 8961 458 -8503
+ Partials 847 134 -713
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| hudicli | `?` | `?` | |
| hudiclient | `?` | `?` | |
| hudicommon | `?` | `?` | |
| hudiflink | `?` | `?` | |
| hudihadoopmr | `?` | `?` | |
| hudisparkdatasource | `?` | `?` | |
| hudisync | `?` | `?` | |
| huditimelineservice | `?` | `?` | |
| hudiutilities | `69.54% <68.18%> (+0.18%)` | `0.00 <5.00> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2227?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `85.84% <68.18%> (-2.94%)` | `20.00 <5.00> (+4.00)` | :arrow_down: |
| [...rg/apache/hudi/common/bloom/SimpleBloomFilter.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2Jsb29tL1NpbXBsZUJsb29tRmlsdGVyLmphdmE=) | | | |
| [...ache/hudi/cli/commands/ArchivedCommitsCommand.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1jbGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpL2NvbW1hbmRzL0FyY2hpdmVkQ29tbWl0c0NvbW1hbmQuamF2YQ==) | | | |
| [...rg/apache/hudi/exception/HoodieIndexException.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhjZXB0aW9uL0hvb2RpZUluZGV4RXhjZXB0aW9uLmphdmE=) | | | |
| [.../apache/hudi/hadoop/RecordReaderValueIterator.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL1JlY29yZFJlYWRlclZhbHVlSXRlcmF0b3IuamF2YQ==) | | | |
| [.../hudi/common/bloom/InternalDynamicBloomFilter.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2Jsb29tL0ludGVybmFsRHluYW1pY0Jsb29tRmlsdGVyLmphdmE=) | | | |
| [.../org/apache/hudi/cli/commands/MetadataCommand.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1jbGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpL2NvbW1hbmRzL01ldGFkYXRhQ29tbWFuZC5qYXZh) | | | |
| [...rg/apache/hudi/common/table/HoodieTableConfig.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL0hvb2RpZVRhYmxlQ29uZmlnLmphdmE=) | | | |
| [.../main/java/org/apache/hudi/util/AvroConvertor.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS91dGlsL0F2cm9Db252ZXJ0b3IuamF2YQ==) | | | |
| [...op/realtime/HoodieCombineRealtimeRecordReader.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3JlYWx0aW1lL0hvb2RpZUNvbWJpbmVSZWFsdGltZVJlY29yZFJlYWRlci5qYXZh) | | | |
| ... and [378 more](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree-more) | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on pull request #2227: [HUDI-1367] Make delastreamer transition from dfsSouce to kafkasouce
Posted by GitBox <gi...@apache.org>.
yanghua commented on pull request #2227:
URL: https://github.com/apache/hudi/pull/2227#issuecomment-743731390
What's the status of this PR? @liujinhui1994
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on a change in pull request #2227: [HUDI-1367] Make deltaStreamer transition from dfsSouce to kafkasouce
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #2227:
URL: https://github.com/apache/hudi/pull/2227#discussion_r571428795
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -156,15 +160,18 @@ public static long totalNewMessages(OffsetRange[] ranges) {
private final HashMap<String, Object> kafkaParams;
private final TypedProperties props;
protected final String topicName;
+ private final KafkaResetOffsetStrategies autoResetValue;
public KafkaOffsetGen(TypedProperties props) {
this.props = props;
+
kafkaParams = new HashMap<>();
for (Object prop : props.keySet()) {
kafkaParams.put(prop.toString(), props.get(prop.toString()));
}
DataSourceUtils.checkRequiredProperties(props, Collections.singletonList(Config.KAFKA_TOPIC_NAME));
topicName = props.getString(Config.KAFKA_TOPIC_NAME);
+ autoResetValue = KafkaResetOffsetStrategies.valueOf(props.getString("auto.offset.reset", Config.DEFAULT_AUTO_RESET_OFFSET.toString()).toUpperCase());
Review comment:
@liujinhui1994 any response?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on pull request #2227: [HUDI-1367] Make delastreamer transition from dfsSouce to kafkasouce
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #2227:
URL: https://github.com/apache/hudi/pull/2227#issuecomment-751914927
yeah we can get this in, if possible
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] wangxianghu commented on a change in pull request #2227: [HUDI-1367] Make delastreamer transition from dfsSouce to kafkasouce
Posted by GitBox <gi...@apache.org>.
wangxianghu commented on a change in pull request #2227:
URL: https://github.com/apache/hudi/pull/2227#discussion_r517075987
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -156,15 +160,18 @@ public static long totalNewMessages(OffsetRange[] ranges) {
private final HashMap<String, Object> kafkaParams;
private final TypedProperties props;
protected final String topicName;
+ private final KafkaResetOffsetStrategies autoResetValue;
public KafkaOffsetGen(TypedProperties props) {
this.props = props;
+
kafkaParams = new HashMap<>();
for (Object prop : props.keySet()) {
kafkaParams.put(prop.toString(), props.get(prop.toString()));
}
DataSourceUtils.checkRequiredProperties(props, Collections.singletonList(Config.KAFKA_TOPIC_NAME));
topicName = props.getString(Config.KAFKA_TOPIC_NAME);
+ autoResetValue = KafkaResetOffsetStrategies.valueOf(props.getString("auto.offset.reset", Config.DEFAULT_AUTO_RESET_OFFSET.toString()).toUpperCase());
Review comment:
I suggest check the string value of `auto.offset.reset` and set default value to `LATEST` if it is not set to `LATEST` or `EARLIEST` before we get the `KafkaResetOffsetStrategies`.
Because `KafkaResetOffsetStrategies` has only two values: `LATEST` and `LATEST`, if the user set it to other value this line will throw an exception.
the lines bellow is useless:
````
default:
throw new HoodieNotSupportedException("Auto reset value must be one of 'earliest' or 'latest' ");
````
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #2227: [HUDI-1367] Make deltaStreamer transition from dfsSouce to kafkasouce
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #2227:
URL: https://github.com/apache/hudi/pull/2227#discussion_r572002870
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -227,12 +232,23 @@ public KafkaOffsetGen(TypedProperties props) {
// else return earliest offsets
private Map<TopicPartition, Long> checkupValidOffsets(KafkaConsumer consumer,
Option<String> lastCheckpointStr, Set<TopicPartition> topicPartitions) {
- Map<TopicPartition, Long> checkpointOffsets = CheckpointUtils.strToOffsets(lastCheckpointStr.get());
Map<TopicPartition, Long> earliestOffsets = consumer.beginningOffsets(topicPartitions);
+ if (checkTopicCheckPoint(lastCheckpointStr)) {
+ Map<TopicPartition, Long> checkpointOffsets = CheckpointUtils.strToOffsets(lastCheckpointStr.get());
+ boolean checkpointOffsetReseter = checkpointOffsets.entrySet().stream()
+ .anyMatch(offset -> offset.getValue() < earliestOffsets.get(offset.getKey()));
+ return checkpointOffsetReseter ? earliestOffsets : checkpointOffsets;
+ }
+
+ switch (autoResetValue) {
+ case EARLIEST:
+ return earliestOffsets;
+ case LATEST:
+ return consumer.endOffsets(topicPartitions);
+ default:
Review comment:
is it intentional for NONE we will throw exception? also, may I know where is NONE(KafkaResetOffsetStrategies) used?
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -257,6 +273,11 @@ public boolean checkTopicExists(KafkaConsumer consumer) {
return result.containsKey(topicName);
}
+ public boolean checkTopicCheckPoint(Option<String> lastCheckpointStr) {
Review comment:
may I know why this is public ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #2227: [HUDI-1367] Make deltaStreamer transition from dfsSouce to kafkasouce
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #2227:
URL: https://github.com/apache/hudi/pull/2227#issuecomment-775865989
@liujinhui1994 : when you get a chance, can you address the feedback and update the PR. Do ping us here once its ready to be reviewed again.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #2227: [HUDI-1367] Make deltaStreamer transition from dfsSouce to kafkasouce
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #2227:
URL: https://github.com/apache/hudi/pull/2227#discussion_r579684280
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -156,15 +163,18 @@ public static long totalNewMessages(OffsetRange[] ranges) {
private final HashMap<String, Object> kafkaParams;
private final TypedProperties props;
protected final String topicName;
+ private final KafkaResetOffsetStrategies autoResetValue;
public KafkaOffsetGen(TypedProperties props) {
this.props = props;
+
kafkaParams = new HashMap<>();
for (Object prop : props.keySet()) {
kafkaParams.put(prop.toString(), props.get(prop.toString()));
}
DataSourceUtils.checkRequiredProperties(props, Collections.singletonList(Config.KAFKA_TOPIC_NAME));
topicName = props.getString(Config.KAFKA_TOPIC_NAME);
+ autoResetValue = KafkaResetOffsetStrategies.valueOf(props.getString("auto.offset.reset", Config.DEFAULT_AUTO_RESET_OFFSET.toString()).toUpperCase());
Review comment:
lets add this to Config in line 154 in this file.
Also, lets name this as your usual way.
"hoodie.deltastreamer.source.kafka.auto.offset.reset"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2227: [HUDI-1367] Make deltaStreamer transition from dfsSouce to kafkasouce
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2227:
URL: https://github.com/apache/hudi/pull/2227#issuecomment-720892015
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2227?src=pr&el=h1) Report
> Merging [#2227](https://codecov.io/gh/apache/hudi/pull/2227?src=pr&el=desc) (8d1257d) into [master](https://codecov.io/gh/apache/hudi/commit/b0010bf3b449a9d2e01955b0746b795e22e577db?el=desc) (b0010bf) will **decrease** coverage by `5.76%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/hudi/pull/2227?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2227 +/- ##
============================================
- Coverage 51.15% 45.39% -5.77%
+ Complexity 3212 2907 -305
============================================
Files 436 438 +2
Lines 19987 20050 +63
Branches 2057 2067 +10
============================================
- Hits 10224 9101 -1123
- Misses 8922 10225 +1303
+ Partials 841 724 -117
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| hudicli | `36.87% <ø> (-0.04%)` | `0.00 <ø> (ø)` | |
| hudiclient | `100.00% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudicommon | `51.39% <ø> (+<0.01%)` | `0.00 <ø> (ø)` | |
| hudiflink | `45.44% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudihadoopmr | `33.16% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudisparkdatasource | `69.75% <ø> (+0.01%)` | `0.00 <ø> (ø)` | |
| hudisync | `48.61% <ø> (ø)` | `0.00 <ø> (ø)` | |
| huditimelineservice | `66.49% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudiutilities | `9.65% <0.00%> (-59.74%)` | `0.00 <0.00> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2227?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `0.00% <0.00%> (-88.78%)` | `0.00 <0.00> (-16.00)` | |
| [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | |
| [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | |
| [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQ3N2REZTU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-10.00%)` | |
| [...g/apache/hudi/utilities/sources/JsonDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkRGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-6.00%)` | |
| [...pache/hudi/utilities/sources/ParquetDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUGFycXVldERGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-5.00%)` | |
| ... and [43 more](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree-more) | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #2227: [HUDI-1367] Make deltaStreamer transition from dfsSouce to kafkasouce
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #2227:
URL: https://github.com/apache/hudi/pull/2227#discussion_r579684280
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -156,15 +163,18 @@ public static long totalNewMessages(OffsetRange[] ranges) {
private final HashMap<String, Object> kafkaParams;
private final TypedProperties props;
protected final String topicName;
+ private final KafkaResetOffsetStrategies autoResetValue;
public KafkaOffsetGen(TypedProperties props) {
this.props = props;
+
kafkaParams = new HashMap<>();
for (Object prop : props.keySet()) {
kafkaParams.put(prop.toString(), props.get(prop.toString()));
}
DataSourceUtils.checkRequiredProperties(props, Collections.singletonList(Config.KAFKA_TOPIC_NAME));
topicName = props.getString(Config.KAFKA_TOPIC_NAME);
+ autoResetValue = KafkaResetOffsetStrategies.valueOf(props.getString("auto.offset.reset", Config.DEFAULT_AUTO_RESET_OFFSET.toString()).toUpperCase());
Review comment:
lets add this to Config (in line 154 in this file)
Also, lets name this as your usual way.
"hoodie.deltastreamer.source.kafka.auto.offset.reset"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan edited a comment on pull request #2227: [HUDI-1367] Make deltaStreamer transition from dfsSouce to kafkasouce
Posted by GitBox <gi...@apache.org>.
nsivabalan edited a comment on pull request #2227:
URL: https://github.com/apache/hudi/pull/2227#issuecomment-782770387
> @nsivabalan Let you take over this PR. Please go ahead.
Sure. By take over you mean, just reviewing or you want me to address all comments by myself and get it to finish line? I have added tests for now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2227: [HUDI-1367] Make deltaStreamer transition from dfsSouce to kafkasouce
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2227:
URL: https://github.com/apache/hudi/pull/2227#issuecomment-720892015
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2227?src=pr&el=h1) Report
> Merging [#2227](https://codecov.io/gh/apache/hudi/pull/2227?src=pr&el=desc) (43d4f7f) into [master](https://codecov.io/gh/apache/hudi/commit/b0010bf3b449a9d2e01955b0746b795e22e577db?el=desc) (b0010bf) will **decrease** coverage by `0.01%`.
> The diff coverage is `46.15%`.
[](https://codecov.io/gh/apache/hudi/pull/2227?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2227 +/- ##
============================================
- Coverage 51.15% 51.14% -0.02%
- Complexity 3212 3216 +4
============================================
Files 436 438 +2
Lines 19987 20050 +63
Branches 2057 2067 +10
============================================
+ Hits 10224 10254 +30
- Misses 8922 8951 +29
- Partials 841 845 +4
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| hudicli | `36.87% <ø> (-0.04%)` | `0.00 <ø> (ø)` | |
| hudiclient | `100.00% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudicommon | `51.39% <ø> (+<0.01%)` | `0.00 <ø> (ø)` | |
| hudiflink | `45.44% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudihadoopmr | `33.16% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudisparkdatasource | `69.75% <ø> (+0.01%)` | `0.00 <ø> (ø)` | |
| hudisync | `48.61% <ø> (ø)` | `0.00 <ø> (ø)` | |
| huditimelineservice | `66.49% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudiutilities | `69.21% <46.15%> (-0.18%)` | `0.00 <4.00> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2227?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `84.76% <46.15%> (-4.02%)` | `17.00 <4.00> (+1.00)` | :arrow_down: |
| [...java/org/apache/hudi/common/util/CleanerUtils.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3V0aWwvQ2xlYW5lclV0aWxzLmphdmE=) | `47.72% <0.00%> (-15.91%)` | `6.00% <0.00%> (ø%)` | |
| [...rg/apache/hudi/cli/commands/SavepointsCommand.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1jbGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpL2NvbW1hbmRzL1NhdmVwb2ludHNDb21tYW5kLmphdmE=) | `13.84% <0.00%> (-0.44%)` | `3.00% <0.00%> (ø%)` | |
| [...i/utilities/deltastreamer/HoodieDeltaStreamer.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvSG9vZGllRGVsdGFTdHJlYW1lci5qYXZh) | `68.72% <0.00%> (-0.26%)` | `18.00% <0.00%> (ø%)` | |
| [...pache/hudi/cli/commands/FileSystemViewCommand.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1jbGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpL2NvbW1hbmRzL0ZpbGVTeXN0ZW1WaWV3Q29tbWFuZC5qYXZh) | `82.45% <0.00%> (-0.16%)` | `19.00% <0.00%> (ø%)` | |
| [...che/hudi/common/table/timeline/HoodieTimeline.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3RpbWVsaW5lL0hvb2RpZVRpbWVsaW5lLmphdmE=) | `91.30% <0.00%> (ø)` | `44.00% <0.00%> (ø%)` | |
| [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `70.00% <0.00%> (ø)` | `50.00% <0.00%> (ø%)` | |
| [...di/common/table/timeline/HoodieActiveTimeline.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3RpbWVsaW5lL0hvb2RpZUFjdGl2ZVRpbWVsaW5lLmphdmE=) | `70.77% <0.00%> (ø)` | `43.00% <0.00%> (ø%)` | |
| [...common/model/HoodieFailedWritesCleaningPolicy.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL0hvb2RpZUZhaWxlZFdyaXRlc0NsZWFuaW5nUG9saWN5LmphdmE=) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (?%)` | |
| [...pache/hudi/exception/HoodieHeartbeatException.java](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhjZXB0aW9uL0hvb2RpZUhlYXJ0YmVhdEV4Y2VwdGlvbi5qYXZh) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (?%)` | |
| ... and [5 more](https://codecov.io/gh/apache/hudi/pull/2227/diff?src=pr&el=tree-more) | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] liujinhui1994 commented on pull request #2227: [HUDI-1367] Make delastreamer transition from dfsSouce to kafkasouce
Posted by GitBox <gi...@apache.org>.
liujinhui1994 commented on pull request #2227:
URL: https://github.com/apache/hudi/pull/2227#issuecomment-751711411
@nsivabalan i will finish tomorrow
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org