You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/03/12 13:30:58 UTC
[GitHub] [hudi] danny0405 opened a new pull request #2669: [HUDI-1684] Tweak hudi-flink-bundle module pom and re-organize the pa…
danny0405 opened a new pull request #2669:
URL: https://github.com/apache/hudi/pull/2669
…cakges for hudi-flink module
## What is the purpose of the pull request
- Add required dependencies for hudi-flink-bundle module
- Some package re-organize of hudi-flink module
## Brief change log
*(for example:)*
- *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2669: [HUDI-1684] Tweak hudi-flink-bundle module pom and reorganize the pa…
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2669:
URL: https://github.com/apache/hudi/pull/2669#issuecomment-797515929
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2669?src=pr&el=h1) Report
> Merging [#2669](https://codecov.io/gh/apache/hudi/pull/2669?src=pr&el=desc) (50d4722) into [master](https://codecov.io/gh/apache/hudi/commit/20786ab8a2a1e7735ab846e92802fb9f4449adc9?el=desc) (20786ab) will **decrease** coverage by `42.48%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/hudi/pull/2669?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2669 +/- ##
============================================
- Coverage 52.00% 9.52% -42.49%
+ Complexity 3579 48 -3531
============================================
Files 465 53 -412
Lines 22268 1963 -20305
Branches 2375 235 -2140
============================================
- Hits 11581 187 -11394
+ Misses 9676 1763 -7913
+ Partials 1011 13 -998
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| hudicli | `?` | `?` | |
| hudiclient | `?` | `?` | |
| hudicommon | `?` | `?` | |
| hudiflink | `?` | `?` | |
| hudihadoopmr | `?` | `?` | |
| hudisparkdatasource | `?` | `?` | |
| hudisync | `?` | `?` | |
| huditimelineservice | `?` | `?` | |
| hudiutilities | `9.52% <ø> (-60.02%)` | `0.00 <ø> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2669?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/2669/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | |
| [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/2669/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | |
| [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/2669/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/2669/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/2669/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2669/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQ3N2REZTU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-10.00%)` | |
| [...g/apache/hudi/utilities/sources/JsonDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2669/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkRGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/2669/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-6.00%)` | |
| [...pache/hudi/utilities/sources/ParquetDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2669/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUGFycXVldERGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-5.00%)` | |
| [...lities/schema/SchemaProviderWithPostProcessor.java](https://codecov.io/gh/apache/hudi/pull/2669/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFQcm92aWRlcldpdGhQb3N0UHJvY2Vzc29yLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| ... and [398 more](https://codecov.io/gh/apache/hudi/pull/2669/diff?src=pr&el=tree-more) | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io commented on pull request #2669: [HUDI-1684] Tweak hudi-flink-bundle module pom and re-organize the pa…
Posted by GitBox <gi...@apache.org>.
codecov-io commented on pull request #2669:
URL: https://github.com/apache/hudi/pull/2669#issuecomment-797515929
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2669?src=pr&el=h1) Report
> Merging [#2669](https://codecov.io/gh/apache/hudi/pull/2669?src=pr&el=desc) (e604fa5) into [master](https://codecov.io/gh/apache/hudi/commit/20786ab8a2a1e7735ab846e92802fb9f4449adc9?el=desc) (20786ab) will **decrease** coverage by `0.01%`.
> The diff coverage is `100.00%`.
[](https://codecov.io/gh/apache/hudi/pull/2669?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2669 +/- ##
============================================
- Coverage 52.00% 51.99% -0.02%
+ Complexity 3579 3578 -1
============================================
Files 465 465
Lines 22268 22268
Branches 2375 2375
============================================
- Hits 11581 11578 -3
- Misses 9676 9677 +1
- Partials 1011 1013 +2
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| hudicli | `37.01% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudiclient | `100.00% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudicommon | `51.49% <ø> (-0.02%)` | `0.00 <ø> (ø)` | |
| hudiflink | `53.57% <100.00%> (ø)` | `0.00 <2.00> (ø)` | |
| hudihadoopmr | `33.44% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudisparkdatasource | `69.84% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudisync | `49.62% <ø> (ø)` | `0.00 <ø> (ø)` | |
| huditimelineservice | `64.36% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudiutilities | `69.48% <ø> (-0.06%)` | `0.00 <ø> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2669?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...va/org/apache/hudi/configuration/FlinkOptions.java](https://codecov.io/gh/apache/hudi/pull/2669/diff?src=pr&el=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9jb25maWd1cmF0aW9uL0ZsaW5rT3B0aW9ucy5qYXZh) | `85.49% <ø> (ø)` | `6.00 <0.00> (?)` | |
| [...rg/apache/hudi/schema/FilebasedSchemaProvider.java](https://codecov.io/gh/apache/hudi/pull/2669/diff?src=pr&el=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zY2hlbWEvRmlsZWJhc2VkU2NoZW1hUHJvdmlkZXIuamF2YQ==) | `29.16% <ø> (ø)` | `2.00 <0.00> (ø)` | |
| [...src/main/java/org/apache/hudi/sink/CommitSink.java](https://codecov.io/gh/apache/hudi/pull/2669/diff?src=pr&el=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL0NvbW1pdFNpbmsuamF2YQ==) | `0.00% <ø> (ø)` | `0.00 <0.00> (ø)` | |
| [.../org/apache/hudi/sink/InstantGenerateOperator.java](https://codecov.io/gh/apache/hudi/pull/2669/diff?src=pr&el=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL0luc3RhbnRHZW5lcmF0ZU9wZXJhdG9yLmphdmE=) | `0.00% <ø> (ø)` | `0.00 <0.00> (?)` | |
| [...rg/apache/hudi/sink/KeyedWriteProcessFunction.java](https://codecov.io/gh/apache/hudi/pull/2669/diff?src=pr&el=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL0tleWVkV3JpdGVQcm9jZXNzRnVuY3Rpb24uamF2YQ==) | `0.00% <ø> (ø)` | `0.00 <0.00> (?)` | |
| [...rg/apache/hudi/sink/KeyedWriteProcessOperator.java](https://codecov.io/gh/apache/hudi/pull/2669/diff?src=pr&el=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL0tleWVkV3JpdGVQcm9jZXNzT3BlcmF0b3IuamF2YQ==) | `0.00% <ø> (ø)` | `0.00 <0.00> (?)` | |
| [...java/org/apache/hudi/sink/StreamWriteOperator.java](https://codecov.io/gh/apache/hudi/pull/2669/diff?src=pr&el=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL1N0cmVhbVdyaXRlT3BlcmF0b3IuamF2YQ==) | `0.00% <ø> (ø)` | `0.00 <0.00> (?)` | |
| [...ache/hudi/sink/StreamWriteOperatorCoordinator.java](https://codecov.io/gh/apache/hudi/pull/2669/diff?src=pr&el=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL1N0cmVhbVdyaXRlT3BlcmF0b3JDb29yZGluYXRvci5qYXZh) | `69.13% <ø> (ø)` | `32.00 <0.00> (?)` | |
| [...g/apache/hudi/sink/StreamWriteOperatorFactory.java](https://codecov.io/gh/apache/hudi/pull/2669/diff?src=pr&el=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL1N0cmVhbVdyaXRlT3BlcmF0b3JGYWN0b3J5LmphdmE=) | `0.00% <ø> (ø)` | `0.00 <0.00> (?)` | |
| [.../org/apache/hudi/sink/compact/CompactFunction.java](https://codecov.io/gh/apache/hudi/pull/2669/diff?src=pr&el=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL2NvbXBhY3QvQ29tcGFjdEZ1bmN0aW9uLmphdmE=) | `100.00% <ø> (ø)` | `4.00 <0.00> (?)` | |
| ... and [36 more](https://codecov.io/gh/apache/hudi/pull/2669/diff?src=pr&el=tree-more) | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2669: [HUDI-1684] Tweak hudi-flink-bundle module pom and reorganize the pa…
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2669:
URL: https://github.com/apache/hudi/pull/2669#issuecomment-797515929
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2669?src=pr&el=h1) Report
> Merging [#2669](https://codecov.io/gh/apache/hudi/pull/2669?src=pr&el=desc) (f632e7a) into [master](https://codecov.io/gh/apache/hudi/commit/20786ab8a2a1e7735ab846e92802fb9f4449adc9?el=desc) (20786ab) will **decrease** coverage by `0.04%`.
> The diff coverage is `100.00%`.
[](https://codecov.io/gh/apache/hudi/pull/2669?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2669 +/- ##
============================================
- Coverage 52.00% 51.96% -0.05%
+ Complexity 3579 3578 -1
============================================
Files 465 466 +1
Lines 22268 22275 +7
Branches 2375 2374 -1
============================================
- Hits 11581 11575 -6
- Misses 9676 9690 +14
+ Partials 1011 1010 -1
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| hudicli | `37.01% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudiclient | `100.00% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudicommon | `51.44% <ø> (-0.08%)` | `0.00 <ø> (ø)` | |
| hudiflink | `53.57% <100.00%> (ø)` | `0.00 <2.00> (ø)` | |
| hudihadoopmr | `33.44% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudisparkdatasource | `69.84% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudisync | `49.62% <ø> (ø)` | `0.00 <ø> (ø)` | |
| huditimelineservice | `64.36% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudiutilities | `69.43% <ø> (-0.11%)` | `0.00 <ø> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2669?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...va/org/apache/hudi/configuration/FlinkOptions.java](https://codecov.io/gh/apache/hudi/pull/2669/diff?src=pr&el=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9jb25maWd1cmF0aW9uL0ZsaW5rT3B0aW9ucy5qYXZh) | `85.49% <ø> (ø)` | `6.00 <0.00> (?)` | |
| [...rg/apache/hudi/schema/FilebasedSchemaProvider.java](https://codecov.io/gh/apache/hudi/pull/2669/diff?src=pr&el=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zY2hlbWEvRmlsZWJhc2VkU2NoZW1hUHJvdmlkZXIuamF2YQ==) | `29.16% <ø> (ø)` | `2.00 <0.00> (ø)` | |
| [...src/main/java/org/apache/hudi/sink/CommitSink.java](https://codecov.io/gh/apache/hudi/pull/2669/diff?src=pr&el=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL0NvbW1pdFNpbmsuamF2YQ==) | `0.00% <ø> (ø)` | `0.00 <0.00> (ø)` | |
| [.../org/apache/hudi/sink/InstantGenerateOperator.java](https://codecov.io/gh/apache/hudi/pull/2669/diff?src=pr&el=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL0luc3RhbnRHZW5lcmF0ZU9wZXJhdG9yLmphdmE=) | `0.00% <ø> (ø)` | `0.00 <0.00> (?)` | |
| [...rg/apache/hudi/sink/KeyedWriteProcessFunction.java](https://codecov.io/gh/apache/hudi/pull/2669/diff?src=pr&el=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL0tleWVkV3JpdGVQcm9jZXNzRnVuY3Rpb24uamF2YQ==) | `0.00% <ø> (ø)` | `0.00 <0.00> (?)` | |
| [...rg/apache/hudi/sink/KeyedWriteProcessOperator.java](https://codecov.io/gh/apache/hudi/pull/2669/diff?src=pr&el=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL0tleWVkV3JpdGVQcm9jZXNzT3BlcmF0b3IuamF2YQ==) | `0.00% <ø> (ø)` | `0.00 <0.00> (?)` | |
| [...java/org/apache/hudi/sink/StreamWriteFunction.java](https://codecov.io/gh/apache/hudi/pull/2669/diff?src=pr&el=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL1N0cmVhbVdyaXRlRnVuY3Rpb24uamF2YQ==) | `84.00% <ø> (ø)` | `22.00 <0.00> (?)` | |
| [...java/org/apache/hudi/sink/StreamWriteOperator.java](https://codecov.io/gh/apache/hudi/pull/2669/diff?src=pr&el=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL1N0cmVhbVdyaXRlT3BlcmF0b3IuamF2YQ==) | `0.00% <ø> (ø)` | `0.00 <0.00> (?)` | |
| [...ache/hudi/sink/StreamWriteOperatorCoordinator.java](https://codecov.io/gh/apache/hudi/pull/2669/diff?src=pr&el=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL1N0cmVhbVdyaXRlT3BlcmF0b3JDb29yZGluYXRvci5qYXZh) | `69.13% <ø> (ø)` | `32.00 <0.00> (?)` | |
| [...g/apache/hudi/sink/StreamWriteOperatorFactory.java](https://codecov.io/gh/apache/hudi/pull/2669/diff?src=pr&el=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL1N0cmVhbVdyaXRlT3BlcmF0b3JGYWN0b3J5LmphdmE=) | `0.00% <ø> (ø)` | `0.00 <0.00> (?)` | |
| ... and [41 more](https://codecov.io/gh/apache/hudi/pull/2669/diff?src=pr&el=tree-more) | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #2669: [HUDI-1684] Tweak hudi-flink-bundle module pom and reorganize the pa…
Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #2669:
URL: https://github.com/apache/hudi/pull/2669#discussion_r594085950
##########
File path: hudi-flink/src/main/java/org/apache/hudi/streamer/HoodieFlinkStreamerV2.java
##########
@@ -19,10 +19,10 @@
package org.apache.hudi.streamer;
import org.apache.hudi.common.model.HoodieRecord;
-import org.apache.hudi.operator.FlinkOptions;
-import org.apache.hudi.operator.StreamWriteOperatorFactory;
-import org.apache.hudi.operator.partitioner.BucketAssignFunction;
-import org.apache.hudi.operator.transform.RowDataToHoodieFunction;
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.sink.StreamWriteOperatorFactory;
+import org.apache.hudi.sink.partitioner.BucketAssignFunction;
+import org.apache.hudi.sink.transform.RowDataToHoodieFunction;
import org.apache.hudi.util.AvroSchemaConverter;
Review comment:
No, this class references Flink, i would prefer to keep the name.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #2669: [HUDI-1684] Tweak hudi-flink-bundle module pom and re-organize the pa…
Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #2669:
URL: https://github.com/apache/hudi/pull/2669#discussion_r594008808
##########
File path: hudi-flink/src/main/java/org/apache/hudi/source/StreamWriteFunction.java
##########
@@ -16,7 +16,7 @@
* limitations under the License.
*/
-package org.apache.hudi.operator;
+package org.apache.hudi.source;
Review comment:
Nope, thanks for the reminder ~
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua merged pull request #2669: [HUDI-1684] Tweak hudi-flink-bundle module pom and reorganize the pa…
Posted by GitBox <gi...@apache.org>.
yanghua merged pull request #2669:
URL: https://github.com/apache/hudi/pull/2669
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on a change in pull request #2669: [HUDI-1684] Tweak hudi-flink-bundle module pom and re-organize the pa…
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #2669:
URL: https://github.com/apache/hudi/pull/2669#discussion_r593996516
##########
File path: hudi-flink/src/main/java/org/apache/hudi/source/StreamWriteFunction.java
##########
@@ -16,7 +16,7 @@
* limitations under the License.
*/
-package org.apache.hudi.operator;
+package org.apache.hudi.source;
Review comment:
Should it be put into the `source` package?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on a change in pull request #2669: [HUDI-1684] Tweak hudi-flink-bundle module pom and reorganize the pa…
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #2669:
URL: https://github.com/apache/hudi/pull/2669#discussion_r594077871
##########
File path: hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java
##########
@@ -16,14 +16,14 @@
* limitations under the License.
*/
-package org.apache.hudi.operator;
+package org.apache.hudi.configuration;
import org.apache.hudi.common.model.HoodieTableType;
-import org.apache.hudi.config.HoodieWriteConfig;
-import org.apache.hudi.streamer.FlinkStreamerConfig;
import org.apache.hudi.common.model.OverwriteWithLatestAvroPayload;
+import org.apache.hudi.config.HoodieWriteConfig;
import org.apache.hudi.keygen.SimpleAvroKeyGenerator;
import org.apache.hudi.keygen.constant.KeyGeneratorOptions;
+import org.apache.hudi.streamer.FlinkStreamerConfig;
Review comment:
We have a `configuration` subpackage, can we put `FlinkStreamerConfig ` into it?
##########
File path: hudi-flink/src/test/java/org/apache/hudi/sink/StreamWriteOperatorCoordinatorTest.java
##########
@@ -16,16 +16,16 @@
* limitations under the License.
*/
-package org.apache.hudi.operator;
+package org.apache.hudi.sink;
import org.apache.hudi.client.WriteStatus;
import org.apache.hudi.common.fs.FSUtils;
import org.apache.hudi.common.model.HoodieWriteStat;
import org.apache.hudi.common.table.HoodieTableMetaClient;
import org.apache.hudi.exception.HoodieException;
-import org.apache.hudi.operator.event.BatchWriteSuccessEvent;
-import org.apache.hudi.operator.utils.TestConfigurations;
+import org.apache.hudi.sink.event.BatchWriteSuccessEvent;
import org.apache.hudi.util.StreamerUtil;
+import org.apache.hudi.utils.TestConfigurations;
Review comment:
`TestStreamWriteOperatorCoordinator` sounds better?
##########
File path: hudi-flink/src/main/java/org/apache/hudi/streamer/HoodieFlinkStreamerV2.java
##########
@@ -19,10 +19,10 @@
package org.apache.hudi.streamer;
import org.apache.hudi.common.model.HoodieRecord;
-import org.apache.hudi.operator.FlinkOptions;
-import org.apache.hudi.operator.StreamWriteOperatorFactory;
-import org.apache.hudi.operator.partitioner.BucketAssignFunction;
-import org.apache.hudi.operator.transform.RowDataToHoodieFunction;
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.sink.StreamWriteOperatorFactory;
+import org.apache.hudi.sink.partitioner.BucketAssignFunction;
+import org.apache.hudi.sink.transform.RowDataToHoodieFunction;
import org.apache.hudi.util.AvroSchemaConverter;
Review comment:
We have some classes that follow these patterns, e.g. `Converter`, `Converters`. Can we choose one?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2669: [HUDI-1684] Tweak hudi-flink-bundle module pom and reorganize the pa…
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2669:
URL: https://github.com/apache/hudi/pull/2669#issuecomment-797515929
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2669?src=pr&el=h1) Report
> Merging [#2669](https://codecov.io/gh/apache/hudi/pull/2669?src=pr&el=desc) (f632e7a) into [master](https://codecov.io/gh/apache/hudi/commit/20786ab8a2a1e7735ab846e92802fb9f4449adc9?el=desc) (20786ab) will **decrease** coverage by `0.19%`.
> The diff coverage is `100.00%`.
[](https://codecov.io/gh/apache/hudi/pull/2669?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2669 +/- ##
============================================
- Coverage 52.00% 51.81% -0.20%
+ Complexity 3579 3388 -191
============================================
Files 465 445 -20
Lines 22268 20764 -1504
Branches 2375 2229 -146
============================================
- Hits 11581 10759 -822
+ Misses 9676 9070 -606
+ Partials 1011 935 -76
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| hudicli | `37.01% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudiclient | `100.00% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudicommon | `51.44% <ø> (-0.08%)` | `0.00 <ø> (ø)` | |
| hudiflink | `53.57% <100.00%> (ø)` | `0.00 <2.00> (ø)` | |
| hudihadoopmr | `33.44% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudisparkdatasource | `69.84% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudisync | `?` | `?` | |
| huditimelineservice | `?` | `?` | |
| hudiutilities | `69.43% <ø> (-0.11%)` | `0.00 <ø> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2669?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...va/org/apache/hudi/configuration/FlinkOptions.java](https://codecov.io/gh/apache/hudi/pull/2669/diff?src=pr&el=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9jb25maWd1cmF0aW9uL0ZsaW5rT3B0aW9ucy5qYXZh) | `85.49% <ø> (ø)` | `6.00 <0.00> (?)` | |
| [...rg/apache/hudi/schema/FilebasedSchemaProvider.java](https://codecov.io/gh/apache/hudi/pull/2669/diff?src=pr&el=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zY2hlbWEvRmlsZWJhc2VkU2NoZW1hUHJvdmlkZXIuamF2YQ==) | `29.16% <ø> (ø)` | `2.00 <0.00> (ø)` | |
| [...src/main/java/org/apache/hudi/sink/CommitSink.java](https://codecov.io/gh/apache/hudi/pull/2669/diff?src=pr&el=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL0NvbW1pdFNpbmsuamF2YQ==) | `0.00% <ø> (ø)` | `0.00 <0.00> (ø)` | |
| [.../org/apache/hudi/sink/InstantGenerateOperator.java](https://codecov.io/gh/apache/hudi/pull/2669/diff?src=pr&el=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL0luc3RhbnRHZW5lcmF0ZU9wZXJhdG9yLmphdmE=) | `0.00% <ø> (ø)` | `0.00 <0.00> (?)` | |
| [...rg/apache/hudi/sink/KeyedWriteProcessFunction.java](https://codecov.io/gh/apache/hudi/pull/2669/diff?src=pr&el=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL0tleWVkV3JpdGVQcm9jZXNzRnVuY3Rpb24uamF2YQ==) | `0.00% <ø> (ø)` | `0.00 <0.00> (?)` | |
| [...rg/apache/hudi/sink/KeyedWriteProcessOperator.java](https://codecov.io/gh/apache/hudi/pull/2669/diff?src=pr&el=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL0tleWVkV3JpdGVQcm9jZXNzT3BlcmF0b3IuamF2YQ==) | `0.00% <ø> (ø)` | `0.00 <0.00> (?)` | |
| [...java/org/apache/hudi/sink/StreamWriteFunction.java](https://codecov.io/gh/apache/hudi/pull/2669/diff?src=pr&el=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL1N0cmVhbVdyaXRlRnVuY3Rpb24uamF2YQ==) | `84.00% <ø> (ø)` | `22.00 <0.00> (?)` | |
| [...java/org/apache/hudi/sink/StreamWriteOperator.java](https://codecov.io/gh/apache/hudi/pull/2669/diff?src=pr&el=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL1N0cmVhbVdyaXRlT3BlcmF0b3IuamF2YQ==) | `0.00% <ø> (ø)` | `0.00 <0.00> (?)` | |
| [...ache/hudi/sink/StreamWriteOperatorCoordinator.java](https://codecov.io/gh/apache/hudi/pull/2669/diff?src=pr&el=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL1N0cmVhbVdyaXRlT3BlcmF0b3JDb29yZGluYXRvci5qYXZh) | `69.13% <ø> (ø)` | `32.00 <0.00> (?)` | |
| [...g/apache/hudi/sink/StreamWriteOperatorFactory.java](https://codecov.io/gh/apache/hudi/pull/2669/diff?src=pr&el=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL1N0cmVhbVdyaXRlT3BlcmF0b3JGYWN0b3J5LmphdmE=) | `0.00% <ø> (ø)` | `0.00 <0.00> (?)` | |
| ... and [62 more](https://codecov.io/gh/apache/hudi/pull/2669/diff?src=pr&el=tree-more) | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #2669: [HUDI-1684] Tweak hudi-flink-bundle module pom and reorganize the pa…
Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #2669:
URL: https://github.com/apache/hudi/pull/2669#discussion_r594086761
##########
File path: hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java
##########
@@ -16,14 +16,14 @@
* limitations under the License.
*/
-package org.apache.hudi.operator;
+package org.apache.hudi.configuration;
import org.apache.hudi.common.model.HoodieTableType;
-import org.apache.hudi.config.HoodieWriteConfig;
-import org.apache.hudi.streamer.FlinkStreamerConfig;
import org.apache.hudi.common.model.OverwriteWithLatestAvroPayload;
+import org.apache.hudi.config.HoodieWriteConfig;
import org.apache.hudi.keygen.SimpleAvroKeyGenerator;
import org.apache.hudi.keygen.constant.KeyGeneratorOptions;
+import org.apache.hudi.streamer.FlinkStreamerConfig;
Review comment:
`FlinkStreamerConfig` is only used for streamer, keep it under `streamer` package is more reasonable.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org