You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/10/06 18:05:10 UTC
[GitHub] [hudi] satishkotha opened a new pull request #2150: [HUDI-1304] Add unit test for testing compaction on replaced file groups
satishkotha opened a new pull request #2150:
URL: https://github.com/apache/hudi/pull/2150
## What is the purpose of the pull request
Add unit test to ensure compaction works on top of replaced file groups
## Brief change log
Add unit test to ensure compaction works on top of replaced file groups
## Verify this pull request
- Test only change
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io commented on pull request #2150: [HUDI-1304] Add unit test for testing compaction on replaced file groups
Posted by GitBox <gi...@apache.org>.
codecov-io commented on pull request #2150:
URL: https://github.com/apache/hudi/pull/2150#issuecomment-704505827
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2150?src=pr&el=h1) Report
> :exclamation: No coverage uploaded for pull request base (`master@fdae388`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/hudi/pull/2150?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2150 +/- ##
=========================================
Coverage ? 52.49%
Complexity ? 2793
=========================================
Files ? 358
Lines ? 16499
Branches ? 1771
=========================================
Hits ? 8661
Misses ? 7097
Partials ? 741
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| #hudicli | `38.37% <ø> (?)` | `193.00 <ø> (?)` | |
| #hudiclient | `100.00% <ø> (?)` | `0.00 <ø> (?)` | |
| #hudicommon | `54.74% <ø> (?)` | `1793.00 <ø> (?)` | |
| #hudihadoopmr | `33.05% <ø> (?)` | `181.00 <ø> (?)` | |
| #hudispark | `65.51% <ø> (?)` | `303.00 <ø> (?)` | |
| #huditimelineservice | `62.29% <ø> (?)` | `50.00 <ø> (?)` | |
| #hudiutilities | `59.60% <ø> (?)` | `273.00 <ø> (?)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2150: [HUDI-1304] Add unit test for testing compaction on replaced file groups
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2150:
URL: https://github.com/apache/hudi/pull/2150#issuecomment-704505827
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2150?src=pr&el=h1) Report
> Merging [#2150](https://codecov.io/gh/apache/hudi/pull/2150?src=pr&el=desc) into [master](https://codecov.io/gh/apache/hudi/commit/fdae388626b8d97acc01191aa0e7075c36a41132?el=desc) will **increase** coverage by `1.83%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/hudi/pull/2150?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2150 +/- ##
============================================
+ Coverage 51.79% 53.63% +1.83%
- Complexity 2532 2848 +316
============================================
Files 318 359 +41
Lines 14417 16545 +2128
Branches 1460 1780 +320
============================================
+ Hits 7468 8874 +1406
- Misses 6354 6912 +558
- Partials 595 759 +164
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| #hudicli | `38.37% <ø> (ø)` | `193.00 <ø> (ø)` | |
| #hudiclient | `100.00% <ø> (ø)` | `0.00 <ø> (ø)` | |
| #hudicommon | `54.75% <ø> (+<0.01%)` | `1795.00 <ø> (+2.00)` | |
| #hudihadoopmr | `33.05% <ø> (ø)` | `181.00 <ø> (ø)` | |
| #hudispark | `65.48% <ø> (?)` | `304.00 <ø> (?)` | |
| #huditimelineservice | `62.29% <ø> (ø)` | `50.00 <ø> (ø)` | |
| #hudiutilities | `70.07% <ø> (+0.60%)` | `325.00 <ø> (+10.00)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2150?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [.../main/java/org/apache/hudi/common/util/Option.java](https://codecov.io/gh/apache/hudi/pull/2150/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3V0aWwvT3B0aW9uLmphdmE=) | `66.66% <0.00%> (-3.61%)` | `23.00% <0.00%> (+1.00%)` | :arrow_down: |
| [...udi/utilities/sources/helpers/DFSPathSelector.java](https://codecov.io/gh/apache/hudi/pull/2150/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9ERlNQYXRoU2VsZWN0b3IuamF2YQ==) | `82.05% <0.00%> (-2.16%)` | `12.00% <0.00%> (ø%)` | |
| [.../org/apache/hudi/common/model/HoodieTableType.java](https://codecov.io/gh/apache/hudi/pull/2150/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL0hvb2RpZVRhYmxlVHlwZS5qYXZh) | `100.00% <0.00%> (ø)` | `1.00% <0.00%> (ø%)` | |
| [...rg/apache/hudi/common/util/SerializationUtils.java](https://codecov.io/gh/apache/hudi/pull/2150/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3V0aWwvU2VyaWFsaXphdGlvblV0aWxzLmphdmE=) | `88.00% <0.00%> (ø)` | `3.00% <0.00%> (ø%)` | |
| [.../apache/hudi/common/table/TableSchemaResolver.java](https://codecov.io/gh/apache/hudi/pull/2150/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL1RhYmxlU2NoZW1hUmVzb2x2ZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (ø%)` | |
| [...pache/hudi/io/storage/HoodieFileReaderFactory.java](https://codecov.io/gh/apache/hudi/pull/2150/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vc3RvcmFnZS9Ib29kaWVGaWxlUmVhZGVyRmFjdG9yeS5qYXZh) | `50.00% <0.00%> (ø)` | `3.00% <0.00%> (ø%)` | |
| [.../hudi/utilities/schema/SchemaRegistryProvider.java](https://codecov.io/gh/apache/hudi/pull/2150/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFSZWdpc3RyeVByb3ZpZGVyLmphdmE=) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (ø%)` | |
| [...i/common/model/OverwriteWithLatestAvroPayload.java](https://codecov.io/gh/apache/hudi/pull/2150/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL092ZXJ3cml0ZVdpdGhMYXRlc3RBdnJvUGF5bG9hZC5qYXZh) | `64.70% <0.00%> (ø)` | `10.00% <0.00%> (ø%)` | |
| [...del/OverwriteNonDefaultsWithLatestAvroPayload.java](https://codecov.io/gh/apache/hudi/pull/2150/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL092ZXJ3cml0ZU5vbkRlZmF1bHRzV2l0aExhdGVzdEF2cm9QYXlsb2FkLmphdmE=) | `78.94% <0.00%> (ø)` | `5.00% <0.00%> (ø%)` | |
| [...s/deltastreamer/HoodieMultiTableDeltaStreamer.java](https://codecov.io/gh/apache/hudi/pull/2150/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvSG9vZGllTXVsdGlUYWJsZURlbHRhU3RyZWFtZXIuamF2YQ==) | `78.39% <0.00%> (ø)` | `18.00% <0.00%> (ø%)` | |
| ... and [47 more](https://codecov.io/gh/apache/hudi/pull/2150/diff?src=pr&el=tree-more) | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on a change in pull request #2150: [HUDI-1304] Add unit test for testing compaction on replaced file groups
Posted by GitBox <gi...@apache.org>.
bvaradar commented on a change in pull request #2150:
URL: https://github.com/apache/hudi/pull/2150#discussion_r501331831
##########
File path: hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/action/compact/TestAsyncCompaction.java
##########
@@ -332,4 +336,51 @@ public void testInterleavedCompaction() throws Exception {
executeCompaction(compactionInstantTime, client, hoodieTable, cfg, numRecs, true);
}
}
+
+ @Test
+ public void testCompactionOnReplacedFiles() throws Exception {
Review comment:
Can you add some comments describing the test case scenario. Would be useful for anyone skimming the class.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] satishkotha commented on a change in pull request #2150: [HUDI-1304] Add unit test for testing compaction on replaced file groups
Posted by GitBox <gi...@apache.org>.
satishkotha commented on a change in pull request #2150:
URL: https://github.com/apache/hudi/pull/2150#discussion_r503565568
##########
File path: hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/action/compact/TestAsyncCompaction.java
##########
@@ -332,4 +336,51 @@ public void testInterleavedCompaction() throws Exception {
executeCompaction(compactionInstantTime, client, hoodieTable, cfg, numRecs, true);
}
}
+
+ @Test
+ public void testCompactionOnReplacedFiles() throws Exception {
Review comment:
@bvaradar Added comment. PTAL
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar merged pull request #2150: [HUDI-1304] Add unit test for testing compaction on replaced file groups
Posted by GitBox <gi...@apache.org>.
vinothchandar merged pull request #2150:
URL: https://github.com/apache/hudi/pull/2150
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2150: [HUDI-1304] Add unit test for testing compaction on replaced file groups
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2150:
URL: https://github.com/apache/hudi/pull/2150#issuecomment-704505827
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2150?src=pr&el=h1) Report
> Merging [#2150](https://codecov.io/gh/apache/hudi/pull/2150?src=pr&el=desc) into [master](https://codecov.io/gh/apache/hudi/commit/fdae388626b8d97acc01191aa0e7075c36a41132?el=desc) will **increase** coverage by `1.83%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/hudi/pull/2150?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2150 +/- ##
============================================
+ Coverage 51.79% 53.63% +1.83%
- Complexity 2532 2848 +316
============================================
Files 318 359 +41
Lines 14417 16545 +2128
Branches 1460 1780 +320
============================================
+ Hits 7468 8874 +1406
- Misses 6354 6912 +558
- Partials 595 759 +164
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| #hudicli | `38.37% <ø> (ø)` | `193.00 <ø> (ø)` | |
| #hudiclient | `100.00% <ø> (ø)` | `0.00 <ø> (ø)` | |
| #hudicommon | `54.75% <ø> (+<0.01%)` | `1795.00 <ø> (+2.00)` | |
| #hudihadoopmr | `33.05% <ø> (ø)` | `181.00 <ø> (ø)` | |
| #hudispark | `65.48% <ø> (?)` | `304.00 <ø> (?)` | |
| #huditimelineservice | `62.29% <ø> (ø)` | `50.00 <ø> (ø)` | |
| #hudiutilities | `70.07% <ø> (+0.60%)` | `325.00 <ø> (+10.00)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2150?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [.../main/java/org/apache/hudi/common/util/Option.java](https://codecov.io/gh/apache/hudi/pull/2150/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3V0aWwvT3B0aW9uLmphdmE=) | `66.66% <0.00%> (-3.61%)` | `23.00% <0.00%> (+1.00%)` | :arrow_down: |
| [...udi/utilities/sources/helpers/DFSPathSelector.java](https://codecov.io/gh/apache/hudi/pull/2150/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9ERlNQYXRoU2VsZWN0b3IuamF2YQ==) | `82.05% <0.00%> (-2.16%)` | `12.00% <0.00%> (ø%)` | |
| [.../org/apache/hudi/common/model/HoodieTableType.java](https://codecov.io/gh/apache/hudi/pull/2150/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL0hvb2RpZVRhYmxlVHlwZS5qYXZh) | `100.00% <0.00%> (ø)` | `1.00% <0.00%> (ø%)` | |
| [...rg/apache/hudi/common/util/SerializationUtils.java](https://codecov.io/gh/apache/hudi/pull/2150/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3V0aWwvU2VyaWFsaXphdGlvblV0aWxzLmphdmE=) | `88.00% <0.00%> (ø)` | `3.00% <0.00%> (ø%)` | |
| [.../apache/hudi/common/table/TableSchemaResolver.java](https://codecov.io/gh/apache/hudi/pull/2150/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL1RhYmxlU2NoZW1hUmVzb2x2ZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (ø%)` | |
| [...pache/hudi/io/storage/HoodieFileReaderFactory.java](https://codecov.io/gh/apache/hudi/pull/2150/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vc3RvcmFnZS9Ib29kaWVGaWxlUmVhZGVyRmFjdG9yeS5qYXZh) | `50.00% <0.00%> (ø)` | `3.00% <0.00%> (ø%)` | |
| [.../hudi/utilities/schema/SchemaRegistryProvider.java](https://codecov.io/gh/apache/hudi/pull/2150/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFSZWdpc3RyeVByb3ZpZGVyLmphdmE=) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (ø%)` | |
| [...i/common/model/OverwriteWithLatestAvroPayload.java](https://codecov.io/gh/apache/hudi/pull/2150/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL092ZXJ3cml0ZVdpdGhMYXRlc3RBdnJvUGF5bG9hZC5qYXZh) | `64.70% <0.00%> (ø)` | `10.00% <0.00%> (ø%)` | |
| [...del/OverwriteNonDefaultsWithLatestAvroPayload.java](https://codecov.io/gh/apache/hudi/pull/2150/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL092ZXJ3cml0ZU5vbkRlZmF1bHRzV2l0aExhdGVzdEF2cm9QYXlsb2FkLmphdmE=) | `78.94% <0.00%> (ø)` | `5.00% <0.00%> (ø%)` | |
| [...s/deltastreamer/HoodieMultiTableDeltaStreamer.java](https://codecov.io/gh/apache/hudi/pull/2150/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvSG9vZGllTXVsdGlUYWJsZURlbHRhU3RyZWFtZXIuamF2YQ==) | `78.39% <0.00%> (ø)` | `18.00% <0.00%> (ø%)` | |
| ... and [47 more](https://codecov.io/gh/apache/hudi/pull/2150/diff?src=pr&el=tree-more) | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org