You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/10/11 14:12:23 UTC
[GitHub] [hudi] nsivabalan opened a new pull request #2172: [HUDI-1338] Adding Delete support to test suite framework
nsivabalan opened a new pull request #2172:
URL: https://github.com/apache/hudi/pull/2172
## What is the purpose of the pull request
Adding Delete support to test suite framework
## Brief change log
- Adding DeleteNode to assist in issuing deletes to hudi in integ test suite framework
## Verify this pull request
Tested using docker set up using complex-dag-cow.yaml
## Committer checklist
- [ x] Has a corresponding JIRA in PR title & commit
- [ x] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2172: [HUDI-1338] Adding Delete support to test suite framework
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2172:
URL: https://github.com/apache/hudi/pull/2172#issuecomment-706720825
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2172?src=pr&el=h1) Report
> Merging [#2172](https://codecov.io/gh/apache/hudi/pull/2172?src=pr&el=desc) into [master](https://codecov.io/gh/apache/hudi/commit/788d236c443eb4ced819f9305ed8e0460b5984b7?el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/hudi/pull/2172?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2172 +/- ##
============================================
- Coverage 53.61% 53.60% -0.02%
- Complexity 2845 2846 +1
============================================
Files 359 359
Lines 16535 16548 +13
Branches 1777 1780 +3
============================================
+ Hits 8866 8870 +4
- Misses 6912 6920 +8
- Partials 757 758 +1
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| #hudicli | `38.37% <ø> (ø)` | `193.00 <ø> (ø)` | |
| #hudiclient | `100.00% <ø> (ø)` | `0.00 <ø> (ø)` | |
| #hudicommon | `54.70% <ø> (-0.04%)` | `1793.00 <ø> (ø)` | |
| #hudihadoopmr | `33.05% <ø> (ø)` | `181.00 <ø> (ø)` | |
| #hudispark | `65.48% <ø> (-0.03%)` | `304.00 <ø> (+1.00)` | :arrow_down: |
| #huditimelineservice | `62.29% <ø> (ø)` | `50.00 <ø> (ø)` | |
| #hudiutilities | `69.98% <ø> (ø)` | `325.00 <ø> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2172?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...in/scala/org/apache/hudi/IncrementalRelation.scala](https://codecov.io/gh/apache/hudi/pull/2172/diff?src=pr&el=tree#diff-aHVkaS1zcGFyay9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2h1ZGkvSW5jcmVtZW50YWxSZWxhdGlvbi5zY2FsYQ==) | `76.19% <0.00%> (-2.30%)` | `20.00% <0.00%> (+1.00%)` | :arrow_down: |
| [.../org/apache/hudi/common/model/HoodieTableType.java](https://codecov.io/gh/apache/hudi/pull/2172/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL0hvb2RpZVRhYmxlVHlwZS5qYXZh) | `100.00% <0.00%> (ø)` | `1.00% <0.00%> (ø%)` | |
| [.../apache/hudi/common/table/TableSchemaResolver.java](https://codecov.io/gh/apache/hudi/pull/2172/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL1RhYmxlU2NoZW1hUmVzb2x2ZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (ø%)` | |
| [...i/common/model/OverwriteWithLatestAvroPayload.java](https://codecov.io/gh/apache/hudi/pull/2172/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL092ZXJ3cml0ZVdpdGhMYXRlc3RBdnJvUGF5bG9hZC5qYXZh) | `64.70% <0.00%> (ø)` | `10.00% <0.00%> (ø%)` | |
| [...del/OverwriteNonDefaultsWithLatestAvroPayload.java](https://codecov.io/gh/apache/hudi/pull/2172/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL092ZXJ3cml0ZU5vbkRlZmF1bHRzV2l0aExhdGVzdEF2cm9QYXlsb2FkLmphdmE=) | `78.94% <0.00%> (ø)` | `5.00% <0.00%> (ø%)` | |
| [...main/scala/org/apache/hudi/DataSourceOptions.scala](https://codecov.io/gh/apache/hudi/pull/2172/diff?src=pr&el=tree#diff-aHVkaS1zcGFyay9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2h1ZGkvRGF0YVNvdXJjZU9wdGlvbnMuc2NhbGE=) | `94.82% <0.00%> (+0.09%)` | `0.00% <0.00%> (ø%)` | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2172: [HUDI-1338] Adding Delete support to test suite framework
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2172:
URL: https://github.com/apache/hudi/pull/2172#issuecomment-706720825
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2172?src=pr&el=h1) Report
> Merging [#2172](https://codecov.io/gh/apache/hudi/pull/2172?src=pr&el=desc) into [master](https://codecov.io/gh/apache/hudi/commit/736a9408549f66014bfdfdc72832be11aa9fd39b?el=desc) will **increase** coverage by `43.30%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/hudi/pull/2172?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2172 +/- ##
=============================================
+ Coverage 10.39% 53.70% +43.30%
- Complexity 48 2850 +2802
=============================================
Files 50 359 +309
Lines 1779 16565 +14786
Branches 211 1782 +1571
=============================================
+ Hits 185 8896 +8711
- Misses 1581 6912 +5331
- Partials 13 757 +744
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| #hudicli | `38.37% <ø> (?)` | `193.00 <ø> (?)` | |
| #hudiclient | `100.00% <ø> (ø)` | `0.00 <ø> (ø)` | |
| #hudicommon | `54.73% <ø> (?)` | `1795.00 <ø> (?)` | |
| #hudihadoopmr | `33.05% <ø> (?)` | `181.00 <ø> (?)` | |
| #hudispark | `65.95% <ø> (?)` | `304.00 <ø> (?)` | |
| #huditimelineservice | `62.29% <ø> (?)` | `50.00 <ø> (?)` | |
| #hudiutilities | `70.09% <ø> (+59.69%)` | `327.00 <ø> (+279.00)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2172?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...ava/org/apache/hudi/cli/commands/UtilsCommand.java](https://codecov.io/gh/apache/hudi/pull/2172/diff?src=pr&el=tree#diff-aHVkaS1jbGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpL2NvbW1hbmRzL1V0aWxzQ29tbWFuZC5qYXZh) | `100.00% <0.00%> (ø)` | `3.00% <0.00%> (?%)` | |
| [...he/hudi/exception/HoodieNotSupportedException.java](https://codecov.io/gh/apache/hudi/pull/2172/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhjZXB0aW9uL0hvb2RpZU5vdFN1cHBvcnRlZEV4Y2VwdGlvbi5qYXZh) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (?%)` | |
| [...rc/main/java/org/apache/hudi/ApiMaturityLevel.java](https://codecov.io/gh/apache/hudi/pull/2172/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvQXBpTWF0dXJpdHlMZXZlbC5qYXZh) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (?%)` | |
| [...rg/apache/hudi/hadoop/HoodieROTablePathFilter.java](https://codecov.io/gh/apache/hudi/pull/2172/diff?src=pr&el=tree#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL0hvb2RpZVJPVGFibGVQYXRoRmlsdGVyLmphdmE=) | `60.81% <0.00%> (ø)` | `11.00% <0.00%> (?%)` | |
| [...mon/table/log/block/HoodieCommandBlockVersion.java](https://codecov.io/gh/apache/hudi/pull/2172/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL2xvZy9ibG9jay9Ib29kaWVDb21tYW5kQmxvY2tWZXJzaW9uLmphdmE=) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (?%)` | |
| [...n/scala/org/apache/hudi/HoodieSparkSqlWriter.scala](https://codecov.io/gh/apache/hudi/pull/2172/diff?src=pr&el=tree#diff-aHVkaS1zcGFyay9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2h1ZGkvSG9vZGllU3BhcmtTcWxXcml0ZXIuc2NhbGE=) | `50.95% <0.00%> (ø)` | `0.00% <0.00%> (?%)` | |
| [...he/hudi/cli/commands/HDFSParquetImportCommand.java](https://codecov.io/gh/apache/hudi/pull/2172/diff?src=pr&el=tree#diff-aHVkaS1jbGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpL2NvbW1hbmRzL0hERlNQYXJxdWV0SW1wb3J0Q29tbWFuZC5qYXZh) | `6.66% <0.00%> (ø)` | `1.00% <0.00%> (?%)` | |
| [...he/hudi/common/util/HoodieRecordSizeEstimator.java](https://codecov.io/gh/apache/hudi/pull/2172/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3V0aWwvSG9vZGllUmVjb3JkU2l6ZUVzdGltYXRvci5qYXZh) | `87.50% <0.00%> (ø)` | `3.00% <0.00%> (?%)` | |
| [...e/hudi/common/util/queue/BoundedInMemoryQueue.java](https://codecov.io/gh/apache/hudi/pull/2172/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3V0aWwvcXVldWUvQm91bmRlZEluTWVtb3J5UXVldWUuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (?%)` | |
| [...che/hudi/common/table/timeline/HoodieTimeline.java](https://codecov.io/gh/apache/hudi/pull/2172/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3RpbWVsaW5lL0hvb2RpZVRpbWVsaW5lLmphdmE=) | `93.18% <0.00%> (ø)` | `43.00% <0.00%> (?%)` | |
| ... and [336 more](https://codecov.io/gh/apache/hudi/pull/2172/diff?src=pr&el=tree-more) | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #2172: [HUDI-1338] Adding Delete support to test suite framework
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #2172:
URL: https://github.com/apache/hudi/pull/2172#issuecomment-718825387
oops, I did rebase yesterday night only. guess some other patch was merged in the mean time. Will rebase and let you know.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash commented on a change in pull request #2172: [HUDI-1338] Adding Delete support to test suite framework
Posted by GitBox <gi...@apache.org>.
n3nash commented on a change in pull request #2172:
URL: https://github.com/apache/hudi/pull/2172#discussion_r511708914
##########
File path: hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/generator/GenericRecordFullPayloadGenerator.java
##########
@@ -130,43 +132,60 @@ public GenericRecord getUpdatePayload(GenericRecord record, List<String> blackli
}
/**
- * Create a new {@link GenericRecord} with random values. Not all the fields have value, it is random, and its value
- * is random too.
+ * Create a new {@link GenericRecord} with random values. Not all the fields have value, it is random, and its value is random too.
*
* @param schema Schema to create with.
* @return A {@link GenericRecord} with random value.
*/
protected GenericRecord convertPartial(Schema schema) {
GenericRecord result = new GenericData.Record(schema);
for (Schema.Field f : schema.getFields()) {
- boolean setNull = random.nextBoolean();
- if (!setNull) {
- result.put(f.name(), typeConvert(f));
+ if (f.name().equals(DEFAULT_HOODIE_IS_DELETED_COL)) {
+ result.put(f.name(), false);
} else {
- result.put(f.name(), null);
+ boolean setNull = random.nextBoolean();
+ if (!setNull) {
+ result.put(f.name(), typeConvert(f));
+ } else {
+ result.put(f.name(), null);
+ }
}
}
// TODO : pack remaining bytes into a complex field
return result;
}
/**
- * Set random value to {@link GenericRecord} according to the schema type of field.
- * The field in blacklist will not be set.
+ * Set random value to {@link GenericRecord} according to the schema type of field. The field in blacklist will not be set.
*
- * @param record GenericRecord to randomize.
+ * @param record GenericRecord to randomize.
* @param blacklistFields blacklistFields where the filed will not be randomized.
* @return Randomized GenericRecord.
*/
protected GenericRecord randomize(GenericRecord record, List<String> blacklistFields) {
for (Schema.Field f : record.getSchema().getFields()) {
- if (blacklistFields == null || !blacklistFields.contains(f.name())) {
- record.put(f.name(), typeConvert(f));
+ if (f.name().equals(DEFAULT_HOODIE_IS_DELETED_COL)) {
+ record.put(f.name(), false);
+ } else {
Review comment:
Make this an inline else-if ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2172: [HUDI-1338] Adding Delete support to test suite framework
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2172:
URL: https://github.com/apache/hudi/pull/2172#issuecomment-706720825
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2172?src=pr&el=h1) Report
> Merging [#2172](https://codecov.io/gh/apache/hudi/pull/2172?src=pr&el=desc) into [master](https://codecov.io/gh/apache/hudi/commit/736a9408549f66014bfdfdc72832be11aa9fd39b?el=desc) will **increase** coverage by `43.28%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/hudi/pull/2172?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2172 +/- ##
=============================================
+ Coverage 10.39% 53.68% +43.28%
- Complexity 48 2849 +2801
=============================================
Files 50 359 +309
Lines 1779 16565 +14786
Branches 211 1782 +1571
=============================================
+ Hits 185 8893 +8708
- Misses 1581 6915 +5334
- Partials 13 757 +744
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| #hudicli | `38.37% <ø> (?)` | `193.00 <ø> (?)` | |
| #hudiclient | `100.00% <ø> (ø)` | `0.00 <ø> (ø)` | |
| #hudicommon | `54.70% <ø> (?)` | `1794.00 <ø> (?)` | |
| #hudihadoopmr | `33.05% <ø> (?)` | `181.00 <ø> (?)` | |
| #hudispark | `65.95% <ø> (?)` | `304.00 <ø> (?)` | |
| #huditimelineservice | `62.29% <ø> (?)` | `50.00 <ø> (?)` | |
| #hudiutilities | `70.09% <ø> (+59.69%)` | `327.00 <ø> (+279.00)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2172?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...udi/common/util/queue/BoundedInMemoryExecutor.java](https://codecov.io/gh/apache/hudi/pull/2172/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3V0aWwvcXVldWUvQm91bmRlZEluTWVtb3J5RXhlY3V0b3IuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (?%)` | |
| [...e/hudi/exception/HoodieCorruptedDataException.java](https://codecov.io/gh/apache/hudi/pull/2172/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhjZXB0aW9uL0hvb2RpZUNvcnJ1cHRlZERhdGFFeGNlcHRpb24uamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (?%)` | |
| [...rg/apache/hudi/common/bloom/SimpleBloomFilter.java](https://codecov.io/gh/apache/hudi/pull/2172/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2Jsb29tL1NpbXBsZUJsb29tRmlsdGVyLmphdmE=) | `56.09% <0.00%> (ø)` | `6.00% <0.00%> (?%)` | |
| [...rg/apache/hudi/common/fs/NoOpConsistencyGuard.java](https://codecov.io/gh/apache/hudi/pull/2172/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2ZzL05vT3BDb25zaXN0ZW5jeUd1YXJkLmphdmE=) | `60.00% <0.00%> (ø)` | `3.00% <0.00%> (?%)` | |
| [...he/hudi/common/util/HoodieRecordSizeEstimator.java](https://codecov.io/gh/apache/hudi/pull/2172/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3V0aWwvSG9vZGllUmVjb3JkU2l6ZUVzdGltYXRvci5qYXZh) | `87.50% <0.00%> (ø)` | `3.00% <0.00%> (?%)` | |
| [...va/org/apache/hudi/keygen/BuiltinKeyGenerator.java](https://codecov.io/gh/apache/hudi/pull/2172/diff?src=pr&el=tree#diff-aHVkaS1zcGFyay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9rZXlnZW4vQnVpbHRpbktleUdlbmVyYXRvci5qYXZh) | `90.24% <0.00%> (ø)` | `21.00% <0.00%> (?%)` | |
| [.../apache/hudi/common/bloom/BloomFilterTypeCode.java](https://codecov.io/gh/apache/hudi/pull/2172/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2Jsb29tL0Jsb29tRmlsdGVyVHlwZUNvZGUuamF2YQ==) | `100.00% <0.00%> (ø)` | `1.00% <0.00%> (?%)` | |
| [...che/hudi/common/util/BufferedRandomAccessFile.java](https://codecov.io/gh/apache/hudi/pull/2172/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3V0aWwvQnVmZmVyZWRSYW5kb21BY2Nlc3NGaWxlLmphdmE=) | `51.75% <0.00%> (ø)` | `22.00% <0.00%> (?%)` | |
| [...n/java/org/apache/hudi/common/metrics/Counter.java](https://codecov.io/gh/apache/hudi/pull/2172/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21ldHJpY3MvQ291bnRlci5qYXZh) | `100.00% <0.00%> (ø)` | `4.00% <0.00%> (?%)` | |
| [...ache/hudi/exception/MetadataNotFoundException.java](https://codecov.io/gh/apache/hudi/pull/2172/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhjZXB0aW9uL01ldGFkYXRhTm90Rm91bmRFeGNlcHRpb24uamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (?%)` | |
| ... and [336 more](https://codecov.io/gh/apache/hudi/pull/2172/diff?src=pr&el=tree-more) | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2172: [HUDI-1338] Adding Delete support to test suite framework
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2172:
URL: https://github.com/apache/hudi/pull/2172#issuecomment-706720825
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2172: [HUDI-1338] Adding Delete support to test suite framework
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2172:
URL: https://github.com/apache/hudi/pull/2172#issuecomment-706720825
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2172: [HUDI-1338] Adding Delete support to test suite framework
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2172:
URL: https://github.com/apache/hudi/pull/2172#issuecomment-706720825
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2172?src=pr&el=h1) Report
> Merging [#2172](https://codecov.io/gh/apache/hudi/pull/2172?src=pr&el=desc) into [master](https://codecov.io/gh/apache/hudi/commit/736a9408549f66014bfdfdc72832be11aa9fd39b?el=desc) will **increase** coverage by `43.30%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/hudi/pull/2172?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2172 +/- ##
=============================================
+ Coverage 10.39% 53.70% +43.30%
- Complexity 48 2850 +2802
=============================================
Files 50 359 +309
Lines 1779 16565 +14786
Branches 211 1782 +1571
=============================================
+ Hits 185 8896 +8711
- Misses 1581 6912 +5331
- Partials 13 757 +744
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| #hudicli | `38.37% <ø> (?)` | `193.00 <ø> (?)` | |
| #hudiclient | `100.00% <ø> (ø)` | `0.00 <ø> (ø)` | |
| #hudicommon | `54.73% <ø> (?)` | `1795.00 <ø> (?)` | |
| #hudihadoopmr | `33.05% <ø> (?)` | `181.00 <ø> (?)` | |
| #hudispark | `65.95% <ø> (?)` | `304.00 <ø> (?)` | |
| #huditimelineservice | `62.29% <ø> (?)` | `50.00 <ø> (?)` | |
| #hudiutilities | `70.09% <ø> (+59.69%)` | `327.00 <ø> (+279.00)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2172?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [.../main/java/org/apache/hudi/common/util/Option.java](https://codecov.io/gh/apache/hudi/pull/2172/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3V0aWwvT3B0aW9uLmphdmE=) | `66.66% <0.00%> (ø)` | `23.00% <0.00%> (?%)` | |
| [.../hudi/async/SparkStreamingAsyncCompactService.java](https://codecov.io/gh/apache/hudi/pull/2172/diff?src=pr&el=tree#diff-aHVkaS1zcGFyay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9hc3luYy9TcGFya1N0cmVhbWluZ0FzeW5jQ29tcGFjdFNlcnZpY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (?%)` | |
| [...che/hudi/common/table/timeline/HoodieTimeline.java](https://codecov.io/gh/apache/hudi/pull/2172/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3RpbWVsaW5lL0hvb2RpZVRpbWVsaW5lLmphdmE=) | `93.18% <0.00%> (ø)` | `43.00% <0.00%> (?%)` | |
| [...apache/hudi/common/model/HoodieRecordLocation.java](https://codecov.io/gh/apache/hudi/pull/2172/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL0hvb2RpZVJlY29yZExvY2F0aW9uLmphdmE=) | `33.33% <0.00%> (ø)` | `3.00% <0.00%> (?%)` | |
| [...di/hadoop/realtime/HoodieRealtimeRecordReader.java](https://codecov.io/gh/apache/hudi/pull/2172/diff?src=pr&el=tree#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3JlYWx0aW1lL0hvb2RpZVJlYWx0aW1lUmVjb3JkUmVhZGVyLmphdmE=) | `70.00% <0.00%> (ø)` | `10.00% <0.00%> (?%)` | |
| [.../apache/hudi/exception/TableNotFoundException.java](https://codecov.io/gh/apache/hudi/pull/2172/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhjZXB0aW9uL1RhYmxlTm90Rm91bmRFeGNlcHRpb24uamF2YQ==) | `15.38% <0.00%> (ø)` | `1.00% <0.00%> (?%)` | |
| [...rg/apache/hudi/common/table/HoodieTableConfig.java](https://codecov.io/gh/apache/hudi/pull/2172/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL0hvb2RpZVRhYmxlQ29uZmlnLmphdmE=) | `46.05% <0.00%> (ø)` | `17.00% <0.00%> (?%)` | |
| [...udi/timeline/service/handlers/TimelineHandler.java](https://codecov.io/gh/apache/hudi/pull/2172/diff?src=pr&el=tree#diff-aHVkaS10aW1lbGluZS1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL3RpbWVsaW5lL3NlcnZpY2UvaGFuZGxlcnMvVGltZWxpbmVIYW5kbGVyLmphdmE=) | `40.00% <0.00%> (ø)` | `1.00% <0.00%> (?%)` | |
| [...apache/hudi/cli/commands/HoodieLogFileCommand.java](https://codecov.io/gh/apache/hudi/pull/2172/diff?src=pr&el=tree#diff-aHVkaS1jbGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpL2NvbW1hbmRzL0hvb2RpZUxvZ0ZpbGVDb21tYW5kLmphdmE=) | `81.57% <0.00%> (ø)` | `17.00% <0.00%> (?%)` | |
| [...va/org/apache/hudi/common/util/TablePathUtils.java](https://codecov.io/gh/apache/hudi/pull/2172/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3V0aWwvVGFibGVQYXRoVXRpbHMuamF2YQ==) | `67.56% <0.00%> (ø)` | `16.00% <0.00%> (?%)` | |
| ... and [336 more](https://codecov.io/gh/apache/hudi/pull/2172/diff?src=pr&el=tree-more) | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash commented on a change in pull request #2172: [HUDI-1338] Adding Delete support to test suite framework
Posted by GitBox <gi...@apache.org>.
n3nash commented on a change in pull request #2172:
URL: https://github.com/apache/hudi/pull/2172#discussion_r511709134
##########
File path: hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/writer/DFSDeltaWriterAdapter.java
##########
@@ -40,10 +40,12 @@ public DFSDeltaWriterAdapter(DeltaInputWriter<GenericRecord> deltaInputGenerator
@Override
public List<DeltaWriteStats> write(Iterator<GenericRecord> input) throws IOException {
while (input.hasNext()) {
+ //GenericRecord next = input.next();
Review comment:
Should these be uncommented or removed ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash commented on a change in pull request #2172: [HUDI-1338] Adding Delete support to test suite framework
Posted by GitBox <gi...@apache.org>.
n3nash commented on a change in pull request #2172:
URL: https://github.com/apache/hudi/pull/2172#discussion_r511707782
##########
File path: hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/dag/nodes/UpsertNode.java
##########
@@ -23,6 +23,7 @@
import org.apache.hudi.integ.testsuite.HoodieTestSuiteWriter;
import org.apache.hudi.integ.testsuite.configuration.DeltaConfig.Config;
import org.apache.hudi.integ.testsuite.generator.DeltaGenerator;
+
Review comment:
Nit: revert whitespace changes
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash commented on a change in pull request #2172: [HUDI-1338] Adding Delete support to test suite framework
Posted by GitBox <gi...@apache.org>.
n3nash commented on a change in pull request #2172:
URL: https://github.com/apache/hudi/pull/2172#discussion_r511707521
##########
File path: docker/demo/config/test-suite/complex-dag-cow.yaml
##########
@@ -93,3 +93,50 @@ second_hive_query:
result2: 11900
type: HiveQueryNode
deps: second_upsert
+fourth_insert:
+ config:
+ record_size: 70000
+ num_insert_partitions: 1
+ repeat_count: 1
+ num_records_insert: 1000
+ deps: second_hive_query
+ type: InsertNode
+third_hive_query:
+ config:
+ hive_props:
+ prop2: "set spark.yarn.queue="
+ prop3: "set hive.strict.checks.large.query=false"
+ prop4: "set hive.stats.autogather=false"
+ hive_queries:
+ query1: "select count(*) from testdb.table1 group by `_row_key` having count(*) > 1"
+ result1: 0
+ query2: "select count(*) from testdb.table1"
+ result2: 12900
+ type: HiveQueryNode
+ deps: fourth_insert
+first_delete:
+ config:
+ record_size: 70000
+ num_partitions_delete: 1
+ num_records_delete: 200
+ deps: third_hive_query
+ type: DeleteNode
+fourth_hive_sync:
+ config:
+ queue_name: "adhoc"
+ engine: "mr"
+ type: HiveSyncNode
+ deps: first_delete
+fourth_hive_query:
+ config:
+ hive_props:
+ prop2: "set spark.yarn.queue="
Review comment:
same here
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash commented on a change in pull request #2172: [HUDI-1338] Adding Delete support to test suite framework
Posted by GitBox <gi...@apache.org>.
n3nash commented on a change in pull request #2172:
URL: https://github.com/apache/hudi/pull/2172#discussion_r511708354
##########
File path: hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/generator/DeltaGenerator.java
##########
@@ -155,6 +156,42 @@ public DeltaGenerator(DeltaConfig deltaOutputConfig, JavaSparkContext jsc, Spark
}
}
+ public JavaRDD<GenericRecord> generateDeletes(Config config) throws IOException {
+ if (deltaOutputConfig.getDeltaOutputMode() == DeltaOutputMode.DFS) {
+ DeltaInputReader deltaInputReader = null;
+ JavaRDD<GenericRecord> adjustedRDD = null;
+
+ if (config.getNumDeletePartitions() < 1) {
+ // randomly generate deletes for a given number of records without regard to partitions and files
+ deltaInputReader = new DFSAvroDeltaInputReader(sparkSession, schemaStr,
+ ((DFSDeltaConfig) deltaOutputConfig).getDeltaBasePath(), Option.empty(), Option.empty());
+ adjustedRDD = deltaInputReader.read(config.getNumRecordsDelete());
Review comment:
Should this be renamed from adjustedRDD to something else ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash commented on pull request #2172: [HUDI-1338] Adding Delete support to test suite framework
Posted by GitBox <gi...@apache.org>.
n3nash commented on pull request #2172:
URL: https://github.com/apache/hudi/pull/2172#issuecomment-718771774
@nsivabalan can you please rebase, I'll merge it then
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #2172: [HUDI-1338] Adding Delete support to test suite framework
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #2172:
URL: https://github.com/apache/hudi/pull/2172#issuecomment-718336881
@n3nash : have addressed all comments. Feel free to take a 2nd pass.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io commented on pull request #2172: [HUDI-1338] Adding Delete support to test suite framework
Posted by GitBox <gi...@apache.org>.
codecov-io commented on pull request #2172:
URL: https://github.com/apache/hudi/pull/2172#issuecomment-706720825
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2172?src=pr&el=h1) Report
> Merging [#2172](https://codecov.io/gh/apache/hudi/pull/2172?src=pr&el=desc) into [master](https://codecov.io/gh/apache/hudi/commit/788d236c443eb4ced819f9305ed8e0460b5984b7?el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/hudi/pull/2172?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2172 +/- ##
============================================
- Coverage 53.61% 53.60% -0.02%
- Complexity 2845 2846 +1
============================================
Files 359 359
Lines 16535 16548 +13
Branches 1777 1780 +3
============================================
+ Hits 8866 8870 +4
- Misses 6912 6920 +8
- Partials 757 758 +1
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| #hudicli | `38.37% <ø> (ø)` | `193.00 <ø> (ø)` | |
| #hudiclient | `100.00% <ø> (ø)` | `0.00 <ø> (ø)` | |
| #hudicommon | `54.70% <ø> (-0.04%)` | `1793.00 <ø> (ø)` | |
| #hudihadoopmr | `33.05% <ø> (ø)` | `181.00 <ø> (ø)` | |
| #hudispark | `65.48% <ø> (-0.03%)` | `304.00 <ø> (+1.00)` | :arrow_down: |
| #huditimelineservice | `62.29% <ø> (ø)` | `50.00 <ø> (ø)` | |
| #hudiutilities | `69.98% <ø> (ø)` | `325.00 <ø> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2172?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...in/scala/org/apache/hudi/IncrementalRelation.scala](https://codecov.io/gh/apache/hudi/pull/2172/diff?src=pr&el=tree#diff-aHVkaS1zcGFyay9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2h1ZGkvSW5jcmVtZW50YWxSZWxhdGlvbi5zY2FsYQ==) | `76.19% <0.00%> (-2.30%)` | `20.00% <0.00%> (+1.00%)` | :arrow_down: |
| [.../org/apache/hudi/common/model/HoodieTableType.java](https://codecov.io/gh/apache/hudi/pull/2172/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL0hvb2RpZVRhYmxlVHlwZS5qYXZh) | `100.00% <0.00%> (ø)` | `1.00% <0.00%> (ø%)` | |
| [.../apache/hudi/common/table/TableSchemaResolver.java](https://codecov.io/gh/apache/hudi/pull/2172/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL1RhYmxlU2NoZW1hUmVzb2x2ZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (ø%)` | |
| [...i/common/model/OverwriteWithLatestAvroPayload.java](https://codecov.io/gh/apache/hudi/pull/2172/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL092ZXJ3cml0ZVdpdGhMYXRlc3RBdnJvUGF5bG9hZC5qYXZh) | `64.70% <0.00%> (ø)` | `10.00% <0.00%> (ø%)` | |
| [...del/OverwriteNonDefaultsWithLatestAvroPayload.java](https://codecov.io/gh/apache/hudi/pull/2172/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL092ZXJ3cml0ZU5vbkRlZmF1bHRzV2l0aExhdGVzdEF2cm9QYXlsb2FkLmphdmE=) | `78.94% <0.00%> (ø)` | `5.00% <0.00%> (ø%)` | |
| [...main/scala/org/apache/hudi/DataSourceOptions.scala](https://codecov.io/gh/apache/hudi/pull/2172/diff?src=pr&el=tree#diff-aHVkaS1zcGFyay9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2h1ZGkvRGF0YVNvdXJjZU9wdGlvbnMuc2NhbGE=) | `94.82% <0.00%> (+0.09%)` | `0.00% <0.00%> (ø%)` | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash commented on a change in pull request #2172: [HUDI-1338] Adding Delete support to test suite framework
Posted by GitBox <gi...@apache.org>.
n3nash commented on a change in pull request #2172:
URL: https://github.com/apache/hudi/pull/2172#discussion_r511707483
##########
File path: docker/demo/config/test-suite/complex-dag-cow.yaml
##########
@@ -93,3 +93,50 @@ second_hive_query:
result2: 11900
type: HiveQueryNode
deps: second_upsert
+fourth_insert:
+ config:
+ record_size: 70000
+ num_insert_partitions: 1
+ repeat_count: 1
+ num_records_insert: 1000
+ deps: second_hive_query
+ type: InsertNode
+third_hive_query:
+ config:
+ hive_props:
+ prop2: "set spark.yarn.queue="
Review comment:
Is this prod needed ? Also the naming should start from prop1 ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash commented on a change in pull request #2172: [HUDI-1338] Adding Delete support to test suite framework
Posted by GitBox <gi...@apache.org>.
n3nash commented on a change in pull request #2172:
URL: https://github.com/apache/hudi/pull/2172#discussion_r511708422
##########
File path: hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/generator/DeltaGenerator.java
##########
@@ -155,6 +156,42 @@ public DeltaGenerator(DeltaConfig deltaOutputConfig, JavaSparkContext jsc, Spark
}
}
+ public JavaRDD<GenericRecord> generateDeletes(Config config) throws IOException {
+ if (deltaOutputConfig.getDeltaOutputMode() == DeltaOutputMode.DFS) {
+ DeltaInputReader deltaInputReader = null;
+ JavaRDD<GenericRecord> adjustedRDD = null;
+
+ if (config.getNumDeletePartitions() < 1) {
+ // randomly generate deletes for a given number of records without regard to partitions and files
+ deltaInputReader = new DFSAvroDeltaInputReader(sparkSession, schemaStr,
+ ((DFSDeltaConfig) deltaOutputConfig).getDeltaBasePath(), Option.empty(), Option.empty());
+ adjustedRDD = deltaInputReader.read(config.getNumRecordsDelete());
Review comment:
Ohh I see, you are trying to reduce number of variables used...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash commented on a change in pull request #2172: [HUDI-1338] Adding Delete support to test suite framework
Posted by GitBox <gi...@apache.org>.
n3nash commented on a change in pull request #2172:
URL: https://github.com/apache/hudi/pull/2172#discussion_r511707633
##########
File path: hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/configuration/DeltaConfig.java
##########
@@ -118,6 +125,10 @@ public int getNumUpsertPartitions() {
return Integer.valueOf(configsMap.getOrDefault(NUM_PARTITIONS_UPSERT, 0).toString());
}
+ public int getNumDeletePartitions() {
+ return Integer.valueOf(configsMap.getOrDefault(NUM_PARTITIONS_DELETE, 1).toString());
Review comment:
What does default value "1" mean in this case ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org