You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/02/09 14:29:34 UTC
[GitHub] [hudi] caidezhi opened a new pull request #2560: [HUDI-1606]fix HoodieJavaWriteClientExample
caidezhi opened a new pull request #2560:
URL: https://github.com/apache/hudi/pull/2560
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contributing.html before opening a pull request.*
## What is the purpose of the pull request
HoodieJavaWriteClientExample fail with exception, the PR is to fix the issue.
## Brief change log
org.apache.hudi.examples.java.HoodieJavaWriteClientExample
## Verify this pull request
This pull request is already covered by existing tests: HoodieJavaWriteClientExample.main()
I manually verified the change by running a job locally.
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #2560: [HUDI-1606]fix HoodieJavaWriteClientExample
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #2560:
URL: https://github.com/apache/hudi/pull/2560#discussion_r573669765
##########
File path: hudi-examples/src/main/java/org/apache/hudi/examples/java/HoodieJavaWriteClientExample.java
##########
@@ -93,7 +94,8 @@ public static void main(String[] args) throws Exception {
List<HoodieRecord<HoodieAvroPayload>> recordsSoFar = new ArrayList<>(records);
List<HoodieRecord<HoodieAvroPayload>> writeRecords =
recordsSoFar.stream().map(r -> new HoodieRecord<HoodieAvroPayload>(r)).collect(Collectors.toList());
- client.upsert(writeRecords, newCommitTime);
+ List<WriteStatus> insertStatus = client.upsert(writeRecords, newCommitTime);
+ client.commit(newCommitTime,insertStatus);
Review comment:
would we just change the BaseJavaCommitActionExecutor#execute method to add `updateIndexAndCommitIfNeeded` method and align with BaseSparkCommitActionExecutor?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf closed pull request #2560: [HUDI-1606]align BaseJavaCommitActionExecuto#execute method with BaseSparkCommitActionExecutor
Posted by GitBox <gi...@apache.org>.
leesf closed pull request #2560:
URL: https://github.com/apache/hudi/pull/2560
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] caidezhi commented on a change in pull request #2560: [HUDI-1606]fix HoodieJavaWriteClientExample
Posted by GitBox <gi...@apache.org>.
caidezhi commented on a change in pull request #2560:
URL: https://github.com/apache/hudi/pull/2560#discussion_r573811921
##########
File path: hudi-examples/src/main/java/org/apache/hudi/examples/java/HoodieJavaWriteClientExample.java
##########
@@ -93,7 +94,8 @@ public static void main(String[] args) throws Exception {
List<HoodieRecord<HoodieAvroPayload>> recordsSoFar = new ArrayList<>(records);
List<HoodieRecord<HoodieAvroPayload>> writeRecords =
recordsSoFar.stream().map(r -> new HoodieRecord<HoodieAvroPayload>(r)).collect(Collectors.toList());
- client.upsert(writeRecords, newCommitTime);
+ List<WriteStatus> insertStatus = client.upsert(writeRecords, newCommitTime);
+ client.commit(newCommitTime,insertStatus);
Review comment:
@leesf many thanks for your review. You suggestion make more sense and i have change the code accordingly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io commented on pull request #2560: [HUDI-1606]fix HoodieJavaWriteClientExample
Posted by GitBox <gi...@apache.org>.
codecov-io commented on pull request #2560:
URL: https://github.com/apache/hudi/pull/2560#issuecomment-775997974
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2560?src=pr&el=h1) Report
> Merging [#2560](https://codecov.io/gh/apache/hudi/pull/2560?src=pr&el=desc) (dc9fdb0) into [master](https://codecov.io/gh/apache/hudi/commit/c30481f4b0d7552a5ac73bb9955f6326f90ba329?el=desc) (c30481f) will **increase** coverage by `18.60%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/hudi/pull/2560?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2560 +/- ##
=============================================
+ Coverage 50.90% 69.51% +18.60%
+ Complexity 3167 357 -2810
=============================================
Files 433 53 -380
Lines 19806 1929 -17877
Branches 2032 229 -1803
=============================================
- Hits 10083 1341 -8742
+ Misses 8904 456 -8448
+ Partials 819 132 -687
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| hudicli | `?` | `?` | |
| hudiclient | `?` | `?` | |
| hudicommon | `?` | `?` | |
| hudiflink | `?` | `?` | |
| hudihadoopmr | `?` | `?` | |
| hudisparkdatasource | `?` | `?` | |
| hudisync | `?` | `?` | |
| huditimelineservice | `?` | `?` | |
| hudiutilities | `69.51% <ø> (+0.05%)` | `0.00 <ø> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2560?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...di/common/bootstrap/index/HFileBootstrapIndex.java](https://codecov.io/gh/apache/hudi/pull/2560/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2Jvb3RzdHJhcC9pbmRleC9IRmlsZUJvb3RzdHJhcEluZGV4LmphdmE=) | | | |
| [...i-cli/src/main/java/org/apache/hudi/cli/Table.java](https://codecov.io/gh/apache/hudi/pull/2560/diff?src=pr&el=tree#diff-aHVkaS1jbGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpL1RhYmxlLmphdmE=) | | | |
| [...che/hudi/exception/InvalidHoodiePathException.java](https://codecov.io/gh/apache/hudi/pull/2560/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhjZXB0aW9uL0ludmFsaWRIb29kaWVQYXRoRXhjZXB0aW9uLmphdmE=) | | | |
| [.../java/org/apache/hudi/common/util/HoodieTimer.java](https://codecov.io/gh/apache/hudi/pull/2560/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3V0aWwvSG9vZGllVGltZXIuamF2YQ==) | | | |
| [.../common/table/log/block/HoodieLogBlockVersion.java](https://codecov.io/gh/apache/hudi/pull/2560/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL2xvZy9ibG9jay9Ib29kaWVMb2dCbG9ja1ZlcnNpb24uamF2YQ==) | | | |
| [...util/jvm/OpenJ9MemoryLayoutSpecification64bit.java](https://codecov.io/gh/apache/hudi/pull/2560/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3V0aWwvanZtL09wZW5KOU1lbW9yeUxheW91dFNwZWNpZmljYXRpb242NGJpdC5qYXZh) | | | |
| [...org/apache/hudi/HoodieDatasetBulkInsertHelper.java](https://codecov.io/gh/apache/hudi/pull/2560/diff?src=pr&el=tree#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3Bhcmsvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvSG9vZGllRGF0YXNldEJ1bGtJbnNlcnRIZWxwZXIuamF2YQ==) | | | |
| [...udi/spark3/internal/HoodieWriterCommitMessage.java](https://codecov.io/gh/apache/hudi/pull/2560/diff?src=pr&el=tree#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3BhcmszL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL3NwYXJrMy9pbnRlcm5hbC9Ib29kaWVXcml0ZXJDb21taXRNZXNzYWdlLmphdmE=) | | | |
| [...udi/timeline/service/handlers/TimelineHandler.java](https://codecov.io/gh/apache/hudi/pull/2560/diff?src=pr&el=tree#diff-aHVkaS10aW1lbGluZS1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL3RpbWVsaW5lL3NlcnZpY2UvaGFuZGxlcnMvVGltZWxpbmVIYW5kbGVyLmphdmE=) | | | |
| [...a/org/apache/hudi/common/util/CompactionUtils.java](https://codecov.io/gh/apache/hudi/pull/2560/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3V0aWwvQ29tcGFjdGlvblV0aWxzLmphdmE=) | | | |
| ... and [371 more](https://codecov.io/gh/apache/hudi/pull/2560/diff?src=pr&el=tree-more) | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #2560: [HUDI-1606]align BaseJavaCommitActionExecuto#execute method with BaseSparkCommitActionExecutor
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #2560:
URL: https://github.com/apache/hudi/pull/2560#discussion_r579808883
##########
File path: hudi-client/hudi-java-client/src/main/java/org/apache/hudi/table/action/commit/BaseJavaCommitActionExecutor.java
##########
@@ -132,6 +132,11 @@ protected void updateIndex(List<WriteStatus> writeStatuses, HoodieWriteMetadata<
result.setWriteStatuses(statuses);
}
+ protected void updateIndexAndCommitIfNeeded(List<WriteStatus> writeStatuses, HoodieWriteMetadata<List<WriteStatus>> result) {
Review comment:
@caidezhi Sorry for the delay, there is a PR https://github.com/apache/hudi/pull/2382 to support COW in Java client which includes the changes here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] caidezhi commented on a change in pull request #2560: [HUDI-1606]align BaseJavaCommitActionExecuto#execute method with BaseSparkCommitActionExecutor
Posted by GitBox <gi...@apache.org>.
caidezhi commented on a change in pull request #2560:
URL: https://github.com/apache/hudi/pull/2560#discussion_r579963408
##########
File path: hudi-client/hudi-java-client/src/main/java/org/apache/hudi/table/action/commit/BaseJavaCommitActionExecutor.java
##########
@@ -132,6 +132,11 @@ protected void updateIndex(List<WriteStatus> writeStatuses, HoodieWriteMetadata<
result.setWriteStatuses(statuses);
}
+ protected void updateIndexAndCommitIfNeeded(List<WriteStatus> writeStatuses, HoodieWriteMetadata<List<WriteStatus>> result) {
Review comment:
@leesf if it is already covered by other PR, please feel free to close this one. Thanks for your review.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2560: [HUDI-1606]align BaseJavaCommitActionExecuto#execute method with BaseSparkCommitActionExecutor
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2560:
URL: https://github.com/apache/hudi/pull/2560#issuecomment-775997974
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2560?src=pr&el=h1) Report
> Merging [#2560](https://codecov.io/gh/apache/hudi/pull/2560?src=pr&el=desc) (835aed0) into [master](https://codecov.io/gh/apache/hudi/commit/b0010bf3b449a9d2e01955b0746b795e22e577db?el=desc) (b0010bf) will **increase** coverage by `0.00%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/hudi/pull/2560?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2560 +/- ##
=========================================
Coverage 51.15% 51.15%
- Complexity 3212 3213 +1
=========================================
Files 436 436
Lines 19987 19987
Branches 2057 2057
=========================================
+ Hits 10224 10225 +1
Misses 8922 8922
+ Partials 841 840 -1
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| hudicli | `36.90% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudiclient | `100.00% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudicommon | `51.39% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudiflink | `45.44% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudihadoopmr | `33.16% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudisparkdatasource | `69.73% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudisync | `48.61% <ø> (ø)` | `0.00 <ø> (ø)` | |
| huditimelineservice | `66.49% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudiutilities | `69.44% <ø> (+0.05%)` | `0.00 <ø> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2560?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/2560/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `70.35% <0.00%> (+0.35%)` | `51.00% <0.00%> (+1.00%)` | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org