You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/09/20 10:33:59 UTC
[GitHub] [hudi] Karl-WangSK opened a new pull request #2096: [HUDI-1284] preCombine all HoodieRecords and update all fields(which is not DefaultValue) according to orderingVal
Karl-WangSK opened a new pull request #2096:
URL: https://github.com/apache/hudi/pull/2096
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contributing.html before opening a pull request.*
## What is the purpose of the pull request
preCombine all HoodieRecords and update all fields(which is not DefaultValue) according to orderingVal
## Brief change log
When more than one HoodieRecord have the same HoodieKey, this function combines all fields(which is not DefaultValue)
before attempting to insert/upsert (if combining turned on in HoodieClientConfig).
eg: 1)
```
In preCombine: (suppose Default value is null)
id name age money ts
1 Karl null 30 0.0 (orderingVal=1)
1 null 18 40 0.0 (orderingVal=2)
```
```
After:
id name age money ts
1 Karl 18 40 0.0
```
## Verify this pull request
Added one test in TestOverwriteWithLatestAvroPayload to verify the change
## Committer checklist
- [x] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Karl-WangSK commented on pull request #2096: [HUDI-1284] preCombine all HoodieRecords and update all fields(which is not DefaultValue) according to orderingVal
Posted by GitBox <gi...@apache.org>.
Karl-WangSK commented on pull request #2096:
URL: https://github.com/apache/hudi/pull/2096#issuecomment-696089004
@vinothchandar hi. Can you look at this pr when you are freeeee?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #2096: [HUDI-1284] preCombine all HoodieRecords and update all fields according to orderingVal
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #2096:
URL: https://github.com/apache/hudi/pull/2096#discussion_r493514951
##########
File path: hudi-client/src/main/java/org/apache/hudi/client/HoodieWriteClient.java
##########
@@ -190,15 +190,27 @@ protected void rollBackInflightBootstrap() {
table.validateUpsertSchema();
setOperationType(WriteOperationType.UPSERT);
this.asyncCleanerService = AsyncCleanerService.startAsyncCleaningIfEnabled(this, instantTime);
- HoodieWriteMetadata result = table.upsert(jsc, instantTime, records);
+ HoodieWriteMetadata result = table.upsert(jsc, instantTime, records, null);
+ if (result.getIndexLookupDuration().isPresent()) {
+ metrics.updateIndexMetrics(LOOKUP_STR, result.getIndexLookupDuration().get().toMillis());
+ }
+ return postWrite(result, instantTime, table);
+ }
+
+ public JavaRDD<WriteStatus> upsert(JavaRDD<HoodieRecord<T>> records, final String instantTime, String schema) {
+ HoodieTable<T> table = getTableAndInitCtx(WriteOperationType.UPSERT, instantTime);
+ table.validateUpsertSchema();
+ setOperationType(WriteOperationType.UPSERT);
+ this.asyncCleanerService = AsyncCleanerService.startAsyncCleaningIfEnabled(this, instantTime);
+ HoodieWriteMetadata result = table.upsert(jsc, instantTime, records, schema);
Review comment:
this is a unrelated change?
##########
File path: hudi-client/src/main/java/org/apache/hudi/client/HoodieWriteClient.java
##########
@@ -190,15 +190,27 @@ protected void rollBackInflightBootstrap() {
table.validateUpsertSchema();
setOperationType(WriteOperationType.UPSERT);
this.asyncCleanerService = AsyncCleanerService.startAsyncCleaningIfEnabled(this, instantTime);
- HoodieWriteMetadata result = table.upsert(jsc, instantTime, records);
+ HoodieWriteMetadata result = table.upsert(jsc, instantTime, records, null);
+ if (result.getIndexLookupDuration().isPresent()) {
+ metrics.updateIndexMetrics(LOOKUP_STR, result.getIndexLookupDuration().get().toMillis());
+ }
+ return postWrite(result, instantTime, table);
+ }
+
+ public JavaRDD<WriteStatus> upsert(JavaRDD<HoodieRecord<T>> records, final String instantTime, String schema) {
+ HoodieTable<T> table = getTableAndInitCtx(WriteOperationType.UPSERT, instantTime);
+ table.validateUpsertSchema();
+ setOperationType(WriteOperationType.UPSERT);
+ this.asyncCleanerService = AsyncCleanerService.startAsyncCleaningIfEnabled(this, instantTime);
+ HoodieWriteMetadata result = table.upsert(jsc, instantTime, records, schema);
Review comment:
this is an unrelated change?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Karl-WangSK commented on pull request #2096: [HUDI-1284] preCombine all HoodieRecords and update all fields according to orderingVal
Posted by GitBox <gi...@apache.org>.
Karl-WangSK commented on pull request #2096:
URL: https://github.com/apache/hudi/pull/2096#issuecomment-697155627
@leesf
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on pull request #2096: [HUDI-1284] preCombine all HoodieRecords and update all fields(which is not DefaultValue) according to orderingVal
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #2096:
URL: https://github.com/apache/hudi/pull/2096#issuecomment-696182257
@Karl-WangSK Will do!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Karl-WangSK commented on pull request #2096: [HUDI-1284] preCombine all HoodieRecords and update all fields(which is not DefaultValue) according to orderingVal
Posted by GitBox <gi...@apache.org>.
Karl-WangSK commented on pull request #2096:
URL: https://github.com/apache/hudi/pull/2096#issuecomment-696089004
@vinothchandar hi. Can you look at this pr when you are freeeee?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on pull request #2096: [HUDI-1284] preCombine all HoodieRecords and update all fields(which is not DefaultValue) according to orderingVal
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #2096:
URL: https://github.com/apache/hudi/pull/2096#issuecomment-696182257
@Karl-WangSK Will do!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Karl-WangSK closed pull request #2096: [HUDI-1284] preCombine all HoodieRecords and update all fields according to orderingVal
Posted by GitBox <gi...@apache.org>.
Karl-WangSK closed pull request #2096:
URL: https://github.com/apache/hudi/pull/2096
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on pull request #2096: [HUDI-1284] preCombine all HoodieRecords and update all fields according to orderingVal
Posted by GitBox <gi...@apache.org>.
leesf commented on pull request #2096:
URL: https://github.com/apache/hudi/pull/2096#issuecomment-697323825
@Karl-WangSK Thanks for your contributing, I see there are lots of unrelated changes, pls rebase to latest master branch, thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #2096: [HUDI-1284] preCombine all HoodieRecords and update all fields according to orderingVal
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #2096:
URL: https://github.com/apache/hudi/pull/2096#discussion_r493515328
##########
File path: hudi-client/src/main/java/org/apache/hudi/client/HoodieWriteClient.java
##########
@@ -190,15 +190,27 @@ protected void rollBackInflightBootstrap() {
table.validateUpsertSchema();
setOperationType(WriteOperationType.UPSERT);
this.asyncCleanerService = AsyncCleanerService.startAsyncCleaningIfEnabled(this, instantTime);
- HoodieWriteMetadata result = table.upsert(jsc, instantTime, records);
+ HoodieWriteMetadata result = table.upsert(jsc, instantTime, records, null);
+ if (result.getIndexLookupDuration().isPresent()) {
+ metrics.updateIndexMetrics(LOOKUP_STR, result.getIndexLookupDuration().get().toMillis());
+ }
+ return postWrite(result, instantTime, table);
+ }
+
+ public JavaRDD<WriteStatus> upsert(JavaRDD<HoodieRecord<T>> records, final String instantTime, String schema) {
+ HoodieTable<T> table = getTableAndInitCtx(WriteOperationType.UPSERT, instantTime);
+ table.validateUpsertSchema();
+ setOperationType(WriteOperationType.UPSERT);
+ this.asyncCleanerService = AsyncCleanerService.startAsyncCleaningIfEnabled(this, instantTime);
+ HoodieWriteMetadata result = table.upsert(jsc, instantTime, records, schema);
if (result.getIndexLookupDuration().isPresent()) {
metrics.updateIndexMetrics(LOOKUP_STR, result.getIndexLookupDuration().get().toMillis());
}
return postWrite(result, instantTime, table);
}
/**
- * Upserts the given prepared records into the Hoodie table, at the supplied instantTime.
+ * Upserts the given prepared records into the Hoodie table,/TestHoodieClientOnCopyOnWriteStorage at the supplied instantTime.
Review comment:
ditto
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org