You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/01/31 12:06:12 UTC
[GitHub] [hudi] stayrascal opened a new pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
stayrascal opened a new pull request #4724:
URL: https://github.com/apache/hudi/pull/4724
…e case
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.*
## What is the purpose of the pull request
Add a customized payload `PartialOverwriteWithLatestAvroPayload` to support partial overwrite records fields.
For example, two incoming record batch A(1, null, 1, 1) and B(2, 2, null, 2) will be combined as the result (2, 2, 1, 2)
*(For example: This pull request adds quick-start document.)*
## Brief change log
- *Add a `PartialOverwriteWithLatestAvroPayload` to support partial overwrite records*
- *Add a config `partial.overwrite.enabled` to control if enable partial overwrite*
- *Change the ValueState of `BucketAssignFunction` from `HoodieRecordGlobalLocation` to `HoodieRecord`*
- Support partition path changed case
- Merge `HoodieRecord` before storing the record in ValueSate
- Copy and sink a new record from state to downstream operator the record partition is changed
- Add `compareTo` to compare two HoodieRecords to chose their sequence, because cannot get the sequence from `#preCombine` method if merge two records in `#preCombine` method
- Add `schema` filed(String) in new AvroPayload, which will be used to `#preCombine` two records
- Change `BootstrapOperator` to load `HoodieRecord` with data instead of only HoddieKey
- Change `RowDataToHoodieFunction` to create payload with schema
## Verify this pull request
- *Added PartialOverwriteWithLatestAvroPayloadTest to verify the change.*
- *Manually verified the change by running a job locally with both `preCombine` and `combineAndGetUpdateValue` methods*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1025731697
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d73d7f503615583bd233b1ba974e7c78125c41bb Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1069123625
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
}, {
"hash" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178",
"triggerID" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"triggerType" : "PUSH"
}, {
"hash" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6320",
"triggerID" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"triggerType" : "PUSH"
}, {
"hash" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6357",
"triggerID" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"triggerType" : "PUSH"
}, {
"hash" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6706",
"triggerID" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"triggerType" : "PUSH"
}, {
"hash" : "d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a86b7ff2d204900374751371cd2c8c36e76ea4b8 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6706)
* d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1075846199
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
}, {
"hash" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178",
"triggerID" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"triggerType" : "PUSH"
}, {
"hash" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6320",
"triggerID" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"triggerType" : "PUSH"
}, {
"hash" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6357",
"triggerID" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"triggerType" : "PUSH"
}, {
"hash" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6706",
"triggerID" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"triggerType" : "PUSH"
}, {
"hash" : "d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7000",
"triggerID" : "d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5",
"triggerType" : "PUSH"
}, {
"hash" : "e89fd60b77fbd366205df5e60777ced881b6d795",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7203",
"triggerID" : "e89fd60b77fbd366205df5e60777ced881b6d795",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e89fd60b77fbd366205df5e60777ced881b6d795 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7203)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1031367162
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 21df6fe5ac130bfca73a7e3f528d8ef1df66c074 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#discussion_r801233272
##########
File path: hudi-flink/src/main/java/org/apache/hudi/sink/partitioner/BucketAssignFunction.java
##########
@@ -141,27 +142,41 @@ public void snapshotState(FunctionSnapshotContext context) {
@Override
public void initializeState(FunctionInitializationContext context) {
- ValueStateDescriptor<HoodieRecordGlobalLocation> indexStateDesc =
+ ValueStateDescriptor<HoodieRecord> indexStateDesc =
Review comment:
This would increase the state size significantly. We better avoid this with better solution, and why we must sore full record instead of index here ?
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/PartialOverwriteWithLatestAvroPayload.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.hudi.common.util.Option;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Objects;
+
+import static org.apache.hudi.avro.HoodieAvroUtils.bytesToAvro;
+
+/**
+ * The only difference with {@link OverwriteNonDefaultsWithLatestAvroPayload} is that it supports
+ * merging the latest non-null partial fields with the old record instead of replacing the whole record.
+ * And merging the non-null fields during preCombine multiple records with same record key instead of choosing the latest record based on ordering field.
+ *
+ * <p> Regarding #combineAndGetUpdateValue, Assuming a {@link GenericRecord} has row schema: (f0 int , f1 int, f2 int).
+ * The first record value is: (1, 2, 3), the second record value is: (4, 5, null) with the field f2 value as null.
+ * Calling the #combineAndGetUpdateValue method of the two records returns record: (4, 5, 3).
+ * Note that field f2 value is ignored because it is null. </p>
+ *
+ * <p> Regarding #preCombine, Assuming a {@link GenericRecord} has row schema: (f0 int , f1 int, f2 int, o1 int),
+ * and initial two {@link PartialOverwriteWithLatestAvroPayload} with different ordering value.
+ * The first record value is (1, null, 1, 1) with the filed f1 value as null, the second value is: (2, 2, null, 2) with the f2 value as null.
+ * Calling the #preCombine method of the two records returns record: (2, 2, 1, 2).
+ * Note:
+ * <ol>
+ * <li>the field f0 value is 2 because the ordering value of second record is bigger.</li>
+ * <li>the filed f1 value is 2 because the f2 value of first record is null.</li>
+ * <li>the filed f2 value is 1 because the f2 value of second record is null.</li>
+ * <li>the filed o1 value is 2 because the ordering value of second record is bigger.</li>
+ * </ol>
+ *
+ * </p>
+ */
+public class PartialOverwriteWithLatestAvroPayload extends OverwriteWithLatestAvroPayload {
+
+ public PartialOverwriteWithLatestAvroPayload(GenericRecord record, Comparable orderingVal) {
+ this(record, orderingVal, null);
+ }
+
+ public PartialOverwriteWithLatestAvroPayload(GenericRecord record, Comparable orderingVal, String schema) {
+ super(record, orderingVal, schema);
+ }
+
+ public PartialOverwriteWithLatestAvroPayload(Option<GenericRecord> record) {
+ super(record); // natural order
+ }
+
+ @Override
+ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord currentValue, Schema schema) throws IOException {
+ if (recordBytes.length == 0) {
+ return Option.empty();
+ }
+
+ GenericRecord incomingRecord = bytesToAvro(recordBytes, schema);
+ if (isDeleteRecord(incomingRecord)) {
+ return Option.empty();
+ }
+
+ GenericRecord currentRecord = (GenericRecord) currentValue;
+ List<Schema.Field> fields = schema.getFields();
+ fields.forEach(field -> {
+ Object value = incomingRecord.get(field.name());
+ if (Objects.nonNull(value)) {
+ currentRecord.put(field.name(), value);
+ }
+ });
+
+ return Option.of(currentRecord);
+ }
+
+ @Override
+ public int compareTo(OverwriteWithLatestAvroPayload oldValue) {
+ return orderingVal.compareTo(oldValue.orderingVal);
+ }
+
+ @Override
+ public OverwriteWithLatestAvroPayload preCombine(OverwriteWithLatestAvroPayload oldValue) {
+ if (null == this.schema || null == oldValue.schema) {
+ return super.preCombine(oldValue);
+ }
+
+ try {
+ Schema schema = new Schema.Parser().parse(this.schema);
+ Option<IndexedRecord> incomingOption = getInsertValue(new Schema.Parser().parse(this.schema));
+ Option<IndexedRecord> insertRecordOption = oldValue.getInsertValue(new Schema.Parser().parse(oldValue.schema));
+
+ if (incomingOption.isPresent() && insertRecordOption.isPresent()) {
+ GenericRecord currentRecord = (GenericRecord) incomingOption.get();
+ GenericRecord insertRecord = (GenericRecord) insertRecordOption.get();
+ boolean chooseCurrent = this.orderingVal.compareTo(oldValue.orderingVal) > 0;
+
+ if (!isDeleteRecord(insertRecord) && !isDeleteRecord(currentRecord)) {
+ schema.getFields().forEach(field -> {
+ Object insertValue = insertRecord.get(field.name());
+ Object currentValue = currentRecord.get(field.name());
+ currentRecord.put(field.name(), mergeValue(currentValue, insertValue, chooseCurrent));
+ });
+ return new PartialOverwriteWithLatestAvroPayload(currentRecord, chooseCurrent ? this.orderingVal : oldValue.orderingVal, this.schema);
+ } else {
+ return isDeleteRecord(insertRecord) ? this : oldValue;
+ }
Review comment:
We should be caution of the `DELETE`s, should we still merge for `DELETE` messages ?
##########
File path: hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/table/action/commit/FlinkWriteHelper.java
##########
@@ -105,7 +105,7 @@ public static FlinkWriteHelper newInstance() {
// we cannot allow the user to change the key or partitionPath, since that will affect
// everything
// so pick it from one of the records.
- boolean choosePrev = data1.equals(reducedData);
+ boolean choosePrev = data2.compareTo(data1) < 0;
HoodieKey reducedKey = choosePrev ? rec1.getKey() : rec2.getKey();
HoodieOperation operation = choosePrev ? rec1.getOperation() : rec2.getOperation();
Review comment:
Why we must need a `compareTo` here ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stayrascal commented on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
stayrascal commented on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1047016868
Hi @danny0405 , regarding the two changes:
- embedding schema in payload might hard to do schema evolution later.
- using flink state to store full HoodieRecord instead of location, will increase the state size.
for first one, I remove the schema filed from payload class, instead, by passing the schema to `preCombine` method, it should be easy to do schema evolution if need.
for second one, I revert my changes about store whole HoodieRecord back to store location, it still has the problem that once the record partition path changed, but i think it should enough for supporting the case that the table don't have partition, or the partition path of record won't changed.
BTW, do you think is it worthy to implement a new bucket assign function(can be controlled by feature toggle/configuration ) which store the full record to support partial update totally ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] LinMingQiang commented on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
LinMingQiang commented on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1049573640
Do I need to modify the preCombine in the HoodieMergedLogRecordScanner.processNextRecord method? What I understand is that when we read a log file, we need to do deduplication and also call preCombine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1052242175
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
}, {
"hash" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178",
"triggerID" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"triggerType" : "PUSH"
}, {
"hash" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6320",
"triggerID" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"triggerType" : "PUSH"
}, {
"hash" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c6f524ea6bf6800a797f9173c4034ce8d111a7d5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6320)
* d3b3e052e3565fe44174dfae73938bfaa44faee4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1030850209
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d73d7f503615583bd233b1ba974e7c78125c41bb Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620)
* 6b6a60fd0f04f236289e500b20951158c2358f51 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1031330512
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6b6a60fd0f04f236289e500b20951158c2358f51 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746)
* 21df6fe5ac130bfca73a7e3f528d8ef1df66c074 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1031332782
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6b6a60fd0f04f236289e500b20951158c2358f51 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746)
* 21df6fe5ac130bfca73a7e3f528d8ef1df66c074 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1031424226
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 21df6fe5ac130bfca73a7e3f528d8ef1df66c074 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stayrascal commented on a change in pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
stayrascal commented on a change in pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#discussion_r814797758
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/PartialOverwriteWithLatestAvroPayload.java
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.hudi.common.util.Option;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Objects;
+import java.util.Properties;
+
+import static org.apache.hudi.avro.HoodieAvroUtils.bytesToAvro;
+
+/**
+ * The only difference with {@link OverwriteNonDefaultsWithLatestAvroPayload} is that it supports
+ * merging the latest non-null partial fields with the old record instead of replacing the whole record.
+ * And merging the non-null fields during preCombine multiple records with same record key instead of choosing the latest record based on ordering field.
+ *
+ * <p> Regarding #combineAndGetUpdateValue, Assuming a {@link GenericRecord} has row schema: (f0 int , f1 int, f2 int).
+ * The first record value is: (1, 2, 3), the second record value is: (4, 5, null) with the field f2 value as null.
+ * Calling the #combineAndGetUpdateValue method of the two records returns record: (4, 5, 3).
+ * Note that field f2 value is ignored because it is null. </p>
+ *
+ * <p> Regarding #preCombine, Assuming a {@link GenericRecord} has row schema: (f0 int , f1 int, f2 int, o1 int),
+ * and initial two {@link PartialOverwriteWithLatestAvroPayload} with different ordering value.
+ * The first record value is (1, null, 1, 1) with the filed f1 value as null, the second value is: (2, 2, null, 2) with the f2 value as null.
+ * Calling the #preCombine method of the two records returns record: (2, 2, 1, 2).
+ * Note:
+ * <ol>
+ * <li>the field f0 value is 2 because the ordering value of second record is bigger.</li>
+ * <li>the filed f1 value is 2 because the f2 value of first record is null.</li>
+ * <li>the filed f2 value is 1 because the f2 value of second record is null.</li>
+ * <li>the filed o1 value is 2 because the ordering value of second record is bigger.</li>
+ * </ol>
+ *
+ * </p>
+ */
+public class PartialOverwriteWithLatestAvroPayload extends OverwriteWithLatestAvroPayload {
+
+ public PartialOverwriteWithLatestAvroPayload(GenericRecord record, Comparable orderingVal) {
+ super(record, orderingVal);
+ }
+
+ public PartialOverwriteWithLatestAvroPayload(Option<GenericRecord> record) {
+ super(record); // natural order
+ }
+
+ @Override
+ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord currentValue, Schema schema) throws IOException {
+ if (recordBytes.length == 0) {
+ return Option.empty();
+ }
+
+ GenericRecord incomingRecord = bytesToAvro(recordBytes, schema);
+ if (isDeleteRecord(incomingRecord)) {
+ return Option.empty();
+ }
+
+ GenericRecord currentRecord = (GenericRecord) currentValue;
+ List<Schema.Field> fields = schema.getFields();
+ fields.forEach(field -> {
+ Object value = incomingRecord.get(field.name());
+ if (Objects.nonNull(value)) {
+ currentRecord.put(field.name(), value);
+ }
+ });
+
+ return Option.of(currentRecord);
+ }
+
+ @Override
+ public int compareTo(OverwriteWithLatestAvroPayload oldValue) {
+ return orderingVal.compareTo(oldValue.orderingVal);
+ }
+
+ @Override
+ public OverwriteWithLatestAvroPayload preCombine(OverwriteWithLatestAvroPayload oldValue, Properties properties, Schema schema) {
+ if (null == schema) {
+ return super.preCombine(oldValue);
+ }
+
+ try {
+ Option<IndexedRecord> incomingOption = getInsertValue(schema);
+ Option<IndexedRecord> oldRecordOption = oldValue.getInsertValue(schema);
+
+ if (incomingOption.isPresent() && oldRecordOption.isPresent()) {
+ GenericRecord incomingRecord = (GenericRecord) incomingOption.get();
+ GenericRecord oldRecord = (GenericRecord) oldRecordOption.get();
+ boolean chooseIncomingRecord = this.orderingVal.compareTo(oldValue.orderingVal) > 0;
Review comment:
Good point here, thx.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stayrascal commented on a change in pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
stayrascal commented on a change in pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#discussion_r811246203
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/PartialOverwriteWithLatestAvroPayload.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.hudi.common.util.Option;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Objects;
+
+import static org.apache.hudi.avro.HoodieAvroUtils.bytesToAvro;
+
+/**
+ * The only difference with {@link OverwriteNonDefaultsWithLatestAvroPayload} is that it supports
+ * merging the latest non-null partial fields with the old record instead of replacing the whole record.
+ * And merging the non-null fields during preCombine multiple records with same record key instead of choosing the latest record based on ordering field.
+ *
+ * <p> Regarding #combineAndGetUpdateValue, Assuming a {@link GenericRecord} has row schema: (f0 int , f1 int, f2 int).
+ * The first record value is: (1, 2, 3), the second record value is: (4, 5, null) with the field f2 value as null.
+ * Calling the #combineAndGetUpdateValue method of the two records returns record: (4, 5, 3).
+ * Note that field f2 value is ignored because it is null. </p>
+ *
+ * <p> Regarding #preCombine, Assuming a {@link GenericRecord} has row schema: (f0 int , f1 int, f2 int, o1 int),
+ * and initial two {@link PartialOverwriteWithLatestAvroPayload} with different ordering value.
+ * The first record value is (1, null, 1, 1) with the filed f1 value as null, the second value is: (2, 2, null, 2) with the f2 value as null.
+ * Calling the #preCombine method of the two records returns record: (2, 2, 1, 2).
+ * Note:
+ * <ol>
+ * <li>the field f0 value is 2 because the ordering value of second record is bigger.</li>
+ * <li>the filed f1 value is 2 because the f2 value of first record is null.</li>
+ * <li>the filed f2 value is 1 because the f2 value of second record is null.</li>
+ * <li>the filed o1 value is 2 because the ordering value of second record is bigger.</li>
+ * </ol>
+ *
+ * </p>
+ */
+public class PartialOverwriteWithLatestAvroPayload extends OverwriteWithLatestAvroPayload {
+
+ public PartialOverwriteWithLatestAvroPayload(GenericRecord record, Comparable orderingVal) {
+ this(record, orderingVal, null);
+ }
+
+ public PartialOverwriteWithLatestAvroPayload(GenericRecord record, Comparable orderingVal, String schema) {
+ super(record, orderingVal, schema);
+ }
+
+ public PartialOverwriteWithLatestAvroPayload(Option<GenericRecord> record) {
+ super(record); // natural order
+ }
+
+ @Override
+ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord currentValue, Schema schema) throws IOException {
+ if (recordBytes.length == 0) {
+ return Option.empty();
+ }
+
+ GenericRecord incomingRecord = bytesToAvro(recordBytes, schema);
+ if (isDeleteRecord(incomingRecord)) {
+ return Option.empty();
+ }
+
+ GenericRecord currentRecord = (GenericRecord) currentValue;
+ List<Schema.Field> fields = schema.getFields();
+ fields.forEach(field -> {
+ Object value = incomingRecord.get(field.name());
Review comment:
current logic will overwrite whole nested field, if the incoming field is not null.
And I think we don't need to support the partial update inner nested field, for example, for Map, List, etc. we should not merge map(1 -> 'a', 2 -> 'b') & map(1 -> ''', 3 -> 'c') to map(1 -> '', 2 -> 'b', 3 -> 'c') incase the upstream want to delete the key '2', if we merge them together, they cannot delete some elements. the List as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1047007541
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
}, {
"hash" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 21df6fe5ac130bfca73a7e3f528d8ef1df66c074 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758)
* 14edef016d98786b713e155e9adec1d87a3500ba Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175)
* ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1046955396
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 21df6fe5ac130bfca73a7e3f528d8ef1df66c074 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758)
* 14edef016d98786b713e155e9adec1d87a3500ba Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#discussion_r818086025
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordPayload.java
##########
@@ -58,6 +58,31 @@ default T preCombine(T oldValue, Properties properties) {
return preCombine(oldValue);
}
+ /**
+ *When more than one HoodieRecord have the same HoodieKey in the incoming batch, this function combines them before attempting to insert/upsert by taking in a property map.
+ *
+ * @param oldValue instance of the old {@link HoodieRecordPayload} to be combined with.
+ * @param properties Payload related properties. For example pass the ordering field(s) name to extract from value in storage.
+ * @param schema Schema used for record
+ * @return the combined value
+ */
+ @PublicAPIMethod(maturity = ApiMaturityLevel.STABLE)
+ default T preCombine(T oldValue, Properties properties, Schema schema) {
Review comment:
CC @rmahindra123 who encountered a necessity to do preCombine but to combine bits and pieces from both records to return a new one. Rajesh: do you wanna go over your use-case may be.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1075962905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
}, {
"hash" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178",
"triggerID" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"triggerType" : "PUSH"
}, {
"hash" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6320",
"triggerID" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"triggerType" : "PUSH"
}, {
"hash" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6357",
"triggerID" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"triggerType" : "PUSH"
}, {
"hash" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6706",
"triggerID" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"triggerType" : "PUSH"
}, {
"hash" : "d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7000",
"triggerID" : "d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5",
"triggerType" : "PUSH"
}, {
"hash" : "e89fd60b77fbd366205df5e60777ced881b6d795",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7203",
"triggerID" : "e89fd60b77fbd366205df5e60777ced881b6d795",
"triggerType" : "PUSH"
}, {
"hash" : "e89fd60b77fbd366205df5e60777ced881b6d795",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7220",
"triggerID" : "1075938530",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* e89fd60b77fbd366205df5e60777ced881b6d795 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7203) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7220)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1052242175
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
}, {
"hash" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178",
"triggerID" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"triggerType" : "PUSH"
}, {
"hash" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6320",
"triggerID" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"triggerType" : "PUSH"
}, {
"hash" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c6f524ea6bf6800a797f9173c4034ce8d111a7d5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6320)
* d3b3e052e3565fe44174dfae73938bfaa44faee4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1051022414
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
}, {
"hash" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178",
"triggerID" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"triggerType" : "PUSH"
}, {
"hash" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6320",
"triggerID" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c6f524ea6bf6800a797f9173c4034ce8d111a7d5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6320)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stayrascal commented on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
stayrascal commented on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1031342598
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1031367162
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 21df6fe5ac130bfca73a7e3f528d8ef1df66c074 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1031343408
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 6b6a60fd0f04f236289e500b20951158c2358f51 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746)
* 21df6fe5ac130bfca73a7e3f528d8ef1df66c074 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1025675312
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d73d7f503615583bd233b1ba974e7c78125c41bb Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stayrascal commented on a change in pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
stayrascal commented on a change in pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#discussion_r811194444
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/PartialOverwriteWithLatestAvroPayload.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.hudi.common.util.Option;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Objects;
+
+import static org.apache.hudi.avro.HoodieAvroUtils.bytesToAvro;
+
+/**
+ * The only difference with {@link OverwriteNonDefaultsWithLatestAvroPayload} is that it supports
+ * merging the latest non-null partial fields with the old record instead of replacing the whole record.
+ * And merging the non-null fields during preCombine multiple records with same record key instead of choosing the latest record based on ordering field.
+ *
+ * <p> Regarding #combineAndGetUpdateValue, Assuming a {@link GenericRecord} has row schema: (f0 int , f1 int, f2 int).
+ * The first record value is: (1, 2, 3), the second record value is: (4, 5, null) with the field f2 value as null.
+ * Calling the #combineAndGetUpdateValue method of the two records returns record: (4, 5, 3).
+ * Note that field f2 value is ignored because it is null. </p>
+ *
+ * <p> Regarding #preCombine, Assuming a {@link GenericRecord} has row schema: (f0 int , f1 int, f2 int, o1 int),
+ * and initial two {@link PartialOverwriteWithLatestAvroPayload} with different ordering value.
+ * The first record value is (1, null, 1, 1) with the filed f1 value as null, the second value is: (2, 2, null, 2) with the f2 value as null.
+ * Calling the #preCombine method of the two records returns record: (2, 2, 1, 2).
+ * Note:
+ * <ol>
+ * <li>the field f0 value is 2 because the ordering value of second record is bigger.</li>
+ * <li>the filed f1 value is 2 because the f2 value of first record is null.</li>
+ * <li>the filed f2 value is 1 because the f2 value of second record is null.</li>
+ * <li>the filed o1 value is 2 because the ordering value of second record is bigger.</li>
+ * </ol>
+ *
+ * </p>
+ */
+public class PartialOverwriteWithLatestAvroPayload extends OverwriteWithLatestAvroPayload {
+
+ public PartialOverwriteWithLatestAvroPayload(GenericRecord record, Comparable orderingVal) {
+ this(record, orderingVal, null);
+ }
+
+ public PartialOverwriteWithLatestAvroPayload(GenericRecord record, Comparable orderingVal, String schema) {
+ super(record, orderingVal, schema);
+ }
+
+ public PartialOverwriteWithLatestAvroPayload(Option<GenericRecord> record) {
+ super(record); // natural order
+ }
+
+ @Override
+ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord currentValue, Schema schema) throws IOException {
+ if (recordBytes.length == 0) {
+ return Option.empty();
+ }
+
+ GenericRecord incomingRecord = bytesToAvro(recordBytes, schema);
+ if (isDeleteRecord(incomingRecord)) {
+ return Option.empty();
+ }
+
+ GenericRecord currentRecord = (GenericRecord) currentValue;
+ List<Schema.Field> fields = schema.getFields();
+ fields.forEach(field -> {
+ Object value = incomingRecord.get(field.name());
+ if (Objects.nonNull(value)) {
+ currentRecord.put(field.name(), value);
+ }
+ });
+
+ return Option.of(currentRecord);
+ }
+
+ @Override
+ public int compareTo(OverwriteWithLatestAvroPayload oldValue) {
+ return orderingVal.compareTo(oldValue.orderingVal);
+ }
+
+ @Override
+ public OverwriteWithLatestAvroPayload preCombine(OverwriteWithLatestAvroPayload oldValue) {
+ if (null == this.schema || null == oldValue.schema) {
+ return super.preCombine(oldValue);
+ }
+
+ try {
+ Schema schema = new Schema.Parser().parse(this.schema);
Review comment:
solved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1046955396
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 21df6fe5ac130bfca73a7e3f528d8ef1df66c074 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758)
* 14edef016d98786b713e155e9adec1d87a3500ba Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1050963505
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
}, {
"hash" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178",
"triggerID" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"triggerType" : "PUSH"
}, {
"hash" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6320",
"triggerID" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178)
* c6f524ea6bf6800a797f9173c4034ce8d111a7d5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6320)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a change in pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on a change in pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#discussion_r813359945
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordPayload.java
##########
@@ -58,6 +58,31 @@ default T preCombine(T oldValue, Properties properties) {
return preCombine(oldValue);
}
+ /**
+ *When more than one HoodieRecord have the same HoodieKey in the incoming batch, this function combines them before attempting to insert/upsert by taking in a property map.
+ *
+ * @param oldValue instance of the old {@link HoodieRecordPayload} to be combined with.
+ * @param properties Payload related properties. For example pass the ordering field(s) name to extract from value in storage.
+ * @param schema Schema used for record
+ * @return the combined value
+ */
+ @PublicAPIMethod(maturity = ApiMaturityLevel.STABLE)
+ default T preCombine(T oldValue, Properties properties, Schema schema) {
Review comment:
I think currently established semantic for `preCombine` -- you select either A or B, but you don't produce new record based on those 2, since it's mostly used to de-dupe records in the incoming batch. I can hardly imagine the case to fuse 2 incoming records into something third. Can you help me understand what use-case you have in mind here?
##########
File path: hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java
##########
@@ -268,6 +268,12 @@ private FlinkOptions() {
.withDescription("Payload class used. Override this, if you like to roll your own merge logic, when upserting/inserting.\n"
+ "This will render any value set for the option in-effective");
+ public static final ConfigOption<Boolean> PARTIAL_OVERWRITE_ENABLED = ConfigOptions
Review comment:
What's the idea for this additional configuration (beside the record payload class)?
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/PartialOverwriteWithLatestAvroPayload.java
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.hudi.common.util.Option;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Objects;
+import java.util.Properties;
+
+import static org.apache.hudi.avro.HoodieAvroUtils.bytesToAvro;
+
+/**
+ * The only difference with {@link OverwriteNonDefaultsWithLatestAvroPayload} is that it supports
+ * merging the latest non-null partial fields with the old record instead of replacing the whole record.
+ * And merging the non-null fields during preCombine multiple records with same record key instead of choosing the latest record based on ordering field.
+ *
+ * <p> Regarding #combineAndGetUpdateValue, Assuming a {@link GenericRecord} has row schema: (f0 int , f1 int, f2 int).
+ * The first record value is: (1, 2, 3), the second record value is: (4, 5, null) with the field f2 value as null.
+ * Calling the #combineAndGetUpdateValue method of the two records returns record: (4, 5, 3).
+ * Note that field f2 value is ignored because it is null. </p>
+ *
+ * <p> Regarding #preCombine, Assuming a {@link GenericRecord} has row schema: (f0 int , f1 int, f2 int, o1 int),
+ * and initial two {@link PartialOverwriteWithLatestAvroPayload} with different ordering value.
+ * The first record value is (1, null, 1, 1) with the filed f1 value as null, the second value is: (2, 2, null, 2) with the f2 value as null.
+ * Calling the #preCombine method of the two records returns record: (2, 2, 1, 2).
+ * Note:
+ * <ol>
+ * <li>the field f0 value is 2 because the ordering value of second record is bigger.</li>
+ * <li>the filed f1 value is 2 because the f2 value of first record is null.</li>
+ * <li>the filed f2 value is 1 because the f2 value of second record is null.</li>
+ * <li>the filed o1 value is 2 because the ordering value of second record is bigger.</li>
+ * </ol>
+ *
+ * </p>
+ */
+public class PartialOverwriteWithLatestAvroPayload extends OverwriteWithLatestAvroPayload {
+
+ public PartialOverwriteWithLatestAvroPayload(GenericRecord record, Comparable orderingVal) {
+ super(record, orderingVal);
+ }
+
+ public PartialOverwriteWithLatestAvroPayload(Option<GenericRecord> record) {
+ super(record); // natural order
+ }
+
+ @Override
+ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord currentValue, Schema schema) throws IOException {
+ if (recordBytes.length == 0) {
+ return Option.empty();
+ }
+
+ GenericRecord incomingRecord = bytesToAvro(recordBytes, schema);
+ if (isDeleteRecord(incomingRecord)) {
+ return Option.empty();
+ }
+
+ GenericRecord currentRecord = (GenericRecord) currentValue;
+ List<Schema.Field> fields = schema.getFields();
+ fields.forEach(field -> {
+ Object value = incomingRecord.get(field.name());
+ if (Objects.nonNull(value)) {
+ currentRecord.put(field.name(), value);
+ }
+ });
+
+ return Option.of(currentRecord);
+ }
+
+ @Override
+ public int compareTo(OverwriteWithLatestAvroPayload oldValue) {
+ return orderingVal.compareTo(oldValue.orderingVal);
+ }
+
+ @Override
+ public OverwriteWithLatestAvroPayload preCombine(OverwriteWithLatestAvroPayload oldValue, Properties properties, Schema schema) {
+ if (null == schema) {
+ return super.preCombine(oldValue);
+ }
+
+ try {
+ Option<IndexedRecord> incomingOption = getInsertValue(schema);
+ Option<IndexedRecord> oldRecordOption = oldValue.getInsertValue(schema);
+
+ if (incomingOption.isPresent() && oldRecordOption.isPresent()) {
Review comment:
In general it's better to express common functionality in a way that would allow it to be re-used and adopted in other places: here for ex, we can reuse the same routine of combining 2 records into one, across 2 methods if we properly abstract it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1078622103
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
}, {
"hash" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178",
"triggerID" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"triggerType" : "PUSH"
}, {
"hash" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6320",
"triggerID" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"triggerType" : "PUSH"
}, {
"hash" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6357",
"triggerID" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"triggerType" : "PUSH"
}, {
"hash" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6706",
"triggerID" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"triggerType" : "PUSH"
}, {
"hash" : "d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7000",
"triggerID" : "d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5",
"triggerType" : "PUSH"
}, {
"hash" : "e89fd60b77fbd366205df5e60777ced881b6d795",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7203",
"triggerID" : "e89fd60b77fbd366205df5e60777ced881b6d795",
"triggerType" : "PUSH"
}, {
"hash" : "e89fd60b77fbd366205df5e60777ced881b6d795",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7220",
"triggerID" : "1075938530",
"triggerType" : "MANUAL"
}, {
"hash" : "3f771d39a403669bf7d0a8d50cd3e5b711b03ebe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3f771d39a403669bf7d0a8d50cd3e5b711b03ebe",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e89fd60b77fbd366205df5e60777ced881b6d795 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7203) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7220)
* 3f771d39a403669bf7d0a8d50cd3e5b711b03ebe UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1030866332
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6b6a60fd0f04f236289e500b20951158c2358f51 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stayrascal commented on a change in pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
stayrascal commented on a change in pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#discussion_r801755729
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/PartialOverwriteWithLatestAvroPayload.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.hudi.common.util.Option;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Objects;
+
+import static org.apache.hudi.avro.HoodieAvroUtils.bytesToAvro;
+
+/**
+ * The only difference with {@link OverwriteNonDefaultsWithLatestAvroPayload} is that it supports
+ * merging the latest non-null partial fields with the old record instead of replacing the whole record.
+ * And merging the non-null fields during preCombine multiple records with same record key instead of choosing the latest record based on ordering field.
+ *
+ * <p> Regarding #combineAndGetUpdateValue, Assuming a {@link GenericRecord} has row schema: (f0 int , f1 int, f2 int).
+ * The first record value is: (1, 2, 3), the second record value is: (4, 5, null) with the field f2 value as null.
+ * Calling the #combineAndGetUpdateValue method of the two records returns record: (4, 5, 3).
+ * Note that field f2 value is ignored because it is null. </p>
+ *
+ * <p> Regarding #preCombine, Assuming a {@link GenericRecord} has row schema: (f0 int , f1 int, f2 int, o1 int),
+ * and initial two {@link PartialOverwriteWithLatestAvroPayload} with different ordering value.
+ * The first record value is (1, null, 1, 1) with the filed f1 value as null, the second value is: (2, 2, null, 2) with the f2 value as null.
+ * Calling the #preCombine method of the two records returns record: (2, 2, 1, 2).
+ * Note:
+ * <ol>
+ * <li>the field f0 value is 2 because the ordering value of second record is bigger.</li>
+ * <li>the filed f1 value is 2 because the f2 value of first record is null.</li>
+ * <li>the filed f2 value is 1 because the f2 value of second record is null.</li>
+ * <li>the filed o1 value is 2 because the ordering value of second record is bigger.</li>
+ * </ol>
+ *
+ * </p>
+ */
+public class PartialOverwriteWithLatestAvroPayload extends OverwriteWithLatestAvroPayload {
+
+ public PartialOverwriteWithLatestAvroPayload(GenericRecord record, Comparable orderingVal) {
+ this(record, orderingVal, null);
+ }
+
+ public PartialOverwriteWithLatestAvroPayload(GenericRecord record, Comparable orderingVal, String schema) {
+ super(record, orderingVal, schema);
+ }
+
+ public PartialOverwriteWithLatestAvroPayload(Option<GenericRecord> record) {
+ super(record); // natural order
+ }
+
+ @Override
+ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord currentValue, Schema schema) throws IOException {
+ if (recordBytes.length == 0) {
+ return Option.empty();
+ }
+
+ GenericRecord incomingRecord = bytesToAvro(recordBytes, schema);
+ if (isDeleteRecord(incomingRecord)) {
+ return Option.empty();
+ }
+
+ GenericRecord currentRecord = (GenericRecord) currentValue;
+ List<Schema.Field> fields = schema.getFields();
+ fields.forEach(field -> {
+ Object value = incomingRecord.get(field.name());
+ if (Objects.nonNull(value)) {
+ currentRecord.put(field.name(), value);
+ }
+ });
+
+ return Option.of(currentRecord);
+ }
+
+ @Override
+ public int compareTo(OverwriteWithLatestAvroPayload oldValue) {
+ return orderingVal.compareTo(oldValue.orderingVal);
+ }
+
+ @Override
+ public OverwriteWithLatestAvroPayload preCombine(OverwriteWithLatestAvroPayload oldValue) {
+ if (null == this.schema || null == oldValue.schema) {
+ return super.preCombine(oldValue);
+ }
+
+ try {
+ Schema schema = new Schema.Parser().parse(this.schema);
+ Option<IndexedRecord> incomingOption = getInsertValue(new Schema.Parser().parse(this.schema));
+ Option<IndexedRecord> insertRecordOption = oldValue.getInsertValue(new Schema.Parser().parse(oldValue.schema));
+
+ if (incomingOption.isPresent() && insertRecordOption.isPresent()) {
+ GenericRecord currentRecord = (GenericRecord) incomingOption.get();
+ GenericRecord insertRecord = (GenericRecord) insertRecordOption.get();
+ boolean chooseCurrent = this.orderingVal.compareTo(oldValue.orderingVal) > 0;
+
+ if (!isDeleteRecord(insertRecord) && !isDeleteRecord(currentRecord)) {
+ schema.getFields().forEach(field -> {
+ Object insertValue = insertRecord.get(field.name());
+ Object currentValue = currentRecord.get(field.name());
+ currentRecord.put(field.name(), mergeValue(currentValue, insertValue, chooseCurrent));
+ });
+ return new PartialOverwriteWithLatestAvroPayload(currentRecord, chooseCurrent ? this.orderingVal : oldValue.orderingVal, this.schema);
+ } else {
+ return isDeleteRecord(insertRecord) ? this : oldValue;
+ }
Review comment:
yeah, if one of record is `DELETE` record, just return themselves directly, no need to merge, the `DELETE` message to delete old record during hudi write. Only when two records are not `DELETE` records, we need to merge them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1047118523
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
}, {
"hash" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178",
"triggerID" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stayrascal commented on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
stayrascal commented on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1050887094
> HoodieMergedLogRecordScanner
As I know, if we also want to achieve partial update purpose, we just need to pass the `Schema` here.(Assume this pull request has been approved).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stayrascal commented on a change in pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
stayrascal commented on a change in pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#discussion_r815265938
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordPayload.java
##########
@@ -58,6 +58,31 @@ default T preCombine(T oldValue, Properties properties) {
return preCombine(oldValue);
}
+ /**
+ *When more than one HoodieRecord have the same HoodieKey in the incoming batch, this function combines them before attempting to insert/upsert by taking in a property map.
+ *
+ * @param oldValue instance of the old {@link HoodieRecordPayload} to be combined with.
+ * @param properties Payload related properties. For example pass the ordering field(s) name to extract from value in storage.
+ * @param schema Schema used for record
+ * @return the combined value
+ */
+ @PublicAPIMethod(maturity = ApiMaturityLevel.STABLE)
+ default T preCombine(T oldValue, Properties properties, Schema schema) {
Review comment:
BTW, thanks a lot for you time, will ping you on slack.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1050963505
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
}, {
"hash" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178",
"triggerID" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"triggerType" : "PUSH"
}, {
"hash" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6320",
"triggerID" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178)
* c6f524ea6bf6800a797f9173c4034ce8d111a7d5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6320)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1052246021
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
}, {
"hash" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178",
"triggerID" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"triggerType" : "PUSH"
}, {
"hash" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6320",
"triggerID" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"triggerType" : "PUSH"
}, {
"hash" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6357",
"triggerID" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c6f524ea6bf6800a797f9173c4034ce8d111a7d5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6320)
* d3b3e052e3565fe44174dfae73938bfaa44faee4 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6357)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1062045316
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
}, {
"hash" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178",
"triggerID" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"triggerType" : "PUSH"
}, {
"hash" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6320",
"triggerID" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"triggerType" : "PUSH"
}, {
"hash" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6357",
"triggerID" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"triggerType" : "PUSH"
}, {
"hash" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6706",
"triggerID" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a86b7ff2d204900374751371cd2c8c36e76ea4b8 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6706)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1069213125
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
}, {
"hash" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178",
"triggerID" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"triggerType" : "PUSH"
}, {
"hash" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6320",
"triggerID" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"triggerType" : "PUSH"
}, {
"hash" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6357",
"triggerID" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"triggerType" : "PUSH"
}, {
"hash" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6706",
"triggerID" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"triggerType" : "PUSH"
}, {
"hash" : "d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7000",
"triggerID" : "d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7000)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1075846199
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
}, {
"hash" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178",
"triggerID" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"triggerType" : "PUSH"
}, {
"hash" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6320",
"triggerID" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"triggerType" : "PUSH"
}, {
"hash" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6357",
"triggerID" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"triggerType" : "PUSH"
}, {
"hash" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6706",
"triggerID" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"triggerType" : "PUSH"
}, {
"hash" : "d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7000",
"triggerID" : "d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5",
"triggerType" : "PUSH"
}, {
"hash" : "e89fd60b77fbd366205df5e60777ced881b6d795",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7203",
"triggerID" : "e89fd60b77fbd366205df5e60777ced881b6d795",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e89fd60b77fbd366205df5e60777ced881b6d795 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7203)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1078623134
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
}, {
"hash" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178",
"triggerID" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"triggerType" : "PUSH"
}, {
"hash" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6320",
"triggerID" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"triggerType" : "PUSH"
}, {
"hash" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6357",
"triggerID" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"triggerType" : "PUSH"
}, {
"hash" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6706",
"triggerID" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"triggerType" : "PUSH"
}, {
"hash" : "d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7000",
"triggerID" : "d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5",
"triggerType" : "PUSH"
}, {
"hash" : "e89fd60b77fbd366205df5e60777ced881b6d795",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7203",
"triggerID" : "e89fd60b77fbd366205df5e60777ced881b6d795",
"triggerType" : "PUSH"
}, {
"hash" : "e89fd60b77fbd366205df5e60777ced881b6d795",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7220",
"triggerID" : "1075938530",
"triggerType" : "MANUAL"
}, {
"hash" : "3f771d39a403669bf7d0a8d50cd3e5b711b03ebe",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7340",
"triggerID" : "3f771d39a403669bf7d0a8d50cd3e5b711b03ebe",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e89fd60b77fbd366205df5e60777ced881b6d795 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7203) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7220)
* 3f771d39a403669bf7d0a8d50cd3e5b711b03ebe Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7340)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1069123625
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
}, {
"hash" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178",
"triggerID" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"triggerType" : "PUSH"
}, {
"hash" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6320",
"triggerID" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"triggerType" : "PUSH"
}, {
"hash" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6357",
"triggerID" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"triggerType" : "PUSH"
}, {
"hash" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6706",
"triggerID" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"triggerType" : "PUSH"
}, {
"hash" : "d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a86b7ff2d204900374751371cd2c8c36e76ea4b8 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6706)
* d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a change in pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on a change in pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#discussion_r815187048
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordPayload.java
##########
@@ -58,6 +58,31 @@ default T preCombine(T oldValue, Properties properties) {
return preCombine(oldValue);
}
+ /**
+ *When more than one HoodieRecord have the same HoodieKey in the incoming batch, this function combines them before attempting to insert/upsert by taking in a property map.
+ *
+ * @param oldValue instance of the old {@link HoodieRecordPayload} to be combined with.
+ * @param properties Payload related properties. For example pass the ordering field(s) name to extract from value in storage.
+ * @param schema Schema used for record
+ * @return the combined value
+ */
+ @PublicAPIMethod(maturity = ApiMaturityLevel.STABLE)
+ default T preCombine(T oldValue, Properties properties, Schema schema) {
Review comment:
Let me try to clarify a few things:
`preCombine` has a _very specific_ semantic: it's de-duplicating by the way of picking "most recent" among records in the batch. Expectation always is that it being handed 2 records it will **have to** return either of them. It could not produce new record. If we want to revisit this semantic this is a far larger change that will surely require writing an RFC and broader discussion regarding the merits of such migration. Please also keep in mind that as of RFC-46 there's an effort underway to abstract whole "record combination/merging" semantic out of `RecordPayload` hierarchy into standalone Combination/Merge Engine API.
> First, from the description of preCombine method, it used for combining multiple records with same HoodieKey before attempting to insert/upsert to disk. The "combine multiple records" might not mean only choosing one of them, we also can combine & merged them to a new one, just depends on how the sub-class implement the preCombine logic(Please correct me if my understanding is wrong :) ). Yeah, it might be a little bit confused that we need Schema if we are trying to merged them.
Please see my comment regarding `preCombine` semantic above. I certainly agree with you that the name is confusing, but i've tried to clear that confusion. Let me know if you have more questions about it.
> Second, I checked when will we call preCombine method is trying to duplicate records with same HoodieKey before insert/update to disk, especially in Flink write case, even through the duplicated logic is choose the latest record, but we need to ensure that one HoodieKey should only contains one record before comparing to existing record and write to disk, otherwise, some records will missed. For example, in HoodieMergeHandle.init(fieId, newRecordsIter), it will convert the record iterator to a map and treat the recordKey as key. So we might not stop de-duping logics and merge them against what is on disk unless we change the logic here. And also we implement another class/method to handle the merge logic, and switch the existing de-duping logic from calling preCombine to new class/method, we have to add an condition to control whether should we call preCombine or not, I think it might not a good way. Instead, we should handle it in preCombine method by different implemented payl
oad.
You're bringing up a good points, let's dive into them one by one: so currently we have 2 mechanisms
1. `preCombine` that allows to select "most recent" record among those having the same key w/in the batch
2. `combineAndGetUpdateValue` that allows to combine previous or "historical" record (on Disk) with the new incoming one (all partial merging semantic is currently implemented in this method)
You rightfully mention some of the invariants are currently that the batch would be de-duped at certain level (b/c we have to maintain PK uniqueness on disk), and so we might need to shift that to accommodate for case that you have. And that's exactly what my question was: if you can elaborate on use-case that you have at hand that you're trying to solve w/ this PR, i would be able to better understand where you're coming from and what's the best path forward for us here.
Questions i'm looking an answers for are basically following:
1. What's nature of your use-case? (domain, record types, frequency, size, etc)
2. Where requirements for partial updates are coming from?
and etc. I'm happy to set some 30min to talk in person regarding this or connect on Slack and discuss it there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1052318257
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
}, {
"hash" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178",
"triggerID" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"triggerType" : "PUSH"
}, {
"hash" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6320",
"triggerID" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"triggerType" : "PUSH"
}, {
"hash" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6357",
"triggerID" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d3b3e052e3565fe44174dfae73938bfaa44faee4 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6357)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a change in pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on a change in pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#discussion_r814333200
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordPayload.java
##########
@@ -58,6 +58,31 @@ default T preCombine(T oldValue, Properties properties) {
return preCombine(oldValue);
}
+ /**
+ *When more than one HoodieRecord have the same HoodieKey in the incoming batch, this function combines them before attempting to insert/upsert by taking in a property map.
+ *
+ * @param oldValue instance of the old {@link HoodieRecordPayload} to be combined with.
+ * @param properties Payload related properties. For example pass the ordering field(s) name to extract from value in storage.
+ * @param schema Schema used for record
+ * @return the combined value
+ */
+ @PublicAPIMethod(maturity = ApiMaturityLevel.STABLE)
+ default T preCombine(T oldValue, Properties properties, Schema schema) {
Review comment:
Right, that's exactly my question: why do you want to implement such semantic w/in `preCombine`? What use-case you're trying to accommodate for here?
Essentially with this change you will introduce a way for 2 records w/in the batch to be combined into 1. But why do you need this?
After all you can achieve the same goal if you just stop de-duping your records, and then subsequently merge them against what is on disk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1051022414
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
}, {
"hash" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178",
"triggerID" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"triggerType" : "PUSH"
}, {
"hash" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6320",
"triggerID" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c6f524ea6bf6800a797f9173c4034ce8d111a7d5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6320)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stayrascal commented on a change in pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
stayrascal commented on a change in pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#discussion_r814533349
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordPayload.java
##########
@@ -58,6 +58,31 @@ default T preCombine(T oldValue, Properties properties) {
return preCombine(oldValue);
}
+ /**
+ *When more than one HoodieRecord have the same HoodieKey in the incoming batch, this function combines them before attempting to insert/upsert by taking in a property map.
+ *
+ * @param oldValue instance of the old {@link HoodieRecordPayload} to be combined with.
+ * @param properties Payload related properties. For example pass the ordering field(s) name to extract from value in storage.
+ * @param schema Schema used for record
+ * @return the combined value
+ */
+ @PublicAPIMethod(maturity = ApiMaturityLevel.STABLE)
+ default T preCombine(T oldValue, Properties properties, Schema schema) {
Review comment:
Hi @alexeykudinkin, I got your point. if we have to combine two records to a combined one, we'd better to implement the combine logics in other place, maybe in some `util` or `helper` classes, right?
Here are some options from mine that `#preCombine` might be a better place to implement these logics, or create new `merge` method in `HoodieRecordPayload` interface.
- First, from the description of `preCombine` method, it used for combining multiple records with same HoodieKey before attempting to insert/upsert to disk. The "combine multiple records" might not mean only choosing one of them, we also can combine & merged them to a new one, just depends on how the sub-class implement the preCombine logic(Please correct me if my understanding is wrong :) ). Yeah, it might be a little bit confused that we need `Schema` if we are trying to merged them.
- Second, I checked when will we call `preCombine` method is trying to duplicate records with same HoodieKey before insert/update to disk, especially in Flink write case, even through the duplicated logic is choose the latest record, but we need to ensure that one HoodieKey should only contains one record before comparing to existing record and write to disk, otherwise, some records will missed. For example, in `HoodieMergeHandle.init(fieId, newRecordsIter)`, it will convert the record iterator to a map and treat the recordKey as key. *So we might not stop de-duping logics and merge them against what is on disk unless we change the logic here.* And also we implement another class/method to handle the merge logic, and switch the existing de-duping logic from calling `preCombine` to new class/method, we have to add an condition to control whether should we call `preCombine` or not, I think it might not a good way. Instead, we should handle it in `preCombine` method by different impl
emented payload.
That's what my thought here, and I'm glad to listen your useful suggestions. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] LinMingQiang commented on a change in pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
LinMingQiang commented on a change in pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#discussion_r814557644
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/PartialOverwriteWithLatestAvroPayload.java
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.hudi.common.util.Option;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Objects;
+import java.util.Properties;
+
+import static org.apache.hudi.avro.HoodieAvroUtils.bytesToAvro;
+
+/**
+ * The only difference with {@link OverwriteNonDefaultsWithLatestAvroPayload} is that it supports
+ * merging the latest non-null partial fields with the old record instead of replacing the whole record.
+ * And merging the non-null fields during preCombine multiple records with same record key instead of choosing the latest record based on ordering field.
+ *
+ * <p> Regarding #combineAndGetUpdateValue, Assuming a {@link GenericRecord} has row schema: (f0 int , f1 int, f2 int).
+ * The first record value is: (1, 2, 3), the second record value is: (4, 5, null) with the field f2 value as null.
+ * Calling the #combineAndGetUpdateValue method of the two records returns record: (4, 5, 3).
+ * Note that field f2 value is ignored because it is null. </p>
+ *
+ * <p> Regarding #preCombine, Assuming a {@link GenericRecord} has row schema: (f0 int , f1 int, f2 int, o1 int),
+ * and initial two {@link PartialOverwriteWithLatestAvroPayload} with different ordering value.
+ * The first record value is (1, null, 1, 1) with the filed f1 value as null, the second value is: (2, 2, null, 2) with the f2 value as null.
+ * Calling the #preCombine method of the two records returns record: (2, 2, 1, 2).
+ * Note:
+ * <ol>
+ * <li>the field f0 value is 2 because the ordering value of second record is bigger.</li>
+ * <li>the filed f1 value is 2 because the f2 value of first record is null.</li>
+ * <li>the filed f2 value is 1 because the f2 value of second record is null.</li>
+ * <li>the filed o1 value is 2 because the ordering value of second record is bigger.</li>
+ * </ol>
+ *
+ * </p>
+ */
+public class PartialOverwriteWithLatestAvroPayload extends OverwriteWithLatestAvroPayload {
+
+ public PartialOverwriteWithLatestAvroPayload(GenericRecord record, Comparable orderingVal) {
+ super(record, orderingVal);
+ }
+
+ public PartialOverwriteWithLatestAvroPayload(Option<GenericRecord> record) {
+ super(record); // natural order
+ }
+
+ @Override
+ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord currentValue, Schema schema) throws IOException {
+ if (recordBytes.length == 0) {
+ return Option.empty();
+ }
+
+ GenericRecord incomingRecord = bytesToAvro(recordBytes, schema);
+ if (isDeleteRecord(incomingRecord)) {
+ return Option.empty();
+ }
+
+ GenericRecord currentRecord = (GenericRecord) currentValue;
+ List<Schema.Field> fields = schema.getFields();
+ fields.forEach(field -> {
+ Object value = incomingRecord.get(field.name());
+ if (Objects.nonNull(value)) {
+ currentRecord.put(field.name(), value);
+ }
+ });
+
+ return Option.of(currentRecord);
+ }
+
+ @Override
+ public int compareTo(OverwriteWithLatestAvroPayload oldValue) {
+ return orderingVal.compareTo(oldValue.orderingVal);
+ }
+
+ @Override
+ public OverwriteWithLatestAvroPayload preCombine(OverwriteWithLatestAvroPayload oldValue, Properties properties, Schema schema) {
+ if (null == schema) {
+ return super.preCombine(oldValue);
+ }
+
+ try {
+ Option<IndexedRecord> incomingOption = getInsertValue(schema);
+ Option<IndexedRecord> oldRecordOption = oldValue.getInsertValue(schema);
+
+ if (incomingOption.isPresent() && oldRecordOption.isPresent()) {
+ GenericRecord incomingRecord = (GenericRecord) incomingOption.get();
+ GenericRecord oldRecord = (GenericRecord) oldRecordOption.get();
+ boolean chooseIncomingRecord = this.orderingVal.compareTo(oldValue.orderingVal) > 0;
Review comment:
This place needs to be changed to >= , because when we do not set the preCombine field, the first data will always be used instead of the latest
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stayrascal commented on a change in pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
stayrascal commented on a change in pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#discussion_r801734866
##########
File path: hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/table/action/commit/FlinkWriteHelper.java
##########
@@ -105,7 +105,7 @@ public static FlinkWriteHelper newInstance() {
// we cannot allow the user to change the key or partitionPath, since that will affect
// everything
// so pick it from one of the records.
- boolean choosePrev = data1.equals(reducedData);
+ boolean choosePrev = data2.compareTo(data1) < 0;
HoodieKey reducedKey = choosePrev ? rec1.getKey() : rec2.getKey();
HoodieOperation operation = choosePrev ? rec1.getOperation() : rec2.getOperation();
Review comment:
Actually, `rec1` and `rec2` should have same HoodieKey here, right, but the HodieOperation might different.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stayrascal commented on a change in pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
stayrascal commented on a change in pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#discussion_r801702428
##########
File path: hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/table/action/commit/FlinkWriteHelper.java
##########
@@ -105,7 +105,7 @@ public static FlinkWriteHelper newInstance() {
// we cannot allow the user to change the key or partitionPath, since that will affect
// everything
// so pick it from one of the records.
- boolean choosePrev = data1.equals(reducedData);
+ boolean choosePrev = data2.compareTo(data1) < 0;
HoodieKey reducedKey = choosePrev ? rec1.getKey() : rec2.getKey();
HoodieOperation operation = choosePrev ? rec1.getOperation() : rec2.getOperation();
Review comment:
The previous logic of `data2.preCombine(data1)` is that return one of data1 or data2 ordering by their `orderVal`. But if we merge/combine data1 and data2 into a new payload(reduceData), the `data1.equals(reduceData)` is always false. In order to get the `HoodieKey` and `HoodieOperation` for new HoodieRecord with `reduceData`, we need to get the latest `HoodieKey` and `HoodieOperation` from `data1` and `data2`， `compareTo` is used for replace `#preCombine` to compare their `orderingVal`.
```
@Override
public int compareTo(OverwriteWithLatestAvroPayload oldValue) {
return orderingVal.compareTo(oldValue.orderingVal);
}
```
```
@Test
public void testCompareFunction() {
GenericRecord record = new GenericData.Record(schema);
record.put("id", "1");
record.put("partition", "partition1");
record.put("ts", 0L);
record.put("_hoodie_is_deleted", false);
record.put("city", "NY0");
record.put("child", Arrays.asList("A"));
PartialOverwriteWithLatestAvroPayload payload1 = new PartialOverwriteWithLatestAvroPayload(record, 1);
PartialOverwriteWithLatestAvroPayload payload2 = new PartialOverwriteWithLatestAvroPayload(record, 2);
assertEquals(payload1.compareTo(payload2), -1);
assertEquals(payload2.compareTo(payload1), 1);
assertEquals(payload1.compareTo(payload1), 0);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1031346059
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 21df6fe5ac130bfca73a7e3f528d8ef1df66c074 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1030849625
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d73d7f503615583bd233b1ba974e7c78125c41bb Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620)
* 6b6a60fd0f04f236289e500b20951158c2358f51 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1047010710
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
}, {
"hash" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 14edef016d98786b713e155e9adec1d87a3500ba Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175)
* ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1052246021
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
}, {
"hash" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178",
"triggerID" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"triggerType" : "PUSH"
}, {
"hash" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6320",
"triggerID" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"triggerType" : "PUSH"
}, {
"hash" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6357",
"triggerID" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c6f524ea6bf6800a797f9173c4034ce8d111a7d5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6320)
* d3b3e052e3565fe44174dfae73938bfaa44faee4 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6357)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1062045316
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
}, {
"hash" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178",
"triggerID" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"triggerType" : "PUSH"
}, {
"hash" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6320",
"triggerID" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"triggerType" : "PUSH"
}, {
"hash" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6357",
"triggerID" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"triggerType" : "PUSH"
}, {
"hash" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6706",
"triggerID" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a86b7ff2d204900374751371cd2c8c36e76ea4b8 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6706)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1075938869
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
}, {
"hash" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178",
"triggerID" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"triggerType" : "PUSH"
}, {
"hash" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6320",
"triggerID" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"triggerType" : "PUSH"
}, {
"hash" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6357",
"triggerID" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"triggerType" : "PUSH"
}, {
"hash" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6706",
"triggerID" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"triggerType" : "PUSH"
}, {
"hash" : "d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7000",
"triggerID" : "d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5",
"triggerType" : "PUSH"
}, {
"hash" : "e89fd60b77fbd366205df5e60777ced881b6d795",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7203",
"triggerID" : "e89fd60b77fbd366205df5e60777ced881b6d795",
"triggerType" : "PUSH"
}, {
"hash" : "e89fd60b77fbd366205df5e60777ced881b6d795",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7220",
"triggerID" : "1075938530",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* e89fd60b77fbd366205df5e60777ced881b6d795 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7203) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7220)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stayrascal commented on a change in pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
stayrascal commented on a change in pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#discussion_r813878115
##########
File path: hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java
##########
@@ -268,6 +268,12 @@ private FlinkOptions() {
.withDescription("Payload class used. Override this, if you like to roll your own merge logic, when upserting/inserting.\n"
+ "This will render any value set for the option in-effective");
+ public static final ConfigOption<Boolean> PARTIAL_OVERWRITE_ENABLED = ConfigOptions
Review comment:
This is the feature toggle to control another change about `BucketAssignFunction` to support the case the record partition path is changed, but I have removed it. so this feature toggle can be removed as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1078622103
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
}, {
"hash" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178",
"triggerID" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"triggerType" : "PUSH"
}, {
"hash" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6320",
"triggerID" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"triggerType" : "PUSH"
}, {
"hash" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6357",
"triggerID" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"triggerType" : "PUSH"
}, {
"hash" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6706",
"triggerID" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"triggerType" : "PUSH"
}, {
"hash" : "d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7000",
"triggerID" : "d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5",
"triggerType" : "PUSH"
}, {
"hash" : "e89fd60b77fbd366205df5e60777ced881b6d795",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7203",
"triggerID" : "e89fd60b77fbd366205df5e60777ced881b6d795",
"triggerType" : "PUSH"
}, {
"hash" : "e89fd60b77fbd366205df5e60777ced881b6d795",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7220",
"triggerID" : "1075938530",
"triggerType" : "MANUAL"
}, {
"hash" : "3f771d39a403669bf7d0a8d50cd3e5b711b03ebe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3f771d39a403669bf7d0a8d50cd3e5b711b03ebe",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e89fd60b77fbd366205df5e60777ced881b6d795 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7203) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7220)
* 3f771d39a403669bf7d0a8d50cd3e5b711b03ebe UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1078623134
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
}, {
"hash" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178",
"triggerID" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"triggerType" : "PUSH"
}, {
"hash" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6320",
"triggerID" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"triggerType" : "PUSH"
}, {
"hash" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6357",
"triggerID" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"triggerType" : "PUSH"
}, {
"hash" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6706",
"triggerID" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"triggerType" : "PUSH"
}, {
"hash" : "d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7000",
"triggerID" : "d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5",
"triggerType" : "PUSH"
}, {
"hash" : "e89fd60b77fbd366205df5e60777ced881b6d795",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7203",
"triggerID" : "e89fd60b77fbd366205df5e60777ced881b6d795",
"triggerType" : "PUSH"
}, {
"hash" : "e89fd60b77fbd366205df5e60777ced881b6d795",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7220",
"triggerID" : "1075938530",
"triggerType" : "MANUAL"
}, {
"hash" : "3f771d39a403669bf7d0a8d50cd3e5b711b03ebe",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7340",
"triggerID" : "3f771d39a403669bf7d0a8d50cd3e5b711b03ebe",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e89fd60b77fbd366205df5e60777ced881b6d795 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7203) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7220)
* 3f771d39a403669bf7d0a8d50cd3e5b711b03ebe Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7340)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1075796441
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
}, {
"hash" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178",
"triggerID" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"triggerType" : "PUSH"
}, {
"hash" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6320",
"triggerID" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"triggerType" : "PUSH"
}, {
"hash" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6357",
"triggerID" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"triggerType" : "PUSH"
}, {
"hash" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6706",
"triggerID" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"triggerType" : "PUSH"
}, {
"hash" : "d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7000",
"triggerID" : "d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5",
"triggerType" : "PUSH"
}, {
"hash" : "e89fd60b77fbd366205df5e60777ced881b6d795",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7203",
"triggerID" : "e89fd60b77fbd366205df5e60777ced881b6d795",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7000)
* e89fd60b77fbd366205df5e60777ced881b6d795 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7203)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1025672450
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d73d7f503615583bd233b1ba974e7c78125c41bb UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1025731697
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d73d7f503615583bd233b1ba974e7c78125c41bb Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1031330512
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6b6a60fd0f04f236289e500b20951158c2358f51 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746)
* 21df6fe5ac130bfca73a7e3f528d8ef1df66c074 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stayrascal commented on a change in pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
stayrascal commented on a change in pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#discussion_r801731047
##########
File path: hudi-flink/src/main/java/org/apache/hudi/sink/partitioner/BucketAssignFunction.java
##########
@@ -141,27 +142,41 @@ public void snapshotState(FunctionSnapshotContext context) {
@Override
public void initializeState(FunctionInitializationContext context) {
- ValueStateDescriptor<HoodieRecordGlobalLocation> indexStateDesc =
+ ValueStateDescriptor<HoodieRecord> indexStateDesc =
Review comment:
Thanks @danny0405 for reviewing this.
Yeah, as mentioned in this issue https://github.com/apache/hudi/issues/4030.
The reason why we need full record here is want to handle the case that the record partition path is changed.
In the original logic, once the record partition path changed, will sink a delete record to old partition file to delete old record, and then sink the incoming record with new partition to new partition file, the final record will only contains the info from incoming record and miss the info from old record. (note the `OverwriteNonDefaultsWithLatestAvroPayload` also have this issue)
So we need to retrieve the old/existing record from base file, and then merge/combine with incoming record, as currently we don't support lookup record from base file, so we have to store the old/existing in somewhere. e.g. fink state. `BucketAssignFunction` is the only place we can store the old/existing record and change its location from old partition file to new partition file.
So the new logic is that:
- store the old record(from source or base file by enable Bootstrap) in Flink state
- once a new record coming with same record key but the partition changed
- sink a delete record to old partition file to delete file
- retrieve & copy old record from state and change its location with new partition, and sink to new partition file
- the copied record and incoming record will be merged by `#preCombine`
the drawback here is that it will increase the state size, but if we don't use the state to store full record, it seems that we don't have approach to merge incoming record with existing record in base file while partition change.
I also consider this problem, what I'm thinking to avoid impact the current logic(overwrite with latest payload) is that create a `updateState` abstract method and treat `indexState` as a abstract filed, different sub class will implement the logic with `ValueState<HoodieRecordGlobalLocation>` or `ValueState<HoodieRecord>`, or a `StateHelper` to handle state operation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#discussion_r810684546
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/PartialOverwriteWithLatestAvroPayload.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.hudi.common.util.Option;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Objects;
+
+import static org.apache.hudi.avro.HoodieAvroUtils.bytesToAvro;
+
+/**
+ * The only difference with {@link OverwriteNonDefaultsWithLatestAvroPayload} is that it supports
+ * merging the latest non-null partial fields with the old record instead of replacing the whole record.
+ * And merging the non-null fields during preCombine multiple records with same record key instead of choosing the latest record based on ordering field.
+ *
+ * <p> Regarding #combineAndGetUpdateValue, Assuming a {@link GenericRecord} has row schema: (f0 int , f1 int, f2 int).
+ * The first record value is: (1, 2, 3), the second record value is: (4, 5, null) with the field f2 value as null.
+ * Calling the #combineAndGetUpdateValue method of the two records returns record: (4, 5, 3).
+ * Note that field f2 value is ignored because it is null. </p>
+ *
+ * <p> Regarding #preCombine, Assuming a {@link GenericRecord} has row schema: (f0 int , f1 int, f2 int, o1 int),
+ * and initial two {@link PartialOverwriteWithLatestAvroPayload} with different ordering value.
+ * The first record value is (1, null, 1, 1) with the filed f1 value as null, the second value is: (2, 2, null, 2) with the f2 value as null.
+ * Calling the #preCombine method of the two records returns record: (2, 2, 1, 2).
+ * Note:
+ * <ol>
+ * <li>the field f0 value is 2 because the ordering value of second record is bigger.</li>
+ * <li>the filed f1 value is 2 because the f2 value of first record is null.</li>
+ * <li>the filed f2 value is 1 because the f2 value of second record is null.</li>
+ * <li>the filed o1 value is 2 because the ordering value of second record is bigger.</li>
+ * </ol>
+ *
+ * </p>
+ */
+public class PartialOverwriteWithLatestAvroPayload extends OverwriteWithLatestAvroPayload {
+
+ public PartialOverwriteWithLatestAvroPayload(GenericRecord record, Comparable orderingVal) {
+ this(record, orderingVal, null);
+ }
+
+ public PartialOverwriteWithLatestAvroPayload(GenericRecord record, Comparable orderingVal, String schema) {
+ super(record, orderingVal, schema);
+ }
+
+ public PartialOverwriteWithLatestAvroPayload(Option<GenericRecord> record) {
+ super(record); // natural order
+ }
+
+ @Override
+ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord currentValue, Schema schema) throws IOException {
+ if (recordBytes.length == 0) {
+ return Option.empty();
+ }
+
+ GenericRecord incomingRecord = bytesToAvro(recordBytes, schema);
+ if (isDeleteRecord(incomingRecord)) {
+ return Option.empty();
+ }
+
+ GenericRecord currentRecord = (GenericRecord) currentValue;
+ List<Schema.Field> fields = schema.getFields();
+ fields.forEach(field -> {
+ Object value = incomingRecord.get(field.name());
Review comment:
do we need to deal w/ nested fields here ?
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/PartialOverwriteWithLatestAvroPayload.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.hudi.common.util.Option;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Objects;
+
+import static org.apache.hudi.avro.HoodieAvroUtils.bytesToAvro;
+
+/**
+ * The only difference with {@link OverwriteNonDefaultsWithLatestAvroPayload} is that it supports
+ * merging the latest non-null partial fields with the old record instead of replacing the whole record.
+ * And merging the non-null fields during preCombine multiple records with same record key instead of choosing the latest record based on ordering field.
+ *
+ * <p> Regarding #combineAndGetUpdateValue, Assuming a {@link GenericRecord} has row schema: (f0 int , f1 int, f2 int).
+ * The first record value is: (1, 2, 3), the second record value is: (4, 5, null) with the field f2 value as null.
+ * Calling the #combineAndGetUpdateValue method of the two records returns record: (4, 5, 3).
+ * Note that field f2 value is ignored because it is null. </p>
+ *
+ * <p> Regarding #preCombine, Assuming a {@link GenericRecord} has row schema: (f0 int , f1 int, f2 int, o1 int),
+ * and initial two {@link PartialOverwriteWithLatestAvroPayload} with different ordering value.
+ * The first record value is (1, null, 1, 1) with the filed f1 value as null, the second value is: (2, 2, null, 2) with the f2 value as null.
+ * Calling the #preCombine method of the two records returns record: (2, 2, 1, 2).
+ * Note:
+ * <ol>
+ * <li>the field f0 value is 2 because the ordering value of second record is bigger.</li>
+ * <li>the filed f1 value is 2 because the f2 value of first record is null.</li>
+ * <li>the filed f2 value is 1 because the f2 value of second record is null.</li>
+ * <li>the filed o1 value is 2 because the ordering value of second record is bigger.</li>
+ * </ol>
+ *
+ * </p>
+ */
+public class PartialOverwriteWithLatestAvroPayload extends OverwriteWithLatestAvroPayload {
+
+ public PartialOverwriteWithLatestAvroPayload(GenericRecord record, Comparable orderingVal) {
+ this(record, orderingVal, null);
+ }
+
+ public PartialOverwriteWithLatestAvroPayload(GenericRecord record, Comparable orderingVal, String schema) {
+ super(record, orderingVal, schema);
+ }
+
+ public PartialOverwriteWithLatestAvroPayload(Option<GenericRecord> record) {
+ super(record); // natural order
+ }
+
+ @Override
+ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord currentValue, Schema schema) throws IOException {
+ if (recordBytes.length == 0) {
+ return Option.empty();
+ }
+
+ GenericRecord incomingRecord = bytesToAvro(recordBytes, schema);
+ if (isDeleteRecord(incomingRecord)) {
+ return Option.empty();
+ }
+
+ GenericRecord currentRecord = (GenericRecord) currentValue;
+ List<Schema.Field> fields = schema.getFields();
+ fields.forEach(field -> {
+ Object value = incomingRecord.get(field.name());
+ if (Objects.nonNull(value)) {
+ currentRecord.put(field.name(), value);
+ }
+ });
+
+ return Option.of(currentRecord);
+ }
+
+ @Override
+ public int compareTo(OverwriteWithLatestAvroPayload oldValue) {
+ return orderingVal.compareTo(oldValue.orderingVal);
+ }
+
+ @Override
+ public OverwriteWithLatestAvroPayload preCombine(OverwriteWithLatestAvroPayload oldValue) {
+ if (null == this.schema || null == oldValue.schema) {
+ return super.preCombine(oldValue);
+ }
+
+ try {
+ Schema schema = new Schema.Parser().parse(this.schema);
Review comment:
argh. this again clashes w/ instance variable "schema". Can we fix the naming.
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/PartialOverwriteWithLatestAvroPayload.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.hudi.common.util.Option;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Objects;
+
+import static org.apache.hudi.avro.HoodieAvroUtils.bytesToAvro;
+
+/**
+ * The only difference with {@link OverwriteNonDefaultsWithLatestAvroPayload} is that it supports
+ * merging the latest non-null partial fields with the old record instead of replacing the whole record.
+ * And merging the non-null fields during preCombine multiple records with same record key instead of choosing the latest record based on ordering field.
+ *
+ * <p> Regarding #combineAndGetUpdateValue, Assuming a {@link GenericRecord} has row schema: (f0 int , f1 int, f2 int).
+ * The first record value is: (1, 2, 3), the second record value is: (4, 5, null) with the field f2 value as null.
+ * Calling the #combineAndGetUpdateValue method of the two records returns record: (4, 5, 3).
+ * Note that field f2 value is ignored because it is null. </p>
+ *
+ * <p> Regarding #preCombine, Assuming a {@link GenericRecord} has row schema: (f0 int , f1 int, f2 int, o1 int),
+ * and initial two {@link PartialOverwriteWithLatestAvroPayload} with different ordering value.
+ * The first record value is (1, null, 1, 1) with the filed f1 value as null, the second value is: (2, 2, null, 2) with the f2 value as null.
+ * Calling the #preCombine method of the two records returns record: (2, 2, 1, 2).
+ * Note:
+ * <ol>
+ * <li>the field f0 value is 2 because the ordering value of second record is bigger.</li>
+ * <li>the filed f1 value is 2 because the f2 value of first record is null.</li>
+ * <li>the filed f2 value is 1 because the f2 value of second record is null.</li>
+ * <li>the filed o1 value is 2 because the ordering value of second record is bigger.</li>
+ * </ol>
+ *
+ * </p>
+ */
+public class PartialOverwriteWithLatestAvroPayload extends OverwriteWithLatestAvroPayload {
+
+ public PartialOverwriteWithLatestAvroPayload(GenericRecord record, Comparable orderingVal) {
+ this(record, orderingVal, null);
+ }
+
+ public PartialOverwriteWithLatestAvroPayload(GenericRecord record, Comparable orderingVal, String schema) {
+ super(record, orderingVal, schema);
+ }
+
+ public PartialOverwriteWithLatestAvroPayload(Option<GenericRecord> record) {
+ super(record); // natural order
+ }
+
+ @Override
+ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord currentValue, Schema schema) throws IOException {
+ if (recordBytes.length == 0) {
+ return Option.empty();
+ }
+
+ GenericRecord incomingRecord = bytesToAvro(recordBytes, schema);
+ if (isDeleteRecord(incomingRecord)) {
+ return Option.empty();
+ }
+
+ GenericRecord currentRecord = (GenericRecord) currentValue;
+ List<Schema.Field> fields = schema.getFields();
+ fields.forEach(field -> {
+ Object value = incomingRecord.get(field.name());
+ if (Objects.nonNull(value)) {
+ currentRecord.put(field.name(), value);
+ }
+ });
+
+ return Option.of(currentRecord);
+ }
+
+ @Override
+ public int compareTo(OverwriteWithLatestAvroPayload oldValue) {
+ return orderingVal.compareTo(oldValue.orderingVal);
+ }
+
+ @Override
+ public OverwriteWithLatestAvroPayload preCombine(OverwriteWithLatestAvroPayload oldValue) {
+ if (null == this.schema || null == oldValue.schema) {
+ return super.preCombine(oldValue);
+ }
+
+ try {
+ Schema schema = new Schema.Parser().parse(this.schema);
+ Option<IndexedRecord> incomingOption = getInsertValue(new Schema.Parser().parse(this.schema));
+ Option<IndexedRecord> insertRecordOption = oldValue.getInsertValue(new Schema.Parser().parse(oldValue.schema));
Review comment:
insertRecordOption -> oldRecordOption
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/OverwriteWithLatestAvroPayload.java
##########
@@ -40,8 +40,18 @@
public class OverwriteWithLatestAvroPayload extends BaseAvroPayload
implements HoodieRecordPayload<OverwriteWithLatestAvroPayload> {
+ /**
+ * the schema of generic record
+ */
+ public final String schema;
Review comment:
but in general, storing schema along w/ payload might have an impact on the performance. and thats why initial payload was designed that way. So, do add a line here for payload implementations setting this schema field might have to watch out for performance.
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/OverwriteWithLatestAvroPayload.java
##########
@@ -40,8 +40,18 @@
public class OverwriteWithLatestAvroPayload extends BaseAvroPayload
implements HoodieRecordPayload<OverwriteWithLatestAvroPayload> {
+ /**
+ * the schema of generic record
+ */
+ public final String schema;
Review comment:
this might be confusing w/ schema arg with combineAndGetUpdateValue. can you fix either of the names.
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/PartialOverwriteWithLatestAvroPayload.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.hudi.common.util.Option;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Objects;
+
+import static org.apache.hudi.avro.HoodieAvroUtils.bytesToAvro;
+
+/**
+ * The only difference with {@link OverwriteNonDefaultsWithLatestAvroPayload} is that it supports
+ * merging the latest non-null partial fields with the old record instead of replacing the whole record.
+ * And merging the non-null fields during preCombine multiple records with same record key instead of choosing the latest record based on ordering field.
+ *
+ * <p> Regarding #combineAndGetUpdateValue, Assuming a {@link GenericRecord} has row schema: (f0 int , f1 int, f2 int).
+ * The first record value is: (1, 2, 3), the second record value is: (4, 5, null) with the field f2 value as null.
+ * Calling the #combineAndGetUpdateValue method of the two records returns record: (4, 5, 3).
+ * Note that field f2 value is ignored because it is null. </p>
+ *
+ * <p> Regarding #preCombine, Assuming a {@link GenericRecord} has row schema: (f0 int , f1 int, f2 int, o1 int),
+ * and initial two {@link PartialOverwriteWithLatestAvroPayload} with different ordering value.
+ * The first record value is (1, null, 1, 1) with the filed f1 value as null, the second value is: (2, 2, null, 2) with the f2 value as null.
+ * Calling the #preCombine method of the two records returns record: (2, 2, 1, 2).
+ * Note:
+ * <ol>
+ * <li>the field f0 value is 2 because the ordering value of second record is bigger.</li>
+ * <li>the filed f1 value is 2 because the f2 value of first record is null.</li>
+ * <li>the filed f2 value is 1 because the f2 value of second record is null.</li>
+ * <li>the filed o1 value is 2 because the ordering value of second record is bigger.</li>
+ * </ol>
+ *
+ * </p>
+ */
+public class PartialOverwriteWithLatestAvroPayload extends OverwriteWithLatestAvroPayload {
+
+ public PartialOverwriteWithLatestAvroPayload(GenericRecord record, Comparable orderingVal) {
+ this(record, orderingVal, null);
+ }
+
+ public PartialOverwriteWithLatestAvroPayload(GenericRecord record, Comparable orderingVal, String schema) {
+ super(record, orderingVal, schema);
+ }
+
+ public PartialOverwriteWithLatestAvroPayload(Option<GenericRecord> record) {
+ super(record); // natural order
+ }
+
+ @Override
+ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord currentValue, Schema schema) throws IOException {
+ if (recordBytes.length == 0) {
+ return Option.empty();
+ }
+
+ GenericRecord incomingRecord = bytesToAvro(recordBytes, schema);
+ if (isDeleteRecord(incomingRecord)) {
+ return Option.empty();
+ }
+
+ GenericRecord currentRecord = (GenericRecord) currentValue;
+ List<Schema.Field> fields = schema.getFields();
Review comment:
guess, this has to be "this.schema.getFields". as I commented earlier, its confusing :) . can we fix the naming of either of them.
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/PartialOverwriteWithLatestAvroPayload.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.hudi.common.util.Option;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Objects;
+
+import static org.apache.hudi.avro.HoodieAvroUtils.bytesToAvro;
+
+/**
+ * The only difference with {@link OverwriteNonDefaultsWithLatestAvroPayload} is that it supports
+ * merging the latest non-null partial fields with the old record instead of replacing the whole record.
+ * And merging the non-null fields during preCombine multiple records with same record key instead of choosing the latest record based on ordering field.
+ *
+ * <p> Regarding #combineAndGetUpdateValue, Assuming a {@link GenericRecord} has row schema: (f0 int , f1 int, f2 int).
+ * The first record value is: (1, 2, 3), the second record value is: (4, 5, null) with the field f2 value as null.
+ * Calling the #combineAndGetUpdateValue method of the two records returns record: (4, 5, 3).
+ * Note that field f2 value is ignored because it is null. </p>
+ *
+ * <p> Regarding #preCombine, Assuming a {@link GenericRecord} has row schema: (f0 int , f1 int, f2 int, o1 int),
+ * and initial two {@link PartialOverwriteWithLatestAvroPayload} with different ordering value.
+ * The first record value is (1, null, 1, 1) with the filed f1 value as null, the second value is: (2, 2, null, 2) with the f2 value as null.
+ * Calling the #preCombine method of the two records returns record: (2, 2, 1, 2).
+ * Note:
+ * <ol>
+ * <li>the field f0 value is 2 because the ordering value of second record is bigger.</li>
+ * <li>the filed f1 value is 2 because the f2 value of first record is null.</li>
+ * <li>the filed f2 value is 1 because the f2 value of second record is null.</li>
+ * <li>the filed o1 value is 2 because the ordering value of second record is bigger.</li>
+ * </ol>
+ *
+ * </p>
+ */
+public class PartialOverwriteWithLatestAvroPayload extends OverwriteWithLatestAvroPayload {
+
+ public PartialOverwriteWithLatestAvroPayload(GenericRecord record, Comparable orderingVal) {
+ this(record, orderingVal, null);
+ }
+
+ public PartialOverwriteWithLatestAvroPayload(GenericRecord record, Comparable orderingVal, String schema) {
+ super(record, orderingVal, schema);
+ }
+
+ public PartialOverwriteWithLatestAvroPayload(Option<GenericRecord> record) {
+ super(record); // natural order
+ }
+
+ @Override
+ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord currentValue, Schema schema) throws IOException {
+ if (recordBytes.length == 0) {
+ return Option.empty();
+ }
+
+ GenericRecord incomingRecord = bytesToAvro(recordBytes, schema);
+ if (isDeleteRecord(incomingRecord)) {
+ return Option.empty();
+ }
+
+ GenericRecord currentRecord = (GenericRecord) currentValue;
+ List<Schema.Field> fields = schema.getFields();
+ fields.forEach(field -> {
+ Object value = incomingRecord.get(field.name());
+ if (Objects.nonNull(value)) {
+ currentRecord.put(field.name(), value);
+ }
+ });
+
+ return Option.of(currentRecord);
+ }
+
+ @Override
+ public int compareTo(OverwriteWithLatestAvroPayload oldValue) {
+ return orderingVal.compareTo(oldValue.orderingVal);
+ }
+
+ @Override
+ public OverwriteWithLatestAvroPayload preCombine(OverwriteWithLatestAvroPayload oldValue) {
Review comment:
instead of storing the schema with payload, did you think about adding a new preCombine method as follows
```
OverwriteWithLatestAvroPayload preCombine(OverwriteWithLatestAvroPayload oldValue, Schema schema);
```
this would make it a lot simpler right. Since preCombine is used only to dedup records within a single batch, both records should have same schema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stayrascal commented on a change in pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
stayrascal commented on a change in pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#discussion_r814533349
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordPayload.java
##########
@@ -58,6 +58,31 @@ default T preCombine(T oldValue, Properties properties) {
return preCombine(oldValue);
}
+ /**
+ *When more than one HoodieRecord have the same HoodieKey in the incoming batch, this function combines them before attempting to insert/upsert by taking in a property map.
+ *
+ * @param oldValue instance of the old {@link HoodieRecordPayload} to be combined with.
+ * @param properties Payload related properties. For example pass the ordering field(s) name to extract from value in storage.
+ * @param schema Schema used for record
+ * @return the combined value
+ */
+ @PublicAPIMethod(maturity = ApiMaturityLevel.STABLE)
+ default T preCombine(T oldValue, Properties properties, Schema schema) {
Review comment:
Hi @alexeykudinkin, I got your point. if we have to combine two records to a combined one, we'd better to implement the combine logics in other place, maybe in some `util` or `helper` classes, or skip the de-duping logic, right?
Here are some options from mine that `#preCombine` might be a better place to implement these logics, or create new `merge` method in `HoodieRecordPayload` interface.
- First, from the description of `preCombine` method, it used for combining multiple records with same HoodieKey before attempting to insert/upsert to disk. The "combine multiple records" might not mean only choosing one of them, we also can combine & merged them to a new one, just depends on how the sub-class implement the preCombine logic(Please correct me if my understanding is wrong :) ). Yeah, it might be a little bit confused that we need `Schema` if we are trying to merged them.
- Second, I checked when will we call `preCombine` method is trying to duplicate records with same HoodieKey before insert/update to disk, especially in Flink write case, even through the duplicated logic is choose the latest record, but we need to ensure that one HoodieKey should only contains one record before comparing to existing record and write to disk, otherwise, some records will missed. For example, in `HoodieMergeHandle.init(fieId, newRecordsIter)`, it will convert the record iterator to a map and treat the recordKey as key. *So we might not stop de-duping logics and merge them against what is on disk unless we change the logic here.* And also we implement another class/method to handle the merge logic, and switch the existing de-duping logic from calling `preCombine` to new class/method, we have to add an condition to control whether should we call `preCombine` or not, I think it might not a good way. Instead, we should handle it in `preCombine` method by different impl
emented payload.
That's what my thought here, and I'm glad to listen your useful suggestions. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1057362722
Just FYI for all interested folks.
Precombine is not just used to dedup two records within same incoming batch, but also to deduce the winner when we [merge records in LogRecordReader.](https://github.com/apache/hudi/blob/907e60c252e80be5ef3a848d773e0f866eb609f9/hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieMergedLogRecordScanner.java#L145)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1061975997
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
}, {
"hash" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178",
"triggerID" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"triggerType" : "PUSH"
}, {
"hash" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6320",
"triggerID" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"triggerType" : "PUSH"
}, {
"hash" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6357",
"triggerID" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"triggerType" : "PUSH"
}, {
"hash" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6706",
"triggerID" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d3b3e052e3565fe44174dfae73938bfaa44faee4 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6357)
* a86b7ff2d204900374751371cd2c8c36e76ea4b8 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6706)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stayrascal commented on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
stayrascal commented on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1075938530
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1075793865
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
}, {
"hash" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178",
"triggerID" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"triggerType" : "PUSH"
}, {
"hash" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6320",
"triggerID" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"triggerType" : "PUSH"
}, {
"hash" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6357",
"triggerID" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"triggerType" : "PUSH"
}, {
"hash" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6706",
"triggerID" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"triggerType" : "PUSH"
}, {
"hash" : "d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7000",
"triggerID" : "d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5",
"triggerType" : "PUSH"
}, {
"hash" : "e89fd60b77fbd366205df5e60777ced881b6d795",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e89fd60b77fbd366205df5e60777ced881b6d795",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7000)
* e89fd60b77fbd366205df5e60777ced881b6d795 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1031369988
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 21df6fe5ac130bfca73a7e3f528d8ef1df66c074 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1031369988
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 21df6fe5ac130bfca73a7e3f528d8ef1df66c074 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1040829413
CC @xushiyan @alexeykudinkin : who might be working on refactoring payload interfaces. and @xiarixiaoyao who might be working on schema evolution story. can you folks take a look at the patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stayrascal commented on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
stayrascal commented on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1041115256
> CC @xushiyan @alexeykudinkin : who might be working on refactoring payload interfaces. and @xiarixiaoyao who might be working on schema evolution story. can you folks take a look at the patch.
Thanks @nsivabalan to follow up. Take a summary, the current implementation has two drawbacks:
- embedding schema in payload might hard to do schema evolution later.
- using flink state to store full HoodieRecord instead of location, will increase the state size.
And there are two challenges need to be solved for supporting partial update.
- how combine/merge two payload in `#preCombine` if without schema.
- how to lookup the record from base/log file during the record partition patch changed, and then combine/merge with the incoming record later.
Any thoughts on these two challenges?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1046951931
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 21df6fe5ac130bfca73a7e3f528d8ef1df66c074 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758)
* 14edef016d98786b713e155e9adec1d87a3500ba UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1046951931
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 21df6fe5ac130bfca73a7e3f528d8ef1df66c074 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758)
* 14edef016d98786b713e155e9adec1d87a3500ba UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stayrascal commented on a change in pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
stayrascal commented on a change in pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#discussion_r811144363
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/PartialOverwriteWithLatestAvroPayload.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.hudi.common.util.Option;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Objects;
+
+import static org.apache.hudi.avro.HoodieAvroUtils.bytesToAvro;
+
+/**
+ * The only difference with {@link OverwriteNonDefaultsWithLatestAvroPayload} is that it supports
+ * merging the latest non-null partial fields with the old record instead of replacing the whole record.
+ * And merging the non-null fields during preCombine multiple records with same record key instead of choosing the latest record based on ordering field.
+ *
+ * <p> Regarding #combineAndGetUpdateValue, Assuming a {@link GenericRecord} has row schema: (f0 int , f1 int, f2 int).
+ * The first record value is: (1, 2, 3), the second record value is: (4, 5, null) with the field f2 value as null.
+ * Calling the #combineAndGetUpdateValue method of the two records returns record: (4, 5, 3).
+ * Note that field f2 value is ignored because it is null. </p>
+ *
+ * <p> Regarding #preCombine, Assuming a {@link GenericRecord} has row schema: (f0 int , f1 int, f2 int, o1 int),
+ * and initial two {@link PartialOverwriteWithLatestAvroPayload} with different ordering value.
+ * The first record value is (1, null, 1, 1) with the filed f1 value as null, the second value is: (2, 2, null, 2) with the f2 value as null.
+ * Calling the #preCombine method of the two records returns record: (2, 2, 1, 2).
+ * Note:
+ * <ol>
+ * <li>the field f0 value is 2 because the ordering value of second record is bigger.</li>
+ * <li>the filed f1 value is 2 because the f2 value of first record is null.</li>
+ * <li>the filed f2 value is 1 because the f2 value of second record is null.</li>
+ * <li>the filed o1 value is 2 because the ordering value of second record is bigger.</li>
+ * </ol>
+ *
+ * </p>
+ */
+public class PartialOverwriteWithLatestAvroPayload extends OverwriteWithLatestAvroPayload {
+
+ public PartialOverwriteWithLatestAvroPayload(GenericRecord record, Comparable orderingVal) {
+ this(record, orderingVal, null);
+ }
+
+ public PartialOverwriteWithLatestAvroPayload(GenericRecord record, Comparable orderingVal, String schema) {
+ super(record, orderingVal, schema);
+ }
+
+ public PartialOverwriteWithLatestAvroPayload(Option<GenericRecord> record) {
+ super(record); // natural order
+ }
+
+ @Override
+ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord currentValue, Schema schema) throws IOException {
+ if (recordBytes.length == 0) {
+ return Option.empty();
+ }
+
+ GenericRecord incomingRecord = bytesToAvro(recordBytes, schema);
+ if (isDeleteRecord(incomingRecord)) {
+ return Option.empty();
+ }
+
+ GenericRecord currentRecord = (GenericRecord) currentValue;
+ List<Schema.Field> fields = schema.getFields();
+ fields.forEach(field -> {
+ Object value = incomingRecord.get(field.name());
+ if (Objects.nonNull(value)) {
+ currentRecord.put(field.name(), value);
+ }
+ });
+
+ return Option.of(currentRecord);
+ }
+
+ @Override
+ public int compareTo(OverwriteWithLatestAvroPayload oldValue) {
+ return orderingVal.compareTo(oldValue.orderingVal);
+ }
+
+ @Override
+ public OverwriteWithLatestAvroPayload preCombine(OverwriteWithLatestAvroPayload oldValue) {
Review comment:
Hi @nsivabalan , thanks a lot for review this.
Regarding adding new `preCombine` method with `Schema`, I considered this, but it means that the method caller who need to get the schema info at first, and currently, it seems that we only can get the schema info from Configuration(from `hoodie.avro.schema` field). Sometimes, the caller might hard to get the schema info, Especially for `FlinkWriteHeler.deduplicateRecords(List<HoodieRecord<T>> records, HoodieIndex<?, ?> index, int parallelism)`.
But compare the performance, it seems that passing the schema in method might be a better approach.
BTW, since we already have had the method `preCombine(T oldValue, Properties properties)`, how about put the schema string in properties, and then parse the schema string to Schema later, so that we don't need to create a new method any more. otherwise, I cannot image when will we will `Properties`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1075962905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
}, {
"hash" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178",
"triggerID" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"triggerType" : "PUSH"
}, {
"hash" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6320",
"triggerID" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"triggerType" : "PUSH"
}, {
"hash" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6357",
"triggerID" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"triggerType" : "PUSH"
}, {
"hash" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6706",
"triggerID" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"triggerType" : "PUSH"
}, {
"hash" : "d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7000",
"triggerID" : "d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5",
"triggerType" : "PUSH"
}, {
"hash" : "e89fd60b77fbd366205df5e60777ced881b6d795",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7203",
"triggerID" : "e89fd60b77fbd366205df5e60777ced881b6d795",
"triggerType" : "PUSH"
}, {
"hash" : "e89fd60b77fbd366205df5e60777ced881b6d795",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7220",
"triggerID" : "1075938530",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* e89fd60b77fbd366205df5e60777ced881b6d795 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7203) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7220)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1061975997
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
}, {
"hash" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178",
"triggerID" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"triggerType" : "PUSH"
}, {
"hash" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6320",
"triggerID" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"triggerType" : "PUSH"
}, {
"hash" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6357",
"triggerID" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"triggerType" : "PUSH"
}, {
"hash" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6706",
"triggerID" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d3b3e052e3565fe44174dfae73938bfaa44faee4 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6357)
* a86b7ff2d204900374751371cd2c8c36e76ea4b8 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6706)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1061973132
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
}, {
"hash" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178",
"triggerID" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"triggerType" : "PUSH"
}, {
"hash" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6320",
"triggerID" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"triggerType" : "PUSH"
}, {
"hash" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6357",
"triggerID" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"triggerType" : "PUSH"
}, {
"hash" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d3b3e052e3565fe44174dfae73938bfaa44faee4 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6357)
* a86b7ff2d204900374751371cd2c8c36e76ea4b8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1061973132
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
}, {
"hash" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178",
"triggerID" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"triggerType" : "PUSH"
}, {
"hash" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6320",
"triggerID" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"triggerType" : "PUSH"
}, {
"hash" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6357",
"triggerID" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"triggerType" : "PUSH"
}, {
"hash" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d3b3e052e3565fe44174dfae73938bfaa44faee4 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6357)
* a86b7ff2d204900374751371cd2c8c36e76ea4b8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stayrascal commented on a change in pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
stayrascal commented on a change in pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#discussion_r828005650
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordPayload.java
##########
@@ -58,6 +58,31 @@ default T preCombine(T oldValue, Properties properties) {
return preCombine(oldValue);
}
+ /**
+ *When more than one HoodieRecord have the same HoodieKey in the incoming batch, this function combines them before attempting to insert/upsert by taking in a property map.
+ *
+ * @param oldValue instance of the old {@link HoodieRecordPayload} to be combined with.
+ * @param properties Payload related properties. For example pass the ordering field(s) name to extract from value in storage.
+ * @param schema Schema used for record
+ * @return the combined value
+ */
+ @PublicAPIMethod(maturity = ApiMaturityLevel.STABLE)
+ default T preCombine(T oldValue, Properties properties, Schema schema) {
Review comment:
@alexeykudinkin , I'm sorry that I still don't find a suitable time to align with online, may i check any thoughts or suggests on this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1075793865
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
}, {
"hash" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178",
"triggerID" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"triggerType" : "PUSH"
}, {
"hash" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6320",
"triggerID" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"triggerType" : "PUSH"
}, {
"hash" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6357",
"triggerID" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"triggerType" : "PUSH"
}, {
"hash" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6706",
"triggerID" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"triggerType" : "PUSH"
}, {
"hash" : "d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7000",
"triggerID" : "d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5",
"triggerType" : "PUSH"
}, {
"hash" : "e89fd60b77fbd366205df5e60777ced881b6d795",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e89fd60b77fbd366205df5e60777ced881b6d795",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7000)
* e89fd60b77fbd366205df5e60777ced881b6d795 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stayrascal commented on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
stayrascal commented on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1057603548
>
yeah, @LinMingQiang has mentioned this one above.
From my understanding, if we want to enable "partial update" feature by defining customized payload class, it should running "partial update" in these three cases:
- merged the incoming batch records before write to disk
- read records from the log file(read from MOR table)
- read records from log file and compact into base file
So I also update the `HoodieMergedLogRecordScanner.processNextRecord ()` by passing Schema info, if the use case is not using "partial update" with other payload class, it will use the default `preCombine()` logic to choose "recent" one.
The current situation is that will we treat `preCombine` return one of two records, or we can merged them to a new record.
> Just FYI for all interested folks. Precombine is not just used to dedup two records within same incoming batch, but also to deduce the winner when we [merge records in LogRecordReader.](https://github.com/apache/hudi/blob/907e60c252e80be5ef3a848d773e0f866eb609f9/hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieMergedLogRecordScanner.java#L145)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1050961037
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
}, {
"hash" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178",
"triggerID" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"triggerType" : "PUSH"
}, {
"hash" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178)
* c6f524ea6bf6800a797f9173c4034ce8d111a7d5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stayrascal commented on a change in pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
stayrascal commented on a change in pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#discussion_r815265857
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordPayload.java
##########
@@ -58,6 +58,31 @@ default T preCombine(T oldValue, Properties properties) {
return preCombine(oldValue);
}
+ /**
+ *When more than one HoodieRecord have the same HoodieKey in the incoming batch, this function combines them before attempting to insert/upsert by taking in a property map.
+ *
+ * @param oldValue instance of the old {@link HoodieRecordPayload} to be combined with.
+ * @param properties Payload related properties. For example pass the ordering field(s) name to extract from value in storage.
+ * @param schema Schema used for record
+ * @return the combined value
+ */
+ @PublicAPIMethod(maturity = ApiMaturityLevel.STABLE)
+ default T preCombine(T oldValue, Properties properties, Schema schema) {
Review comment:
Hi @alexeykudinkin , Thanks a lot for you detail clarification.
1. Regarding the design of `preCombine`, I'm clear now. I'm sorry I don't know the detail of RFC-46, and also I didn't find the link RFC-46 from [here](https://cwiki.apache.org/confluence/display/HUDI/RFC+Process), cloud you please share the link?
2. and regarding the requirements for partial updates/overwrite, I saw some same requirements from community. In my case, generally, we want to build a customer profile with multiple attributes, these attributes might come from different systems, one system might only provides some attributes in a event/record, and two systems might the events/records with different attributes, we should not only choose the recent one, we need to merged them before writing to disk. Otherwise, we have to keep all change logs, and then start a new job to dedup & merge these attributes among the change logs. For example, we have 10 attributes a1-a10(all of them are optional), source system A only has the a1-a5, source system B only has a6-a10, what result we expect is that the final record contains a1-a10, not only a1-a5 or a6-a10. And because we might receive two events/records in same time, they might be in a same batch, that's why we want to merge them before `combineAndGetUpdateValue `.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1047007541
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
}, {
"hash" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 21df6fe5ac130bfca73a7e3f528d8ef1df66c074 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758)
* 14edef016d98786b713e155e9adec1d87a3500ba Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175)
* ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stayrascal commented on a change in pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
stayrascal commented on a change in pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#discussion_r813849545
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordPayload.java
##########
@@ -58,6 +58,31 @@ default T preCombine(T oldValue, Properties properties) {
return preCombine(oldValue);
}
+ /**
+ *When more than one HoodieRecord have the same HoodieKey in the incoming batch, this function combines them before attempting to insert/upsert by taking in a property map.
+ *
+ * @param oldValue instance of the old {@link HoodieRecordPayload} to be combined with.
+ * @param properties Payload related properties. For example pass the ordering field(s) name to extract from value in storage.
+ * @param schema Schema used for record
+ * @return the combined value
+ */
+ @PublicAPIMethod(maturity = ApiMaturityLevel.STABLE)
+ default T preCombine(T oldValue, Properties properties, Schema schema) {
Review comment:
Thanks @alexeykudinkin for reviewing this.
What we are trying to do is implement partial update purpose. For example, let's assume the record schema is `(f0 int , f1 int, f2 int)`, The first record value is: (1, 2, 3), the second record value is: (4, 5, null) with the field f2 value as null. We hope that the result after run `preCombine` is `(4, 5, 3)`, which means we need to combine/merge two records to a third one, not only choose one of them.
Actually, what we want to implement is similar with `#combineAndGetUpdateValue(IndexedRecord currentValue, Schema schema)` which used for combine the incoming record with existing record from base/log file.
But `#preCombine` will be used for combing/merging two incoming records in a batch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1047017097
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
}, {
"hash" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178",
"triggerID" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 14edef016d98786b713e155e9adec1d87a3500ba Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175)
* ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1075796441
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
}, {
"hash" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178",
"triggerID" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"triggerType" : "PUSH"
}, {
"hash" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6320",
"triggerID" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"triggerType" : "PUSH"
}, {
"hash" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6357",
"triggerID" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"triggerType" : "PUSH"
}, {
"hash" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6706",
"triggerID" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"triggerType" : "PUSH"
}, {
"hash" : "d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7000",
"triggerID" : "d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5",
"triggerType" : "PUSH"
}, {
"hash" : "e89fd60b77fbd366205df5e60777ced881b6d795",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7203",
"triggerID" : "e89fd60b77fbd366205df5e60777ced881b6d795",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7000)
* e89fd60b77fbd366205df5e60777ced881b6d795 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7203)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1075938869
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
}, {
"hash" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178",
"triggerID" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"triggerType" : "PUSH"
}, {
"hash" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6320",
"triggerID" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"triggerType" : "PUSH"
}, {
"hash" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6357",
"triggerID" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"triggerType" : "PUSH"
}, {
"hash" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6706",
"triggerID" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"triggerType" : "PUSH"
}, {
"hash" : "d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7000",
"triggerID" : "d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5",
"triggerType" : "PUSH"
}, {
"hash" : "e89fd60b77fbd366205df5e60777ced881b6d795",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7203",
"triggerID" : "e89fd60b77fbd366205df5e60777ced881b6d795",
"triggerType" : "PUSH"
}, {
"hash" : "e89fd60b77fbd366205df5e60777ced881b6d795",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7220",
"triggerID" : "1075938530",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* e89fd60b77fbd366205df5e60777ced881b6d795 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7203) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7220)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1025672450
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d73d7f503615583bd233b1ba974e7c78125c41bb UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1030849625
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d73d7f503615583bd233b1ba974e7c78125c41bb Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620)
* 6b6a60fd0f04f236289e500b20951158c2358f51 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1030850209
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d73d7f503615583bd233b1ba974e7c78125c41bb Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620)
* 6b6a60fd0f04f236289e500b20951158c2358f51 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1031346059
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 21df6fe5ac130bfca73a7e3f528d8ef1df66c074 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1031343408
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 6b6a60fd0f04f236289e500b20951158c2358f51 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746)
* 21df6fe5ac130bfca73a7e3f528d8ef1df66c074 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stayrascal commented on a change in pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
stayrascal commented on a change in pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#discussion_r818260180
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/PartialOverwriteWithLatestAvroPayload.java
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.hudi.common.util.Option;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Objects;
+import java.util.Properties;
+
+import static org.apache.hudi.avro.HoodieAvroUtils.bytesToAvro;
+
+/**
+ * The only difference with {@link OverwriteNonDefaultsWithLatestAvroPayload} is that it supports
+ * merging the latest non-null partial fields with the old record instead of replacing the whole record.
+ * And merging the non-null fields during preCombine multiple records with same record key instead of choosing the latest record based on ordering field.
+ *
+ * <p> Regarding #combineAndGetUpdateValue, Assuming a {@link GenericRecord} has row schema: (f0 int , f1 int, f2 int).
+ * The first record value is: (1, 2, 3), the second record value is: (4, 5, null) with the field f2 value as null.
+ * Calling the #combineAndGetUpdateValue method of the two records returns record: (4, 5, 3).
+ * Note that field f2 value is ignored because it is null. </p>
+ *
+ * <p> Regarding #preCombine, Assuming a {@link GenericRecord} has row schema: (f0 int , f1 int, f2 int, o1 int),
+ * and initial two {@link PartialOverwriteWithLatestAvroPayload} with different ordering value.
+ * The first record value is (1, null, 1, 1) with the filed f1 value as null, the second value is: (2, 2, null, 2) with the f2 value as null.
+ * Calling the #preCombine method of the two records returns record: (2, 2, 1, 2).
+ * Note:
+ * <ol>
+ * <li>the field f0 value is 2 because the ordering value of second record is bigger.</li>
+ * <li>the filed f1 value is 2 because the f2 value of first record is null.</li>
+ * <li>the filed f2 value is 1 because the f2 value of second record is null.</li>
+ * <li>the filed o1 value is 2 because the ordering value of second record is bigger.</li>
+ * </ol>
+ *
+ * </p>
+ */
+public class PartialOverwriteWithLatestAvroPayload extends OverwriteWithLatestAvroPayload {
+
+ public PartialOverwriteWithLatestAvroPayload(GenericRecord record, Comparable orderingVal) {
+ super(record, orderingVal);
+ }
+
+ public PartialOverwriteWithLatestAvroPayload(Option<GenericRecord> record) {
+ super(record); // natural order
+ }
+
+ @Override
+ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord currentValue, Schema schema) throws IOException {
+ if (recordBytes.length == 0) {
+ return Option.empty();
+ }
+
+ GenericRecord incomingRecord = bytesToAvro(recordBytes, schema);
+ if (isDeleteRecord(incomingRecord)) {
+ return Option.empty();
+ }
+
+ GenericRecord currentRecord = (GenericRecord) currentValue;
+ List<Schema.Field> fields = schema.getFields();
+ fields.forEach(field -> {
+ Object value = incomingRecord.get(field.name());
+ if (Objects.nonNull(value)) {
+ currentRecord.put(field.name(), value);
+ }
+ });
+
+ return Option.of(currentRecord);
+ }
+
+ @Override
+ public int compareTo(OverwriteWithLatestAvroPayload oldValue) {
+ return orderingVal.compareTo(oldValue.orderingVal);
+ }
+
+ @Override
+ public OverwriteWithLatestAvroPayload preCombine(OverwriteWithLatestAvroPayload oldValue, Properties properties, Schema schema) {
+ if (null == schema) {
+ return super.preCombine(oldValue);
+ }
+
+ try {
+ Option<IndexedRecord> incomingOption = getInsertValue(schema);
+ Option<IndexedRecord> oldRecordOption = oldValue.getInsertValue(schema);
+
+ if (incomingOption.isPresent() && oldRecordOption.isPresent()) {
Review comment:
abstracted the merge method, but still in current class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1052318257
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
}, {
"hash" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178",
"triggerID" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"triggerType" : "PUSH"
}, {
"hash" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6320",
"triggerID" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"triggerType" : "PUSH"
}, {
"hash" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6357",
"triggerID" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d3b3e052e3565fe44174dfae73938bfaa44faee4 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6357)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1031332782
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6b6a60fd0f04f236289e500b20951158c2358f51 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746)
* 21df6fe5ac130bfca73a7e3f528d8ef1df66c074 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stayrascal commented on a change in pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
stayrascal commented on a change in pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#discussion_r801702428
##########
File path: hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/table/action/commit/FlinkWriteHelper.java
##########
@@ -105,7 +105,7 @@ public static FlinkWriteHelper newInstance() {
// we cannot allow the user to change the key or partitionPath, since that will affect
// everything
// so pick it from one of the records.
- boolean choosePrev = data1.equals(reducedData);
+ boolean choosePrev = data2.compareTo(data1) < 0;
HoodieKey reducedKey = choosePrev ? rec1.getKey() : rec2.getKey();
HoodieOperation operation = choosePrev ? rec1.getOperation() : rec2.getOperation();
Review comment:
The previous logic of `data2.preCombine(data1)` is that return one of data1 or data2 ordering by their `orderVal`. But if we merge/combine data1 and data2 into a new payload(reduceData), the `data1.equals(reduceData)` is always false. In order to get the `HoodieKey` and `HoodieOperation` for new HoodieRecord with `reduceData`, we need to get the latest `HoodieKey` and `HoodieOperation` from `data1` and `data2`， `compareTo` is used for replace `@preCombine` to compare their `orderingVal`.
```
@Override
public int compareTo(OverwriteWithLatestAvroPayload oldValue) {
return orderingVal.compareTo(oldValue.orderingVal);
}
```
```
@Test
public void testCompareFunction() {
GenericRecord record = new GenericData.Record(schema);
record.put("id", "1");
record.put("partition", "partition1");
record.put("ts", 0L);
record.put("_hoodie_is_deleted", false);
record.put("city", "NY0");
record.put("child", Arrays.asList("A"));
PartialOverwriteWithLatestAvroPayload payload1 = new PartialOverwriteWithLatestAvroPayload(record, 1);
PartialOverwriteWithLatestAvroPayload payload2 = new PartialOverwriteWithLatestAvroPayload(record, 2);
assertEquals(payload1.compareTo(payload2), -1);
assertEquals(payload2.compareTo(payload1), 1);
assertEquals(payload1.compareTo(payload1), 0);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stayrascal commented on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
stayrascal commented on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1032725182
> Thanks for the contribution ~ I kind of think we better do this after id-based schema evolution is supported, only after that, we have more light-wright solution to support per-record schema.
>
> Generally, take schema with record seems not a good solution.
Yeah, agree with taking schema with record is not a good solution, it couple the byte data with schema and hard to evolute schema later, but it seems that we cannot merge two `Payload` without schema. I also tired if we can pass the `schema` to `#preCombine` method, just similar with `#combineAndGetUpdateValue`. But we only can get the schema info from flink configuration, the `FlinkWriteHelper` doesn't(and should not) support flink configuration to support retrieve schema from configuration, so I have to inject schema via constructor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1030866332
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6b6a60fd0f04f236289e500b20951158c2358f51 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stayrascal commented on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
stayrascal commented on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1031365957
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1025675312
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d73d7f503615583bd233b1ba974e7c78125c41bb Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1069126921
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
}, {
"hash" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178",
"triggerID" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"triggerType" : "PUSH"
}, {
"hash" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6320",
"triggerID" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"triggerType" : "PUSH"
}, {
"hash" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6357",
"triggerID" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"triggerType" : "PUSH"
}, {
"hash" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6706",
"triggerID" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"triggerType" : "PUSH"
}, {
"hash" : "d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7000",
"triggerID" : "d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a86b7ff2d204900374751371cd2c8c36e76ea4b8 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6706)
* d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7000)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1050961037
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
}, {
"hash" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178",
"triggerID" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"triggerType" : "PUSH"
}, {
"hash" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178)
* c6f524ea6bf6800a797f9173c4034ce8d111a7d5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1047118523
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
}, {
"hash" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178",
"triggerID" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stayrascal commented on a change in pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
stayrascal commented on a change in pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#discussion_r811194029
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/PartialOverwriteWithLatestAvroPayload.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.hudi.common.util.Option;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Objects;
+
+import static org.apache.hudi.avro.HoodieAvroUtils.bytesToAvro;
+
+/**
+ * The only difference with {@link OverwriteNonDefaultsWithLatestAvroPayload} is that it supports
+ * merging the latest non-null partial fields with the old record instead of replacing the whole record.
+ * And merging the non-null fields during preCombine multiple records with same record key instead of choosing the latest record based on ordering field.
+ *
+ * <p> Regarding #combineAndGetUpdateValue, Assuming a {@link GenericRecord} has row schema: (f0 int , f1 int, f2 int).
+ * The first record value is: (1, 2, 3), the second record value is: (4, 5, null) with the field f2 value as null.
+ * Calling the #combineAndGetUpdateValue method of the two records returns record: (4, 5, 3).
+ * Note that field f2 value is ignored because it is null. </p>
+ *
+ * <p> Regarding #preCombine, Assuming a {@link GenericRecord} has row schema: (f0 int , f1 int, f2 int, o1 int),
+ * and initial two {@link PartialOverwriteWithLatestAvroPayload} with different ordering value.
+ * The first record value is (1, null, 1, 1) with the filed f1 value as null, the second value is: (2, 2, null, 2) with the f2 value as null.
+ * Calling the #preCombine method of the two records returns record: (2, 2, 1, 2).
+ * Note:
+ * <ol>
+ * <li>the field f0 value is 2 because the ordering value of second record is bigger.</li>
+ * <li>the filed f1 value is 2 because the f2 value of first record is null.</li>
+ * <li>the filed f2 value is 1 because the f2 value of second record is null.</li>
+ * <li>the filed o1 value is 2 because the ordering value of second record is bigger.</li>
+ * </ol>
+ *
+ * </p>
+ */
+public class PartialOverwriteWithLatestAvroPayload extends OverwriteWithLatestAvroPayload {
+
+ public PartialOverwriteWithLatestAvroPayload(GenericRecord record, Comparable orderingVal) {
+ this(record, orderingVal, null);
+ }
+
+ public PartialOverwriteWithLatestAvroPayload(GenericRecord record, Comparable orderingVal, String schema) {
+ super(record, orderingVal, schema);
+ }
+
+ public PartialOverwriteWithLatestAvroPayload(Option<GenericRecord> record) {
+ super(record); // natural order
+ }
+
+ @Override
+ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord currentValue, Schema schema) throws IOException {
+ if (recordBytes.length == 0) {
+ return Option.empty();
+ }
+
+ GenericRecord incomingRecord = bytesToAvro(recordBytes, schema);
+ if (isDeleteRecord(incomingRecord)) {
+ return Option.empty();
+ }
+
+ GenericRecord currentRecord = (GenericRecord) currentValue;
+ List<Schema.Field> fields = schema.getFields();
+ fields.forEach(field -> {
+ Object value = incomingRecord.get(field.name());
+ if (Objects.nonNull(value)) {
+ currentRecord.put(field.name(), value);
+ }
+ });
+
+ return Option.of(currentRecord);
+ }
+
+ @Override
+ public int compareTo(OverwriteWithLatestAvroPayload oldValue) {
+ return orderingVal.compareTo(oldValue.orderingVal);
+ }
+
+ @Override
+ public OverwriteWithLatestAvroPayload preCombine(OverwriteWithLatestAvroPayload oldValue) {
+ if (null == this.schema || null == oldValue.schema) {
+ return super.preCombine(oldValue);
+ }
+
+ try {
+ Schema schema = new Schema.Parser().parse(this.schema);
+ Option<IndexedRecord> incomingOption = getInsertValue(new Schema.Parser().parse(this.schema));
+ Option<IndexedRecord> insertRecordOption = oldValue.getInsertValue(new Schema.Parser().parse(oldValue.schema));
Review comment:
solved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stayrascal commented on a change in pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
stayrascal commented on a change in pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#discussion_r811193683
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/PartialOverwriteWithLatestAvroPayload.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.hudi.common.util.Option;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Objects;
+
+import static org.apache.hudi.avro.HoodieAvroUtils.bytesToAvro;
+
+/**
+ * The only difference with {@link OverwriteNonDefaultsWithLatestAvroPayload} is that it supports
+ * merging the latest non-null partial fields with the old record instead of replacing the whole record.
+ * And merging the non-null fields during preCombine multiple records with same record key instead of choosing the latest record based on ordering field.
+ *
+ * <p> Regarding #combineAndGetUpdateValue, Assuming a {@link GenericRecord} has row schema: (f0 int , f1 int, f2 int).
+ * The first record value is: (1, 2, 3), the second record value is: (4, 5, null) with the field f2 value as null.
+ * Calling the #combineAndGetUpdateValue method of the two records returns record: (4, 5, 3).
+ * Note that field f2 value is ignored because it is null. </p>
+ *
+ * <p> Regarding #preCombine, Assuming a {@link GenericRecord} has row schema: (f0 int , f1 int, f2 int, o1 int),
+ * and initial two {@link PartialOverwriteWithLatestAvroPayload} with different ordering value.
+ * The first record value is (1, null, 1, 1) with the filed f1 value as null, the second value is: (2, 2, null, 2) with the f2 value as null.
+ * Calling the #preCombine method of the two records returns record: (2, 2, 1, 2).
+ * Note:
+ * <ol>
+ * <li>the field f0 value is 2 because the ordering value of second record is bigger.</li>
+ * <li>the filed f1 value is 2 because the f2 value of first record is null.</li>
+ * <li>the filed f2 value is 1 because the f2 value of second record is null.</li>
+ * <li>the filed o1 value is 2 because the ordering value of second record is bigger.</li>
+ * </ol>
+ *
+ * </p>
+ */
+public class PartialOverwriteWithLatestAvroPayload extends OverwriteWithLatestAvroPayload {
+
+ public PartialOverwriteWithLatestAvroPayload(GenericRecord record, Comparable orderingVal) {
+ this(record, orderingVal, null);
+ }
+
+ public PartialOverwriteWithLatestAvroPayload(GenericRecord record, Comparable orderingVal, String schema) {
+ super(record, orderingVal, schema);
+ }
+
+ public PartialOverwriteWithLatestAvroPayload(Option<GenericRecord> record) {
+ super(record); // natural order
+ }
+
+ @Override
+ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord currentValue, Schema schema) throws IOException {
+ if (recordBytes.length == 0) {
+ return Option.empty();
+ }
+
+ GenericRecord incomingRecord = bytesToAvro(recordBytes, schema);
+ if (isDeleteRecord(incomingRecord)) {
+ return Option.empty();
+ }
+
+ GenericRecord currentRecord = (GenericRecord) currentValue;
+ List<Schema.Field> fields = schema.getFields();
Review comment:
fixed the schema name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1031424226
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 21df6fe5ac130bfca73a7e3f528d8ef1df66c074 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stayrascal commented on a change in pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
stayrascal commented on a change in pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#discussion_r811193217
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/OverwriteWithLatestAvroPayload.java
##########
@@ -40,8 +40,18 @@
public class OverwriteWithLatestAvroPayload extends BaseAvroPayload
implements HoodieRecordPayload<OverwriteWithLatestAvroPayload> {
+ /**
+ * the schema of generic record
+ */
+ public final String schema;
Review comment:
removed this field.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1047010710
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
}, {
"hash" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 14edef016d98786b713e155e9adec1d87a3500ba Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175)
* ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1069126921
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
}, {
"hash" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178",
"triggerID" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"triggerType" : "PUSH"
}, {
"hash" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6320",
"triggerID" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"triggerType" : "PUSH"
}, {
"hash" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6357",
"triggerID" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"triggerType" : "PUSH"
}, {
"hash" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6706",
"triggerID" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"triggerType" : "PUSH"
}, {
"hash" : "d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7000",
"triggerID" : "d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a86b7ff2d204900374751371cd2c8c36e76ea4b8 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6706)
* d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7000)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1069213125
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
}, {
"hash" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178",
"triggerID" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"triggerType" : "PUSH"
}, {
"hash" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6320",
"triggerID" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"triggerType" : "PUSH"
}, {
"hash" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6357",
"triggerID" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"triggerType" : "PUSH"
}, {
"hash" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6706",
"triggerID" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"triggerType" : "PUSH"
}, {
"hash" : "d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7000",
"triggerID" : "d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7000)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1047017097
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
}, {
"hash" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178",
"triggerID" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 14edef016d98786b713e155e9adec1d87a3500ba Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175)
* ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4724:
URL: https://github.com/apache/hudi/pull/4724#issuecomment-1078895292
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5620",
"triggerID" : "d73d7f503615583bd233b1ba974e7c78125c41bb",
"triggerType" : "PUSH"
}, {
"hash" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5746",
"triggerID" : "6b6a60fd0f04f236289e500b20951158c2358f51",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"triggerType" : "PUSH"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5757",
"triggerID" : "1031342598",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5759",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "21df6fe5ac130bfca73a7e3f528d8ef1df66c074",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5758",
"triggerID" : "1031365957",
"triggerType" : "MANUAL"
}, {
"hash" : "14edef016d98786b713e155e9adec1d87a3500ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6175",
"triggerID" : "14edef016d98786b713e155e9adec1d87a3500ba",
"triggerType" : "PUSH"
}, {
"hash" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6178",
"triggerID" : "ce561bc7ed79af0b0cc6cc627ed3747a1d2fd75e",
"triggerType" : "PUSH"
}, {
"hash" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6320",
"triggerID" : "c6f524ea6bf6800a797f9173c4034ce8d111a7d5",
"triggerType" : "PUSH"
}, {
"hash" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6357",
"triggerID" : "d3b3e052e3565fe44174dfae73938bfaa44faee4",
"triggerType" : "PUSH"
}, {
"hash" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6706",
"triggerID" : "a86b7ff2d204900374751371cd2c8c36e76ea4b8",
"triggerType" : "PUSH"
}, {
"hash" : "d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7000",
"triggerID" : "d11c670e816a0a38f10e20c8cdbd8bb1cfc1afe5",
"triggerType" : "PUSH"
}, {
"hash" : "e89fd60b77fbd366205df5e60777ced881b6d795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7203",
"triggerID" : "e89fd60b77fbd366205df5e60777ced881b6d795",
"triggerType" : "PUSH"
}, {
"hash" : "e89fd60b77fbd366205df5e60777ced881b6d795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7220",
"triggerID" : "1075938530",
"triggerType" : "MANUAL"
}, {
"hash" : "3f771d39a403669bf7d0a8d50cd3e5b711b03ebe",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7340",
"triggerID" : "3f771d39a403669bf7d0a8d50cd3e5b711b03ebe",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3f771d39a403669bf7d0a8d50cd3e5b711b03ebe Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7340)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org