You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "bschell (via GitHub)" <gi...@apache.org> on 2023/01/26 22:43:26 UTC
[GitHub] [hudi] bschell opened a new pull request, #7759: [HUDI-5624] Fix HoodieAvroRecordMerger to use new precombine API
bschell opened a new pull request, #7759:
URL: https://github.com/apache/hudi/pull/7759
Updates the HoodieAvroRecordMerger to use the new precombine API instead of the deprecated one. This fixes issues with backwards compatibility with certain payloads.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #7759: [HUDI-5624] Fix HoodieAvroRecordMerger to use new precombine API
Posted by "alexeykudinkin (via GitHub)" <gi...@apache.org>.
alexeykudinkin commented on code in PR #7759:
URL: https://github.com/apache/hudi/pull/7759#discussion_r1089478769
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestPartialUpdateAvroPayloadE2E.scala:
##########
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.functional
+
+import java.util.function.Consumer
+
+import org.apache.hadoop.fs.FileSystem
+import org.apache.hudi.HoodieConversionUtils.toJavaOption
+import org.apache.hudi.{DataSourceWriteOptions, QuickstartUtils}
+import org.apache.hudi.QuickstartUtils.{convertToStringList, getQuickstartWriteConfigs}
+import org.apache.hudi.common.model.HoodieTableType
+import org.apache.hudi.common.util
+import org.apache.hudi.config.HoodieWriteConfig
+import org.apache.hudi.testutils.HoodieClientTestBase
+import org.apache.hudi.util.JFunction
+import org.apache.spark.sql._
+import org.apache.spark.sql.functions.{lit, typedLit}
+import org.apache.spark.sql.hudi.HoodieSparkSessionExtension
+import org.apache.spark.sql.types.{DoubleType, StringType}
+import org.junit.jupiter.api.Assertions.assertEquals
+import org.junit.jupiter.api.{AfterEach, BeforeEach}
+import org.junit.jupiter.params.ParameterizedTest
+import org.junit.jupiter.params.provider.EnumSource
+
+import scala.collection.JavaConversions._
+
+class TestPartialUpdateAvroPayloadE2E extends HoodieClientTestBase {
Review Comment:
Thanks for adding the test @bschell!
We can drop the E2E suffix for it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7759: [HUDI-5624] Fix HoodieAvroRecordMerger to use new precombine API
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7759:
URL: https://github.com/apache/hudi/pull/7759#issuecomment-1407116064
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "92e6187265bbd13d9eed14b4f797a216fb16aebd",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14660",
"triggerID" : "92e6187265bbd13d9eed14b4f797a216fb16aebd",
"triggerType" : "PUSH"
}, {
"hash" : "3027c666e5b04bfc25be6bb85b011bfeae792688",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3027c666e5b04bfc25be6bb85b011bfeae792688",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 92e6187265bbd13d9eed14b4f797a216fb16aebd Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14660)
* 3027c666e5b04bfc25be6bb85b011bfeae792688 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on pull request #7759: [HUDI-5624] Fix HoodieAvroRecordMerger to use new precombine API
Posted by "alexeykudinkin (via GitHub)" <gi...@apache.org>.
alexeykudinkin commented on PR #7759:
URL: https://github.com/apache/hudi/pull/7759#issuecomment-1407479690
CI is green:
<img width="1632" alt="Screenshot 2023-01-28 at 12 28 50 PM" src="https://user-images.githubusercontent.com/428277/215289413-e55fd6ba-68f1-44d1-84cb-84a65cbe347e.png">
https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=14698&view=results
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7759: [HUDI-5624] Fix HoodieAvroRecordMerger to use new precombine API
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7759:
URL: https://github.com/apache/hudi/pull/7759#issuecomment-1405813634
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "92e6187265bbd13d9eed14b4f797a216fb16aebd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92e6187265bbd13d9eed14b4f797a216fb16aebd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 92e6187265bbd13d9eed14b4f797a216fb16aebd UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #7759: [HUDI-5624] Fix HoodieAvroRecordMerger to use new precombine API
Posted by "alexeykudinkin (via GitHub)" <gi...@apache.org>.
alexeykudinkin commented on code in PR #7759:
URL: https://github.com/apache/hudi/pull/7759#discussion_r1089555248
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieAvroRecordMerger.java:
##########
@@ -71,10 +71,10 @@ public HoodieRecordType getRecordType() {
return HoodieRecordType.AVRO;
}
- private HoodieRecord preCombine(HoodieRecord older, HoodieRecord newer) {
- HoodieRecordPayload picked = unsafeCast(((HoodieAvroRecord) newer).getData().preCombine(((HoodieAvroRecord) older).getData()));
- if (picked instanceof HoodieMetadataPayload) {
- // NOTE: HoodieMetadataPayload return a new payload
+ private HoodieRecord preCombine(HoodieRecord older, HoodieRecord newer, Schema schema, Properties props) {
+ HoodieRecordPayload picked = unsafeCast(((HoodieAvroRecord) newer).getData().preCombine(((HoodieAvroRecord) older).getData(), schema, props));
+ if (picked instanceof HoodieMetadataPayload || picked instanceof OverwriteNonDefaultsWithLatestAvroPayload) {
+ // NOTE: When payload class returns a new payload instead of choosing from older/newer payload, create a new record.
return new HoodieAvroRecord(newer.getKey(), picked, newer.getOperation());
}
return picked.equals(((HoodieAvroRecord) newer).getData()) ? newer : older;
Review Comment:
I think we can nuke this optimization avoiding creation of the new record and just always create it in that case (to make sure this doesn't affect all random payloads out there)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7759: [HUDI-5624] Fix HoodieAvroRecordMerger to use new precombine API
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7759:
URL: https://github.com/apache/hudi/pull/7759#issuecomment-1405939710
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "92e6187265bbd13d9eed14b4f797a216fb16aebd",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14660",
"triggerID" : "92e6187265bbd13d9eed14b4f797a216fb16aebd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 92e6187265bbd13d9eed14b4f797a216fb16aebd Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14660)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7759: [HUDI-5624] Fix HoodieAvroRecordMerger to use new precombine API
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7759:
URL: https://github.com/apache/hudi/pull/7759#issuecomment-1405818327
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "92e6187265bbd13d9eed14b4f797a216fb16aebd",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14660",
"triggerID" : "92e6187265bbd13d9eed14b4f797a216fb16aebd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 92e6187265bbd13d9eed14b4f797a216fb16aebd Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14660)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7759: [HUDI-5624] Fix HoodieAvroRecordMerger to use new precombine API
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7759:
URL: https://github.com/apache/hudi/pull/7759#issuecomment-1407236841
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "92e6187265bbd13d9eed14b4f797a216fb16aebd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14660",
"triggerID" : "92e6187265bbd13d9eed14b4f797a216fb16aebd",
"triggerType" : "PUSH"
}, {
"hash" : "3027c666e5b04bfc25be6bb85b011bfeae792688",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14674",
"triggerID" : "3027c666e5b04bfc25be6bb85b011bfeae792688",
"triggerType" : "PUSH"
}, {
"hash" : "8bb795346fd54da170b3282a3a647bbe64c818cb",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14679",
"triggerID" : "8bb795346fd54da170b3282a3a647bbe64c818cb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3027c666e5b04bfc25be6bb85b011bfeae792688 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14674)
* 8bb795346fd54da170b3282a3a647bbe64c818cb Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14679)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ys8 commented on pull request #7759: [HUDI-5624] Fix HoodieAvroRecordMerger to use new precombine API
Posted by "ys8 (via GitHub)" <gi...@apache.org>.
ys8 commented on PR #7759:
URL: https://github.com/apache/hudi/pull/7759#issuecomment-1625709567
I believe it introduced a regression. The new record is always picking `newer.getOperation()` even if the older payload was picked. There is no guarantee that the operation is same between older and newer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin merged pull request #7759: [HUDI-5624] Fix HoodieAvroRecordMerger to use new precombine API
Posted by "alexeykudinkin (via GitHub)" <gi...@apache.org>.
alexeykudinkin merged PR #7759:
URL: https://github.com/apache/hudi/pull/7759
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7759: [HUDI-5624] Fix HoodieAvroRecordMerger to use new precombine API
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7759:
URL: https://github.com/apache/hudi/pull/7759#issuecomment-1407277156
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "92e6187265bbd13d9eed14b4f797a216fb16aebd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14660",
"triggerID" : "92e6187265bbd13d9eed14b4f797a216fb16aebd",
"triggerType" : "PUSH"
}, {
"hash" : "3027c666e5b04bfc25be6bb85b011bfeae792688",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14674",
"triggerID" : "3027c666e5b04bfc25be6bb85b011bfeae792688",
"triggerType" : "PUSH"
}, {
"hash" : "8bb795346fd54da170b3282a3a647bbe64c818cb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14679",
"triggerID" : "8bb795346fd54da170b3282a3a647bbe64c818cb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8bb795346fd54da170b3282a3a647bbe64c818cb Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14679)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #7759: [HUDI-5624] Fix HoodieAvroRecordMerger to use new precombine API
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #7759:
URL: https://github.com/apache/hudi/pull/7759#issuecomment-1626486629
Yes, it is fixed in https://github.com/apache/hudi/pull/8953
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7759: [HUDI-5624] Fix HoodieAvroRecordMerger to use new precombine API
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7759:
URL: https://github.com/apache/hudi/pull/7759#issuecomment-1407361095
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "92e6187265bbd13d9eed14b4f797a216fb16aebd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14660",
"triggerID" : "92e6187265bbd13d9eed14b4f797a216fb16aebd",
"triggerType" : "PUSH"
}, {
"hash" : "3027c666e5b04bfc25be6bb85b011bfeae792688",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14674",
"triggerID" : "3027c666e5b04bfc25be6bb85b011bfeae792688",
"triggerType" : "PUSH"
}, {
"hash" : "8bb795346fd54da170b3282a3a647bbe64c818cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14679",
"triggerID" : "8bb795346fd54da170b3282a3a647bbe64c818cb",
"triggerType" : "PUSH"
}, {
"hash" : "9f247d90caf948a2de54be2cf109151453b2a57d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14698",
"triggerID" : "9f247d90caf948a2de54be2cf109151453b2a57d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9f247d90caf948a2de54be2cf109151453b2a57d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14698)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7759: [HUDI-5624] Fix HoodieAvroRecordMerger to use new precombine API
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7759:
URL: https://github.com/apache/hudi/pull/7759#issuecomment-1407230815
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "92e6187265bbd13d9eed14b4f797a216fb16aebd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14660",
"triggerID" : "92e6187265bbd13d9eed14b4f797a216fb16aebd",
"triggerType" : "PUSH"
}, {
"hash" : "3027c666e5b04bfc25be6bb85b011bfeae792688",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14674",
"triggerID" : "3027c666e5b04bfc25be6bb85b011bfeae792688",
"triggerType" : "PUSH"
}, {
"hash" : "8bb795346fd54da170b3282a3a647bbe64c818cb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8bb795346fd54da170b3282a3a647bbe64c818cb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3027c666e5b04bfc25be6bb85b011bfeae792688 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14674)
* 8bb795346fd54da170b3282a3a647bbe64c818cb UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7759: [HUDI-5624] Fix HoodieAvroRecordMerger to use new precombine API
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7759:
URL: https://github.com/apache/hudi/pull/7759#issuecomment-1407227714
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "92e6187265bbd13d9eed14b4f797a216fb16aebd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14660",
"triggerID" : "92e6187265bbd13d9eed14b4f797a216fb16aebd",
"triggerType" : "PUSH"
}, {
"hash" : "3027c666e5b04bfc25be6bb85b011bfeae792688",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14674",
"triggerID" : "3027c666e5b04bfc25be6bb85b011bfeae792688",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3027c666e5b04bfc25be6bb85b011bfeae792688 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14674)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7759: [HUDI-5624] Fix HoodieAvroRecordMerger to use new precombine API
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7759:
URL: https://github.com/apache/hudi/pull/7759#issuecomment-1407312504
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "92e6187265bbd13d9eed14b4f797a216fb16aebd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14660",
"triggerID" : "92e6187265bbd13d9eed14b4f797a216fb16aebd",
"triggerType" : "PUSH"
}, {
"hash" : "3027c666e5b04bfc25be6bb85b011bfeae792688",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14674",
"triggerID" : "3027c666e5b04bfc25be6bb85b011bfeae792688",
"triggerType" : "PUSH"
}, {
"hash" : "8bb795346fd54da170b3282a3a647bbe64c818cb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14679",
"triggerID" : "8bb795346fd54da170b3282a3a647bbe64c818cb",
"triggerType" : "PUSH"
}, {
"hash" : "9f247d90caf948a2de54be2cf109151453b2a57d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9f247d90caf948a2de54be2cf109151453b2a57d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8bb795346fd54da170b3282a3a647bbe64c818cb Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14679)
* 9f247d90caf948a2de54be2cf109151453b2a57d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on pull request #7759: [HUDI-5624] Fix HoodieAvroRecordMerger to use new precombine API
Posted by "alexeykudinkin (via GitHub)" <gi...@apache.org>.
alexeykudinkin commented on PR #7759:
URL: https://github.com/apache/hudi/pull/7759#issuecomment-1405841612
@bschell did this fail any test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7759: [HUDI-5624] Fix HoodieAvroRecordMerger to use new precombine API
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7759:
URL: https://github.com/apache/hudi/pull/7759#issuecomment-1407313576
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "92e6187265bbd13d9eed14b4f797a216fb16aebd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14660",
"triggerID" : "92e6187265bbd13d9eed14b4f797a216fb16aebd",
"triggerType" : "PUSH"
}, {
"hash" : "3027c666e5b04bfc25be6bb85b011bfeae792688",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14674",
"triggerID" : "3027c666e5b04bfc25be6bb85b011bfeae792688",
"triggerType" : "PUSH"
}, {
"hash" : "8bb795346fd54da170b3282a3a647bbe64c818cb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14679",
"triggerID" : "8bb795346fd54da170b3282a3a647bbe64c818cb",
"triggerType" : "PUSH"
}, {
"hash" : "9f247d90caf948a2de54be2cf109151453b2a57d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14698",
"triggerID" : "9f247d90caf948a2de54be2cf109151453b2a57d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8bb795346fd54da170b3282a3a647bbe64c818cb Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14679)
* 9f247d90caf948a2de54be2cf109151453b2a57d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14698)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bschell commented on pull request #7759: [HUDI-5624] Fix HoodieAvroRecordMerger to use new precombine API
Posted by "bschell (via GitHub)" <gi...@apache.org>.
bschell commented on PR #7759:
URL: https://github.com/apache/hudi/pull/7759#issuecomment-1407069668
@alexeykudinkin I added an end-to-end test for PartialUpdateAvroPayload that catches this regression. While doing so I found and fixed another bug in HoodieAvroRecordMerger here: https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/common/model/HoodieAvroRecordMerger.java#L76
Basically HoodieMetadataPayload is not the only payload class that returns an entirely new payload. This breaks any of the "merge" payloads. I was thinking it might be even better to add this check as an API to HoodieRecordPayload so that we do not have to hardcode all the exceptions here but I kept the smaller change for now as we are deprecating this path later anyway.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7759: [HUDI-5624] Fix HoodieAvroRecordMerger to use new precombine API
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7759:
URL: https://github.com/apache/hudi/pull/7759#issuecomment-1407121537
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "92e6187265bbd13d9eed14b4f797a216fb16aebd",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14660",
"triggerID" : "92e6187265bbd13d9eed14b4f797a216fb16aebd",
"triggerType" : "PUSH"
}, {
"hash" : "3027c666e5b04bfc25be6bb85b011bfeae792688",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14674",
"triggerID" : "3027c666e5b04bfc25be6bb85b011bfeae792688",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 92e6187265bbd13d9eed14b4f797a216fb16aebd Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14660)
* 3027c666e5b04bfc25be6bb85b011bfeae792688 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14674)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #7759: [HUDI-5624] Fix HoodieAvroRecordMerger to use new precombine API
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #7759:
URL: https://github.com/apache/hudi/pull/7759#issuecomment-1407307608
rebased w/ laster master. we landed a flaky test fix which was causing failures w/ CI runs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org