You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "linliu-code (via GitHub)" <gi...@apache.org> on 2023/12/08 17:48:18 UTC
[PR] [HUDI-7200] Fix bugs of Avro record merger for event time merging [hudi]
linliu-code opened a new pull request, #10286:
URL: https://github.com/apache/hudi/pull/10286
### Change Logs
Support operation patterns like insert/delete/update.
### Impact
Fix the broken logic.
### Risk level (write none, low medium or high below)
Low.
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7200] Fix bugs of Avro record merger for event time merging [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10286:
URL: https://github.com/apache/hudi/pull/10286#issuecomment-1847859635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "980dbb9b0027593ebb49d32872e656b1b42a0769",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21409",
"triggerID" : "980dbb9b0027593ebb49d32872e656b1b42a0769",
"triggerType" : "PUSH"
}, {
"hash" : "90b5a084cf7e1a65d143f605653f5eb661fe9fd3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "90b5a084cf7e1a65d143f605653f5eb661fe9fd3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 980dbb9b0027593ebb49d32872e656b1b42a0769 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21409)
* 90b5a084cf7e1a65d143f605653f5eb661fe9fd3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7200] Fix bugs of Avro record merger for event time merging [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on PR #10286:
URL: https://github.com/apache/hudi/pull/10286#issuecomment-1852793324
After talked with @yihua , we decided to use HoodieAvroRecord with EmptyHoodieRecordPayload to indicate a delete record; so this PR can be dropped for now. We may need to revisit this problem when we decide to depreciate payload hierarchy in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7200] Fix bugs of Avro record merger for event time merging [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10286:
URL: https://github.com/apache/hudi/pull/10286#issuecomment-1847706763
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "980dbb9b0027593ebb49d32872e656b1b42a0769",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21409",
"triggerID" : "980dbb9b0027593ebb49d32872e656b1b42a0769",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 980dbb9b0027593ebb49d32872e656b1b42a0769 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21409)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7200] Fix bugs of Avro record merger for event time merging [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #10286:
URL: https://github.com/apache/hudi/pull/10286#discussion_r1424512086
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieAvroRecordMerger.java:
##########
@@ -59,6 +59,13 @@ private Option<IndexedRecord> combineAndGetUpdateValue(HoodieRecord older, Hoodi
return Option.empty();
}
+ if (newer.isDelete(schema, props)) {
+ if (newer.getOrderingValue(schema, props).compareTo(older.getOrderingValue(schema, props)) >= 0) {
+ return Option.empty();
Review Comment:
Meanwhile, this bug blocks some tests I am working on. So I will try to fix this for general Avro payload first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7200] Fix bugs of Avro record merger for event time merging [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #10286:
URL: https://github.com/apache/hudi/pull/10286#discussion_r1424510797
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieAvroRecordMerger.java:
##########
@@ -59,6 +59,13 @@ private Option<IndexedRecord> combineAndGetUpdateValue(HoodieRecord older, Hoodi
return Option.empty();
}
+ if (newer.isDelete(schema, props)) {
+ if (newer.getOrderingValue(schema, props).compareTo(older.getOrderingValue(schema, props)) >= 0) {
+ return Option.empty();
Review Comment:
My rough idea is that if we are going to remove all kinds of payload classes, then we should start to move these logics bit by bit. I will document this idea somewhere.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7200] Fix bugs of Avro record merger for event time merging [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #10286:
URL: https://github.com/apache/hudi/pull/10286#discussion_r1424512510
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieAvroRecordMerger.java:
##########
@@ -59,6 +59,13 @@ private Option<IndexedRecord> combineAndGetUpdateValue(HoodieRecord older, Hoodi
return Option.empty();
}
+ if (newer.isDelete(schema, props)) {
+ if (newer.getOrderingValue(schema, props).compareTo(older.getOrderingValue(schema, props)) >= 0) {
+ return Option.empty();
Review Comment:
@yihua, @codope, do you guys have any comments?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7200] Fix bugs of Avro record merger for event time merging [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10286:
URL: https://github.com/apache/hudi/pull/10286#issuecomment-1847866801
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "980dbb9b0027593ebb49d32872e656b1b42a0769",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21409",
"triggerID" : "980dbb9b0027593ebb49d32872e656b1b42a0769",
"triggerType" : "PUSH"
}, {
"hash" : "90b5a084cf7e1a65d143f605653f5eb661fe9fd3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21417",
"triggerID" : "90b5a084cf7e1a65d143f605653f5eb661fe9fd3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 980dbb9b0027593ebb49d32872e656b1b42a0769 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21409)
* 90b5a084cf7e1a65d143f605653f5eb661fe9fd3 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21417)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7200] Fix bugs of Avro record merger for event time merging [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10286:
URL: https://github.com/apache/hudi/pull/10286#issuecomment-1847996546
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "980dbb9b0027593ebb49d32872e656b1b42a0769",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21409",
"triggerID" : "980dbb9b0027593ebb49d32872e656b1b42a0769",
"triggerType" : "PUSH"
}, {
"hash" : "90b5a084cf7e1a65d143f605653f5eb661fe9fd3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21417",
"triggerID" : "90b5a084cf7e1a65d143f605653f5eb661fe9fd3",
"triggerType" : "PUSH"
}, {
"hash" : "2d0867b6095e0ff05d23823fa99e42c331785e45",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21419",
"triggerID" : "2d0867b6095e0ff05d23823fa99e42c331785e45",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 90b5a084cf7e1a65d143f605653f5eb661fe9fd3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21417)
* 2d0867b6095e0ff05d23823fa99e42c331785e45 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21419)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7200] Fix bugs of Avro record merger for event time merging [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10286:
URL: https://github.com/apache/hudi/pull/10286#issuecomment-1848068494
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "980dbb9b0027593ebb49d32872e656b1b42a0769",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21409",
"triggerID" : "980dbb9b0027593ebb49d32872e656b1b42a0769",
"triggerType" : "PUSH"
}, {
"hash" : "90b5a084cf7e1a65d143f605653f5eb661fe9fd3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21417",
"triggerID" : "90b5a084cf7e1a65d143f605653f5eb661fe9fd3",
"triggerType" : "PUSH"
}, {
"hash" : "2d0867b6095e0ff05d23823fa99e42c331785e45",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21419",
"triggerID" : "2d0867b6095e0ff05d23823fa99e42c331785e45",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2d0867b6095e0ff05d23823fa99e42c331785e45 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21419)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7200] Fix bugs of Avro record merger for event time merging [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10286:
URL: https://github.com/apache/hudi/pull/10286#issuecomment-1847645628
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "980dbb9b0027593ebb49d32872e656b1b42a0769",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21409",
"triggerID" : "980dbb9b0027593ebb49d32872e656b1b42a0769",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 980dbb9b0027593ebb49d32872e656b1b42a0769 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21409)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7200] Fix bugs of Avro record merger for event time merging [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code closed pull request #10286: [HUDI-7200] Fix bugs of Avro record merger for event time merging
URL: https://github.com/apache/hudi/pull/10286
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7200] Fix bugs of Avro record merger for event time merging [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10286:
URL: https://github.com/apache/hudi/pull/10286#issuecomment-1847635859
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "980dbb9b0027593ebb49d32872e656b1b42a0769",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "980dbb9b0027593ebb49d32872e656b1b42a0769",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 980dbb9b0027593ebb49d32872e656b1b42a0769 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7200] Fix bugs of Avro record merger for event time merging [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on PR #10286:
URL: https://github.com/apache/hudi/pull/10286#issuecomment-1847835537
> @linliu-code : I am not sure if there is an issue here. Can you kindly check the caller to see if it is handled there ? Also, Can you add a test-case to reproduce the setup ?
Hi @bvaradar, I am building a test framework for file group reader level tests; as an example, I am adding the event time merging tests, and this issue is found by adding the tests. You can check the test in the PR draft: https://github.com/apache/hudi/pull/10277/files#diff-0fe37af8d2444e330aca2694c0babf38ab7fc9a7de603acc450d6201b2e2c495
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7200] Fix bugs of Avro record merger for event time merging [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10286:
URL: https://github.com/apache/hudi/pull/10286#issuecomment-1847926627
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "980dbb9b0027593ebb49d32872e656b1b42a0769",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21409",
"triggerID" : "980dbb9b0027593ebb49d32872e656b1b42a0769",
"triggerType" : "PUSH"
}, {
"hash" : "90b5a084cf7e1a65d143f605653f5eb661fe9fd3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21417",
"triggerID" : "90b5a084cf7e1a65d143f605653f5eb661fe9fd3",
"triggerType" : "PUSH"
}, {
"hash" : "2d0867b6095e0ff05d23823fa99e42c331785e45",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21419",
"triggerID" : "2d0867b6095e0ff05d23823fa99e42c331785e45",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 980dbb9b0027593ebb49d32872e656b1b42a0769 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21409)
* 90b5a084cf7e1a65d143f605653f5eb661fe9fd3 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21417)
* 2d0867b6095e0ff05d23823fa99e42c331785e45 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21419)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7200] Fix bugs of Avro record merger for event time merging [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10286:
URL: https://github.com/apache/hudi/pull/10286#issuecomment-1847920668
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "980dbb9b0027593ebb49d32872e656b1b42a0769",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21409",
"triggerID" : "980dbb9b0027593ebb49d32872e656b1b42a0769",
"triggerType" : "PUSH"
}, {
"hash" : "90b5a084cf7e1a65d143f605653f5eb661fe9fd3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21417",
"triggerID" : "90b5a084cf7e1a65d143f605653f5eb661fe9fd3",
"triggerType" : "PUSH"
}, {
"hash" : "2d0867b6095e0ff05d23823fa99e42c331785e45",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2d0867b6095e0ff05d23823fa99e42c331785e45",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 980dbb9b0027593ebb49d32872e656b1b42a0769 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21409)
* 90b5a084cf7e1a65d143f605653f5eb661fe9fd3 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21417)
* 2d0867b6095e0ff05d23823fa99e42c331785e45 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7200] Fix bugs of Avro record merger for event time merging [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #10286:
URL: https://github.com/apache/hudi/pull/10286#discussion_r1421220175
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieAvroRecordMerger.java:
##########
@@ -59,6 +59,13 @@ private Option<IndexedRecord> combineAndGetUpdateValue(HoodieRecord older, Hoodi
return Option.empty();
}
+ if (newer.isDelete(schema, props)) {
+ if (newer.getOrderingValue(schema, props).compareTo(older.getOrderingValue(schema, props)) >= 0) {
+ return Option.empty();
Review Comment:
In some payload class like `EventTimeAvroPayload`, we already handle the deletion logic, but it lost the even time semantics somehow:
```java
if (recordBytes.length == 0 || isDeletedRecord) {
return Option.empty();
}
```
Should we remove the deletion logic in all kinds of payload classes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7200] Fix bugs of Avro record merger for event time merging [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #10286:
URL: https://github.com/apache/hudi/pull/10286#discussion_r1421220175
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieAvroRecordMerger.java:
##########
@@ -59,6 +59,13 @@ private Option<IndexedRecord> combineAndGetUpdateValue(HoodieRecord older, Hoodi
return Option.empty();
}
+ if (newer.isDelete(schema, props)) {
+ if (newer.getOrderingValue(schema, props).compareTo(older.getOrderingValue(schema, props)) >= 0) {
+ return Option.empty();
Review Comment:
In some payload class like `EventTimeAvroPayload`, we already handle the deletion logic, but it lost the even time semantics somehow:
```java
if (recordBytes.length == 0 || isDeletedRecord) {
return Option.empty();
}
```
We better still maintain the logic in the payload, not the merger.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org