You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "linliu-code (via GitHub)" <gi...@apache.org> on 2025/02/14 21:10:36 UTC
[PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
linliu-code opened a new pull request, #12843:
URL: https://github.com/apache/hudi/pull/12843
### Change Logs
Log data block may contain delete records labelled by the custom delete marker.
Currently these records will go through the non-delete record process path.
This PR identifies these records in the fg reader, and process them as normal delete records.
### Impact
Support custom delete records for MOR table.
### Risk level (write none, low medium or high below)
Low.
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none"._
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2660322484
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645)
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* 2eacf85acc06405bb4a11f54050bcfb647e2691f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2660320439
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645)
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #12843:
URL: https://github.com/apache/hudi/pull/12843#discussion_r2004205194
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/PositionBasedFileGroupRecordBuffer.java:
##########
@@ -61,23 +63,23 @@
* Here the position means that record position in the base file. The records from the base file is accessed from an iterator object. These records are merged when the
* {@link #hasNext} method is called.
*/
-public class HoodiePositionBasedFileGroupRecordBuffer<T> extends HoodieKeyBasedFileGroupRecordBuffer<T> {
- private static final Logger LOG = LoggerFactory.getLogger(HoodiePositionBasedFileGroupRecordBuffer.class);
+public class PositionBasedFileGroupRecordBuffer<T> extends KeyBasedFileGroupRecordBuffer<T> {
Review Comment:
ticket: https://issues.apache.org/jira/browse/HUDI-9200
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2739311336
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
}, {
"hash" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974",
"triggerID" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"triggerType" : "PUSH"
}, {
"hash" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3981",
"triggerID" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"triggerType" : "PUSH"
}, {
"hash" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3984",
"triggerID" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"triggerType" : "PUSH"
}, {
"hash" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"triggerType" : "PUSH"
}, {
"hash" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4100",
"triggerID" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"triggerType" : "PUSH"
}, {
"hash" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4181",
"triggerID" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"triggerType" : "PUSH"
}, {
"hash" : "16be88870989aa1e54cf03f15ff61654d590e736",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4274",
"triggerID" : "16be88870989aa1e54cf03f15ff61654d590e736",
"triggerType" : "PUSH"
}, {
"hash" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"triggerType" : "PUSH"
}, {
"hash" : "a5227479708c9669e82835056cc515043539114a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4279",
"triggerID" : "a5227479708c9669e82835056cc515043539114a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* 3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616 UNKNOWN
* 16be88870989aa1e54cf03f15ff61654d590e736 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4274)
* 49c310b28459d5414dce0cfdcba59ea0dac1eb6f UNKNOWN
* a5227479708c9669e82835056cc515043539114a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4279)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2739415083
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
}, {
"hash" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974",
"triggerID" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"triggerType" : "PUSH"
}, {
"hash" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3981",
"triggerID" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"triggerType" : "PUSH"
}, {
"hash" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3984",
"triggerID" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"triggerType" : "PUSH"
}, {
"hash" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"triggerType" : "PUSH"
}, {
"hash" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4100",
"triggerID" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"triggerType" : "PUSH"
}, {
"hash" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4181",
"triggerID" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"triggerType" : "PUSH"
}, {
"hash" : "16be88870989aa1e54cf03f15ff61654d590e736",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4274",
"triggerID" : "16be88870989aa1e54cf03f15ff61654d590e736",
"triggerType" : "PUSH"
}, {
"hash" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"triggerType" : "PUSH"
}, {
"hash" : "a5227479708c9669e82835056cc515043539114a",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4279",
"triggerID" : "a5227479708c9669e82835056cc515043539114a",
"triggerType" : "PUSH"
}, {
"hash" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"triggerType" : "PUSH"
}, {
"hash" : "8ebbd8b0698af9d7dadc71007805e8b6fed1c464",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4280",
"triggerID" : "8ebbd8b0698af9d7dadc71007805e8b6fed1c464",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* 3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616 UNKNOWN
* 49c310b28459d5414dce0cfdcba59ea0dac1eb6f UNKNOWN
* a5227479708c9669e82835056cc515043539114a Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4279)
* 4cf517aa15396f2766bbc35c6b6b59400f1fbebd UNKNOWN
* 8ebbd8b0698af9d7dadc71007805e8b6fed1c464 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4280)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #12843:
URL: https://github.com/apache/hudi/pull/12843#discussion_r2002139125
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -143,6 +148,29 @@ public HoodieBaseFileGroupRecordBuffer(HoodieReaderContext<T> readerContext,
} catch (IOException e) {
throw new HoodieIOException("IOException when creating ExternalSpillableMap at " + spillableMapBasePath, e);
}
+ this.shouldCheckCustomDeleteMarker = hasCustomDeleteConfigs(props, readerSchema);
+ if (shouldCheckCustomDeleteMarker) {
+ this.customDeleteMarkerKey = props.getProperty(DELETE_KEY);
+ this.customDeleteMarkerValue = props.getProperty(DELETE_MARKER);
+ }
+ }
+
+ protected boolean hasCustomDeleteConfigs(TypedProperties props, Schema schema) {
+ // DELETE_KEY and DELETE_MARKER both should be set.
+ if (StringUtils.isNullOrEmpty(props.getProperty(DELETE_KEY))
+ || StringUtils.isNullOrEmpty(props.getProperty(DELETE_MARKER))) {
+ return false;
Review Comment:
Change to: Either one of them is missing, and throw.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2739421387
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
}, {
"hash" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974",
"triggerID" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"triggerType" : "PUSH"
}, {
"hash" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3981",
"triggerID" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"triggerType" : "PUSH"
}, {
"hash" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3984",
"triggerID" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"triggerType" : "PUSH"
}, {
"hash" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"triggerType" : "PUSH"
}, {
"hash" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4100",
"triggerID" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"triggerType" : "PUSH"
}, {
"hash" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4181",
"triggerID" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"triggerType" : "PUSH"
}, {
"hash" : "16be88870989aa1e54cf03f15ff61654d590e736",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4274",
"triggerID" : "16be88870989aa1e54cf03f15ff61654d590e736",
"triggerType" : "PUSH"
}, {
"hash" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"triggerType" : "PUSH"
}, {
"hash" : "a5227479708c9669e82835056cc515043539114a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4279",
"triggerID" : "a5227479708c9669e82835056cc515043539114a",
"triggerType" : "PUSH"
}, {
"hash" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"triggerType" : "PUSH"
}, {
"hash" : "8ebbd8b0698af9d7dadc71007805e8b6fed1c464",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4280",
"triggerID" : "8ebbd8b0698af9d7dadc71007805e8b6fed1c464",
"triggerType" : "PUSH"
}, {
"hash" : "d711c309c285da81e8a214ebe670c23c01d01f27",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d711c309c285da81e8a214ebe670c23c01d01f27",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* 3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616 UNKNOWN
* 49c310b28459d5414dce0cfdcba59ea0dac1eb6f UNKNOWN
* 4cf517aa15396f2766bbc35c6b6b59400f1fbebd UNKNOWN
* 8ebbd8b0698af9d7dadc71007805e8b6fed1c464 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4280)
* d711c309c285da81e8a214ebe670c23c01d01f27 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2739880307
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
}, {
"hash" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974",
"triggerID" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"triggerType" : "PUSH"
}, {
"hash" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3981",
"triggerID" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"triggerType" : "PUSH"
}, {
"hash" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3984",
"triggerID" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"triggerType" : "PUSH"
}, {
"hash" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"triggerType" : "PUSH"
}, {
"hash" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4100",
"triggerID" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"triggerType" : "PUSH"
}, {
"hash" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4181",
"triggerID" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"triggerType" : "PUSH"
}, {
"hash" : "16be88870989aa1e54cf03f15ff61654d590e736",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4274",
"triggerID" : "16be88870989aa1e54cf03f15ff61654d590e736",
"triggerType" : "PUSH"
}, {
"hash" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"triggerType" : "PUSH"
}, {
"hash" : "a5227479708c9669e82835056cc515043539114a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4279",
"triggerID" : "a5227479708c9669e82835056cc515043539114a",
"triggerType" : "PUSH"
}, {
"hash" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"triggerType" : "PUSH"
}, {
"hash" : "8ebbd8b0698af9d7dadc71007805e8b6fed1c464",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4280",
"triggerID" : "8ebbd8b0698af9d7dadc71007805e8b6fed1c464",
"triggerType" : "PUSH"
}, {
"hash" : "d711c309c285da81e8a214ebe670c23c01d01f27",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4281",
"triggerID" : "d711c309c285da81e8a214ebe670c23c01d01f27",
"triggerType" : "PUSH"
}, {
"hash" : "1f50e01a909603339fd95c7972af791625d8616d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4282",
"triggerID" : "1f50e01a909603339fd95c7972af791625d8616d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* 3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616 UNKNOWN
* 49c310b28459d5414dce0cfdcba59ea0dac1eb6f UNKNOWN
* 4cf517aa15396f2766bbc35c6b6b59400f1fbebd UNKNOWN
* 1f50e01a909603339fd95c7972af791625d8616d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4282)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #12843:
URL: https://github.com/apache/hudi/pull/12843#discussion_r2004325657
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/PositionBasedFileGroupRecordBuffer.java:
##########
@@ -135,13 +137,13 @@ public void processDataBlock(HoodieDataBlock dataBlock, Option<KeySpec> keySpecO
}
long recordPosition = recordPositions.get(recordIndex++);
-
T evolvedNextRecord = schemaTransformerWithEvolvedSchema.getLeft().apply(nextRecord);
- processNextDataRecord(
- evolvedNextRecord,
- readerContext.generateMetadataForRecord(evolvedNextRecord, schema),
- recordPosition
- );
+ Map<String, Object> metadata = readerContext.generateMetadataForRecord(evolvedNextRecord, schema);
+ if (shouldCheckCustomDeleteMarker && isCustomDeleteRecord(evolvedNextRecord)) {
+ processCustomDeleteRecord(evolvedNextRecord, metadata, recordPosition);
Review Comment:
https://issues.apache.org/jira/browse/HUDI-9201
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2739388824
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
}, {
"hash" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974",
"triggerID" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"triggerType" : "PUSH"
}, {
"hash" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3981",
"triggerID" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"triggerType" : "PUSH"
}, {
"hash" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3984",
"triggerID" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"triggerType" : "PUSH"
}, {
"hash" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"triggerType" : "PUSH"
}, {
"hash" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4100",
"triggerID" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"triggerType" : "PUSH"
}, {
"hash" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4181",
"triggerID" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"triggerType" : "PUSH"
}, {
"hash" : "16be88870989aa1e54cf03f15ff61654d590e736",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4274",
"triggerID" : "16be88870989aa1e54cf03f15ff61654d590e736",
"triggerType" : "PUSH"
}, {
"hash" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"triggerType" : "PUSH"
}, {
"hash" : "a5227479708c9669e82835056cc515043539114a",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4279",
"triggerID" : "a5227479708c9669e82835056cc515043539114a",
"triggerType" : "PUSH"
}, {
"hash" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"triggerType" : "PUSH"
}, {
"hash" : "8ebbd8b0698af9d7dadc71007805e8b6fed1c464",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8ebbd8b0698af9d7dadc71007805e8b6fed1c464",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* 3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616 UNKNOWN
* 49c310b28459d5414dce0cfdcba59ea0dac1eb6f UNKNOWN
* a5227479708c9669e82835056cc515043539114a Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4279)
* 4cf517aa15396f2766bbc35c6b6b59400f1fbebd UNKNOWN
* 8ebbd8b0698af9d7dadc71007805e8b6fed1c464 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2751934238
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b3f0a89eda2a4bb6f6b21edd3133cc77acdf89bc",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4375",
"triggerID" : "b3f0a89eda2a4bb6f6b21edd3133cc77acdf89bc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b3f0a89eda2a4bb6f6b21edd3133cc77acdf89bc Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4375)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12843:
URL: https://github.com/apache/hudi/pull/12843#discussion_r2011191079
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/KeyBasedFileGroupRecordBuffer.java:
##########
@@ -123,6 +132,18 @@ public void processNextDeletedRecord(DeleteRecord deleteRecord, Serializable rec
}
}
+ protected void processCustomDeleteRecord(T record, Map<String, Object> metadata) {
+ DeleteRecord deleteRecord = DeleteRecord.create(
+ new HoodieKey(
+ (String) metadata.get(INTERNAL_META_RECORD_KEY),
+ null),
Review Comment:
Added.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/PositionBasedFileGroupRecordBuffer.java:
##########
@@ -202,6 +203,15 @@ public void processDeleteBlock(HoodieDeleteBlock deleteBlock) throws IOException
}
}
+ protected void processCustomDeleteRecord(T record, Map<String, Object> metadata, long recordPosition) {
+ DeleteRecord deleteRecord = DeleteRecord.create(
+ new HoodieKey(
+ (String) metadata.get(INTERNAL_META_RECORD_KEY), null),
Review Comment:
Addressed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2691902012
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
}, {
"hash" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974",
"triggerID" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"triggerType" : "PUSH"
}, {
"hash" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3981",
"triggerID" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"triggerType" : "PUSH"
}, {
"hash" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3984",
"triggerID" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* fa3151a7387c16435a20c3f1d0a72a1b9f31e75d Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3984)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2691521109
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
}, {
"hash" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* 2eacf85acc06405bb4a11f54050bcfb647e2691f Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646)
* 7a82eff1dfdbc4f4e6181b689218a7726a32f03b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2749974417
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
}, {
"hash" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974",
"triggerID" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"triggerType" : "PUSH"
}, {
"hash" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3981",
"triggerID" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"triggerType" : "PUSH"
}, {
"hash" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3984",
"triggerID" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"triggerType" : "PUSH"
}, {
"hash" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"triggerType" : "PUSH"
}, {
"hash" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4100",
"triggerID" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"triggerType" : "PUSH"
}, {
"hash" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4181",
"triggerID" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"triggerType" : "PUSH"
}, {
"hash" : "16be88870989aa1e54cf03f15ff61654d590e736",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4274",
"triggerID" : "16be88870989aa1e54cf03f15ff61654d590e736",
"triggerType" : "PUSH"
}, {
"hash" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"triggerType" : "PUSH"
}, {
"hash" : "a5227479708c9669e82835056cc515043539114a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4279",
"triggerID" : "a5227479708c9669e82835056cc515043539114a",
"triggerType" : "PUSH"
}, {
"hash" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"triggerType" : "PUSH"
}, {
"hash" : "8ebbd8b0698af9d7dadc71007805e8b6fed1c464",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4280",
"triggerID" : "8ebbd8b0698af9d7dadc71007805e8b6fed1c464",
"triggerType" : "PUSH"
}, {
"hash" : "d711c309c285da81e8a214ebe670c23c01d01f27",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4281",
"triggerID" : "d711c309c285da81e8a214ebe670c23c01d01f27",
"triggerType" : "PUSH"
}, {
"hash" : "1f50e01a909603339fd95c7972af791625d8616d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4282",
"triggerID" : "1f50e01a909603339fd95c7972af791625d8616d",
"triggerType" : "PUSH"
}, {
"hash" : "16edc580ea76612ea2a8f0e6243f6ede5fdf9546",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4289",
"triggerID" : "16edc580ea76612ea2a8f0e6243f6ede5fdf9546",
"triggerType" : "PUSH"
}, {
"hash" : "6c6788f3c699fc556ccd25c0f37518fc0cf7b0a5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4318",
"triggerID" : "6c6788f3c699fc556ccd25c0f37518fc0cf7b0a5",
"triggerType" : "PUSH"
}, {
"hash" : "dd0c492908ae259b4df9c537f2c518628756d6aa",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4369",
"triggerID" : "dd0c492908ae259b4df9c537f2c518628756d6aa",
"triggerType" : "PUSH"
}, {
"hash" : "19815dab31ee8c45968c2aa0d4b92fd6b3cc587f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "19815dab31ee8c45968c2aa0d4b92fd6b3cc587f",
"triggerType" : "PUSH"
}, {
"hash" : "e486a5f141c8e78e1dc63184e95c1d960b067c5a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4373",
"triggerID" : "e486a5f141c8e78e1dc63184e95c1d960b067c5a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* 3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616 UNKNOWN
* 49c310b28459d5414dce0cfdcba59ea0dac1eb6f UNKNOWN
* 4cf517aa15396f2766bbc35c6b6b59400f1fbebd UNKNOWN
* dd0c492908ae259b4df9c537f2c518628756d6aa Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4369)
* 19815dab31ee8c45968c2aa0d4b92fd6b3cc587f UNKNOWN
* e486a5f141c8e78e1dc63184e95c1d960b067c5a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4373)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12843:
URL: https://github.com/apache/hudi/pull/12843#discussion_r2012539863
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -244,8 +280,7 @@ protected Option<Pair<Option<T>, Map<String, Object>>> doProcessNextDataRecord(T
Comparable incomingOrderingValue = readerContext.getOrderingValue(
Option.of(record), metadata, readerSchema, orderingFieldName);
if (incomingOrderingValue.compareTo(existingOrderingValue) > 0) {
- return Option.of(Pair.of(isDeleteRecord(Option.of(record), readerContext.getSchemaFromMetadata(metadata))
- ? Option.empty() : Option.of(record), metadata));
+ return Option.of(Pair.of(Option.of(record), metadata));
Review Comment:
No. We consolidated the delete handling as you suggested before so that all types of delete records are handled by `KeyBasedFileGroupRecordBuffer#processDeleteRecord` and `#processNextDeletedRecord`. This method only handles non-deletes so there is no need to check whether the record is a delete or not I this method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2749897521
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
}, {
"hash" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974",
"triggerID" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"triggerType" : "PUSH"
}, {
"hash" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3981",
"triggerID" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"triggerType" : "PUSH"
}, {
"hash" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3984",
"triggerID" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"triggerType" : "PUSH"
}, {
"hash" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"triggerType" : "PUSH"
}, {
"hash" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4100",
"triggerID" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"triggerType" : "PUSH"
}, {
"hash" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4181",
"triggerID" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"triggerType" : "PUSH"
}, {
"hash" : "16be88870989aa1e54cf03f15ff61654d590e736",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4274",
"triggerID" : "16be88870989aa1e54cf03f15ff61654d590e736",
"triggerType" : "PUSH"
}, {
"hash" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"triggerType" : "PUSH"
}, {
"hash" : "a5227479708c9669e82835056cc515043539114a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4279",
"triggerID" : "a5227479708c9669e82835056cc515043539114a",
"triggerType" : "PUSH"
}, {
"hash" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"triggerType" : "PUSH"
}, {
"hash" : "8ebbd8b0698af9d7dadc71007805e8b6fed1c464",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4280",
"triggerID" : "8ebbd8b0698af9d7dadc71007805e8b6fed1c464",
"triggerType" : "PUSH"
}, {
"hash" : "d711c309c285da81e8a214ebe670c23c01d01f27",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4281",
"triggerID" : "d711c309c285da81e8a214ebe670c23c01d01f27",
"triggerType" : "PUSH"
}, {
"hash" : "1f50e01a909603339fd95c7972af791625d8616d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4282",
"triggerID" : "1f50e01a909603339fd95c7972af791625d8616d",
"triggerType" : "PUSH"
}, {
"hash" : "16edc580ea76612ea2a8f0e6243f6ede5fdf9546",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4289",
"triggerID" : "16edc580ea76612ea2a8f0e6243f6ede5fdf9546",
"triggerType" : "PUSH"
}, {
"hash" : "6c6788f3c699fc556ccd25c0f37518fc0cf7b0a5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4318",
"triggerID" : "6c6788f3c699fc556ccd25c0f37518fc0cf7b0a5",
"triggerType" : "PUSH"
}, {
"hash" : "dd0c492908ae259b4df9c537f2c518628756d6aa",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4369",
"triggerID" : "dd0c492908ae259b4df9c537f2c518628756d6aa",
"triggerType" : "PUSH"
}, {
"hash" : "19815dab31ee8c45968c2aa0d4b92fd6b3cc587f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "19815dab31ee8c45968c2aa0d4b92fd6b3cc587f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* 3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616 UNKNOWN
* 49c310b28459d5414dce0cfdcba59ea0dac1eb6f UNKNOWN
* 4cf517aa15396f2766bbc35c6b6b59400f1fbebd UNKNOWN
* dd0c492908ae259b4df9c537f2c518628756d6aa Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4369)
* 19815dab31ee8c45968c2aa0d4b92fd6b3cc587f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12843:
URL: https://github.com/apache/hudi/pull/12843#discussion_r2010925325
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -143,6 +148,29 @@ public HoodieBaseFileGroupRecordBuffer(HoodieReaderContext<T> readerContext,
} catch (IOException e) {
throw new HoodieIOException("IOException when creating ExternalSpillableMap at " + spillableMapBasePath, e);
}
+ this.shouldCheckCustomDeleteMarker = hasCustomDeleteConfigs(props, readerSchema);
+ if (shouldCheckCustomDeleteMarker) {
+ this.customDeleteMarkerKey = props.getProperty(DELETE_KEY);
+ this.customDeleteMarkerValue = props.getProperty(DELETE_MARKER);
+ }
+ }
+
+ protected boolean hasCustomDeleteConfigs(TypedProperties props, Schema schema) {
+ // DELETE_KEY and DELETE_MARKER both should be set.
+ if (StringUtils.isNullOrEmpty(props.getProperty(DELETE_KEY))
Review Comment:
This one still needs to be improved. I'll take it up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2660367301
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645)
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* 2eacf85acc06405bb4a11f54050bcfb647e2691f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2691834804
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
}, {
"hash" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974",
"triggerID" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"triggerType" : "PUSH"
}, {
"hash" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3981",
"triggerID" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"triggerType" : "PUSH"
}, {
"hash" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3984",
"triggerID" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* 26f9abed586abb54ac4a404dac8a0b62aab1026e Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3981)
* fa3151a7387c16435a20c3f1d0a72a1b9f31e75d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3984)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12843:
URL: https://github.com/apache/hudi/pull/12843#discussion_r1978416459
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/BaseHoodieFileGroupRecordBuffer.java:
##########
@@ -72,13 +72,15 @@
import static org.apache.hudi.common.config.HoodieMemoryConfig.SPILLABLE_MAP_BASE_PATH;
import static org.apache.hudi.common.engine.HoodieReaderContext.INTERNAL_META_PARTITION_PATH;
import static org.apache.hudi.common.engine.HoodieReaderContext.INTERNAL_META_RECORD_KEY;
+import static org.apache.hudi.common.model.DefaultHoodieRecordPayload.DELETE_KEY;
+import static org.apache.hudi.common.model.DefaultHoodieRecordPayload.DELETE_MARKER;
import static org.apache.hudi.common.model.HoodieRecord.DEFAULT_ORDERING_VALUE;
import static org.apache.hudi.common.model.HoodieRecord.HOODIE_IS_DELETED_FIELD;
import static org.apache.hudi.common.model.HoodieRecord.OPERATION_METADATA_FIELD;
import static org.apache.hudi.common.model.HoodieRecordMerger.PAYLOAD_BASED_MERGE_STRATEGY_UUID;
import static org.apache.hudi.common.table.log.block.HoodieLogBlock.HeaderMetadataType.INSTANT_TIME;
-public abstract class HoodieBaseFileGroupRecordBuffer<T> implements HoodieFileGroupRecordBuffer<T> {
+public abstract class BaseHoodieFileGroupRecordBuffer<T> implements HoodieFileGroupRecordBuffer<T> {
Review Comment:
For simplicity we can name this class as `FileGroupRecordBuffer`
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/KeyBasedHoodieFileGroupRecordBuffer.java:
##########
@@ -40,15 +41,18 @@
import java.util.Iterator;
import java.util.Map;
+import static org.apache.hudi.common.engine.HoodieReaderContext.INTERNAL_META_PARTITION_PATH;
+import static org.apache.hudi.common.engine.HoodieReaderContext.INTERNAL_META_RECORD_KEY;
+
/**
* A buffer that is used to store log records by {@link org.apache.hudi.common.table.log.HoodieMergedLogRecordReader}
* by calling the {@link #processDataBlock} and {@link #processDeleteBlock} methods into a record key based map.
* The records from the base file is accessed from an iterator object. These records are merged when the
* {@link #hasNext} method is called.
*/
-public class HoodieKeyBasedFileGroupRecordBuffer<T> extends HoodieBaseFileGroupRecordBuffer<T> {
+public class KeyBasedHoodieFileGroupRecordBuffer<T> extends BaseHoodieFileGroupRecordBuffer<T> {
Review Comment:
Similarly, remove `Hoodie` in the class name for simplicity
```suggestion
public class KeyBasedFileGroupRecordBuffer<T> extends BaseHoodieFileGroupRecordBuffer<T> {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2749807666
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
}, {
"hash" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974",
"triggerID" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"triggerType" : "PUSH"
}, {
"hash" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3981",
"triggerID" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"triggerType" : "PUSH"
}, {
"hash" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3984",
"triggerID" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"triggerType" : "PUSH"
}, {
"hash" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"triggerType" : "PUSH"
}, {
"hash" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4100",
"triggerID" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"triggerType" : "PUSH"
}, {
"hash" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4181",
"triggerID" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"triggerType" : "PUSH"
}, {
"hash" : "16be88870989aa1e54cf03f15ff61654d590e736",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4274",
"triggerID" : "16be88870989aa1e54cf03f15ff61654d590e736",
"triggerType" : "PUSH"
}, {
"hash" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"triggerType" : "PUSH"
}, {
"hash" : "a5227479708c9669e82835056cc515043539114a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4279",
"triggerID" : "a5227479708c9669e82835056cc515043539114a",
"triggerType" : "PUSH"
}, {
"hash" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"triggerType" : "PUSH"
}, {
"hash" : "8ebbd8b0698af9d7dadc71007805e8b6fed1c464",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4280",
"triggerID" : "8ebbd8b0698af9d7dadc71007805e8b6fed1c464",
"triggerType" : "PUSH"
}, {
"hash" : "d711c309c285da81e8a214ebe670c23c01d01f27",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4281",
"triggerID" : "d711c309c285da81e8a214ebe670c23c01d01f27",
"triggerType" : "PUSH"
}, {
"hash" : "1f50e01a909603339fd95c7972af791625d8616d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4282",
"triggerID" : "1f50e01a909603339fd95c7972af791625d8616d",
"triggerType" : "PUSH"
}, {
"hash" : "16edc580ea76612ea2a8f0e6243f6ede5fdf9546",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4289",
"triggerID" : "16edc580ea76612ea2a8f0e6243f6ede5fdf9546",
"triggerType" : "PUSH"
}, {
"hash" : "6c6788f3c699fc556ccd25c0f37518fc0cf7b0a5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4318",
"triggerID" : "6c6788f3c699fc556ccd25c0f37518fc0cf7b0a5",
"triggerType" : "PUSH"
}, {
"hash" : "dd0c492908ae259b4df9c537f2c518628756d6aa",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4369",
"triggerID" : "dd0c492908ae259b4df9c537f2c518628756d6aa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* 3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616 UNKNOWN
* 49c310b28459d5414dce0cfdcba59ea0dac1eb6f UNKNOWN
* 4cf517aa15396f2766bbc35c6b6b59400f1fbebd UNKNOWN
* 6c6788f3c699fc556ccd25c0f37518fc0cf7b0a5 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4318)
* dd0c492908ae259b4df9c537f2c518628756d6aa Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4369)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #12843:
URL: https://github.com/apache/hudi/pull/12843#discussion_r2003346568
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -143,6 +148,29 @@ public HoodieBaseFileGroupRecordBuffer(HoodieReaderContext<T> readerContext,
} catch (IOException e) {
throw new HoodieIOException("IOException when creating ExternalSpillableMap at " + spillableMapBasePath, e);
}
+ this.shouldCheckCustomDeleteMarker = hasCustomDeleteConfigs(props, readerSchema);
+ if (shouldCheckCustomDeleteMarker) {
+ this.customDeleteMarkerKey = props.getProperty(DELETE_KEY);
+ this.customDeleteMarkerValue = props.getProperty(DELETE_MARKER);
+ }
+ }
+
+ protected boolean hasCustomDeleteConfigs(TypedProperties props, Schema schema) {
+ // DELETE_KEY and DELETE_MARKER both should be set.
+ if (StringUtils.isNullOrEmpty(props.getProperty(DELETE_KEY))
+ || StringUtils.isNullOrEmpty(props.getProperty(DELETE_MARKER))) {
+ return false;
+ }
+ // Schema should have the DELETE_KEY field.
+ String deleteKeyField = props.getProperty(DELETE_KEY);
+ return schema.getField(deleteKeyField) != null;
+ }
+
+ protected boolean isCustomDeleteRecord(T record) {
+ Object deleteMarkerValue =
+ readerContext.getValue(record, readerSchema, customDeleteMarkerKey);
+ return deleteMarkerValue != null
+ && customDeleteMarkerValue.equals(deleteMarkerValue.toString());
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #12843:
URL: https://github.com/apache/hudi/pull/12843#discussion_r2003350854
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -139,26 +139,26 @@ public HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
/**
* Initialize correct record buffer
*/
- private static HoodieFileGroupRecordBuffer getRecordBuffer(HoodieReaderContext readerContext,
- HoodieTableMetaClient hoodieTableMetaClient,
- RecordMergeMode recordMergeMode,
- TypedProperties props,
- Option<HoodieBaseFile> baseFileOption,
- boolean hasNoLogFiles,
- boolean isSkipMerge,
- boolean shouldUseRecordPosition,
- HoodieReadStats readStats) {
+ private static FileGroupRecordBuffer getRecordBuffer(HoodieReaderContext readerContext,
+ HoodieTableMetaClient hoodieTableMetaClient,
+ RecordMergeMode recordMergeMode,
+ TypedProperties props,
+ Option<HoodieBaseFile> baseFileOption,
+ boolean hasNoLogFiles,
+ boolean isSkipMerge,
+ boolean shouldUseRecordPosition,
+ HoodieReadStats readStats) {
if (hasNoLogFiles) {
return null;
} else if (isSkipMerge) {
- return new HoodieUnmergedFileGroupRecordBuffer<>(
+ return new UnmergedHoodieFileGroupRecordBuffer<>(
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2738872221
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
}, {
"hash" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974",
"triggerID" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"triggerType" : "PUSH"
}, {
"hash" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3981",
"triggerID" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"triggerType" : "PUSH"
}, {
"hash" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3984",
"triggerID" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"triggerType" : "PUSH"
}, {
"hash" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"triggerType" : "PUSH"
}, {
"hash" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4100",
"triggerID" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"triggerType" : "PUSH"
}, {
"hash" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4181",
"triggerID" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"triggerType" : "PUSH"
}, {
"hash" : "16be88870989aa1e54cf03f15ff61654d590e736",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4274",
"triggerID" : "16be88870989aa1e54cf03f15ff61654d590e736",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* 3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616 UNKNOWN
* 16be88870989aa1e54cf03f15ff61654d590e736 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4274)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2739271557
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
}, {
"hash" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974",
"triggerID" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"triggerType" : "PUSH"
}, {
"hash" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3981",
"triggerID" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"triggerType" : "PUSH"
}, {
"hash" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3984",
"triggerID" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"triggerType" : "PUSH"
}, {
"hash" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"triggerType" : "PUSH"
}, {
"hash" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4100",
"triggerID" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"triggerType" : "PUSH"
}, {
"hash" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4181",
"triggerID" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"triggerType" : "PUSH"
}, {
"hash" : "16be88870989aa1e54cf03f15ff61654d590e736",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4274",
"triggerID" : "16be88870989aa1e54cf03f15ff61654d590e736",
"triggerType" : "PUSH"
}, {
"hash" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"triggerType" : "PUSH"
}, {
"hash" : "a5227479708c9669e82835056cc515043539114a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a5227479708c9669e82835056cc515043539114a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* 3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616 UNKNOWN
* 16be88870989aa1e54cf03f15ff61654d590e736 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4274)
* 49c310b28459d5414dce0cfdcba59ea0dac1eb6f UNKNOWN
* a5227479708c9669e82835056cc515043539114a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #12843:
URL: https://github.com/apache/hudi/pull/12843#discussion_r2004414897
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/common/table/read/TestCustomDeleteRecords.scala:
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read
+
+import org.apache.hudi.{DataSourceWriteOptions, DefaultSparkRecordMerger, OverwriteWithLatestSparkRecordMerger}
+import org.apache.hudi.DataSourceWriteOptions.{OPERATION, PRECOMBINE_FIELD, RECORDKEY_FIELD, TABLE_TYPE}
+import org.apache.hudi.common.config.{HoodieReaderConfig, HoodieStorageConfig, RecordMergeMode}
+import org.apache.hudi.common.model.{HoodieAvroRecordMerger, HoodieRecordMerger, OverwriteWithLatestMerger}
+import org.apache.hudi.common.model.DefaultHoodieRecordPayload.{DELETE_KEY, DELETE_MARKER}
+import org.apache.hudi.config.{HoodieCompactionConfig, HoodieWriteConfig}
+import org.apache.hudi.testutils.SparkClientFunctionalTestHarness
+
+import org.apache.spark.sql.{Dataset, Row, SaveMode}
+import org.junit.jupiter.api.Assertions.assertTrue
+import org.junit.jupiter.params.ParameterizedTest
+import org.junit.jupiter.params.provider.{Arguments, MethodSource}
+
+class TestCustomDeleteRecords extends SparkClientFunctionalTestHarness {
+ val expectedEventTimeBased: Seq[(Int, String, String, String, Double, String)] = Seq(
+ (10, "5", "rider-E", "driver-E", 17.85, "i"),
+ (10, "3", "rider-C", "driver-C", 33.9, "i"),
+ (10, "2", "rider-B", "driver-B", 27.7, "i"))
+ val expectedCommitTimeBased: Seq[(Int, String, String, String, Double, String)] = Seq(
+ (10, "5", "rider-E", "driver-E", 17.85, "i"),
+ (10, "3", "rider-C", "driver-C", 33.9, "i"))
+
+ @ParameterizedTest
+ @MethodSource(Array("provideParams"))
+ def testCustomDelete(useFgReader: String,
+ tableType: String,
+ recordType: String,
+ positionUsed: String,
+ mergeMode: String): Unit = {
+ val sparkMergeClasses = List(
+ classOf[DefaultSparkRecordMerger].getName,
+ classOf[OverwriteWithLatestSparkRecordMerger].getName).mkString(",")
+ val avroMergerClasses = List(
+ classOf[HoodieAvroRecordMerger].getName,
+ classOf[OverwriteWithLatestMerger].getName).mkString(",")
+
+ val mergeStrategy = if (mergeMode.equals(RecordMergeMode.EVENT_TIME_ORDERING.name)) {
+ HoodieRecordMerger.EVENT_TIME_BASED_MERGE_STRATEGY_UUID
+ } else {
+ HoodieRecordMerger.COMMIT_TIME_BASED_MERGE_STRATEGY_UUID
+ }
+ val mergeOpts: Map[String, String] = Map(
+ HoodieStorageConfig.LOGFILE_DATA_BLOCK_FORMAT.key -> "parquet",
+ HoodieWriteConfig.RECORD_MERGE_IMPL_CLASSES.key ->
+ (if (recordType.equals("SPARK")) sparkMergeClasses else avroMergerClasses),
+ HoodieWriteConfig.RECORD_MERGE_STRATEGY_ID.key -> mergeStrategy)
+ val fgReaderOpts: Map[String, String] = Map(
+ HoodieReaderConfig.FILE_GROUP_READER_ENABLED.key -> useFgReader,
+ HoodieReaderConfig.MERGE_USE_RECORD_POSITIONS.key -> positionUsed,
+ HoodieWriteConfig.RECORD_MERGE_MODE.key -> mergeMode
+ )
+ val deleteOpts: Map[String, String] = Map(
+ DELETE_KEY -> "op", DELETE_MARKER -> "d")
+ val opts = mergeOpts ++ fgReaderOpts ++ deleteOpts
+ val columns = Seq("ts", "key", "rider", "driver", "fare", "op")
+
+ val data = Seq(
+ (10, "1", "rider-A", "driver-A", 19.10, "i"),
+ (10, "2", "rider-B", "driver-B", 27.70, "i"),
+ (10, "3", "rider-C", "driver-C", 33.90, "i"),
+ (10, "4", "rider-D", "driver-D", 34.15, "i"),
+ (10, "5", "rider-E", "driver-E", 17.85, "i"))
+ val inserts = spark.createDataFrame(data).toDF(columns: _*)
+ inserts.write.format("hudi").
+ option(RECORDKEY_FIELD.key(), "key").
+ option(PRECOMBINE_FIELD.key(), "ts").
+ option(TABLE_TYPE.key(), tableType).
+ option(DataSourceWriteOptions.TABLE_NAME.key(), "test_table").
+ option(HoodieCompactionConfig.INLINE_COMPACT.key(), "false").
+ options(opts).
+ mode(SaveMode.Overwrite).
+ save(basePath)
+
+ // Delete using delete markers.
+ val updateData = Seq(
+ (11, "1", "rider-X", "driver-X", 19.10, "d"),
+ (9, "2", "rider-Y", "driver-Y", 27.70, "d"))
+ val updates = spark.createDataFrame(updateData).toDF(columns: _*)
+ updates.write.format("hudi").
+ option(OPERATION.key(), "upsert").
+ option(HoodieCompactionConfig.INLINE_COMPACT.key(), "false").
+ options(opts).
+ mode(SaveMode.Append).
+ save(basePath)
+
+ // Delete from operation.
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2750120609
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
}, {
"hash" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974",
"triggerID" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"triggerType" : "PUSH"
}, {
"hash" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3981",
"triggerID" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"triggerType" : "PUSH"
}, {
"hash" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3984",
"triggerID" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"triggerType" : "PUSH"
}, {
"hash" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"triggerType" : "PUSH"
}, {
"hash" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4100",
"triggerID" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"triggerType" : "PUSH"
}, {
"hash" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4181",
"triggerID" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"triggerType" : "PUSH"
}, {
"hash" : "16be88870989aa1e54cf03f15ff61654d590e736",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4274",
"triggerID" : "16be88870989aa1e54cf03f15ff61654d590e736",
"triggerType" : "PUSH"
}, {
"hash" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"triggerType" : "PUSH"
}, {
"hash" : "a5227479708c9669e82835056cc515043539114a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4279",
"triggerID" : "a5227479708c9669e82835056cc515043539114a",
"triggerType" : "PUSH"
}, {
"hash" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"triggerType" : "PUSH"
}, {
"hash" : "8ebbd8b0698af9d7dadc71007805e8b6fed1c464",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4280",
"triggerID" : "8ebbd8b0698af9d7dadc71007805e8b6fed1c464",
"triggerType" : "PUSH"
}, {
"hash" : "d711c309c285da81e8a214ebe670c23c01d01f27",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4281",
"triggerID" : "d711c309c285da81e8a214ebe670c23c01d01f27",
"triggerType" : "PUSH"
}, {
"hash" : "1f50e01a909603339fd95c7972af791625d8616d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4282",
"triggerID" : "1f50e01a909603339fd95c7972af791625d8616d",
"triggerType" : "PUSH"
}, {
"hash" : "16edc580ea76612ea2a8f0e6243f6ede5fdf9546",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4289",
"triggerID" : "16edc580ea76612ea2a8f0e6243f6ede5fdf9546",
"triggerType" : "PUSH"
}, {
"hash" : "6c6788f3c699fc556ccd25c0f37518fc0cf7b0a5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4318",
"triggerID" : "6c6788f3c699fc556ccd25c0f37518fc0cf7b0a5",
"triggerType" : "PUSH"
}, {
"hash" : "dd0c492908ae259b4df9c537f2c518628756d6aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4369",
"triggerID" : "dd0c492908ae259b4df9c537f2c518628756d6aa",
"triggerType" : "PUSH"
}, {
"hash" : "19815dab31ee8c45968c2aa0d4b92fd6b3cc587f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "19815dab31ee8c45968c2aa0d4b92fd6b3cc587f",
"triggerType" : "PUSH"
}, {
"hash" : "e486a5f141c8e78e1dc63184e95c1d960b067c5a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4373",
"triggerID" : "e486a5f141c8e78e1dc63184e95c1d960b067c5a",
"triggerType" : "PUSH"
}, {
"hash" : "b3f0a89eda2a4bb6f6b21edd3133cc77acdf89bc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4375",
"triggerID" : "b3f0a89eda2a4bb6f6b21edd3133cc77acdf89bc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* 3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616 UNKNOWN
* 49c310b28459d5414dce0cfdcba59ea0dac1eb6f UNKNOWN
* 4cf517aa15396f2766bbc35c6b6b59400f1fbebd UNKNOWN
* 19815dab31ee8c45968c2aa0d4b92fd6b3cc587f UNKNOWN
* b3f0a89eda2a4bb6f6b21edd3133cc77acdf89bc Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4375)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12843:
URL: https://github.com/apache/hudi/pull/12843#discussion_r2012539863
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -244,8 +280,7 @@ protected Option<Pair<Option<T>, Map<String, Object>>> doProcessNextDataRecord(T
Comparable incomingOrderingValue = readerContext.getOrderingValue(
Option.of(record), metadata, readerSchema, orderingFieldName);
if (incomingOrderingValue.compareTo(existingOrderingValue) > 0) {
- return Option.of(Pair.of(isDeleteRecord(Option.of(record), readerContext.getSchemaFromMetadata(metadata))
- ? Option.empty() : Option.of(record), metadata));
+ return Option.of(Pair.of(Option.of(record), metadata));
Review Comment:
No. We consolidated the delete handling as you suggested before so that all types of delete records are handled by another method. This method only handles non-deletes so there is no need to check whether the record is a delete or not I this method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers in FG reader [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #12843:
URL: https://github.com/apache/hudi/pull/12843
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12843:
URL: https://github.com/apache/hudi/pull/12843#discussion_r2010931567
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/PositionBasedFileGroupRecordBuffer.java:
##########
@@ -202,6 +203,15 @@ public void processDeleteBlock(HoodieDeleteBlock deleteBlock) throws IOException
}
}
+ protected void processCustomDeleteRecord(T record, Map<String, Object> metadata, long recordPosition) {
+ DeleteRecord deleteRecord = DeleteRecord.create(
+ new HoodieKey(
+ (String) metadata.get(INTERNAL_META_RECORD_KEY), null),
Review Comment:
similar on documenting the `null` here.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -143,6 +149,69 @@ public HoodieBaseFileGroupRecordBuffer(HoodieReaderContext<T> readerContext,
} catch (IOException e) {
throw new HoodieIOException("IOException when creating ExternalSpillableMap at " + spillableMapBasePath, e);
}
+ // Initialize custom delete marker related variables.
+ initCustomDeleteConfigs();
+ }
+
+ protected void initCustomDeleteConfigs() {
+ this.shouldCheckCustomDeleteMarker = hasCustomDeleteConfigs(props, readerSchema);
+ if (shouldCheckCustomDeleteMarker) {
+ this.customDeleteMarkerKey = props.getProperty(DELETE_KEY);
+ this.customDeleteMarkerValue = props.getProperty(DELETE_MARKER);
+ }
+ this.shouldCheckHoodieDeleteMarker = hasHoodieDeleteField(readerSchema);
+ }
+
+ /**
+ * Check if
+ * 1. delete key and delete markers are set properly.
+ * 2. schema contains delete key column.
+ */
+ static boolean hasCustomDeleteConfigs(TypedProperties props, Schema schema) {
+ String deleteKey = props.getProperty(DELETE_KEY);
+ String deleteMarker = props.getProperty(DELETE_MARKER);
+ boolean deleteKeyExists = !StringUtils.isNullOrEmpty(deleteKey);
+ boolean deleteMarkerExists = !StringUtils.isNullOrEmpty(deleteMarker);
+
+ // DELETE_KEY and DELETE_MARKER both should be set.
+ if (deleteKeyExists && deleteMarkerExists) {
+ // DELETE_KEY field exists in the schema.
+ return schema.getField(deleteKey) != null;
+ } else if (!deleteKeyExists && !deleteMarkerExists) {
+ // Normal case.
+ return false;
+ } else {
+ throw new RuntimeException("Either custom delete key or marker is not specified");
Review Comment:
Use `IllegalArgumentException` instead of `RuntimeException`
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/KeyBasedFileGroupRecordBuffer.java:
##########
@@ -123,6 +132,18 @@ public void processNextDeletedRecord(DeleteRecord deleteRecord, Serializable rec
}
}
+ protected void processCustomDeleteRecord(T record, Map<String, Object> metadata) {
+ DeleteRecord deleteRecord = DeleteRecord.create(
+ new HoodieKey(
+ (String) metadata.get(INTERNAL_META_RECORD_KEY),
+ null),
Review Comment:
nit: add a comment on the `null` here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12843:
URL: https://github.com/apache/hudi/pull/12843#discussion_r2011257677
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -244,8 +280,7 @@ protected Option<Pair<Option<T>, Map<String, Object>>> doProcessNextDataRecord(T
Comparable incomingOrderingValue = readerContext.getOrderingValue(
Option.of(record), metadata, readerSchema, orderingFieldName);
if (incomingOrderingValue.compareTo(existingOrderingValue) > 0) {
- return Option.of(Pair.of(isDeleteRecord(Option.of(record), readerContext.getSchemaFromMetadata(metadata))
- ? Option.empty() : Option.of(record), metadata));
+ return Option.of(Pair.of(Option.of(record), metadata));
Review Comment:
So this is a bug fix?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2751929854
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b3f0a89eda2a4bb6f6b21edd3133cc77acdf89bc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b3f0a89eda2a4bb6f6b21edd3133cc77acdf89bc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b3f0a89eda2a4bb6f6b21edd3133cc77acdf89bc UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12843:
URL: https://github.com/apache/hudi/pull/12843#discussion_r1999268772
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/KeyBasedFileGroupRecordBuffer.java:
##########
@@ -123,6 +132,18 @@ public void processNextDeletedRecord(DeleteRecord deleteRecord, Serializable rec
}
}
+ protected void processCustomDeleteRecord(T record, Map<String, Object> metadata) {
+ DeleteRecord deleteRecord = DeleteRecord.create(
+ new HoodieKey(
+ (String) metadata.get(INTERNAL_META_RECORD_KEY),
+ (String) metadata.get(INTERNAL_META_PARTITION_PATH)),
Review Comment:
It looks like `INTERNAL_META_PARTITION_PATH` is not filled by `readerContext.generateMetadataForRecord(nextRecord, schema)` and now this logic uses it. We should fix that.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/PositionBasedFileGroupRecordBuffer.java:
##########
@@ -135,13 +137,13 @@ public void processDataBlock(HoodieDataBlock dataBlock, Option<KeySpec> keySpecO
}
long recordPosition = recordPositions.get(recordIndex++);
-
T evolvedNextRecord = schemaTransformerWithEvolvedSchema.getLeft().apply(nextRecord);
- processNextDataRecord(
- evolvedNextRecord,
- readerContext.generateMetadataForRecord(evolvedNextRecord, schema),
- recordPosition
- );
+ Map<String, Object> metadata = readerContext.generateMetadataForRecord(evolvedNextRecord, schema);
+ if (shouldCheckCustomDeleteMarker && isCustomDeleteRecord(evolvedNextRecord)) {
+ processCustomDeleteRecord(evolvedNextRecord, metadata, recordPosition);
Review Comment:
Also, check CUSTOM merge to make sure deletes are handled properly in the new flow.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/PositionBasedFileGroupRecordBuffer.java:
##########
@@ -202,6 +204,16 @@ public void processDeleteBlock(HoodieDeleteBlock deleteBlock) throws IOException
}
}
+ protected void processCustomDeleteRecord(T record, Map<String, Object> metadata, long recordPosition) {
+ DeleteRecord deleteRecord = DeleteRecord.create(
+ new HoodieKey(
+ (String) metadata.get(INTERNAL_META_RECORD_KEY),
+ (String) metadata.get(INTERNAL_META_PARTITION_PATH)),
Review Comment:
Similar here to revisit the partition path fetching.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/PositionBasedFileGroupRecordBuffer.java:
##########
@@ -135,13 +137,13 @@ public void processDataBlock(HoodieDataBlock dataBlock, Option<KeySpec> keySpecO
}
long recordPosition = recordPositions.get(recordIndex++);
-
T evolvedNextRecord = schemaTransformerWithEvolvedSchema.getLeft().apply(nextRecord);
- processNextDataRecord(
- evolvedNextRecord,
- readerContext.generateMetadataForRecord(evolvedNextRecord, schema),
- recordPosition
- );
+ Map<String, Object> metadata = readerContext.generateMetadataForRecord(evolvedNextRecord, schema);
+ if (shouldCheckCustomDeleteMarker && isCustomDeleteRecord(evolvedNextRecord)) {
+ processCustomDeleteRecord(evolvedNextRecord, metadata, recordPosition);
Review Comment:
Since all deletes now go through `doProcessNextDeletedRecord`, could you simplify `doProcessNextDataRecord` to only consider non-delete records?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12843:
URL: https://github.com/apache/hudi/pull/12843#discussion_r1999295986
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/KeyBasedFileGroupRecordBuffer.java:
##########
@@ -82,7 +86,12 @@ public void processDataBlock(HoodieDataBlock dataBlock, Option<KeySpec> keySpecO
Map<String, Object> metadata = readerContext.generateMetadataForRecord(
nextRecord, schema);
String recordKey = (String) metadata.get(HoodieReaderContext.INTERNAL_META_RECORD_KEY);
- processNextDataRecord(nextRecord, metadata, recordKey);
+
+ if (shouldCheckCustomDeleteMarker && isCustomDeleteRecord(nextRecord)) {
Review Comment:
Could you move all delete checks here including the `_hoodie_is_deleted`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2739642679
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
}, {
"hash" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974",
"triggerID" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"triggerType" : "PUSH"
}, {
"hash" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3981",
"triggerID" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"triggerType" : "PUSH"
}, {
"hash" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3984",
"triggerID" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"triggerType" : "PUSH"
}, {
"hash" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"triggerType" : "PUSH"
}, {
"hash" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4100",
"triggerID" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"triggerType" : "PUSH"
}, {
"hash" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4181",
"triggerID" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"triggerType" : "PUSH"
}, {
"hash" : "16be88870989aa1e54cf03f15ff61654d590e736",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4274",
"triggerID" : "16be88870989aa1e54cf03f15ff61654d590e736",
"triggerType" : "PUSH"
}, {
"hash" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"triggerType" : "PUSH"
}, {
"hash" : "a5227479708c9669e82835056cc515043539114a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4279",
"triggerID" : "a5227479708c9669e82835056cc515043539114a",
"triggerType" : "PUSH"
}, {
"hash" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"triggerType" : "PUSH"
}, {
"hash" : "8ebbd8b0698af9d7dadc71007805e8b6fed1c464",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4280",
"triggerID" : "8ebbd8b0698af9d7dadc71007805e8b6fed1c464",
"triggerType" : "PUSH"
}, {
"hash" : "d711c309c285da81e8a214ebe670c23c01d01f27",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4281",
"triggerID" : "d711c309c285da81e8a214ebe670c23c01d01f27",
"triggerType" : "PUSH"
}, {
"hash" : "1f50e01a909603339fd95c7972af791625d8616d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1f50e01a909603339fd95c7972af791625d8616d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* 3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616 UNKNOWN
* 49c310b28459d5414dce0cfdcba59ea0dac1eb6f UNKNOWN
* 4cf517aa15396f2766bbc35c6b6b59400f1fbebd UNKNOWN
* d711c309c285da81e8a214ebe670c23c01d01f27 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4281)
* 1f50e01a909603339fd95c7972af791625d8616d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #12843:
URL: https://github.com/apache/hudi/pull/12843#discussion_r1999785970
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/PositionBasedFileGroupRecordBuffer.java:
##########
@@ -61,23 +63,23 @@
* Here the position means that record position in the base file. The records from the base file is accessed from an iterator object. These records are merged when the
* {@link #hasNext} method is called.
*/
-public class HoodiePositionBasedFileGroupRecordBuffer<T> extends HoodieKeyBasedFileGroupRecordBuffer<T> {
- private static final Logger LOG = LoggerFactory.getLogger(HoodiePositionBasedFileGroupRecordBuffer.class);
+public class PositionBasedFileGroupRecordBuffer<T> extends KeyBasedFileGroupRecordBuffer<T> {
Review Comment:
Make some functions final.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #12843:
URL: https://github.com/apache/hudi/pull/12843#discussion_r2004326972
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/PositionBasedFileGroupRecordBuffer.java:
##########
@@ -202,6 +204,16 @@ public void processDeleteBlock(HoodieDeleteBlock deleteBlock) throws IOException
}
}
+ protected void processCustomDeleteRecord(T record, Map<String, Object> metadata, long recordPosition) {
+ DeleteRecord deleteRecord = DeleteRecord.create(
+ new HoodieKey(
+ (String) metadata.get(INTERNAL_META_RECORD_KEY),
+ (String) metadata.get(INTERNAL_META_PARTITION_PATH)),
Review Comment:
Made it explicitly "null"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2744020645
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
}, {
"hash" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974",
"triggerID" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"triggerType" : "PUSH"
}, {
"hash" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3981",
"triggerID" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"triggerType" : "PUSH"
}, {
"hash" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3984",
"triggerID" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"triggerType" : "PUSH"
}, {
"hash" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"triggerType" : "PUSH"
}, {
"hash" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4100",
"triggerID" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"triggerType" : "PUSH"
}, {
"hash" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4181",
"triggerID" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"triggerType" : "PUSH"
}, {
"hash" : "16be88870989aa1e54cf03f15ff61654d590e736",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4274",
"triggerID" : "16be88870989aa1e54cf03f15ff61654d590e736",
"triggerType" : "PUSH"
}, {
"hash" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"triggerType" : "PUSH"
}, {
"hash" : "a5227479708c9669e82835056cc515043539114a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4279",
"triggerID" : "a5227479708c9669e82835056cc515043539114a",
"triggerType" : "PUSH"
}, {
"hash" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"triggerType" : "PUSH"
}, {
"hash" : "8ebbd8b0698af9d7dadc71007805e8b6fed1c464",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4280",
"triggerID" : "8ebbd8b0698af9d7dadc71007805e8b6fed1c464",
"triggerType" : "PUSH"
}, {
"hash" : "d711c309c285da81e8a214ebe670c23c01d01f27",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4281",
"triggerID" : "d711c309c285da81e8a214ebe670c23c01d01f27",
"triggerType" : "PUSH"
}, {
"hash" : "1f50e01a909603339fd95c7972af791625d8616d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4282",
"triggerID" : "1f50e01a909603339fd95c7972af791625d8616d",
"triggerType" : "PUSH"
}, {
"hash" : "16edc580ea76612ea2a8f0e6243f6ede5fdf9546",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4289",
"triggerID" : "16edc580ea76612ea2a8f0e6243f6ede5fdf9546",
"triggerType" : "PUSH"
}, {
"hash" : "6c6788f3c699fc556ccd25c0f37518fc0cf7b0a5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6c6788f3c699fc556ccd25c0f37518fc0cf7b0a5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* 3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616 UNKNOWN
* 49c310b28459d5414dce0cfdcba59ea0dac1eb6f UNKNOWN
* 4cf517aa15396f2766bbc35c6b6b59400f1fbebd UNKNOWN
* 16edc580ea76612ea2a8f0e6243f6ede5fdf9546 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4289)
* 6c6788f3c699fc556ccd25c0f37518fc0cf7b0a5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2707505427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
}, {
"hash" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974",
"triggerID" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"triggerType" : "PUSH"
}, {
"hash" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3981",
"triggerID" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"triggerType" : "PUSH"
}, {
"hash" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3984",
"triggerID" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"triggerType" : "PUSH"
}, {
"hash" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* fa3151a7387c16435a20c3f1d0a72a1b9f31e75d Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3984)
* 3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2691791545
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
}, {
"hash" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974",
"triggerID" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"triggerType" : "PUSH"
}, {
"hash" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3981",
"triggerID" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* 7a82eff1dfdbc4f4e6181b689218a7726a32f03b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974)
* 26f9abed586abb54ac4a404dac8a0b62aab1026e Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3981)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2691822321
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
}, {
"hash" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974",
"triggerID" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"triggerType" : "PUSH"
}, {
"hash" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3981",
"triggerID" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"triggerType" : "PUSH"
}, {
"hash" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* 7a82eff1dfdbc4f4e6181b689218a7726a32f03b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974)
* 26f9abed586abb54ac4a404dac8a0b62aab1026e Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3981)
* fa3151a7387c16435a20c3f1d0a72a1b9f31e75d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12843:
URL: https://github.com/apache/hudi/pull/12843#discussion_r1999329139
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/common/table/read/TestCustomDeleteRecords.scala:
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read
+
+import org.apache.hudi.{DataSourceWriteOptions, DefaultSparkRecordMerger, OverwriteWithLatestSparkRecordMerger}
+import org.apache.hudi.DataSourceWriteOptions.{OPERATION, PRECOMBINE_FIELD, RECORDKEY_FIELD, TABLE_TYPE}
+import org.apache.hudi.common.config.{HoodieReaderConfig, HoodieStorageConfig, RecordMergeMode}
+import org.apache.hudi.common.model.{HoodieAvroRecordMerger, HoodieRecordMerger, OverwriteWithLatestMerger}
+import org.apache.hudi.common.model.DefaultHoodieRecordPayload.{DELETE_KEY, DELETE_MARKER}
+import org.apache.hudi.config.{HoodieCompactionConfig, HoodieWriteConfig}
+import org.apache.hudi.testutils.SparkClientFunctionalTestHarness
+
+import org.apache.spark.sql.{Dataset, Row, SaveMode}
+import org.junit.jupiter.api.Assertions.assertTrue
+import org.junit.jupiter.params.ParameterizedTest
+import org.junit.jupiter.params.provider.{Arguments, MethodSource}
+
+class TestCustomDeleteRecords extends SparkClientFunctionalTestHarness {
+ val expectedEventTimeBased: Seq[(Int, String, String, String, Double, String)] = Seq(
+ (10, "5", "rider-E", "driver-E", 17.85, "i"),
+ (10, "3", "rider-C", "driver-C", 33.9, "i"),
+ (10, "2", "rider-B", "driver-B", 27.7, "i"))
+ val expectedCommitTimeBased: Seq[(Int, String, String, String, Double, String)] = Seq(
+ (10, "5", "rider-E", "driver-E", 17.85, "i"),
+ (10, "3", "rider-C", "driver-C", 33.9, "i"))
+
+ @ParameterizedTest
+ @MethodSource(Array("provideParams"))
+ def testCustomDelete(useFgReader: String,
+ tableType: String,
+ recordType: String,
+ positionUsed: String,
+ mergeMode: String): Unit = {
+ val sparkMergeClasses = List(
+ classOf[DefaultSparkRecordMerger].getName,
+ classOf[OverwriteWithLatestSparkRecordMerger].getName).mkString(",")
+ val avroMergerClasses = List(
+ classOf[HoodieAvroRecordMerger].getName,
+ classOf[OverwriteWithLatestMerger].getName).mkString(",")
+
+ val mergeStrategy = if (mergeMode.equals(RecordMergeMode.EVENT_TIME_ORDERING.name)) {
+ HoodieRecordMerger.EVENT_TIME_BASED_MERGE_STRATEGY_UUID
+ } else {
+ HoodieRecordMerger.COMMIT_TIME_BASED_MERGE_STRATEGY_UUID
+ }
+ val mergeOpts: Map[String, String] = Map(
+ HoodieStorageConfig.LOGFILE_DATA_BLOCK_FORMAT.key -> "parquet",
+ HoodieWriteConfig.RECORD_MERGE_IMPL_CLASSES.key ->
+ (if (recordType.equals("SPARK")) sparkMergeClasses else avroMergerClasses),
+ HoodieWriteConfig.RECORD_MERGE_STRATEGY_ID.key -> mergeStrategy)
+ val fgReaderOpts: Map[String, String] = Map(
+ HoodieReaderConfig.FILE_GROUP_READER_ENABLED.key -> useFgReader,
+ HoodieReaderConfig.MERGE_USE_RECORD_POSITIONS.key -> positionUsed,
+ HoodieWriteConfig.RECORD_MERGE_MODE.key -> mergeMode
+ )
+ val deleteOpts: Map[String, String] = Map(
+ DELETE_KEY -> "op", DELETE_MARKER -> "d")
+ val opts = mergeOpts ++ fgReaderOpts ++ deleteOpts
+ val columns = Seq("ts", "key", "rider", "driver", "fare", "op")
+
+ val data = Seq(
+ (10, "1", "rider-A", "driver-A", 19.10, "i"),
+ (10, "2", "rider-B", "driver-B", 27.70, "i"),
+ (10, "3", "rider-C", "driver-C", 33.90, "i"),
+ (10, "4", "rider-D", "driver-D", 34.15, "i"),
+ (10, "5", "rider-E", "driver-E", 17.85, "i"))
+ val inserts = spark.createDataFrame(data).toDF(columns: _*)
+ inserts.write.format("hudi").
+ option(RECORDKEY_FIELD.key(), "key").
+ option(PRECOMBINE_FIELD.key(), "ts").
+ option(TABLE_TYPE.key(), tableType).
+ option(DataSourceWriteOptions.TABLE_NAME.key(), "test_table").
+ option(HoodieCompactionConfig.INLINE_COMPACT.key(), "false").
+ options(opts).
+ mode(SaveMode.Overwrite).
+ save(basePath)
+
+ // Delete using delete markers.
+ val updateData = Seq(
+ (11, "1", "rider-X", "driver-X", 19.10, "d"),
+ (9, "2", "rider-Y", "driver-Y", 27.70, "d"))
+ val updates = spark.createDataFrame(updateData).toDF(columns: _*)
+ updates.write.format("hudi").
+ option(OPERATION.key(), "upsert").
+ option(HoodieCompactionConfig.INLINE_COMPACT.key(), "false").
+ options(opts).
+ mode(SaveMode.Append).
+ save(basePath)
+
+ // Delete from operation.
+ val deletesData = Seq((-5, "4", "rider-D", "driver-D", 34.15, 6))
+ val deletes = spark.createDataFrame(deletesData).toDF(columns: _*)
+ deletes.write.format("hudi").
+ option(OPERATION.key(), "DELETE").
+ option(HoodieCompactionConfig.INLINE_COMPACT.key(), "false").
+ options(opts).
+ mode(SaveMode.Append).
+ save(basePath)
+
+ // Validate in the end.
+ val df = spark.read.format("hudi").options(opts).load(basePath)
Review Comment:
Avoid adding `RECORD_MERGE_IMPL_CLASSES` and `RECORD_MERGE_STRATEGY_ID` on the reader side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #12843:
URL: https://github.com/apache/hudi/pull/12843#discussion_r1985738769
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/BaseHoodieFileGroupRecordBuffer.java:
##########
@@ -72,13 +72,15 @@
import static org.apache.hudi.common.config.HoodieMemoryConfig.SPILLABLE_MAP_BASE_PATH;
import static org.apache.hudi.common.engine.HoodieReaderContext.INTERNAL_META_PARTITION_PATH;
import static org.apache.hudi.common.engine.HoodieReaderContext.INTERNAL_META_RECORD_KEY;
+import static org.apache.hudi.common.model.DefaultHoodieRecordPayload.DELETE_KEY;
+import static org.apache.hudi.common.model.DefaultHoodieRecordPayload.DELETE_MARKER;
import static org.apache.hudi.common.model.HoodieRecord.DEFAULT_ORDERING_VALUE;
import static org.apache.hudi.common.model.HoodieRecord.HOODIE_IS_DELETED_FIELD;
import static org.apache.hudi.common.model.HoodieRecord.OPERATION_METADATA_FIELD;
import static org.apache.hudi.common.model.HoodieRecordMerger.PAYLOAD_BASED_MERGE_STRATEGY_UUID;
import static org.apache.hudi.common.table.log.block.HoodieLogBlock.HeaderMetadataType.INSTANT_TIME;
-public abstract class HoodieBaseFileGroupRecordBuffer<T> implements HoodieFileGroupRecordBuffer<T> {
+public abstract class BaseHoodieFileGroupRecordBuffer<T> implements HoodieFileGroupRecordBuffer<T> {
Review Comment:
Done.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/KeyBasedHoodieFileGroupRecordBuffer.java:
##########
@@ -40,15 +41,18 @@
import java.util.Iterator;
import java.util.Map;
+import static org.apache.hudi.common.engine.HoodieReaderContext.INTERNAL_META_PARTITION_PATH;
+import static org.apache.hudi.common.engine.HoodieReaderContext.INTERNAL_META_RECORD_KEY;
+
/**
* A buffer that is used to store log records by {@link org.apache.hudi.common.table.log.HoodieMergedLogRecordReader}
* by calling the {@link #processDataBlock} and {@link #processDeleteBlock} methods into a record key based map.
* The records from the base file is accessed from an iterator object. These records are merged when the
* {@link #hasNext} method is called.
*/
-public class HoodieKeyBasedFileGroupRecordBuffer<T> extends HoodieBaseFileGroupRecordBuffer<T> {
+public class KeyBasedHoodieFileGroupRecordBuffer<T> extends BaseHoodieFileGroupRecordBuffer<T> {
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2719502652
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
}, {
"hash" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974",
"triggerID" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"triggerType" : "PUSH"
}, {
"hash" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3981",
"triggerID" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"triggerType" : "PUSH"
}, {
"hash" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3984",
"triggerID" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"triggerType" : "PUSH"
}, {
"hash" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"triggerType" : "PUSH"
}, {
"hash" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4100",
"triggerID" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"triggerType" : "PUSH"
}, {
"hash" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4181",
"triggerID" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* 3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616 UNKNOWN
* e888cdfc0f2bf40a5422d9290b8dbeb9e643d564 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4100)
* a9e67270ca6cc81e56c3a752326f9d884620f1ed Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4181)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2660278292
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2660318229
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645)
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2749756831
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
}, {
"hash" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974",
"triggerID" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"triggerType" : "PUSH"
}, {
"hash" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3981",
"triggerID" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"triggerType" : "PUSH"
}, {
"hash" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3984",
"triggerID" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"triggerType" : "PUSH"
}, {
"hash" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"triggerType" : "PUSH"
}, {
"hash" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4100",
"triggerID" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"triggerType" : "PUSH"
}, {
"hash" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4181",
"triggerID" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"triggerType" : "PUSH"
}, {
"hash" : "16be88870989aa1e54cf03f15ff61654d590e736",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4274",
"triggerID" : "16be88870989aa1e54cf03f15ff61654d590e736",
"triggerType" : "PUSH"
}, {
"hash" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"triggerType" : "PUSH"
}, {
"hash" : "a5227479708c9669e82835056cc515043539114a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4279",
"triggerID" : "a5227479708c9669e82835056cc515043539114a",
"triggerType" : "PUSH"
}, {
"hash" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"triggerType" : "PUSH"
}, {
"hash" : "8ebbd8b0698af9d7dadc71007805e8b6fed1c464",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4280",
"triggerID" : "8ebbd8b0698af9d7dadc71007805e8b6fed1c464",
"triggerType" : "PUSH"
}, {
"hash" : "d711c309c285da81e8a214ebe670c23c01d01f27",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4281",
"triggerID" : "d711c309c285da81e8a214ebe670c23c01d01f27",
"triggerType" : "PUSH"
}, {
"hash" : "1f50e01a909603339fd95c7972af791625d8616d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4282",
"triggerID" : "1f50e01a909603339fd95c7972af791625d8616d",
"triggerType" : "PUSH"
}, {
"hash" : "16edc580ea76612ea2a8f0e6243f6ede5fdf9546",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4289",
"triggerID" : "16edc580ea76612ea2a8f0e6243f6ede5fdf9546",
"triggerType" : "PUSH"
}, {
"hash" : "6c6788f3c699fc556ccd25c0f37518fc0cf7b0a5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4318",
"triggerID" : "6c6788f3c699fc556ccd25c0f37518fc0cf7b0a5",
"triggerType" : "PUSH"
}, {
"hash" : "dd0c492908ae259b4df9c537f2c518628756d6aa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "dd0c492908ae259b4df9c537f2c518628756d6aa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* 3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616 UNKNOWN
* 49c310b28459d5414dce0cfdcba59ea0dac1eb6f UNKNOWN
* 4cf517aa15396f2766bbc35c6b6b59400f1fbebd UNKNOWN
* 6c6788f3c699fc556ccd25c0f37518fc0cf7b0a5 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4318)
* dd0c492908ae259b4df9c537f2c518628756d6aa UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12843:
URL: https://github.com/apache/hudi/pull/12843#discussion_r2010944971
##########
hudi-common/src/main/java/org/apache/hudi/common/engine/HoodieReaderContext.java:
##########
@@ -215,6 +215,19 @@ public ClosableIterator<T> getFileRecordIterator(
*/
public abstract Object getValue(T record, Schema schema, String fieldName);
+ /**
+ * Cast to Java boolean value.
+ * If the object is not compatible with boolean type, throws.
+ */
+ public boolean castToBoolean(Object value) {
Review Comment:
nit: This is only used in one place. No need for a new API in reader context which can cause confusion.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReaderSchemaHandler.java:
##########
@@ -207,17 +213,22 @@ private static String[] getMandatoryFieldsForMerging(HoodieTableConfig cfg, Type
requiredFields.addAll(Arrays.asList(fields.get()));
}
}
-
+ // Add precombine field for event time ordering merge mode.
if (mergingConfigs.getLeft() == RecordMergeMode.EVENT_TIME_ORDERING) {
String preCombine = cfg.getPreCombineField();
if (!StringUtils.isNullOrEmpty(preCombine)) {
requiredFields.add(preCombine);
}
}
-
+ // Add `HOODIE_IS_DELETED_FIELD` field if exists.
if (dataSchema.getField(HoodieRecord.HOODIE_IS_DELETED_FIELD) != null) {
requiredFields.add(HoodieRecord.HOODIE_IS_DELETED_FIELD);
}
+ // Add custom delete key field if exists.
+ if (FileGroupRecordBuffer.hasCustomDeleteConfigs(props, dataSchema)) {
+ requiredFields.add(props.getProperty(DELETE_KEY));
Review Comment:
Could we fetch delete marker configs in schema handler once and pass it to record buffer?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2739223749
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
}, {
"hash" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974",
"triggerID" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"triggerType" : "PUSH"
}, {
"hash" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3981",
"triggerID" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"triggerType" : "PUSH"
}, {
"hash" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3984",
"triggerID" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"triggerType" : "PUSH"
}, {
"hash" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"triggerType" : "PUSH"
}, {
"hash" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4100",
"triggerID" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"triggerType" : "PUSH"
}, {
"hash" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4181",
"triggerID" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"triggerType" : "PUSH"
}, {
"hash" : "16be88870989aa1e54cf03f15ff61654d590e736",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4274",
"triggerID" : "16be88870989aa1e54cf03f15ff61654d590e736",
"triggerType" : "PUSH"
}, {
"hash" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* 3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616 UNKNOWN
* 16be88870989aa1e54cf03f15ff61654d590e736 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4274)
* 49c310b28459d5414dce0cfdcba59ea0dac1eb6f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #12843:
URL: https://github.com/apache/hudi/pull/12843#discussion_r2003424892
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/KeyBasedFileGroupRecordBuffer.java:
##########
@@ -123,6 +132,18 @@ public void processNextDeletedRecord(DeleteRecord deleteRecord, Serializable rec
}
}
+ protected void processCustomDeleteRecord(T record, Map<String, Object> metadata) {
+ DeleteRecord deleteRecord = DeleteRecord.create(
+ new HoodieKey(
+ (String) metadata.get(INTERNAL_META_RECORD_KEY),
+ (String) metadata.get(INTERNAL_META_PARTITION_PATH)),
Review Comment:
As discussed offline, here we pass "null" directly to avoid confusion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2738727518
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
}, {
"hash" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974",
"triggerID" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"triggerType" : "PUSH"
}, {
"hash" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3981",
"triggerID" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"triggerType" : "PUSH"
}, {
"hash" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3984",
"triggerID" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"triggerType" : "PUSH"
}, {
"hash" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"triggerType" : "PUSH"
}, {
"hash" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4100",
"triggerID" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"triggerType" : "PUSH"
}, {
"hash" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4181",
"triggerID" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"triggerType" : "PUSH"
}, {
"hash" : "16be88870989aa1e54cf03f15ff61654d590e736",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4274",
"triggerID" : "16be88870989aa1e54cf03f15ff61654d590e736",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* 3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616 UNKNOWN
* a9e67270ca6cc81e56c3a752326f9d884620f1ed Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4181)
* 16be88870989aa1e54cf03f15ff61654d590e736 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4274)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2739638453
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
}, {
"hash" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974",
"triggerID" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"triggerType" : "PUSH"
}, {
"hash" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3981",
"triggerID" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"triggerType" : "PUSH"
}, {
"hash" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3984",
"triggerID" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"triggerType" : "PUSH"
}, {
"hash" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"triggerType" : "PUSH"
}, {
"hash" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4100",
"triggerID" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"triggerType" : "PUSH"
}, {
"hash" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4181",
"triggerID" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"triggerType" : "PUSH"
}, {
"hash" : "16be88870989aa1e54cf03f15ff61654d590e736",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4274",
"triggerID" : "16be88870989aa1e54cf03f15ff61654d590e736",
"triggerType" : "PUSH"
}, {
"hash" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"triggerType" : "PUSH"
}, {
"hash" : "a5227479708c9669e82835056cc515043539114a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4279",
"triggerID" : "a5227479708c9669e82835056cc515043539114a",
"triggerType" : "PUSH"
}, {
"hash" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"triggerType" : "PUSH"
}, {
"hash" : "8ebbd8b0698af9d7dadc71007805e8b6fed1c464",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4280",
"triggerID" : "8ebbd8b0698af9d7dadc71007805e8b6fed1c464",
"triggerType" : "PUSH"
}, {
"hash" : "d711c309c285da81e8a214ebe670c23c01d01f27",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4281",
"triggerID" : "d711c309c285da81e8a214ebe670c23c01d01f27",
"triggerType" : "PUSH"
}, {
"hash" : "1f50e01a909603339fd95c7972af791625d8616d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1f50e01a909603339fd95c7972af791625d8616d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* 3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616 UNKNOWN
* 49c310b28459d5414dce0cfdcba59ea0dac1eb6f UNKNOWN
* 4cf517aa15396f2766bbc35c6b6b59400f1fbebd UNKNOWN
* 8ebbd8b0698af9d7dadc71007805e8b6fed1c464 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4280)
* d711c309c285da81e8a214ebe670c23c01d01f27 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4281)
* 1f50e01a909603339fd95c7972af791625d8616d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2741164149
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
}, {
"hash" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974",
"triggerID" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"triggerType" : "PUSH"
}, {
"hash" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3981",
"triggerID" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"triggerType" : "PUSH"
}, {
"hash" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3984",
"triggerID" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"triggerType" : "PUSH"
}, {
"hash" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"triggerType" : "PUSH"
}, {
"hash" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4100",
"triggerID" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"triggerType" : "PUSH"
}, {
"hash" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4181",
"triggerID" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"triggerType" : "PUSH"
}, {
"hash" : "16be88870989aa1e54cf03f15ff61654d590e736",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4274",
"triggerID" : "16be88870989aa1e54cf03f15ff61654d590e736",
"triggerType" : "PUSH"
}, {
"hash" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"triggerType" : "PUSH"
}, {
"hash" : "a5227479708c9669e82835056cc515043539114a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4279",
"triggerID" : "a5227479708c9669e82835056cc515043539114a",
"triggerType" : "PUSH"
}, {
"hash" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"triggerType" : "PUSH"
}, {
"hash" : "8ebbd8b0698af9d7dadc71007805e8b6fed1c464",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4280",
"triggerID" : "8ebbd8b0698af9d7dadc71007805e8b6fed1c464",
"triggerType" : "PUSH"
}, {
"hash" : "d711c309c285da81e8a214ebe670c23c01d01f27",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4281",
"triggerID" : "d711c309c285da81e8a214ebe670c23c01d01f27",
"triggerType" : "PUSH"
}, {
"hash" : "1f50e01a909603339fd95c7972af791625d8616d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4282",
"triggerID" : "1f50e01a909603339fd95c7972af791625d8616d",
"triggerType" : "PUSH"
}, {
"hash" : "16edc580ea76612ea2a8f0e6243f6ede5fdf9546",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "16edc580ea76612ea2a8f0e6243f6ede5fdf9546",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* 3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616 UNKNOWN
* 49c310b28459d5414dce0cfdcba59ea0dac1eb6f UNKNOWN
* 4cf517aa15396f2766bbc35c6b6b59400f1fbebd UNKNOWN
* 1f50e01a909603339fd95c7972af791625d8616d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4282)
* 16edc580ea76612ea2a8f0e6243f6ede5fdf9546 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2719438051
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
}, {
"hash" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974",
"triggerID" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"triggerType" : "PUSH"
}, {
"hash" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3981",
"triggerID" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"triggerType" : "PUSH"
}, {
"hash" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3984",
"triggerID" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"triggerType" : "PUSH"
}, {
"hash" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"triggerType" : "PUSH"
}, {
"hash" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4100",
"triggerID" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"triggerType" : "PUSH"
}, {
"hash" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* 3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616 UNKNOWN
* e888cdfc0f2bf40a5422d9290b8dbeb9e643d564 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4100)
* a9e67270ca6cc81e56c3a752326f9d884620f1ed UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2719707752
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
}, {
"hash" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974",
"triggerID" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"triggerType" : "PUSH"
}, {
"hash" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3981",
"triggerID" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"triggerType" : "PUSH"
}, {
"hash" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3984",
"triggerID" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"triggerType" : "PUSH"
}, {
"hash" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"triggerType" : "PUSH"
}, {
"hash" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4100",
"triggerID" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"triggerType" : "PUSH"
}, {
"hash" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4181",
"triggerID" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* 3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616 UNKNOWN
* a9e67270ca6cc81e56c3a752326f9d884620f1ed Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4181)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #12843:
URL: https://github.com/apache/hudi/pull/12843#discussion_r2002132006
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -143,6 +148,29 @@ public HoodieBaseFileGroupRecordBuffer(HoodieReaderContext<T> readerContext,
} catch (IOException e) {
throw new HoodieIOException("IOException when creating ExternalSpillableMap at " + spillableMapBasePath, e);
}
+ this.shouldCheckCustomDeleteMarker = hasCustomDeleteConfigs(props, readerSchema);
+ if (shouldCheckCustomDeleteMarker) {
+ this.customDeleteMarkerKey = props.getProperty(DELETE_KEY);
+ this.customDeleteMarkerValue = props.getProperty(DELETE_MARKER);
+ }
Review Comment:
Done.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -143,6 +148,29 @@ public HoodieBaseFileGroupRecordBuffer(HoodieReaderContext<T> readerContext,
} catch (IOException e) {
throw new HoodieIOException("IOException when creating ExternalSpillableMap at " + spillableMapBasePath, e);
}
+ this.shouldCheckCustomDeleteMarker = hasCustomDeleteConfigs(props, readerSchema);
+ if (shouldCheckCustomDeleteMarker) {
+ this.customDeleteMarkerKey = props.getProperty(DELETE_KEY);
+ this.customDeleteMarkerValue = props.getProperty(DELETE_MARKER);
+ }
+ }
+
+ protected boolean hasCustomDeleteConfigs(TypedProperties props, Schema schema) {
+ // DELETE_KEY and DELETE_MARKER both should be set.
+ if (StringUtils.isNullOrEmpty(props.getProperty(DELETE_KEY))
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #12843:
URL: https://github.com/apache/hudi/pull/12843#discussion_r2004463270
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/common/table/read/TestCustomDeleteRecords.scala:
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read
+
+import org.apache.hudi.{DataSourceWriteOptions, DefaultSparkRecordMerger, OverwriteWithLatestSparkRecordMerger}
+import org.apache.hudi.DataSourceWriteOptions.{OPERATION, PRECOMBINE_FIELD, RECORDKEY_FIELD, TABLE_TYPE}
+import org.apache.hudi.common.config.{HoodieReaderConfig, HoodieStorageConfig, RecordMergeMode}
+import org.apache.hudi.common.model.{HoodieAvroRecordMerger, HoodieRecordMerger, OverwriteWithLatestMerger}
+import org.apache.hudi.common.model.DefaultHoodieRecordPayload.{DELETE_KEY, DELETE_MARKER}
+import org.apache.hudi.config.{HoodieCompactionConfig, HoodieWriteConfig}
+import org.apache.hudi.testutils.SparkClientFunctionalTestHarness
+
+import org.apache.spark.sql.{Dataset, Row, SaveMode}
+import org.junit.jupiter.api.Assertions.assertTrue
+import org.junit.jupiter.params.ParameterizedTest
+import org.junit.jupiter.params.provider.{Arguments, MethodSource}
+
+class TestCustomDeleteRecords extends SparkClientFunctionalTestHarness {
+ val expectedEventTimeBased: Seq[(Int, String, String, String, Double, String)] = Seq(
+ (10, "5", "rider-E", "driver-E", 17.85, "i"),
+ (10, "3", "rider-C", "driver-C", 33.9, "i"),
+ (10, "2", "rider-B", "driver-B", 27.7, "i"))
+ val expectedCommitTimeBased: Seq[(Int, String, String, String, Double, String)] = Seq(
+ (10, "5", "rider-E", "driver-E", 17.85, "i"),
+ (10, "3", "rider-C", "driver-C", 33.9, "i"))
+
+ @ParameterizedTest
+ @MethodSource(Array("provideParams"))
+ def testCustomDelete(useFgReader: String,
+ tableType: String,
+ recordType: String,
+ positionUsed: String,
+ mergeMode: String): Unit = {
+ val sparkMergeClasses = List(
+ classOf[DefaultSparkRecordMerger].getName,
+ classOf[OverwriteWithLatestSparkRecordMerger].getName).mkString(",")
+ val avroMergerClasses = List(
+ classOf[HoodieAvroRecordMerger].getName,
+ classOf[OverwriteWithLatestMerger].getName).mkString(",")
+
+ val mergeStrategy = if (mergeMode.equals(RecordMergeMode.EVENT_TIME_ORDERING.name)) {
+ HoodieRecordMerger.EVENT_TIME_BASED_MERGE_STRATEGY_UUID
+ } else {
+ HoodieRecordMerger.COMMIT_TIME_BASED_MERGE_STRATEGY_UUID
+ }
+ val mergeOpts: Map[String, String] = Map(
+ HoodieStorageConfig.LOGFILE_DATA_BLOCK_FORMAT.key -> "parquet",
+ HoodieWriteConfig.RECORD_MERGE_IMPL_CLASSES.key ->
+ (if (recordType.equals("SPARK")) sparkMergeClasses else avroMergerClasses),
+ HoodieWriteConfig.RECORD_MERGE_STRATEGY_ID.key -> mergeStrategy)
+ val fgReaderOpts: Map[String, String] = Map(
+ HoodieReaderConfig.FILE_GROUP_READER_ENABLED.key -> useFgReader,
+ HoodieReaderConfig.MERGE_USE_RECORD_POSITIONS.key -> positionUsed,
+ HoodieWriteConfig.RECORD_MERGE_MODE.key -> mergeMode
+ )
+ val deleteOpts: Map[String, String] = Map(
+ DELETE_KEY -> "op", DELETE_MARKER -> "d")
+ val opts = mergeOpts ++ fgReaderOpts ++ deleteOpts
+ val columns = Seq("ts", "key", "rider", "driver", "fare", "op")
+
+ val data = Seq(
+ (10, "1", "rider-A", "driver-A", 19.10, "i"),
+ (10, "2", "rider-B", "driver-B", 27.70, "i"),
+ (10, "3", "rider-C", "driver-C", 33.90, "i"),
+ (10, "4", "rider-D", "driver-D", 34.15, "i"),
+ (10, "5", "rider-E", "driver-E", 17.85, "i"))
+ val inserts = spark.createDataFrame(data).toDF(columns: _*)
+ inserts.write.format("hudi").
+ option(RECORDKEY_FIELD.key(), "key").
+ option(PRECOMBINE_FIELD.key(), "ts").
+ option(TABLE_TYPE.key(), tableType).
+ option(DataSourceWriteOptions.TABLE_NAME.key(), "test_table").
+ option(HoodieCompactionConfig.INLINE_COMPACT.key(), "false").
+ options(opts).
+ mode(SaveMode.Overwrite).
+ save(basePath)
+
+ // Delete using delete markers.
+ val updateData = Seq(
+ (11, "1", "rider-X", "driver-X", 19.10, "d"),
+ (9, "2", "rider-Y", "driver-Y", 27.70, "d"))
+ val updates = spark.createDataFrame(updateData).toDF(columns: _*)
+ updates.write.format("hudi").
+ option(OPERATION.key(), "upsert").
+ option(HoodieCompactionConfig.INLINE_COMPACT.key(), "false").
+ options(opts).
+ mode(SaveMode.Append).
+ save(basePath)
+
+ // Delete from operation.
+ val deletesData = Seq((-5, "4", "rider-D", "driver-D", 34.15, 6))
+ val deletes = spark.createDataFrame(deletesData).toDF(columns: _*)
+ deletes.write.format("hudi").
+ option(OPERATION.key(), "DELETE").
+ option(HoodieCompactionConfig.INLINE_COMPACT.key(), "false").
+ options(opts).
+ mode(SaveMode.Append).
+ save(basePath)
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #12843:
URL: https://github.com/apache/hudi/pull/12843#discussion_r2004485060
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/common/table/read/TestCustomDeleteRecords.scala:
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read
+
+import org.apache.hudi.{DataSourceWriteOptions, DefaultSparkRecordMerger, OverwriteWithLatestSparkRecordMerger}
+import org.apache.hudi.DataSourceWriteOptions.{OPERATION, PRECOMBINE_FIELD, RECORDKEY_FIELD, TABLE_TYPE}
+import org.apache.hudi.common.config.{HoodieReaderConfig, HoodieStorageConfig, RecordMergeMode}
+import org.apache.hudi.common.model.{HoodieAvroRecordMerger, HoodieRecordMerger, OverwriteWithLatestMerger}
+import org.apache.hudi.common.model.DefaultHoodieRecordPayload.{DELETE_KEY, DELETE_MARKER}
+import org.apache.hudi.config.{HoodieCompactionConfig, HoodieWriteConfig}
+import org.apache.hudi.testutils.SparkClientFunctionalTestHarness
+
+import org.apache.spark.sql.{Dataset, Row, SaveMode}
+import org.junit.jupiter.api.Assertions.assertTrue
+import org.junit.jupiter.params.ParameterizedTest
+import org.junit.jupiter.params.provider.{Arguments, MethodSource}
+
+class TestCustomDeleteRecords extends SparkClientFunctionalTestHarness {
+ val expectedEventTimeBased: Seq[(Int, String, String, String, Double, String)] = Seq(
+ (10, "5", "rider-E", "driver-E", 17.85, "i"),
+ (10, "3", "rider-C", "driver-C", 33.9, "i"),
+ (10, "2", "rider-B", "driver-B", 27.7, "i"))
+ val expectedCommitTimeBased: Seq[(Int, String, String, String, Double, String)] = Seq(
+ (10, "5", "rider-E", "driver-E", 17.85, "i"),
+ (10, "3", "rider-C", "driver-C", 33.9, "i"))
+
+ @ParameterizedTest
+ @MethodSource(Array("provideParams"))
+ def testCustomDelete(useFgReader: String,
+ tableType: String,
+ recordType: String,
+ positionUsed: String,
+ mergeMode: String): Unit = {
+ val sparkMergeClasses = List(
+ classOf[DefaultSparkRecordMerger].getName,
+ classOf[OverwriteWithLatestSparkRecordMerger].getName).mkString(",")
+ val avroMergerClasses = List(
+ classOf[HoodieAvroRecordMerger].getName,
+ classOf[OverwriteWithLatestMerger].getName).mkString(",")
+
+ val mergeStrategy = if (mergeMode.equals(RecordMergeMode.EVENT_TIME_ORDERING.name)) {
+ HoodieRecordMerger.EVENT_TIME_BASED_MERGE_STRATEGY_UUID
+ } else {
+ HoodieRecordMerger.COMMIT_TIME_BASED_MERGE_STRATEGY_UUID
+ }
+ val mergeOpts: Map[String, String] = Map(
+ HoodieStorageConfig.LOGFILE_DATA_BLOCK_FORMAT.key -> "parquet",
+ HoodieWriteConfig.RECORD_MERGE_IMPL_CLASSES.key ->
+ (if (recordType.equals("SPARK")) sparkMergeClasses else avroMergerClasses),
+ HoodieWriteConfig.RECORD_MERGE_STRATEGY_ID.key -> mergeStrategy)
+ val fgReaderOpts: Map[String, String] = Map(
+ HoodieReaderConfig.FILE_GROUP_READER_ENABLED.key -> useFgReader,
+ HoodieReaderConfig.MERGE_USE_RECORD_POSITIONS.key -> positionUsed,
+ HoodieWriteConfig.RECORD_MERGE_MODE.key -> mergeMode
+ )
+ val deleteOpts: Map[String, String] = Map(
+ DELETE_KEY -> "op", DELETE_MARKER -> "d")
+ val opts = mergeOpts ++ fgReaderOpts ++ deleteOpts
+ val columns = Seq("ts", "key", "rider", "driver", "fare", "op")
+
+ val data = Seq(
+ (10, "1", "rider-A", "driver-A", 19.10, "i"),
+ (10, "2", "rider-B", "driver-B", 27.70, "i"),
+ (10, "3", "rider-C", "driver-C", 33.90, "i"),
+ (10, "4", "rider-D", "driver-D", 34.15, "i"),
+ (10, "5", "rider-E", "driver-E", 17.85, "i"))
+ val inserts = spark.createDataFrame(data).toDF(columns: _*)
+ inserts.write.format("hudi").
+ option(RECORDKEY_FIELD.key(), "key").
+ option(PRECOMBINE_FIELD.key(), "ts").
+ option(TABLE_TYPE.key(), tableType).
+ option(DataSourceWriteOptions.TABLE_NAME.key(), "test_table").
+ option(HoodieCompactionConfig.INLINE_COMPACT.key(), "false").
+ options(opts).
+ mode(SaveMode.Overwrite).
+ save(basePath)
+
+ // Delete using delete markers.
+ val updateData = Seq(
+ (11, "1", "rider-X", "driver-X", 19.10, "d"),
+ (9, "2", "rider-Y", "driver-Y", 27.70, "d"))
+ val updates = spark.createDataFrame(updateData).toDF(columns: _*)
+ updates.write.format("hudi").
+ option(OPERATION.key(), "upsert").
+ option(HoodieCompactionConfig.INLINE_COMPACT.key(), "false").
+ options(opts).
+ mode(SaveMode.Append).
+ save(basePath)
+
+ // Delete from operation.
+ val deletesData = Seq((-5, "4", "rider-D", "driver-D", 34.15, 6))
+ val deletes = spark.createDataFrame(deletesData).toDF(columns: _*)
+ deletes.write.format("hudi").
+ option(OPERATION.key(), "DELETE").
+ option(HoodieCompactionConfig.INLINE_COMPACT.key(), "false").
+ options(opts).
+ mode(SaveMode.Append).
+ save(basePath)
+
+ // Validate in the end.
+ val df = spark.read.format("hudi").options(opts).load(basePath)
+ val finalDf = df.select("ts", "key", "rider", "driver", "fare", "op").sort("key")
+ finalDf.show(false)
+ val expected = if (mergeMode == RecordMergeMode.COMMIT_TIME_ORDERING.name()) {
+ expectedCommitTimeBased
+ } else {
+ expectedEventTimeBased
+ }
+ val expectedDf = spark.createDataFrame(expected).toDF(columns: _*).sort("key")
+ TestCustomDeleteRecords.validate(expectedDf, finalDf)
+ }
+}
+
+object TestCustomDeleteRecords {
+ def provideParams(): java.util.List[Arguments] = {
+ java.util.Arrays.asList(
+ Arguments.of("true", "MERGE_ON_READ", "AVRO", "false", "EVENT_TIME_ORDERING"),
+ Arguments.of("true", "MERGE_ON_READ", "AVRO", "true", "EVENT_TIME_ORDERING"),
+ Arguments.of("true", "MERGE_ON_READ", "AVRO", "false", "COMMIT_TIME_ORDERING"),
+ Arguments.of("true", "MERGE_ON_READ", "AVRO", "true", "COMMIT_TIME_ORDERING"),
+ Arguments.of("true", "MERGE_ON_READ", "SPARK", "false", "EVENT_TIME_ORDERING"),
+ Arguments.of("true", "MERGE_ON_READ", "SPARK", "true", "EVENT_TIME_ORDERING"),
+ Arguments.of("true", "MERGE_ON_READ", "SPARK", "false", "COMMIT_TIME_ORDERING"),
+ Arguments.of("true", "MERGE_ON_READ", "SPARK", "true", "COMMIT_TIME_ORDERING"))
+ }
+
+ def validate(expectedDf: Dataset[Row], actualDf: Dataset[Row]): Unit = {
Review Comment:
Done.
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/common/table/read/TestCustomDeleteRecords.scala:
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read
+
+import org.apache.hudi.{DataSourceWriteOptions, DefaultSparkRecordMerger, OverwriteWithLatestSparkRecordMerger}
+import org.apache.hudi.DataSourceWriteOptions.{OPERATION, PRECOMBINE_FIELD, RECORDKEY_FIELD, TABLE_TYPE}
+import org.apache.hudi.common.config.{HoodieReaderConfig, HoodieStorageConfig, RecordMergeMode}
+import org.apache.hudi.common.model.{HoodieAvroRecordMerger, HoodieRecordMerger, OverwriteWithLatestMerger}
+import org.apache.hudi.common.model.DefaultHoodieRecordPayload.{DELETE_KEY, DELETE_MARKER}
+import org.apache.hudi.config.{HoodieCompactionConfig, HoodieWriteConfig}
+import org.apache.hudi.testutils.SparkClientFunctionalTestHarness
+
+import org.apache.spark.sql.{Dataset, Row, SaveMode}
+import org.junit.jupiter.api.Assertions.assertTrue
+import org.junit.jupiter.params.ParameterizedTest
+import org.junit.jupiter.params.provider.{Arguments, MethodSource}
+
+class TestCustomDeleteRecords extends SparkClientFunctionalTestHarness {
+ val expectedEventTimeBased: Seq[(Int, String, String, String, Double, String)] = Seq(
+ (10, "5", "rider-E", "driver-E", 17.85, "i"),
+ (10, "3", "rider-C", "driver-C", 33.9, "i"),
+ (10, "2", "rider-B", "driver-B", 27.7, "i"))
+ val expectedCommitTimeBased: Seq[(Int, String, String, String, Double, String)] = Seq(
+ (10, "5", "rider-E", "driver-E", 17.85, "i"),
+ (10, "3", "rider-C", "driver-C", 33.9, "i"))
+
+ @ParameterizedTest
+ @MethodSource(Array("provideParams"))
+ def testCustomDelete(useFgReader: String,
+ tableType: String,
+ recordType: String,
+ positionUsed: String,
+ mergeMode: String): Unit = {
+ val sparkMergeClasses = List(
+ classOf[DefaultSparkRecordMerger].getName,
+ classOf[OverwriteWithLatestSparkRecordMerger].getName).mkString(",")
+ val avroMergerClasses = List(
+ classOf[HoodieAvroRecordMerger].getName,
+ classOf[OverwriteWithLatestMerger].getName).mkString(",")
+
+ val mergeStrategy = if (mergeMode.equals(RecordMergeMode.EVENT_TIME_ORDERING.name)) {
+ HoodieRecordMerger.EVENT_TIME_BASED_MERGE_STRATEGY_UUID
+ } else {
+ HoodieRecordMerger.COMMIT_TIME_BASED_MERGE_STRATEGY_UUID
+ }
+ val mergeOpts: Map[String, String] = Map(
+ HoodieStorageConfig.LOGFILE_DATA_BLOCK_FORMAT.key -> "parquet",
+ HoodieWriteConfig.RECORD_MERGE_IMPL_CLASSES.key ->
+ (if (recordType.equals("SPARK")) sparkMergeClasses else avroMergerClasses),
+ HoodieWriteConfig.RECORD_MERGE_STRATEGY_ID.key -> mergeStrategy)
+ val fgReaderOpts: Map[String, String] = Map(
+ HoodieReaderConfig.FILE_GROUP_READER_ENABLED.key -> useFgReader,
+ HoodieReaderConfig.MERGE_USE_RECORD_POSITIONS.key -> positionUsed,
+ HoodieWriteConfig.RECORD_MERGE_MODE.key -> mergeMode
+ )
+ val deleteOpts: Map[String, String] = Map(
+ DELETE_KEY -> "op", DELETE_MARKER -> "d")
+ val opts = mergeOpts ++ fgReaderOpts ++ deleteOpts
+ val columns = Seq("ts", "key", "rider", "driver", "fare", "op")
+
+ val data = Seq(
+ (10, "1", "rider-A", "driver-A", 19.10, "i"),
+ (10, "2", "rider-B", "driver-B", 27.70, "i"),
+ (10, "3", "rider-C", "driver-C", 33.90, "i"),
+ (10, "4", "rider-D", "driver-D", 34.15, "i"),
+ (10, "5", "rider-E", "driver-E", 17.85, "i"))
+ val inserts = spark.createDataFrame(data).toDF(columns: _*)
+ inserts.write.format("hudi").
+ option(RECORDKEY_FIELD.key(), "key").
+ option(PRECOMBINE_FIELD.key(), "ts").
+ option(TABLE_TYPE.key(), tableType).
+ option(DataSourceWriteOptions.TABLE_NAME.key(), "test_table").
+ option(HoodieCompactionConfig.INLINE_COMPACT.key(), "false").
+ options(opts).
+ mode(SaveMode.Overwrite).
+ save(basePath)
+
+ // Delete using delete markers.
+ val updateData = Seq(
+ (11, "1", "rider-X", "driver-X", 19.10, "d"),
+ (9, "2", "rider-Y", "driver-Y", 27.70, "d"))
+ val updates = spark.createDataFrame(updateData).toDF(columns: _*)
+ updates.write.format("hudi").
+ option(OPERATION.key(), "upsert").
+ option(HoodieCompactionConfig.INLINE_COMPACT.key(), "false").
+ options(opts).
+ mode(SaveMode.Append).
+ save(basePath)
+
+ // Delete from operation.
+ val deletesData = Seq((-5, "4", "rider-D", "driver-D", 34.15, 6))
+ val deletes = spark.createDataFrame(deletesData).toDF(columns: _*)
+ deletes.write.format("hudi").
+ option(OPERATION.key(), "DELETE").
+ option(HoodieCompactionConfig.INLINE_COMPACT.key(), "false").
+ options(opts).
+ mode(SaveMode.Append).
+ save(basePath)
+
+ // Validate in the end.
+ val df = spark.read.format("hudi").options(opts).load(basePath)
+ val finalDf = df.select("ts", "key", "rider", "driver", "fare", "op").sort("key")
+ finalDf.show(false)
+ val expected = if (mergeMode == RecordMergeMode.COMMIT_TIME_ORDERING.name()) {
+ expectedCommitTimeBased
+ } else {
+ expectedEventTimeBased
+ }
+ val expectedDf = spark.createDataFrame(expected).toDF(columns: _*).sort("key")
+ TestCustomDeleteRecords.validate(expectedDf, finalDf)
+ }
+}
+
+object TestCustomDeleteRecords {
+ def provideParams(): java.util.List[Arguments] = {
+ java.util.Arrays.asList(
+ Arguments.of("true", "MERGE_ON_READ", "AVRO", "false", "EVENT_TIME_ORDERING"),
+ Arguments.of("true", "MERGE_ON_READ", "AVRO", "true", "EVENT_TIME_ORDERING"),
+ Arguments.of("true", "MERGE_ON_READ", "AVRO", "false", "COMMIT_TIME_ORDERING"),
+ Arguments.of("true", "MERGE_ON_READ", "AVRO", "true", "COMMIT_TIME_ORDERING"),
+ Arguments.of("true", "MERGE_ON_READ", "SPARK", "false", "EVENT_TIME_ORDERING"),
+ Arguments.of("true", "MERGE_ON_READ", "SPARK", "true", "EVENT_TIME_ORDERING"),
+ Arguments.of("true", "MERGE_ON_READ", "SPARK", "false", "COMMIT_TIME_ORDERING"),
+ Arguments.of("true", "MERGE_ON_READ", "SPARK", "true", "COMMIT_TIME_ORDERING"))
+ }
+
+ def validate(expectedDf: Dataset[Row], actualDf: Dataset[Row]): Unit = {
+ val expectedMinusActual = expectedDf.except(actualDf)
+ val actualMinusExpected = actualDf.except(expectedDf)
+ expectedDf.show(false)
+ actualDf.show(false)
Review Comment:
Done.
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/common/table/read/TestCustomDeleteRecords.scala:
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read
+
+import org.apache.hudi.{DataSourceWriteOptions, DefaultSparkRecordMerger, OverwriteWithLatestSparkRecordMerger}
+import org.apache.hudi.DataSourceWriteOptions.{OPERATION, PRECOMBINE_FIELD, RECORDKEY_FIELD, TABLE_TYPE}
+import org.apache.hudi.common.config.{HoodieReaderConfig, HoodieStorageConfig, RecordMergeMode}
+import org.apache.hudi.common.model.{HoodieAvroRecordMerger, HoodieRecordMerger, OverwriteWithLatestMerger}
+import org.apache.hudi.common.model.DefaultHoodieRecordPayload.{DELETE_KEY, DELETE_MARKER}
+import org.apache.hudi.config.{HoodieCompactionConfig, HoodieWriteConfig}
+import org.apache.hudi.testutils.SparkClientFunctionalTestHarness
+
+import org.apache.spark.sql.{Dataset, Row, SaveMode}
+import org.junit.jupiter.api.Assertions.assertTrue
+import org.junit.jupiter.params.ParameterizedTest
+import org.junit.jupiter.params.provider.{Arguments, MethodSource}
+
+class TestCustomDeleteRecords extends SparkClientFunctionalTestHarness {
+ val expectedEventTimeBased: Seq[(Int, String, String, String, Double, String)] = Seq(
+ (10, "5", "rider-E", "driver-E", 17.85, "i"),
+ (10, "3", "rider-C", "driver-C", 33.9, "i"),
+ (10, "2", "rider-B", "driver-B", 27.7, "i"))
+ val expectedCommitTimeBased: Seq[(Int, String, String, String, Double, String)] = Seq(
+ (10, "5", "rider-E", "driver-E", 17.85, "i"),
+ (10, "3", "rider-C", "driver-C", 33.9, "i"))
+
+ @ParameterizedTest
+ @MethodSource(Array("provideParams"))
+ def testCustomDelete(useFgReader: String,
+ tableType: String,
+ recordType: String,
+ positionUsed: String,
+ mergeMode: String): Unit = {
+ val sparkMergeClasses = List(
+ classOf[DefaultSparkRecordMerger].getName,
+ classOf[OverwriteWithLatestSparkRecordMerger].getName).mkString(",")
+ val avroMergerClasses = List(
+ classOf[HoodieAvroRecordMerger].getName,
+ classOf[OverwriteWithLatestMerger].getName).mkString(",")
+
+ val mergeStrategy = if (mergeMode.equals(RecordMergeMode.EVENT_TIME_ORDERING.name)) {
+ HoodieRecordMerger.EVENT_TIME_BASED_MERGE_STRATEGY_UUID
+ } else {
+ HoodieRecordMerger.COMMIT_TIME_BASED_MERGE_STRATEGY_UUID
+ }
+ val mergeOpts: Map[String, String] = Map(
+ HoodieStorageConfig.LOGFILE_DATA_BLOCK_FORMAT.key -> "parquet",
+ HoodieWriteConfig.RECORD_MERGE_IMPL_CLASSES.key ->
+ (if (recordType.equals("SPARK")) sparkMergeClasses else avroMergerClasses),
+ HoodieWriteConfig.RECORD_MERGE_STRATEGY_ID.key -> mergeStrategy)
+ val fgReaderOpts: Map[String, String] = Map(
+ HoodieReaderConfig.FILE_GROUP_READER_ENABLED.key -> useFgReader,
+ HoodieReaderConfig.MERGE_USE_RECORD_POSITIONS.key -> positionUsed,
+ HoodieWriteConfig.RECORD_MERGE_MODE.key -> mergeMode
+ )
+ val deleteOpts: Map[String, String] = Map(
+ DELETE_KEY -> "op", DELETE_MARKER -> "d")
+ val opts = mergeOpts ++ fgReaderOpts ++ deleteOpts
+ val columns = Seq("ts", "key", "rider", "driver", "fare", "op")
+
+ val data = Seq(
+ (10, "1", "rider-A", "driver-A", 19.10, "i"),
+ (10, "2", "rider-B", "driver-B", 27.70, "i"),
+ (10, "3", "rider-C", "driver-C", 33.90, "i"),
+ (10, "4", "rider-D", "driver-D", 34.15, "i"),
+ (10, "5", "rider-E", "driver-E", 17.85, "i"))
+ val inserts = spark.createDataFrame(data).toDF(columns: _*)
+ inserts.write.format("hudi").
+ option(RECORDKEY_FIELD.key(), "key").
+ option(PRECOMBINE_FIELD.key(), "ts").
+ option(TABLE_TYPE.key(), tableType).
+ option(DataSourceWriteOptions.TABLE_NAME.key(), "test_table").
+ option(HoodieCompactionConfig.INLINE_COMPACT.key(), "false").
+ options(opts).
+ mode(SaveMode.Overwrite).
+ save(basePath)
+
+ // Delete using delete markers.
+ val updateData = Seq(
+ (11, "1", "rider-X", "driver-X", 19.10, "d"),
+ (9, "2", "rider-Y", "driver-Y", 27.70, "d"))
+ val updates = spark.createDataFrame(updateData).toDF(columns: _*)
+ updates.write.format("hudi").
+ option(OPERATION.key(), "upsert").
+ option(HoodieCompactionConfig.INLINE_COMPACT.key(), "false").
+ options(opts).
+ mode(SaveMode.Append).
+ save(basePath)
+
+ // Delete from operation.
+ val deletesData = Seq((-5, "4", "rider-D", "driver-D", 34.15, 6))
+ val deletes = spark.createDataFrame(deletesData).toDF(columns: _*)
+ deletes.write.format("hudi").
+ option(OPERATION.key(), "DELETE").
+ option(HoodieCompactionConfig.INLINE_COMPACT.key(), "false").
+ options(opts).
+ mode(SaveMode.Append).
+ save(basePath)
+
+ // Validate in the end.
+ val df = spark.read.format("hudi").options(opts).load(basePath)
+ val finalDf = df.select("ts", "key", "rider", "driver", "fare", "op").sort("key")
+ finalDf.show(false)
+ val expected = if (mergeMode == RecordMergeMode.COMMIT_TIME_ORDERING.name()) {
+ expectedCommitTimeBased
+ } else {
+ expectedEventTimeBased
+ }
+ val expectedDf = spark.createDataFrame(expected).toDF(columns: _*).sort("key")
+ TestCustomDeleteRecords.validate(expectedDf, finalDf)
+ }
+}
+
+object TestCustomDeleteRecords {
+ def provideParams(): java.util.List[Arguments] = {
+ java.util.Arrays.asList(
+ Arguments.of("true", "MERGE_ON_READ", "AVRO", "false", "EVENT_TIME_ORDERING"),
+ Arguments.of("true", "MERGE_ON_READ", "AVRO", "true", "EVENT_TIME_ORDERING"),
+ Arguments.of("true", "MERGE_ON_READ", "AVRO", "false", "COMMIT_TIME_ORDERING"),
+ Arguments.of("true", "MERGE_ON_READ", "AVRO", "true", "COMMIT_TIME_ORDERING"),
+ Arguments.of("true", "MERGE_ON_READ", "SPARK", "false", "EVENT_TIME_ORDERING"),
+ Arguments.of("true", "MERGE_ON_READ", "SPARK", "true", "EVENT_TIME_ORDERING"),
+ Arguments.of("true", "MERGE_ON_READ", "SPARK", "false", "COMMIT_TIME_ORDERING"),
+ Arguments.of("true", "MERGE_ON_READ", "SPARK", "true", "COMMIT_TIME_ORDERING"))
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2739337698
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
}, {
"hash" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974",
"triggerID" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"triggerType" : "PUSH"
}, {
"hash" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3981",
"triggerID" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"triggerType" : "PUSH"
}, {
"hash" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3984",
"triggerID" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"triggerType" : "PUSH"
}, {
"hash" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"triggerType" : "PUSH"
}, {
"hash" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4100",
"triggerID" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"triggerType" : "PUSH"
}, {
"hash" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4181",
"triggerID" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"triggerType" : "PUSH"
}, {
"hash" : "16be88870989aa1e54cf03f15ff61654d590e736",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4274",
"triggerID" : "16be88870989aa1e54cf03f15ff61654d590e736",
"triggerType" : "PUSH"
}, {
"hash" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"triggerType" : "PUSH"
}, {
"hash" : "a5227479708c9669e82835056cc515043539114a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4279",
"triggerID" : "a5227479708c9669e82835056cc515043539114a",
"triggerType" : "PUSH"
}, {
"hash" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* 3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616 UNKNOWN
* 16be88870989aa1e54cf03f15ff61654d590e736 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4274)
* 49c310b28459d5414dce0cfdcba59ea0dac1eb6f UNKNOWN
* a5227479708c9669e82835056cc515043539114a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4279)
* 4cf517aa15396f2766bbc35c6b6b59400f1fbebd UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #12843:
URL: https://github.com/apache/hudi/pull/12843#discussion_r2003426198
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/KeyBasedFileGroupRecordBuffer.java:
##########
@@ -123,6 +132,18 @@ public void processNextDeletedRecord(DeleteRecord deleteRecord, Serializable rec
}
}
+ protected void processCustomDeleteRecord(T record, Map<String, Object> metadata) {
+ DeleteRecord deleteRecord = DeleteRecord.create(
+ new HoodieKey(
+ (String) metadata.get(INTERNAL_META_RECORD_KEY),
+ (String) metadata.get(INTERNAL_META_PARTITION_PATH)),
Review Comment:
done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2749988708
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
}, {
"hash" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974",
"triggerID" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"triggerType" : "PUSH"
}, {
"hash" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3981",
"triggerID" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"triggerType" : "PUSH"
}, {
"hash" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3984",
"triggerID" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"triggerType" : "PUSH"
}, {
"hash" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"triggerType" : "PUSH"
}, {
"hash" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4100",
"triggerID" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"triggerType" : "PUSH"
}, {
"hash" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4181",
"triggerID" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"triggerType" : "PUSH"
}, {
"hash" : "16be88870989aa1e54cf03f15ff61654d590e736",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4274",
"triggerID" : "16be88870989aa1e54cf03f15ff61654d590e736",
"triggerType" : "PUSH"
}, {
"hash" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"triggerType" : "PUSH"
}, {
"hash" : "a5227479708c9669e82835056cc515043539114a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4279",
"triggerID" : "a5227479708c9669e82835056cc515043539114a",
"triggerType" : "PUSH"
}, {
"hash" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"triggerType" : "PUSH"
}, {
"hash" : "8ebbd8b0698af9d7dadc71007805e8b6fed1c464",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4280",
"triggerID" : "8ebbd8b0698af9d7dadc71007805e8b6fed1c464",
"triggerType" : "PUSH"
}, {
"hash" : "d711c309c285da81e8a214ebe670c23c01d01f27",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4281",
"triggerID" : "d711c309c285da81e8a214ebe670c23c01d01f27",
"triggerType" : "PUSH"
}, {
"hash" : "1f50e01a909603339fd95c7972af791625d8616d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4282",
"triggerID" : "1f50e01a909603339fd95c7972af791625d8616d",
"triggerType" : "PUSH"
}, {
"hash" : "16edc580ea76612ea2a8f0e6243f6ede5fdf9546",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4289",
"triggerID" : "16edc580ea76612ea2a8f0e6243f6ede5fdf9546",
"triggerType" : "PUSH"
}, {
"hash" : "6c6788f3c699fc556ccd25c0f37518fc0cf7b0a5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4318",
"triggerID" : "6c6788f3c699fc556ccd25c0f37518fc0cf7b0a5",
"triggerType" : "PUSH"
}, {
"hash" : "dd0c492908ae259b4df9c537f2c518628756d6aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4369",
"triggerID" : "dd0c492908ae259b4df9c537f2c518628756d6aa",
"triggerType" : "PUSH"
}, {
"hash" : "19815dab31ee8c45968c2aa0d4b92fd6b3cc587f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "19815dab31ee8c45968c2aa0d4b92fd6b3cc587f",
"triggerType" : "PUSH"
}, {
"hash" : "e486a5f141c8e78e1dc63184e95c1d960b067c5a",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4373",
"triggerID" : "e486a5f141c8e78e1dc63184e95c1d960b067c5a",
"triggerType" : "PUSH"
}, {
"hash" : "b3f0a89eda2a4bb6f6b21edd3133cc77acdf89bc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b3f0a89eda2a4bb6f6b21edd3133cc77acdf89bc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* 3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616 UNKNOWN
* 49c310b28459d5414dce0cfdcba59ea0dac1eb6f UNKNOWN
* 4cf517aa15396f2766bbc35c6b6b59400f1fbebd UNKNOWN
* 19815dab31ee8c45968c2aa0d4b92fd6b3cc587f UNKNOWN
* e486a5f141c8e78e1dc63184e95c1d960b067c5a Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4373)
* b3f0a89eda2a4bb6f6b21edd3133cc77acdf89bc UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2691823842
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
}, {
"hash" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974",
"triggerID" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"triggerType" : "PUSH"
}, {
"hash" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3981",
"triggerID" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"triggerType" : "PUSH"
}, {
"hash" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* 26f9abed586abb54ac4a404dac8a0b62aab1026e Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3981)
* fa3151a7387c16435a20c3f1d0a72a1b9f31e75d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2739524271
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
}, {
"hash" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974",
"triggerID" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"triggerType" : "PUSH"
}, {
"hash" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3981",
"triggerID" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"triggerType" : "PUSH"
}, {
"hash" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3984",
"triggerID" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"triggerType" : "PUSH"
}, {
"hash" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"triggerType" : "PUSH"
}, {
"hash" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4100",
"triggerID" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"triggerType" : "PUSH"
}, {
"hash" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4181",
"triggerID" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"triggerType" : "PUSH"
}, {
"hash" : "16be88870989aa1e54cf03f15ff61654d590e736",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4274",
"triggerID" : "16be88870989aa1e54cf03f15ff61654d590e736",
"triggerType" : "PUSH"
}, {
"hash" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"triggerType" : "PUSH"
}, {
"hash" : "a5227479708c9669e82835056cc515043539114a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4279",
"triggerID" : "a5227479708c9669e82835056cc515043539114a",
"triggerType" : "PUSH"
}, {
"hash" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"triggerType" : "PUSH"
}, {
"hash" : "8ebbd8b0698af9d7dadc71007805e8b6fed1c464",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4280",
"triggerID" : "8ebbd8b0698af9d7dadc71007805e8b6fed1c464",
"triggerType" : "PUSH"
}, {
"hash" : "d711c309c285da81e8a214ebe670c23c01d01f27",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4281",
"triggerID" : "d711c309c285da81e8a214ebe670c23c01d01f27",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* 3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616 UNKNOWN
* 49c310b28459d5414dce0cfdcba59ea0dac1eb6f UNKNOWN
* 4cf517aa15396f2766bbc35c6b6b59400f1fbebd UNKNOWN
* 8ebbd8b0698af9d7dadc71007805e8b6fed1c464 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4280)
* d711c309c285da81e8a214ebe670c23c01d01f27 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4281)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2660466014
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* 2eacf85acc06405bb4a11f54050bcfb647e2691f Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2739677474
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
}, {
"hash" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974",
"triggerID" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"triggerType" : "PUSH"
}, {
"hash" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3981",
"triggerID" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"triggerType" : "PUSH"
}, {
"hash" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3984",
"triggerID" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"triggerType" : "PUSH"
}, {
"hash" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"triggerType" : "PUSH"
}, {
"hash" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4100",
"triggerID" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"triggerType" : "PUSH"
}, {
"hash" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4181",
"triggerID" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"triggerType" : "PUSH"
}, {
"hash" : "16be88870989aa1e54cf03f15ff61654d590e736",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4274",
"triggerID" : "16be88870989aa1e54cf03f15ff61654d590e736",
"triggerType" : "PUSH"
}, {
"hash" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"triggerType" : "PUSH"
}, {
"hash" : "a5227479708c9669e82835056cc515043539114a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4279",
"triggerID" : "a5227479708c9669e82835056cc515043539114a",
"triggerType" : "PUSH"
}, {
"hash" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"triggerType" : "PUSH"
}, {
"hash" : "8ebbd8b0698af9d7dadc71007805e8b6fed1c464",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4280",
"triggerID" : "8ebbd8b0698af9d7dadc71007805e8b6fed1c464",
"triggerType" : "PUSH"
}, {
"hash" : "d711c309c285da81e8a214ebe670c23c01d01f27",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4281",
"triggerID" : "d711c309c285da81e8a214ebe670c23c01d01f27",
"triggerType" : "PUSH"
}, {
"hash" : "1f50e01a909603339fd95c7972af791625d8616d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4282",
"triggerID" : "1f50e01a909603339fd95c7972af791625d8616d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* 3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616 UNKNOWN
* 49c310b28459d5414dce0cfdcba59ea0dac1eb6f UNKNOWN
* 4cf517aa15396f2766bbc35c6b6b59400f1fbebd UNKNOWN
* d711c309c285da81e8a214ebe670c23c01d01f27 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4281)
* 1f50e01a909603339fd95c7972af791625d8616d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4282)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12843:
URL: https://github.com/apache/hudi/pull/12843#discussion_r1999259219
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -143,6 +148,29 @@ public HoodieBaseFileGroupRecordBuffer(HoodieReaderContext<T> readerContext,
} catch (IOException e) {
throw new HoodieIOException("IOException when creating ExternalSpillableMap at " + spillableMapBasePath, e);
}
+ this.shouldCheckCustomDeleteMarker = hasCustomDeleteConfigs(props, readerSchema);
+ if (shouldCheckCustomDeleteMarker) {
+ this.customDeleteMarkerKey = props.getProperty(DELETE_KEY);
+ this.customDeleteMarkerValue = props.getProperty(DELETE_MARKER);
+ }
+ }
+
+ protected boolean hasCustomDeleteConfigs(TypedProperties props, Schema schema) {
+ // DELETE_KEY and DELETE_MARKER both should be set.
+ if (StringUtils.isNullOrEmpty(props.getProperty(DELETE_KEY))
+ || StringUtils.isNullOrEmpty(props.getProperty(DELETE_MARKER))) {
+ return false;
+ }
+ // Schema should have the DELETE_KEY field.
+ String deleteKeyField = props.getProperty(DELETE_KEY);
+ return schema.getField(deleteKeyField) != null;
+ }
+
+ protected boolean isCustomDeleteRecord(T record) {
+ Object deleteMarkerValue =
+ readerContext.getValue(record, readerSchema, customDeleteMarkerKey);
+ return deleteMarkerValue != null
+ && customDeleteMarkerValue.equals(deleteMarkerValue.toString());
Review Comment:
We should document in javadocs that only string typed field can be used for delete marker.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12843:
URL: https://github.com/apache/hudi/pull/12843#discussion_r1999306636
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestHoodieFileGroupRecordBuffer.java:
##########
@@ -53,4 +101,89 @@ private void mockDeleteRecord(DeleteRecord deleteRecord,
Comparable orderingValue) {
when(deleteRecord.getOrderingValue()).thenReturn(orderingValue);
}
+
+ @Test
+ void testHasCustomDeleteConfigs() {
+ KeyBasedFileGroupRecordBuffer keyBasedBuffer =
+ new KeyBasedFileGroupRecordBuffer(
+ readerContext,
+ hoodieTableMetaClient,
+ RecordMergeMode.COMMIT_TIME_ORDERING,
+ partitionNameOverrideOpt,
+ partitionPathFieldOpt,
+ props,
+ readStats);
+
+ assertFalse(keyBasedBuffer.hasCustomDeleteConfigs(props, schema));
+ props.setProperty(DELETE_KEY, "op");
+ assertFalse(keyBasedBuffer.hasCustomDeleteConfigs(props, schema));
+ props.setProperty(DELETE_MARKER, "d");
+ assertThrows(
+ NullPointerException.class,
+ () -> keyBasedBuffer.hasCustomDeleteConfigs(props, null));
+ assertTrue(keyBasedBuffer.hasCustomDeleteConfigs(props, schema));
+ }
+
+ @Test
+ void testIsCustomDeleteRecord() {
+ GenericRecord record = new GenericData.Record(schema);
+ record.put("id", "12345");
+ record.put("ts", System.currentTimeMillis());
+ record.put("op", "d");
+
+ KeyBasedFileGroupRecordBuffer keyBasedBuffer =
+ new KeyBasedFileGroupRecordBuffer(
+ readerContext,
+ hoodieTableMetaClient,
+ RecordMergeMode.COMMIT_TIME_ORDERING,
+ partitionNameOverrideOpt,
+ partitionPathFieldOpt,
+ props,
+ readStats);
+ when(readerContext.getValue(any(), any(), any())).thenReturn(null);
+ assertFalse(keyBasedBuffer.isCustomDeleteRecord(record));
+
+ props.setProperty(DELETE_KEY, "op");
+ props.setProperty(DELETE_MARKER, "d");
+ keyBasedBuffer = new KeyBasedFileGroupRecordBuffer(
+ readerContext,
+ hoodieTableMetaClient,
+ RecordMergeMode.COMMIT_TIME_ORDERING,
+ partitionNameOverrideOpt,
+ partitionPathFieldOpt,
+ props,
+ readStats);
+ when(readerContext.getValue(any(), any(), any())).thenReturn("i");
+ assertFalse(keyBasedBuffer.isCustomDeleteRecord(record));
+ when(readerContext.getValue(any(), any(), any())).thenReturn("d");
+ assertTrue(keyBasedBuffer.isCustomDeleteRecord(record));
+ }
+
+ @Test
+ void testProcessCustomDeleteRecord() {
+ GenericRecord record = new GenericData.Record(schema);
+ record.put("id", "12345");
+ record.put("ts", System.currentTimeMillis());
+ record.put("op", "d");
+
+ KeyBasedFileGroupRecordBuffer keyBasedBuffer =
+ new KeyBasedFileGroupRecordBuffer(
+ readerContext,
+ hoodieTableMetaClient,
+ RecordMergeMode.COMMIT_TIME_ORDERING,
+ partitionNameOverrideOpt,
+ partitionPathFieldOpt,
+ props,
+ readStats);
+
+ Map<String, Object> metadata = new HashMap<>();
+ metadata.put(INTERNAL_META_RECORD_KEY, "12345");
+ metadata.put(INTERNAL_META_PARTITION_PATH, "partition1");
+ when(readerContext.getOrderingValue(any(), any(), any(), any())).thenReturn(1);
+ keyBasedBuffer.processCustomDeleteRecord(record, metadata);
+ Map<Serializable, Pair<Option<GenericRecord>, Map<String, Object>>> records =
+ keyBasedBuffer.getLogRecords();
+ assertEquals(1, records.size());
+ assertTrue(records.containsKey("12345"));
Review Comment:
Could you assert the whole delete record here, i.e., `records.get(recordKey)`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #12843:
URL: https://github.com/apache/hudi/pull/12843#discussion_r2004322820
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/PositionBasedFileGroupRecordBuffer.java:
##########
@@ -135,13 +137,13 @@ public void processDataBlock(HoodieDataBlock dataBlock, Option<KeySpec> keySpecO
}
long recordPosition = recordPositions.get(recordIndex++);
-
T evolvedNextRecord = schemaTransformerWithEvolvedSchema.getLeft().apply(nextRecord);
- processNextDataRecord(
- evolvedNextRecord,
- readerContext.generateMetadataForRecord(evolvedNextRecord, schema),
- recordPosition
- );
+ Map<String, Object> metadata = readerContext.generateMetadataForRecord(evolvedNextRecord, schema);
+ if (shouldCheckCustomDeleteMarker && isCustomDeleteRecord(evolvedNextRecord)) {
+ processCustomDeleteRecord(evolvedNextRecord, metadata, recordPosition);
Review Comment:
I have made some change to the Event time ordering. For CUSTOM, no obvious change. But I do have some ideas to validate or simplify the logic of these doProcessNextDataRecord logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #12843:
URL: https://github.com/apache/hudi/pull/12843#discussion_r2004864672
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/common/table/read/TestCustomDeleteRecords.scala:
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read
+
+import org.apache.hudi.{DataSourceWriteOptions, DefaultSparkRecordMerger, OverwriteWithLatestSparkRecordMerger}
+import org.apache.hudi.DataSourceWriteOptions.{OPERATION, PRECOMBINE_FIELD, RECORDKEY_FIELD, TABLE_TYPE}
+import org.apache.hudi.common.config.{HoodieReaderConfig, HoodieStorageConfig, RecordMergeMode}
+import org.apache.hudi.common.model.{HoodieAvroRecordMerger, HoodieRecordMerger, OverwriteWithLatestMerger}
+import org.apache.hudi.common.model.DefaultHoodieRecordPayload.{DELETE_KEY, DELETE_MARKER}
+import org.apache.hudi.config.{HoodieCompactionConfig, HoodieWriteConfig}
+import org.apache.hudi.testutils.SparkClientFunctionalTestHarness
+
+import org.apache.spark.sql.{Dataset, Row, SaveMode}
+import org.junit.jupiter.api.Assertions.assertTrue
+import org.junit.jupiter.params.ParameterizedTest
+import org.junit.jupiter.params.provider.{Arguments, MethodSource}
+
+class TestCustomDeleteRecords extends SparkClientFunctionalTestHarness {
+ val expectedEventTimeBased: Seq[(Int, String, String, String, Double, String)] = Seq(
+ (10, "5", "rider-E", "driver-E", 17.85, "i"),
+ (10, "3", "rider-C", "driver-C", 33.9, "i"),
+ (10, "2", "rider-B", "driver-B", 27.7, "i"))
+ val expectedCommitTimeBased: Seq[(Int, String, String, String, Double, String)] = Seq(
+ (10, "5", "rider-E", "driver-E", 17.85, "i"),
+ (10, "3", "rider-C", "driver-C", 33.9, "i"))
+
+ @ParameterizedTest
+ @MethodSource(Array("provideParams"))
+ def testCustomDelete(useFgReader: String,
+ tableType: String,
+ recordType: String,
+ positionUsed: String,
+ mergeMode: String): Unit = {
+ val sparkMergeClasses = List(
+ classOf[DefaultSparkRecordMerger].getName,
+ classOf[OverwriteWithLatestSparkRecordMerger].getName).mkString(",")
+ val avroMergerClasses = List(
+ classOf[HoodieAvroRecordMerger].getName,
+ classOf[OverwriteWithLatestMerger].getName).mkString(",")
+
+ val mergeStrategy = if (mergeMode.equals(RecordMergeMode.EVENT_TIME_ORDERING.name)) {
+ HoodieRecordMerger.EVENT_TIME_BASED_MERGE_STRATEGY_UUID
+ } else {
+ HoodieRecordMerger.COMMIT_TIME_BASED_MERGE_STRATEGY_UUID
+ }
+ val mergeOpts: Map[String, String] = Map(
+ HoodieStorageConfig.LOGFILE_DATA_BLOCK_FORMAT.key -> "parquet",
+ HoodieWriteConfig.RECORD_MERGE_IMPL_CLASSES.key ->
+ (if (recordType.equals("SPARK")) sparkMergeClasses else avroMergerClasses),
+ HoodieWriteConfig.RECORD_MERGE_STRATEGY_ID.key -> mergeStrategy)
+ val fgReaderOpts: Map[String, String] = Map(
+ HoodieReaderConfig.FILE_GROUP_READER_ENABLED.key -> useFgReader,
+ HoodieReaderConfig.MERGE_USE_RECORD_POSITIONS.key -> positionUsed,
+ HoodieWriteConfig.RECORD_MERGE_MODE.key -> mergeMode
+ )
+ val deleteOpts: Map[String, String] = Map(
+ DELETE_KEY -> "op", DELETE_MARKER -> "d")
+ val opts = mergeOpts ++ fgReaderOpts ++ deleteOpts
+ val columns = Seq("ts", "key", "rider", "driver", "fare", "op")
+
+ val data = Seq(
+ (10, "1", "rider-A", "driver-A", 19.10, "i"),
+ (10, "2", "rider-B", "driver-B", 27.70, "i"),
+ (10, "3", "rider-C", "driver-C", 33.90, "i"),
+ (10, "4", "rider-D", "driver-D", 34.15, "i"),
+ (10, "5", "rider-E", "driver-E", 17.85, "i"))
+ val inserts = spark.createDataFrame(data).toDF(columns: _*)
+ inserts.write.format("hudi").
+ option(RECORDKEY_FIELD.key(), "key").
+ option(PRECOMBINE_FIELD.key(), "ts").
+ option(TABLE_TYPE.key(), tableType).
+ option(DataSourceWriteOptions.TABLE_NAME.key(), "test_table").
+ option(HoodieCompactionConfig.INLINE_COMPACT.key(), "false").
+ options(opts).
+ mode(SaveMode.Overwrite).
+ save(basePath)
+
+ // Delete using delete markers.
+ val updateData = Seq(
+ (11, "1", "rider-X", "driver-X", 19.10, "d"),
+ (9, "2", "rider-Y", "driver-Y", 27.70, "d"))
+ val updates = spark.createDataFrame(updateData).toDF(columns: _*)
+ updates.write.format("hudi").
+ option(OPERATION.key(), "upsert").
+ option(HoodieCompactionConfig.INLINE_COMPACT.key(), "false").
+ options(opts).
+ mode(SaveMode.Append).
+ save(basePath)
+
+ // Delete from operation.
+ val deletesData = Seq((-5, "4", "rider-D", "driver-D", 34.15, 6))
+ val deletes = spark.createDataFrame(deletesData).toDF(columns: _*)
+ deletes.write.format("hudi").
+ option(OPERATION.key(), "DELETE").
+ option(HoodieCompactionConfig.INLINE_COMPACT.key(), "false").
+ options(opts).
+ mode(SaveMode.Append).
+ save(basePath)
+
+ // Validate in the end.
+ val df = spark.read.format("hudi").options(opts).load(basePath)
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #12843:
URL: https://github.com/apache/hudi/pull/12843#discussion_r2004936799
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/testutils/SparkClientFunctionalTestHarness.java:
##########
@@ -456,11 +458,8 @@ public static boolean areDataframesEqual(Dataset<Row> expectedDf, Dataset<Row> a
Dataset<Row> df1Normalized = expectedDf.selectExpr(sortedColumnNames);
Dataset<Row> df2Normalized = actualDf.selectExpr(sortedColumnNames);
- // Sort rows
- Dataset<Row> df1Sorted = df1Normalized.sort("_row_key");
- Dataset<Row> df2Sorted = df2Normalized.sort("_row_key");
-
Review Comment:
"_row_key" may not exist; removed it. both data frames can be sorted by all columns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2744340348
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
}, {
"hash" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974",
"triggerID" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"triggerType" : "PUSH"
}, {
"hash" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3981",
"triggerID" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"triggerType" : "PUSH"
}, {
"hash" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3984",
"triggerID" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"triggerType" : "PUSH"
}, {
"hash" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"triggerType" : "PUSH"
}, {
"hash" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4100",
"triggerID" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"triggerType" : "PUSH"
}, {
"hash" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4181",
"triggerID" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"triggerType" : "PUSH"
}, {
"hash" : "16be88870989aa1e54cf03f15ff61654d590e736",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4274",
"triggerID" : "16be88870989aa1e54cf03f15ff61654d590e736",
"triggerType" : "PUSH"
}, {
"hash" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"triggerType" : "PUSH"
}, {
"hash" : "a5227479708c9669e82835056cc515043539114a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4279",
"triggerID" : "a5227479708c9669e82835056cc515043539114a",
"triggerType" : "PUSH"
}, {
"hash" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"triggerType" : "PUSH"
}, {
"hash" : "8ebbd8b0698af9d7dadc71007805e8b6fed1c464",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4280",
"triggerID" : "8ebbd8b0698af9d7dadc71007805e8b6fed1c464",
"triggerType" : "PUSH"
}, {
"hash" : "d711c309c285da81e8a214ebe670c23c01d01f27",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4281",
"triggerID" : "d711c309c285da81e8a214ebe670c23c01d01f27",
"triggerType" : "PUSH"
}, {
"hash" : "1f50e01a909603339fd95c7972af791625d8616d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4282",
"triggerID" : "1f50e01a909603339fd95c7972af791625d8616d",
"triggerType" : "PUSH"
}, {
"hash" : "16edc580ea76612ea2a8f0e6243f6ede5fdf9546",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4289",
"triggerID" : "16edc580ea76612ea2a8f0e6243f6ede5fdf9546",
"triggerType" : "PUSH"
}, {
"hash" : "6c6788f3c699fc556ccd25c0f37518fc0cf7b0a5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4318",
"triggerID" : "6c6788f3c699fc556ccd25c0f37518fc0cf7b0a5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* 3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616 UNKNOWN
* 49c310b28459d5414dce0cfdcba59ea0dac1eb6f UNKNOWN
* 4cf517aa15396f2766bbc35c6b6b59400f1fbebd UNKNOWN
* 6c6788f3c699fc556ccd25c0f37518fc0cf7b0a5 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4318)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2741167768
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
}, {
"hash" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974",
"triggerID" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"triggerType" : "PUSH"
}, {
"hash" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3981",
"triggerID" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"triggerType" : "PUSH"
}, {
"hash" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3984",
"triggerID" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"triggerType" : "PUSH"
}, {
"hash" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"triggerType" : "PUSH"
}, {
"hash" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4100",
"triggerID" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"triggerType" : "PUSH"
}, {
"hash" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4181",
"triggerID" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"triggerType" : "PUSH"
}, {
"hash" : "16be88870989aa1e54cf03f15ff61654d590e736",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4274",
"triggerID" : "16be88870989aa1e54cf03f15ff61654d590e736",
"triggerType" : "PUSH"
}, {
"hash" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"triggerType" : "PUSH"
}, {
"hash" : "a5227479708c9669e82835056cc515043539114a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4279",
"triggerID" : "a5227479708c9669e82835056cc515043539114a",
"triggerType" : "PUSH"
}, {
"hash" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"triggerType" : "PUSH"
}, {
"hash" : "8ebbd8b0698af9d7dadc71007805e8b6fed1c464",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4280",
"triggerID" : "8ebbd8b0698af9d7dadc71007805e8b6fed1c464",
"triggerType" : "PUSH"
}, {
"hash" : "d711c309c285da81e8a214ebe670c23c01d01f27",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4281",
"triggerID" : "d711c309c285da81e8a214ebe670c23c01d01f27",
"triggerType" : "PUSH"
}, {
"hash" : "1f50e01a909603339fd95c7972af791625d8616d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4282",
"triggerID" : "1f50e01a909603339fd95c7972af791625d8616d",
"triggerType" : "PUSH"
}, {
"hash" : "16edc580ea76612ea2a8f0e6243f6ede5fdf9546",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4289",
"triggerID" : "16edc580ea76612ea2a8f0e6243f6ede5fdf9546",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* 3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616 UNKNOWN
* 49c310b28459d5414dce0cfdcba59ea0dac1eb6f UNKNOWN
* 4cf517aa15396f2766bbc35c6b6b59400f1fbebd UNKNOWN
* 1f50e01a909603339fd95c7972af791625d8616d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4282)
* 16edc580ea76612ea2a8f0e6243f6ede5fdf9546 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4289)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2708463448
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
}, {
"hash" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974",
"triggerID" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"triggerType" : "PUSH"
}, {
"hash" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3981",
"triggerID" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"triggerType" : "PUSH"
}, {
"hash" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3984",
"triggerID" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"triggerType" : "PUSH"
}, {
"hash" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"triggerType" : "PUSH"
}, {
"hash" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4100",
"triggerID" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* 3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616 UNKNOWN
* e888cdfc0f2bf40a5422d9290b8dbeb9e643d564 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4100)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2739418174
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
}, {
"hash" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974",
"triggerID" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"triggerType" : "PUSH"
}, {
"hash" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3981",
"triggerID" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"triggerType" : "PUSH"
}, {
"hash" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3984",
"triggerID" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"triggerType" : "PUSH"
}, {
"hash" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"triggerType" : "PUSH"
}, {
"hash" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4100",
"triggerID" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"triggerType" : "PUSH"
}, {
"hash" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4181",
"triggerID" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"triggerType" : "PUSH"
}, {
"hash" : "16be88870989aa1e54cf03f15ff61654d590e736",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4274",
"triggerID" : "16be88870989aa1e54cf03f15ff61654d590e736",
"triggerType" : "PUSH"
}, {
"hash" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"triggerType" : "PUSH"
}, {
"hash" : "a5227479708c9669e82835056cc515043539114a",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4279",
"triggerID" : "a5227479708c9669e82835056cc515043539114a",
"triggerType" : "PUSH"
}, {
"hash" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"triggerType" : "PUSH"
}, {
"hash" : "8ebbd8b0698af9d7dadc71007805e8b6fed1c464",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4280",
"triggerID" : "8ebbd8b0698af9d7dadc71007805e8b6fed1c464",
"triggerType" : "PUSH"
}, {
"hash" : "d711c309c285da81e8a214ebe670c23c01d01f27",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d711c309c285da81e8a214ebe670c23c01d01f27",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* 3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616 UNKNOWN
* 49c310b28459d5414dce0cfdcba59ea0dac1eb6f UNKNOWN
* a5227479708c9669e82835056cc515043539114a Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4279)
* 4cf517aa15396f2766bbc35c6b6b59400f1fbebd UNKNOWN
* 8ebbd8b0698af9d7dadc71007805e8b6fed1c464 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4280)
* d711c309c285da81e8a214ebe670c23c01d01f27 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #12843:
URL: https://github.com/apache/hudi/pull/12843#discussion_r2004342735
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestHoodieFileGroupRecordBuffer.java:
##########
@@ -53,4 +101,89 @@ private void mockDeleteRecord(DeleteRecord deleteRecord,
Comparable orderingValue) {
when(deleteRecord.getOrderingValue()).thenReturn(orderingValue);
}
+
+ @Test
+ void testHasCustomDeleteConfigs() {
+ KeyBasedFileGroupRecordBuffer keyBasedBuffer =
+ new KeyBasedFileGroupRecordBuffer(
+ readerContext,
+ hoodieTableMetaClient,
+ RecordMergeMode.COMMIT_TIME_ORDERING,
+ partitionNameOverrideOpt,
+ partitionPathFieldOpt,
+ props,
+ readStats);
+
+ assertFalse(keyBasedBuffer.hasCustomDeleteConfigs(props, schema));
+ props.setProperty(DELETE_KEY, "op");
+ assertFalse(keyBasedBuffer.hasCustomDeleteConfigs(props, schema));
+ props.setProperty(DELETE_MARKER, "d");
+ assertThrows(
+ NullPointerException.class,
+ () -> keyBasedBuffer.hasCustomDeleteConfigs(props, null));
+ assertTrue(keyBasedBuffer.hasCustomDeleteConfigs(props, schema));
+ }
+
+ @Test
+ void testIsCustomDeleteRecord() {
+ GenericRecord record = new GenericData.Record(schema);
+ record.put("id", "12345");
+ record.put("ts", System.currentTimeMillis());
+ record.put("op", "d");
+
+ KeyBasedFileGroupRecordBuffer keyBasedBuffer =
+ new KeyBasedFileGroupRecordBuffer(
+ readerContext,
+ hoodieTableMetaClient,
+ RecordMergeMode.COMMIT_TIME_ORDERING,
+ partitionNameOverrideOpt,
+ partitionPathFieldOpt,
+ props,
+ readStats);
+ when(readerContext.getValue(any(), any(), any())).thenReturn(null);
+ assertFalse(keyBasedBuffer.isCustomDeleteRecord(record));
+
+ props.setProperty(DELETE_KEY, "op");
+ props.setProperty(DELETE_MARKER, "d");
+ keyBasedBuffer = new KeyBasedFileGroupRecordBuffer(
+ readerContext,
+ hoodieTableMetaClient,
+ RecordMergeMode.COMMIT_TIME_ORDERING,
+ partitionNameOverrideOpt,
+ partitionPathFieldOpt,
+ props,
+ readStats);
+ when(readerContext.getValue(any(), any(), any())).thenReturn("i");
+ assertFalse(keyBasedBuffer.isCustomDeleteRecord(record));
+ when(readerContext.getValue(any(), any(), any())).thenReturn("d");
+ assertTrue(keyBasedBuffer.isCustomDeleteRecord(record));
+ }
+
+ @Test
+ void testProcessCustomDeleteRecord() {
+ GenericRecord record = new GenericData.Record(schema);
+ record.put("id", "12345");
+ record.put("ts", System.currentTimeMillis());
+ record.put("op", "d");
+
+ KeyBasedFileGroupRecordBuffer keyBasedBuffer =
+ new KeyBasedFileGroupRecordBuffer(
+ readerContext,
+ hoodieTableMetaClient,
+ RecordMergeMode.COMMIT_TIME_ORDERING,
+ partitionNameOverrideOpt,
+ partitionPathFieldOpt,
+ props,
+ readStats);
+
+ Map<String, Object> metadata = new HashMap<>();
+ metadata.put(INTERNAL_META_RECORD_KEY, "12345");
+ metadata.put(INTERNAL_META_PARTITION_PATH, "partition1");
+ when(readerContext.getOrderingValue(any(), any(), any(), any())).thenReturn(1);
+ keyBasedBuffer.processCustomDeleteRecord(record, metadata);
+ Map<Serializable, Pair<Option<GenericRecord>, Map<String, Object>>> records =
+ keyBasedBuffer.getLogRecords();
+ assertEquals(1, records.size());
+ assertTrue(records.containsKey("12345"));
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #12843:
URL: https://github.com/apache/hudi/pull/12843#discussion_r2004329500
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestHoodieFileGroupRecordBuffer.java:
##########
@@ -19,21 +19,69 @@
package org.apache.hudi.common.table.read;
+import org.apache.hudi.common.config.RecordMergeMode;
+import org.apache.hudi.common.config.TypedProperties;
import org.apache.hudi.common.engine.HoodieReaderContext;
import org.apache.hudi.common.model.DeleteRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
+import java.io.Serializable;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.hudi.common.engine.HoodieReaderContext.INTERNAL_META_PARTITION_PATH;
+import static org.apache.hudi.common.engine.HoodieReaderContext.INTERNAL_META_RECORD_KEY;
+import static org.apache.hudi.common.model.DefaultHoodieRecordPayload.DELETE_KEY;
+import static org.apache.hudi.common.model.DefaultHoodieRecordPayload.DELETE_MARKER;
import static org.apache.hudi.common.model.HoodieRecord.DEFAULT_ORDERING_VALUE;
-import static org.apache.hudi.common.table.read.HoodieBaseFileGroupRecordBuffer.getOrderingValue;
+import static org.apache.hudi.common.table.read.FileGroupRecordBuffer.getOrderingValue;
import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
/**
- * Tests {@link HoodieBaseFileGroupRecordBuffer}
+ * Tests {@link FileGroupRecordBuffer}
*/
public class TestHoodieFileGroupRecordBuffer {
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #12843:
URL: https://github.com/apache/hudi/pull/12843#discussion_r2004507774
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/common/table/read/TestCustomDeleteRecords.scala:
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read
+
+import org.apache.hudi.{DataSourceWriteOptions, DefaultSparkRecordMerger, OverwriteWithLatestSparkRecordMerger}
+import org.apache.hudi.DataSourceWriteOptions.{OPERATION, PRECOMBINE_FIELD, RECORDKEY_FIELD, TABLE_TYPE}
+import org.apache.hudi.common.config.{HoodieReaderConfig, HoodieStorageConfig, RecordMergeMode}
+import org.apache.hudi.common.model.{HoodieAvroRecordMerger, HoodieRecordMerger, OverwriteWithLatestMerger}
+import org.apache.hudi.common.model.DefaultHoodieRecordPayload.{DELETE_KEY, DELETE_MARKER}
+import org.apache.hudi.config.{HoodieCompactionConfig, HoodieWriteConfig}
+import org.apache.hudi.testutils.SparkClientFunctionalTestHarness
+
+import org.apache.spark.sql.{Dataset, Row, SaveMode}
+import org.junit.jupiter.api.Assertions.assertTrue
+import org.junit.jupiter.params.ParameterizedTest
+import org.junit.jupiter.params.provider.{Arguments, MethodSource}
+
+class TestCustomDeleteRecords extends SparkClientFunctionalTestHarness {
+ val expectedEventTimeBased: Seq[(Int, String, String, String, Double, String)] = Seq(
+ (10, "5", "rider-E", "driver-E", 17.85, "i"),
+ (10, "3", "rider-C", "driver-C", 33.9, "i"),
+ (10, "2", "rider-B", "driver-B", 27.7, "i"))
+ val expectedCommitTimeBased: Seq[(Int, String, String, String, Double, String)] = Seq(
+ (10, "5", "rider-E", "driver-E", 17.85, "i"),
+ (10, "3", "rider-C", "driver-C", 33.9, "i"))
+
+ @ParameterizedTest
+ @MethodSource(Array("provideParams"))
+ def testCustomDelete(useFgReader: String,
+ tableType: String,
+ recordType: String,
+ positionUsed: String,
+ mergeMode: String): Unit = {
+ val sparkMergeClasses = List(
+ classOf[DefaultSparkRecordMerger].getName,
+ classOf[OverwriteWithLatestSparkRecordMerger].getName).mkString(",")
+ val avroMergerClasses = List(
+ classOf[HoodieAvroRecordMerger].getName,
+ classOf[OverwriteWithLatestMerger].getName).mkString(",")
+
+ val mergeStrategy = if (mergeMode.equals(RecordMergeMode.EVENT_TIME_ORDERING.name)) {
+ HoodieRecordMerger.EVENT_TIME_BASED_MERGE_STRATEGY_UUID
+ } else {
+ HoodieRecordMerger.COMMIT_TIME_BASED_MERGE_STRATEGY_UUID
+ }
+ val mergeOpts: Map[String, String] = Map(
+ HoodieStorageConfig.LOGFILE_DATA_BLOCK_FORMAT.key -> "parquet",
Review Comment:
Not quite relevant. Let me remove it at all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #12843:
URL: https://github.com/apache/hudi/pull/12843#discussion_r2004327818
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/UnmergedHoodieFileGroupRecordBuffer.java:
##########
@@ -41,12 +41,12 @@
import java.util.Iterator;
import java.util.Map;
-public class HoodieUnmergedFileGroupRecordBuffer<T> extends HoodieBaseFileGroupRecordBuffer<T> {
+public class UnmergedHoodieFileGroupRecordBuffer<T> extends FileGroupRecordBuffer<T> {
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12843:
URL: https://github.com/apache/hudi/pull/12843#discussion_r2011189639
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestHoodieFileGroupRecordBuffer.java:
##########
@@ -53,4 +101,89 @@ private void mockDeleteRecord(DeleteRecord deleteRecord,
Comparable orderingValue) {
when(deleteRecord.getOrderingValue()).thenReturn(orderingValue);
}
+
+ @Test
+ void testHasCustomDeleteConfigs() {
+ KeyBasedFileGroupRecordBuffer keyBasedBuffer =
+ new KeyBasedFileGroupRecordBuffer(
+ readerContext,
+ hoodieTableMetaClient,
+ RecordMergeMode.COMMIT_TIME_ORDERING,
+ partitionNameOverrideOpt,
+ partitionPathFieldOpt,
+ props,
+ readStats);
+
+ assertFalse(keyBasedBuffer.hasCustomDeleteConfigs(props, schema));
+ props.setProperty(DELETE_KEY, "op");
+ assertFalse(keyBasedBuffer.hasCustomDeleteConfigs(props, schema));
+ props.setProperty(DELETE_MARKER, "d");
+ assertThrows(
+ NullPointerException.class,
+ () -> keyBasedBuffer.hasCustomDeleteConfigs(props, null));
+ assertTrue(keyBasedBuffer.hasCustomDeleteConfigs(props, schema));
+ }
+
+ @Test
+ void testIsCustomDeleteRecord() {
+ GenericRecord record = new GenericData.Record(schema);
+ record.put("id", "12345");
+ record.put("ts", System.currentTimeMillis());
+ record.put("op", "d");
+
+ KeyBasedFileGroupRecordBuffer keyBasedBuffer =
+ new KeyBasedFileGroupRecordBuffer(
+ readerContext,
+ hoodieTableMetaClient,
+ RecordMergeMode.COMMIT_TIME_ORDERING,
+ partitionNameOverrideOpt,
+ partitionPathFieldOpt,
+ props,
+ readStats);
+ when(readerContext.getValue(any(), any(), any())).thenReturn(null);
+ assertFalse(keyBasedBuffer.isCustomDeleteRecord(record));
+
+ props.setProperty(DELETE_KEY, "op");
+ props.setProperty(DELETE_MARKER, "d");
+ keyBasedBuffer = new KeyBasedFileGroupRecordBuffer(
+ readerContext,
+ hoodieTableMetaClient,
+ RecordMergeMode.COMMIT_TIME_ORDERING,
+ partitionNameOverrideOpt,
+ partitionPathFieldOpt,
+ props,
+ readStats);
+ when(readerContext.getValue(any(), any(), any())).thenReturn("i");
+ assertFalse(keyBasedBuffer.isCustomDeleteRecord(record));
+ when(readerContext.getValue(any(), any(), any())).thenReturn("d");
+ assertTrue(keyBasedBuffer.isCustomDeleteRecord(record));
+ }
+
+ @Test
+ void testProcessCustomDeleteRecord() {
+ GenericRecord record = new GenericData.Record(schema);
+ record.put("id", "12345");
+ record.put("ts", System.currentTimeMillis());
+ record.put("op", "d");
+
+ KeyBasedFileGroupRecordBuffer keyBasedBuffer =
+ new KeyBasedFileGroupRecordBuffer(
+ readerContext,
+ hoodieTableMetaClient,
+ RecordMergeMode.COMMIT_TIME_ORDERING,
+ partitionNameOverrideOpt,
+ partitionPathFieldOpt,
+ props,
+ readStats);
+
+ Map<String, Object> metadata = new HashMap<>();
+ metadata.put(INTERNAL_META_RECORD_KEY, "12345");
+ metadata.put(INTERNAL_META_PARTITION_PATH, "partition1");
+ when(readerContext.getOrderingValue(any(), any(), any(), any())).thenReturn(1);
+ keyBasedBuffer.processCustomDeleteRecord(record, metadata);
+ Map<Serializable, Pair<Option<GenericRecord>, Map<String, Object>>> records =
+ keyBasedBuffer.getLogRecords();
+ assertEquals(1, records.size());
+ assertTrue(records.containsKey("12345"));
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2749940805
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
}, {
"hash" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974",
"triggerID" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"triggerType" : "PUSH"
}, {
"hash" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3981",
"triggerID" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"triggerType" : "PUSH"
}, {
"hash" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3984",
"triggerID" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"triggerType" : "PUSH"
}, {
"hash" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"triggerType" : "PUSH"
}, {
"hash" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4100",
"triggerID" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"triggerType" : "PUSH"
}, {
"hash" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4181",
"triggerID" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"triggerType" : "PUSH"
}, {
"hash" : "16be88870989aa1e54cf03f15ff61654d590e736",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4274",
"triggerID" : "16be88870989aa1e54cf03f15ff61654d590e736",
"triggerType" : "PUSH"
}, {
"hash" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"triggerType" : "PUSH"
}, {
"hash" : "a5227479708c9669e82835056cc515043539114a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4279",
"triggerID" : "a5227479708c9669e82835056cc515043539114a",
"triggerType" : "PUSH"
}, {
"hash" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"triggerType" : "PUSH"
}, {
"hash" : "8ebbd8b0698af9d7dadc71007805e8b6fed1c464",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4280",
"triggerID" : "8ebbd8b0698af9d7dadc71007805e8b6fed1c464",
"triggerType" : "PUSH"
}, {
"hash" : "d711c309c285da81e8a214ebe670c23c01d01f27",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4281",
"triggerID" : "d711c309c285da81e8a214ebe670c23c01d01f27",
"triggerType" : "PUSH"
}, {
"hash" : "1f50e01a909603339fd95c7972af791625d8616d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4282",
"triggerID" : "1f50e01a909603339fd95c7972af791625d8616d",
"triggerType" : "PUSH"
}, {
"hash" : "16edc580ea76612ea2a8f0e6243f6ede5fdf9546",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4289",
"triggerID" : "16edc580ea76612ea2a8f0e6243f6ede5fdf9546",
"triggerType" : "PUSH"
}, {
"hash" : "6c6788f3c699fc556ccd25c0f37518fc0cf7b0a5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4318",
"triggerID" : "6c6788f3c699fc556ccd25c0f37518fc0cf7b0a5",
"triggerType" : "PUSH"
}, {
"hash" : "dd0c492908ae259b4df9c537f2c518628756d6aa",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4369",
"triggerID" : "dd0c492908ae259b4df9c537f2c518628756d6aa",
"triggerType" : "PUSH"
}, {
"hash" : "19815dab31ee8c45968c2aa0d4b92fd6b3cc587f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "19815dab31ee8c45968c2aa0d4b92fd6b3cc587f",
"triggerType" : "PUSH"
}, {
"hash" : "e486a5f141c8e78e1dc63184e95c1d960b067c5a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e486a5f141c8e78e1dc63184e95c1d960b067c5a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* 3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616 UNKNOWN
* 49c310b28459d5414dce0cfdcba59ea0dac1eb6f UNKNOWN
* 4cf517aa15396f2766bbc35c6b6b59400f1fbebd UNKNOWN
* dd0c492908ae259b4df9c537f2c518628756d6aa Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4369)
* 19815dab31ee8c45968c2aa0d4b92fd6b3cc587f UNKNOWN
* e486a5f141c8e78e1dc63184e95c1d960b067c5a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2749986898
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
}, {
"hash" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974",
"triggerID" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"triggerType" : "PUSH"
}, {
"hash" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3981",
"triggerID" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"triggerType" : "PUSH"
}, {
"hash" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3984",
"triggerID" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"triggerType" : "PUSH"
}, {
"hash" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"triggerType" : "PUSH"
}, {
"hash" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4100",
"triggerID" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"triggerType" : "PUSH"
}, {
"hash" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4181",
"triggerID" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"triggerType" : "PUSH"
}, {
"hash" : "16be88870989aa1e54cf03f15ff61654d590e736",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4274",
"triggerID" : "16be88870989aa1e54cf03f15ff61654d590e736",
"triggerType" : "PUSH"
}, {
"hash" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"triggerType" : "PUSH"
}, {
"hash" : "a5227479708c9669e82835056cc515043539114a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4279",
"triggerID" : "a5227479708c9669e82835056cc515043539114a",
"triggerType" : "PUSH"
}, {
"hash" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"triggerType" : "PUSH"
}, {
"hash" : "8ebbd8b0698af9d7dadc71007805e8b6fed1c464",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4280",
"triggerID" : "8ebbd8b0698af9d7dadc71007805e8b6fed1c464",
"triggerType" : "PUSH"
}, {
"hash" : "d711c309c285da81e8a214ebe670c23c01d01f27",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4281",
"triggerID" : "d711c309c285da81e8a214ebe670c23c01d01f27",
"triggerType" : "PUSH"
}, {
"hash" : "1f50e01a909603339fd95c7972af791625d8616d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4282",
"triggerID" : "1f50e01a909603339fd95c7972af791625d8616d",
"triggerType" : "PUSH"
}, {
"hash" : "16edc580ea76612ea2a8f0e6243f6ede5fdf9546",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4289",
"triggerID" : "16edc580ea76612ea2a8f0e6243f6ede5fdf9546",
"triggerType" : "PUSH"
}, {
"hash" : "6c6788f3c699fc556ccd25c0f37518fc0cf7b0a5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4318",
"triggerID" : "6c6788f3c699fc556ccd25c0f37518fc0cf7b0a5",
"triggerType" : "PUSH"
}, {
"hash" : "dd0c492908ae259b4df9c537f2c518628756d6aa",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4369",
"triggerID" : "dd0c492908ae259b4df9c537f2c518628756d6aa",
"triggerType" : "PUSH"
}, {
"hash" : "19815dab31ee8c45968c2aa0d4b92fd6b3cc587f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "19815dab31ee8c45968c2aa0d4b92fd6b3cc587f",
"triggerType" : "PUSH"
}, {
"hash" : "e486a5f141c8e78e1dc63184e95c1d960b067c5a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4373",
"triggerID" : "e486a5f141c8e78e1dc63184e95c1d960b067c5a",
"triggerType" : "PUSH"
}, {
"hash" : "b3f0a89eda2a4bb6f6b21edd3133cc77acdf89bc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b3f0a89eda2a4bb6f6b21edd3133cc77acdf89bc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* 3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616 UNKNOWN
* 49c310b28459d5414dce0cfdcba59ea0dac1eb6f UNKNOWN
* 4cf517aa15396f2766bbc35c6b6b59400f1fbebd UNKNOWN
* dd0c492908ae259b4df9c537f2c518628756d6aa Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4369)
* 19815dab31ee8c45968c2aa0d4b92fd6b3cc587f UNKNOWN
* e486a5f141c8e78e1dc63184e95c1d960b067c5a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4373)
* b3f0a89eda2a4bb6f6b21edd3133cc77acdf89bc UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2691781161
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
}, {
"hash" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974",
"triggerID" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"triggerType" : "PUSH"
}, {
"hash" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* 7a82eff1dfdbc4f4e6181b689218a7726a32f03b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974)
* 26f9abed586abb54ac4a404dac8a0b62aab1026e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2707528365
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
}, {
"hash" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974",
"triggerID" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"triggerType" : "PUSH"
}, {
"hash" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3981",
"triggerID" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"triggerType" : "PUSH"
}, {
"hash" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3984",
"triggerID" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"triggerType" : "PUSH"
}, {
"hash" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"triggerType" : "PUSH"
}, {
"hash" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* fa3151a7387c16435a20c3f1d0a72a1b9f31e75d Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3984)
* 3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616 UNKNOWN
* e888cdfc0f2bf40a5422d9290b8dbeb9e643d564 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12843:
URL: https://github.com/apache/hudi/pull/12843#discussion_r1999213711
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -139,26 +139,26 @@ public HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
/**
* Initialize correct record buffer
*/
- private static HoodieFileGroupRecordBuffer getRecordBuffer(HoodieReaderContext readerContext,
- HoodieTableMetaClient hoodieTableMetaClient,
- RecordMergeMode recordMergeMode,
- TypedProperties props,
- Option<HoodieBaseFile> baseFileOption,
- boolean hasNoLogFiles,
- boolean isSkipMerge,
- boolean shouldUseRecordPosition,
- HoodieReadStats readStats) {
+ private static FileGroupRecordBuffer getRecordBuffer(HoodieReaderContext readerContext,
+ HoodieTableMetaClient hoodieTableMetaClient,
+ RecordMergeMode recordMergeMode,
+ TypedProperties props,
+ Option<HoodieBaseFile> baseFileOption,
+ boolean hasNoLogFiles,
+ boolean isSkipMerge,
+ boolean shouldUseRecordPosition,
+ HoodieReadStats readStats) {
if (hasNoLogFiles) {
return null;
} else if (isSkipMerge) {
- return new HoodieUnmergedFileGroupRecordBuffer<>(
+ return new UnmergedHoodieFileGroupRecordBuffer<>(
Review Comment:
```suggestion
return new UnmergedFileGroupRecordBuffer<>(
```
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/UnmergedHoodieFileGroupRecordBuffer.java:
##########
@@ -41,12 +41,12 @@
import java.util.Iterator;
import java.util.Map;
-public class HoodieUnmergedFileGroupRecordBuffer<T> extends HoodieBaseFileGroupRecordBuffer<T> {
+public class UnmergedHoodieFileGroupRecordBuffer<T> extends FileGroupRecordBuffer<T> {
Review Comment:
```suggestion
public class UnmergedFileGroupRecordBuffer<T> extends FileGroupRecordBuffer<T> {
```
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -143,6 +148,29 @@ public HoodieBaseFileGroupRecordBuffer(HoodieReaderContext<T> readerContext,
} catch (IOException e) {
throw new HoodieIOException("IOException when creating ExternalSpillableMap at " + spillableMapBasePath, e);
}
+ this.shouldCheckCustomDeleteMarker = hasCustomDeleteConfigs(props, readerSchema);
+ if (shouldCheckCustomDeleteMarker) {
+ this.customDeleteMarkerKey = props.getProperty(DELETE_KEY);
+ this.customDeleteMarkerValue = props.getProperty(DELETE_MARKER);
+ }
+ }
+
+ protected boolean hasCustomDeleteConfigs(TypedProperties props, Schema schema) {
+ // DELETE_KEY and DELETE_MARKER both should be set.
+ if (StringUtils.isNullOrEmpty(props.getProperty(DELETE_KEY))
Review Comment:
nit: fetch `props.getProperty(DELETE_KEY)` once instead of calling `props.getProperty(DELETE_KEY)` twice in this method
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -143,6 +148,29 @@ public HoodieBaseFileGroupRecordBuffer(HoodieReaderContext<T> readerContext,
} catch (IOException e) {
throw new HoodieIOException("IOException when creating ExternalSpillableMap at " + spillableMapBasePath, e);
}
+ this.shouldCheckCustomDeleteMarker = hasCustomDeleteConfigs(props, readerSchema);
+ if (shouldCheckCustomDeleteMarker) {
+ this.customDeleteMarkerKey = props.getProperty(DELETE_KEY);
+ this.customDeleteMarkerValue = props.getProperty(DELETE_MARKER);
+ }
Review Comment:
Could we have these assigning delete marker member variables in a single method?
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -143,6 +148,29 @@ public HoodieBaseFileGroupRecordBuffer(HoodieReaderContext<T> readerContext,
} catch (IOException e) {
throw new HoodieIOException("IOException when creating ExternalSpillableMap at " + spillableMapBasePath, e);
}
+ this.shouldCheckCustomDeleteMarker = hasCustomDeleteConfigs(props, readerSchema);
+ if (shouldCheckCustomDeleteMarker) {
+ this.customDeleteMarkerKey = props.getProperty(DELETE_KEY);
+ this.customDeleteMarkerValue = props.getProperty(DELETE_MARKER);
+ }
+ }
+
+ protected boolean hasCustomDeleteConfigs(TypedProperties props, Schema schema) {
+ // DELETE_KEY and DELETE_MARKER both should be set.
+ if (StringUtils.isNullOrEmpty(props.getProperty(DELETE_KEY))
+ || StringUtils.isNullOrEmpty(props.getProperty(DELETE_MARKER))) {
+ return false;
Review Comment:
Log a warning message if `DELETE_KEY` is set but `DELETE_MARKER` is not set.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12843:
URL: https://github.com/apache/hudi/pull/12843#discussion_r1999301020
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestHoodieFileGroupRecordBuffer.java:
##########
@@ -19,21 +19,69 @@
package org.apache.hudi.common.table.read;
+import org.apache.hudi.common.config.RecordMergeMode;
+import org.apache.hudi.common.config.TypedProperties;
import org.apache.hudi.common.engine.HoodieReaderContext;
import org.apache.hudi.common.model.DeleteRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
+import java.io.Serializable;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.hudi.common.engine.HoodieReaderContext.INTERNAL_META_PARTITION_PATH;
+import static org.apache.hudi.common.engine.HoodieReaderContext.INTERNAL_META_RECORD_KEY;
+import static org.apache.hudi.common.model.DefaultHoodieRecordPayload.DELETE_KEY;
+import static org.apache.hudi.common.model.DefaultHoodieRecordPayload.DELETE_MARKER;
import static org.apache.hudi.common.model.HoodieRecord.DEFAULT_ORDERING_VALUE;
-import static org.apache.hudi.common.table.read.HoodieBaseFileGroupRecordBuffer.getOrderingValue;
+import static org.apache.hudi.common.table.read.FileGroupRecordBuffer.getOrderingValue;
import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
/**
- * Tests {@link HoodieBaseFileGroupRecordBuffer}
+ * Tests {@link FileGroupRecordBuffer}
*/
public class TestHoodieFileGroupRecordBuffer {
Review Comment:
Rename the test classes accordingly.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/PositionBasedFileGroupRecordBuffer.java:
##########
@@ -61,23 +63,23 @@
* Here the position means that record position in the base file. The records from the base file is accessed from an iterator object. These records are merged when the
* {@link #hasNext} method is called.
*/
-public class HoodiePositionBasedFileGroupRecordBuffer<T> extends HoodieKeyBasedFileGroupRecordBuffer<T> {
- private static final Logger LOG = LoggerFactory.getLogger(HoodiePositionBasedFileGroupRecordBuffer.class);
+public class PositionBasedFileGroupRecordBuffer<T> extends KeyBasedFileGroupRecordBuffer<T> {
Review Comment:
Let's file a follow-up JIRA ticket to revisit the hierarchy of `PositionBasedFileGroupRecordBuffer` and `KeyBasedFileGroupRecordBuffer` and make sure that position-based file group record buffer does not accidentally override shared logic (e.g., extracting common logic and make them final).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12843:
URL: https://github.com/apache/hudi/pull/12843#discussion_r1999310157
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/common/table/read/TestCustomDeleteRecords.scala:
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read
+
+import org.apache.hudi.{DataSourceWriteOptions, DefaultSparkRecordMerger, OverwriteWithLatestSparkRecordMerger}
+import org.apache.hudi.DataSourceWriteOptions.{OPERATION, PRECOMBINE_FIELD, RECORDKEY_FIELD, TABLE_TYPE}
+import org.apache.hudi.common.config.{HoodieReaderConfig, HoodieStorageConfig, RecordMergeMode}
+import org.apache.hudi.common.model.{HoodieAvroRecordMerger, HoodieRecordMerger, OverwriteWithLatestMerger}
+import org.apache.hudi.common.model.DefaultHoodieRecordPayload.{DELETE_KEY, DELETE_MARKER}
+import org.apache.hudi.config.{HoodieCompactionConfig, HoodieWriteConfig}
+import org.apache.hudi.testutils.SparkClientFunctionalTestHarness
+
+import org.apache.spark.sql.{Dataset, Row, SaveMode}
+import org.junit.jupiter.api.Assertions.assertTrue
+import org.junit.jupiter.params.ParameterizedTest
+import org.junit.jupiter.params.provider.{Arguments, MethodSource}
+
+class TestCustomDeleteRecords extends SparkClientFunctionalTestHarness {
+ val expectedEventTimeBased: Seq[(Int, String, String, String, Double, String)] = Seq(
+ (10, "5", "rider-E", "driver-E", 17.85, "i"),
+ (10, "3", "rider-C", "driver-C", 33.9, "i"),
+ (10, "2", "rider-B", "driver-B", 27.7, "i"))
+ val expectedCommitTimeBased: Seq[(Int, String, String, String, Double, String)] = Seq(
+ (10, "5", "rider-E", "driver-E", 17.85, "i"),
+ (10, "3", "rider-C", "driver-C", 33.9, "i"))
+
+ @ParameterizedTest
+ @MethodSource(Array("provideParams"))
+ def testCustomDelete(useFgReader: String,
+ tableType: String,
+ recordType: String,
+ positionUsed: String,
+ mergeMode: String): Unit = {
+ val sparkMergeClasses = List(
+ classOf[DefaultSparkRecordMerger].getName,
+ classOf[OverwriteWithLatestSparkRecordMerger].getName).mkString(",")
+ val avroMergerClasses = List(
+ classOf[HoodieAvroRecordMerger].getName,
+ classOf[OverwriteWithLatestMerger].getName).mkString(",")
+
+ val mergeStrategy = if (mergeMode.equals(RecordMergeMode.EVENT_TIME_ORDERING.name)) {
+ HoodieRecordMerger.EVENT_TIME_BASED_MERGE_STRATEGY_UUID
+ } else {
+ HoodieRecordMerger.COMMIT_TIME_BASED_MERGE_STRATEGY_UUID
+ }
+ val mergeOpts: Map[String, String] = Map(
+ HoodieStorageConfig.LOGFILE_DATA_BLOCK_FORMAT.key -> "parquet",
+ HoodieWriteConfig.RECORD_MERGE_IMPL_CLASSES.key ->
+ (if (recordType.equals("SPARK")) sparkMergeClasses else avroMergerClasses),
+ HoodieWriteConfig.RECORD_MERGE_STRATEGY_ID.key -> mergeStrategy)
+ val fgReaderOpts: Map[String, String] = Map(
+ HoodieReaderConfig.FILE_GROUP_READER_ENABLED.key -> useFgReader,
+ HoodieReaderConfig.MERGE_USE_RECORD_POSITIONS.key -> positionUsed,
+ HoodieWriteConfig.RECORD_MERGE_MODE.key -> mergeMode
+ )
+ val deleteOpts: Map[String, String] = Map(
+ DELETE_KEY -> "op", DELETE_MARKER -> "d")
+ val opts = mergeOpts ++ fgReaderOpts ++ deleteOpts
+ val columns = Seq("ts", "key", "rider", "driver", "fare", "op")
+
+ val data = Seq(
+ (10, "1", "rider-A", "driver-A", 19.10, "i"),
+ (10, "2", "rider-B", "driver-B", 27.70, "i"),
+ (10, "3", "rider-C", "driver-C", 33.90, "i"),
+ (10, "4", "rider-D", "driver-D", 34.15, "i"),
+ (10, "5", "rider-E", "driver-E", 17.85, "i"))
+ val inserts = spark.createDataFrame(data).toDF(columns: _*)
+ inserts.write.format("hudi").
+ option(RECORDKEY_FIELD.key(), "key").
+ option(PRECOMBINE_FIELD.key(), "ts").
+ option(TABLE_TYPE.key(), tableType).
+ option(DataSourceWriteOptions.TABLE_NAME.key(), "test_table").
+ option(HoodieCompactionConfig.INLINE_COMPACT.key(), "false").
+ options(opts).
+ mode(SaveMode.Overwrite).
+ save(basePath)
+
+ // Delete using delete markers.
+ val updateData = Seq(
+ (11, "1", "rider-X", "driver-X", 19.10, "d"),
+ (9, "2", "rider-Y", "driver-Y", 27.70, "d"))
+ val updates = spark.createDataFrame(updateData).toDF(columns: _*)
+ updates.write.format("hudi").
+ option(OPERATION.key(), "upsert").
+ option(HoodieCompactionConfig.INLINE_COMPACT.key(), "false").
+ options(opts).
+ mode(SaveMode.Append).
+ save(basePath)
+
+ // Delete from operation.
+ val deletesData = Seq((-5, "4", "rider-D", "driver-D", 34.15, 6))
+ val deletes = spark.createDataFrame(deletesData).toDF(columns: _*)
+ deletes.write.format("hudi").
+ option(OPERATION.key(), "DELETE").
+ option(HoodieCompactionConfig.INLINE_COMPACT.key(), "false").
+ options(opts).
+ mode(SaveMode.Append).
+ save(basePath)
+
+ // Validate in the end.
+ val df = spark.read.format("hudi").options(opts).load(basePath)
+ val finalDf = df.select("ts", "key", "rider", "driver", "fare", "op").sort("key")
+ finalDf.show(false)
+ val expected = if (mergeMode == RecordMergeMode.COMMIT_TIME_ORDERING.name()) {
+ expectedCommitTimeBased
+ } else {
+ expectedEventTimeBased
+ }
+ val expectedDf = spark.createDataFrame(expected).toDF(columns: _*).sort("key")
+ TestCustomDeleteRecords.validate(expectedDf, finalDf)
+ }
+}
+
+object TestCustomDeleteRecords {
+ def provideParams(): java.util.List[Arguments] = {
+ java.util.Arrays.asList(
+ Arguments.of("true", "MERGE_ON_READ", "AVRO", "false", "EVENT_TIME_ORDERING"),
+ Arguments.of("true", "MERGE_ON_READ", "AVRO", "true", "EVENT_TIME_ORDERING"),
+ Arguments.of("true", "MERGE_ON_READ", "AVRO", "false", "COMMIT_TIME_ORDERING"),
+ Arguments.of("true", "MERGE_ON_READ", "AVRO", "true", "COMMIT_TIME_ORDERING"),
+ Arguments.of("true", "MERGE_ON_READ", "SPARK", "false", "EVENT_TIME_ORDERING"),
+ Arguments.of("true", "MERGE_ON_READ", "SPARK", "true", "EVENT_TIME_ORDERING"),
+ Arguments.of("true", "MERGE_ON_READ", "SPARK", "false", "COMMIT_TIME_ORDERING"),
+ Arguments.of("true", "MERGE_ON_READ", "SPARK", "true", "COMMIT_TIME_ORDERING"))
Review Comment:
Test CUSTOM merge mode too?
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/common/table/read/TestCustomDeleteRecords.scala:
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read
+
+import org.apache.hudi.{DataSourceWriteOptions, DefaultSparkRecordMerger, OverwriteWithLatestSparkRecordMerger}
+import org.apache.hudi.DataSourceWriteOptions.{OPERATION, PRECOMBINE_FIELD, RECORDKEY_FIELD, TABLE_TYPE}
+import org.apache.hudi.common.config.{HoodieReaderConfig, HoodieStorageConfig, RecordMergeMode}
+import org.apache.hudi.common.model.{HoodieAvroRecordMerger, HoodieRecordMerger, OverwriteWithLatestMerger}
+import org.apache.hudi.common.model.DefaultHoodieRecordPayload.{DELETE_KEY, DELETE_MARKER}
+import org.apache.hudi.config.{HoodieCompactionConfig, HoodieWriteConfig}
+import org.apache.hudi.testutils.SparkClientFunctionalTestHarness
+
+import org.apache.spark.sql.{Dataset, Row, SaveMode}
+import org.junit.jupiter.api.Assertions.assertTrue
+import org.junit.jupiter.params.ParameterizedTest
+import org.junit.jupiter.params.provider.{Arguments, MethodSource}
+
+class TestCustomDeleteRecords extends SparkClientFunctionalTestHarness {
+ val expectedEventTimeBased: Seq[(Int, String, String, String, Double, String)] = Seq(
+ (10, "5", "rider-E", "driver-E", 17.85, "i"),
+ (10, "3", "rider-C", "driver-C", 33.9, "i"),
+ (10, "2", "rider-B", "driver-B", 27.7, "i"))
+ val expectedCommitTimeBased: Seq[(Int, String, String, String, Double, String)] = Seq(
+ (10, "5", "rider-E", "driver-E", 17.85, "i"),
+ (10, "3", "rider-C", "driver-C", 33.9, "i"))
+
+ @ParameterizedTest
+ @MethodSource(Array("provideParams"))
+ def testCustomDelete(useFgReader: String,
+ tableType: String,
+ recordType: String,
+ positionUsed: String,
+ mergeMode: String): Unit = {
+ val sparkMergeClasses = List(
+ classOf[DefaultSparkRecordMerger].getName,
+ classOf[OverwriteWithLatestSparkRecordMerger].getName).mkString(",")
+ val avroMergerClasses = List(
+ classOf[HoodieAvroRecordMerger].getName,
+ classOf[OverwriteWithLatestMerger].getName).mkString(",")
+
+ val mergeStrategy = if (mergeMode.equals(RecordMergeMode.EVENT_TIME_ORDERING.name)) {
+ HoodieRecordMerger.EVENT_TIME_BASED_MERGE_STRATEGY_UUID
+ } else {
+ HoodieRecordMerger.COMMIT_TIME_BASED_MERGE_STRATEGY_UUID
+ }
Review Comment:
On the writer side, these are automatically inferred so there is no need to add these.
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/common/table/read/TestCustomDeleteRecords.scala:
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read
+
+import org.apache.hudi.{DataSourceWriteOptions, DefaultSparkRecordMerger, OverwriteWithLatestSparkRecordMerger}
+import org.apache.hudi.DataSourceWriteOptions.{OPERATION, PRECOMBINE_FIELD, RECORDKEY_FIELD, TABLE_TYPE}
+import org.apache.hudi.common.config.{HoodieReaderConfig, HoodieStorageConfig, RecordMergeMode}
+import org.apache.hudi.common.model.{HoodieAvroRecordMerger, HoodieRecordMerger, OverwriteWithLatestMerger}
+import org.apache.hudi.common.model.DefaultHoodieRecordPayload.{DELETE_KEY, DELETE_MARKER}
+import org.apache.hudi.config.{HoodieCompactionConfig, HoodieWriteConfig}
+import org.apache.hudi.testutils.SparkClientFunctionalTestHarness
+
+import org.apache.spark.sql.{Dataset, Row, SaveMode}
+import org.junit.jupiter.api.Assertions.assertTrue
+import org.junit.jupiter.params.ParameterizedTest
+import org.junit.jupiter.params.provider.{Arguments, MethodSource}
+
+class TestCustomDeleteRecords extends SparkClientFunctionalTestHarness {
+ val expectedEventTimeBased: Seq[(Int, String, String, String, Double, String)] = Seq(
+ (10, "5", "rider-E", "driver-E", 17.85, "i"),
+ (10, "3", "rider-C", "driver-C", 33.9, "i"),
+ (10, "2", "rider-B", "driver-B", 27.7, "i"))
+ val expectedCommitTimeBased: Seq[(Int, String, String, String, Double, String)] = Seq(
+ (10, "5", "rider-E", "driver-E", 17.85, "i"),
+ (10, "3", "rider-C", "driver-C", 33.9, "i"))
+
+ @ParameterizedTest
+ @MethodSource(Array("provideParams"))
+ def testCustomDelete(useFgReader: String,
+ tableType: String,
+ recordType: String,
+ positionUsed: String,
+ mergeMode: String): Unit = {
+ val sparkMergeClasses = List(
+ classOf[DefaultSparkRecordMerger].getName,
+ classOf[OverwriteWithLatestSparkRecordMerger].getName).mkString(",")
+ val avroMergerClasses = List(
+ classOf[HoodieAvroRecordMerger].getName,
+ classOf[OverwriteWithLatestMerger].getName).mkString(",")
+
+ val mergeStrategy = if (mergeMode.equals(RecordMergeMode.EVENT_TIME_ORDERING.name)) {
+ HoodieRecordMerger.EVENT_TIME_BASED_MERGE_STRATEGY_UUID
+ } else {
+ HoodieRecordMerger.COMMIT_TIME_BASED_MERGE_STRATEGY_UUID
+ }
+ val mergeOpts: Map[String, String] = Map(
+ HoodieStorageConfig.LOGFILE_DATA_BLOCK_FORMAT.key -> "parquet",
+ HoodieWriteConfig.RECORD_MERGE_IMPL_CLASSES.key ->
+ (if (recordType.equals("SPARK")) sparkMergeClasses else avroMergerClasses),
+ HoodieWriteConfig.RECORD_MERGE_STRATEGY_ID.key -> mergeStrategy)
+ val fgReaderOpts: Map[String, String] = Map(
+ HoodieReaderConfig.FILE_GROUP_READER_ENABLED.key -> useFgReader,
+ HoodieReaderConfig.MERGE_USE_RECORD_POSITIONS.key -> positionUsed,
+ HoodieWriteConfig.RECORD_MERGE_MODE.key -> mergeMode
+ )
+ val deleteOpts: Map[String, String] = Map(
+ DELETE_KEY -> "op", DELETE_MARKER -> "d")
+ val opts = mergeOpts ++ fgReaderOpts ++ deleteOpts
+ val columns = Seq("ts", "key", "rider", "driver", "fare", "op")
+
+ val data = Seq(
+ (10, "1", "rider-A", "driver-A", 19.10, "i"),
+ (10, "2", "rider-B", "driver-B", 27.70, "i"),
+ (10, "3", "rider-C", "driver-C", 33.90, "i"),
+ (10, "4", "rider-D", "driver-D", 34.15, "i"),
+ (10, "5", "rider-E", "driver-E", 17.85, "i"))
+ val inserts = spark.createDataFrame(data).toDF(columns: _*)
+ inserts.write.format("hudi").
+ option(RECORDKEY_FIELD.key(), "key").
+ option(PRECOMBINE_FIELD.key(), "ts").
+ option(TABLE_TYPE.key(), tableType).
+ option(DataSourceWriteOptions.TABLE_NAME.key(), "test_table").
+ option(HoodieCompactionConfig.INLINE_COMPACT.key(), "false").
+ options(opts).
+ mode(SaveMode.Overwrite).
+ save(basePath)
+
+ // Delete using delete markers.
+ val updateData = Seq(
+ (11, "1", "rider-X", "driver-X", 19.10, "d"),
+ (9, "2", "rider-Y", "driver-Y", 27.70, "d"))
+ val updates = spark.createDataFrame(updateData).toDF(columns: _*)
+ updates.write.format("hudi").
+ option(OPERATION.key(), "upsert").
+ option(HoodieCompactionConfig.INLINE_COMPACT.key(), "false").
+ options(opts).
+ mode(SaveMode.Append).
+ save(basePath)
+
+ // Delete from operation.
+ val deletesData = Seq((-5, "4", "rider-D", "driver-D", 34.15, 6))
+ val deletes = spark.createDataFrame(deletesData).toDF(columns: _*)
+ deletes.write.format("hudi").
+ option(OPERATION.key(), "DELETE").
+ option(HoodieCompactionConfig.INLINE_COMPACT.key(), "false").
+ options(opts).
+ mode(SaveMode.Append).
+ save(basePath)
Review Comment:
Add validation that after each write operation, deletes are written to the log files.
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/common/table/read/TestCustomDeleteRecords.scala:
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read
+
+import org.apache.hudi.{DataSourceWriteOptions, DefaultSparkRecordMerger, OverwriteWithLatestSparkRecordMerger}
+import org.apache.hudi.DataSourceWriteOptions.{OPERATION, PRECOMBINE_FIELD, RECORDKEY_FIELD, TABLE_TYPE}
+import org.apache.hudi.common.config.{HoodieReaderConfig, HoodieStorageConfig, RecordMergeMode}
+import org.apache.hudi.common.model.{HoodieAvroRecordMerger, HoodieRecordMerger, OverwriteWithLatestMerger}
+import org.apache.hudi.common.model.DefaultHoodieRecordPayload.{DELETE_KEY, DELETE_MARKER}
+import org.apache.hudi.config.{HoodieCompactionConfig, HoodieWriteConfig}
+import org.apache.hudi.testutils.SparkClientFunctionalTestHarness
+
+import org.apache.spark.sql.{Dataset, Row, SaveMode}
+import org.junit.jupiter.api.Assertions.assertTrue
+import org.junit.jupiter.params.ParameterizedTest
+import org.junit.jupiter.params.provider.{Arguments, MethodSource}
+
+class TestCustomDeleteRecords extends SparkClientFunctionalTestHarness {
+ val expectedEventTimeBased: Seq[(Int, String, String, String, Double, String)] = Seq(
+ (10, "5", "rider-E", "driver-E", 17.85, "i"),
+ (10, "3", "rider-C", "driver-C", 33.9, "i"),
+ (10, "2", "rider-B", "driver-B", 27.7, "i"))
+ val expectedCommitTimeBased: Seq[(Int, String, String, String, Double, String)] = Seq(
+ (10, "5", "rider-E", "driver-E", 17.85, "i"),
+ (10, "3", "rider-C", "driver-C", 33.9, "i"))
+
+ @ParameterizedTest
+ @MethodSource(Array("provideParams"))
+ def testCustomDelete(useFgReader: String,
+ tableType: String,
+ recordType: String,
+ positionUsed: String,
+ mergeMode: String): Unit = {
+ val sparkMergeClasses = List(
+ classOf[DefaultSparkRecordMerger].getName,
+ classOf[OverwriteWithLatestSparkRecordMerger].getName).mkString(",")
+ val avroMergerClasses = List(
+ classOf[HoodieAvroRecordMerger].getName,
+ classOf[OverwriteWithLatestMerger].getName).mkString(",")
+
+ val mergeStrategy = if (mergeMode.equals(RecordMergeMode.EVENT_TIME_ORDERING.name)) {
+ HoodieRecordMerger.EVENT_TIME_BASED_MERGE_STRATEGY_UUID
+ } else {
+ HoodieRecordMerger.COMMIT_TIME_BASED_MERGE_STRATEGY_UUID
+ }
+ val mergeOpts: Map[String, String] = Map(
+ HoodieStorageConfig.LOGFILE_DATA_BLOCK_FORMAT.key -> "parquet",
+ HoodieWriteConfig.RECORD_MERGE_IMPL_CLASSES.key ->
+ (if (recordType.equals("SPARK")) sparkMergeClasses else avroMergerClasses),
+ HoodieWriteConfig.RECORD_MERGE_STRATEGY_ID.key -> mergeStrategy)
+ val fgReaderOpts: Map[String, String] = Map(
+ HoodieReaderConfig.FILE_GROUP_READER_ENABLED.key -> useFgReader,
+ HoodieReaderConfig.MERGE_USE_RECORD_POSITIONS.key -> positionUsed,
+ HoodieWriteConfig.RECORD_MERGE_MODE.key -> mergeMode
+ )
+ val deleteOpts: Map[String, String] = Map(
+ DELETE_KEY -> "op", DELETE_MARKER -> "d")
+ val opts = mergeOpts ++ fgReaderOpts ++ deleteOpts
+ val columns = Seq("ts", "key", "rider", "driver", "fare", "op")
+
+ val data = Seq(
+ (10, "1", "rider-A", "driver-A", 19.10, "i"),
+ (10, "2", "rider-B", "driver-B", 27.70, "i"),
+ (10, "3", "rider-C", "driver-C", 33.90, "i"),
+ (10, "4", "rider-D", "driver-D", 34.15, "i"),
+ (10, "5", "rider-E", "driver-E", 17.85, "i"))
+ val inserts = spark.createDataFrame(data).toDF(columns: _*)
+ inserts.write.format("hudi").
+ option(RECORDKEY_FIELD.key(), "key").
+ option(PRECOMBINE_FIELD.key(), "ts").
+ option(TABLE_TYPE.key(), tableType).
+ option(DataSourceWriteOptions.TABLE_NAME.key(), "test_table").
+ option(HoodieCompactionConfig.INLINE_COMPACT.key(), "false").
+ options(opts).
+ mode(SaveMode.Overwrite).
+ save(basePath)
+
+ // Delete using delete markers.
+ val updateData = Seq(
+ (11, "1", "rider-X", "driver-X", 19.10, "d"),
+ (9, "2", "rider-Y", "driver-Y", 27.70, "d"))
+ val updates = spark.createDataFrame(updateData).toDF(columns: _*)
+ updates.write.format("hudi").
+ option(OPERATION.key(), "upsert").
+ option(HoodieCompactionConfig.INLINE_COMPACT.key(), "false").
+ options(opts).
+ mode(SaveMode.Append).
+ save(basePath)
+
+ // Delete from operation.
+ val deletesData = Seq((-5, "4", "rider-D", "driver-D", 34.15, 6))
+ val deletes = spark.createDataFrame(deletesData).toDF(columns: _*)
+ deletes.write.format("hudi").
+ option(OPERATION.key(), "DELETE").
+ option(HoodieCompactionConfig.INLINE_COMPACT.key(), "false").
+ options(opts).
+ mode(SaveMode.Append).
+ save(basePath)
+
+ // Validate in the end.
+ val df = spark.read.format("hudi").options(opts).load(basePath)
+ val finalDf = df.select("ts", "key", "rider", "driver", "fare", "op").sort("key")
+ finalDf.show(false)
+ val expected = if (mergeMode == RecordMergeMode.COMMIT_TIME_ORDERING.name()) {
+ expectedCommitTimeBased
+ } else {
+ expectedEventTimeBased
+ }
+ val expectedDf = spark.createDataFrame(expected).toDF(columns: _*).sort("key")
+ TestCustomDeleteRecords.validate(expectedDf, finalDf)
+ }
+}
+
+object TestCustomDeleteRecords {
+ def provideParams(): java.util.List[Arguments] = {
+ java.util.Arrays.asList(
+ Arguments.of("true", "MERGE_ON_READ", "AVRO", "false", "EVENT_TIME_ORDERING"),
+ Arguments.of("true", "MERGE_ON_READ", "AVRO", "true", "EVENT_TIME_ORDERING"),
+ Arguments.of("true", "MERGE_ON_READ", "AVRO", "false", "COMMIT_TIME_ORDERING"),
+ Arguments.of("true", "MERGE_ON_READ", "AVRO", "true", "COMMIT_TIME_ORDERING"),
+ Arguments.of("true", "MERGE_ON_READ", "SPARK", "false", "EVENT_TIME_ORDERING"),
+ Arguments.of("true", "MERGE_ON_READ", "SPARK", "true", "EVENT_TIME_ORDERING"),
+ Arguments.of("true", "MERGE_ON_READ", "SPARK", "false", "COMMIT_TIME_ORDERING"),
+ Arguments.of("true", "MERGE_ON_READ", "SPARK", "true", "COMMIT_TIME_ORDERING"))
+ }
+
+ def validate(expectedDf: Dataset[Row], actualDf: Dataset[Row]): Unit = {
+ val expectedMinusActual = expectedDf.except(actualDf)
+ val actualMinusExpected = actualDf.except(expectedDf)
+ expectedDf.show(false)
+ actualDf.show(false)
Review Comment:
Remove these to avoid spurious logs
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/common/table/read/TestCustomDeleteRecords.scala:
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read
+
+import org.apache.hudi.{DataSourceWriteOptions, DefaultSparkRecordMerger, OverwriteWithLatestSparkRecordMerger}
+import org.apache.hudi.DataSourceWriteOptions.{OPERATION, PRECOMBINE_FIELD, RECORDKEY_FIELD, TABLE_TYPE}
+import org.apache.hudi.common.config.{HoodieReaderConfig, HoodieStorageConfig, RecordMergeMode}
+import org.apache.hudi.common.model.{HoodieAvroRecordMerger, HoodieRecordMerger, OverwriteWithLatestMerger}
+import org.apache.hudi.common.model.DefaultHoodieRecordPayload.{DELETE_KEY, DELETE_MARKER}
+import org.apache.hudi.config.{HoodieCompactionConfig, HoodieWriteConfig}
+import org.apache.hudi.testutils.SparkClientFunctionalTestHarness
+
+import org.apache.spark.sql.{Dataset, Row, SaveMode}
+import org.junit.jupiter.api.Assertions.assertTrue
+import org.junit.jupiter.params.ParameterizedTest
+import org.junit.jupiter.params.provider.{Arguments, MethodSource}
+
+class TestCustomDeleteRecords extends SparkClientFunctionalTestHarness {
Review Comment:
Could you put the custom delete records in existing file group reader tests? It does not scale to add new tests for every changes.
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/common/table/read/TestCustomDeleteRecords.scala:
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read
+
+import org.apache.hudi.{DataSourceWriteOptions, DefaultSparkRecordMerger, OverwriteWithLatestSparkRecordMerger}
+import org.apache.hudi.DataSourceWriteOptions.{OPERATION, PRECOMBINE_FIELD, RECORDKEY_FIELD, TABLE_TYPE}
+import org.apache.hudi.common.config.{HoodieReaderConfig, HoodieStorageConfig, RecordMergeMode}
+import org.apache.hudi.common.model.{HoodieAvroRecordMerger, HoodieRecordMerger, OverwriteWithLatestMerger}
+import org.apache.hudi.common.model.DefaultHoodieRecordPayload.{DELETE_KEY, DELETE_MARKER}
+import org.apache.hudi.config.{HoodieCompactionConfig, HoodieWriteConfig}
+import org.apache.hudi.testutils.SparkClientFunctionalTestHarness
+
+import org.apache.spark.sql.{Dataset, Row, SaveMode}
+import org.junit.jupiter.api.Assertions.assertTrue
+import org.junit.jupiter.params.ParameterizedTest
+import org.junit.jupiter.params.provider.{Arguments, MethodSource}
+
+class TestCustomDeleteRecords extends SparkClientFunctionalTestHarness {
+ val expectedEventTimeBased: Seq[(Int, String, String, String, Double, String)] = Seq(
+ (10, "5", "rider-E", "driver-E", 17.85, "i"),
+ (10, "3", "rider-C", "driver-C", 33.9, "i"),
+ (10, "2", "rider-B", "driver-B", 27.7, "i"))
+ val expectedCommitTimeBased: Seq[(Int, String, String, String, Double, String)] = Seq(
+ (10, "5", "rider-E", "driver-E", 17.85, "i"),
+ (10, "3", "rider-C", "driver-C", 33.9, "i"))
+
+ @ParameterizedTest
+ @MethodSource(Array("provideParams"))
+ def testCustomDelete(useFgReader: String,
+ tableType: String,
+ recordType: String,
+ positionUsed: String,
+ mergeMode: String): Unit = {
+ val sparkMergeClasses = List(
+ classOf[DefaultSparkRecordMerger].getName,
+ classOf[OverwriteWithLatestSparkRecordMerger].getName).mkString(",")
+ val avroMergerClasses = List(
+ classOf[HoodieAvroRecordMerger].getName,
+ classOf[OverwriteWithLatestMerger].getName).mkString(",")
+
+ val mergeStrategy = if (mergeMode.equals(RecordMergeMode.EVENT_TIME_ORDERING.name)) {
+ HoodieRecordMerger.EVENT_TIME_BASED_MERGE_STRATEGY_UUID
+ } else {
+ HoodieRecordMerger.COMMIT_TIME_BASED_MERGE_STRATEGY_UUID
+ }
+ val mergeOpts: Map[String, String] = Map(
+ HoodieStorageConfig.LOGFILE_DATA_BLOCK_FORMAT.key -> "parquet",
+ HoodieWriteConfig.RECORD_MERGE_IMPL_CLASSES.key ->
+ (if (recordType.equals("SPARK")) sparkMergeClasses else avroMergerClasses),
+ HoodieWriteConfig.RECORD_MERGE_STRATEGY_ID.key -> mergeStrategy)
+ val fgReaderOpts: Map[String, String] = Map(
+ HoodieReaderConfig.FILE_GROUP_READER_ENABLED.key -> useFgReader,
+ HoodieReaderConfig.MERGE_USE_RECORD_POSITIONS.key -> positionUsed,
+ HoodieWriteConfig.RECORD_MERGE_MODE.key -> mergeMode
+ )
+ val deleteOpts: Map[String, String] = Map(
+ DELETE_KEY -> "op", DELETE_MARKER -> "d")
+ val opts = mergeOpts ++ fgReaderOpts ++ deleteOpts
+ val columns = Seq("ts", "key", "rider", "driver", "fare", "op")
+
+ val data = Seq(
+ (10, "1", "rider-A", "driver-A", 19.10, "i"),
+ (10, "2", "rider-B", "driver-B", 27.70, "i"),
+ (10, "3", "rider-C", "driver-C", 33.90, "i"),
+ (10, "4", "rider-D", "driver-D", 34.15, "i"),
+ (10, "5", "rider-E", "driver-E", 17.85, "i"))
+ val inserts = spark.createDataFrame(data).toDF(columns: _*)
+ inserts.write.format("hudi").
+ option(RECORDKEY_FIELD.key(), "key").
+ option(PRECOMBINE_FIELD.key(), "ts").
+ option(TABLE_TYPE.key(), tableType).
+ option(DataSourceWriteOptions.TABLE_NAME.key(), "test_table").
+ option(HoodieCompactionConfig.INLINE_COMPACT.key(), "false").
+ options(opts).
+ mode(SaveMode.Overwrite).
+ save(basePath)
+
+ // Delete using delete markers.
+ val updateData = Seq(
+ (11, "1", "rider-X", "driver-X", 19.10, "d"),
+ (9, "2", "rider-Y", "driver-Y", 27.70, "d"))
+ val updates = spark.createDataFrame(updateData).toDF(columns: _*)
+ updates.write.format("hudi").
+ option(OPERATION.key(), "upsert").
+ option(HoodieCompactionConfig.INLINE_COMPACT.key(), "false").
+ options(opts).
+ mode(SaveMode.Append).
+ save(basePath)
+
+ // Delete from operation.
Review Comment:
Could you also add a second batch of custom deletes with different ordering values to validate that ordering values are considered by `EVENT_TIME_ORDERING` and not by `COMMIT_TIME_ORDERING`?
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/common/table/read/TestCustomDeleteRecords.scala:
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read
+
+import org.apache.hudi.{DataSourceWriteOptions, DefaultSparkRecordMerger, OverwriteWithLatestSparkRecordMerger}
+import org.apache.hudi.DataSourceWriteOptions.{OPERATION, PRECOMBINE_FIELD, RECORDKEY_FIELD, TABLE_TYPE}
+import org.apache.hudi.common.config.{HoodieReaderConfig, HoodieStorageConfig, RecordMergeMode}
+import org.apache.hudi.common.model.{HoodieAvroRecordMerger, HoodieRecordMerger, OverwriteWithLatestMerger}
+import org.apache.hudi.common.model.DefaultHoodieRecordPayload.{DELETE_KEY, DELETE_MARKER}
+import org.apache.hudi.config.{HoodieCompactionConfig, HoodieWriteConfig}
+import org.apache.hudi.testutils.SparkClientFunctionalTestHarness
+
+import org.apache.spark.sql.{Dataset, Row, SaveMode}
+import org.junit.jupiter.api.Assertions.assertTrue
+import org.junit.jupiter.params.ParameterizedTest
+import org.junit.jupiter.params.provider.{Arguments, MethodSource}
+
+class TestCustomDeleteRecords extends SparkClientFunctionalTestHarness {
+ val expectedEventTimeBased: Seq[(Int, String, String, String, Double, String)] = Seq(
+ (10, "5", "rider-E", "driver-E", 17.85, "i"),
+ (10, "3", "rider-C", "driver-C", 33.9, "i"),
+ (10, "2", "rider-B", "driver-B", 27.7, "i"))
+ val expectedCommitTimeBased: Seq[(Int, String, String, String, Double, String)] = Seq(
+ (10, "5", "rider-E", "driver-E", 17.85, "i"),
+ (10, "3", "rider-C", "driver-C", 33.9, "i"))
+
+ @ParameterizedTest
+ @MethodSource(Array("provideParams"))
+ def testCustomDelete(useFgReader: String,
+ tableType: String,
+ recordType: String,
+ positionUsed: String,
+ mergeMode: String): Unit = {
+ val sparkMergeClasses = List(
+ classOf[DefaultSparkRecordMerger].getName,
+ classOf[OverwriteWithLatestSparkRecordMerger].getName).mkString(",")
+ val avroMergerClasses = List(
+ classOf[HoodieAvroRecordMerger].getName,
+ classOf[OverwriteWithLatestMerger].getName).mkString(",")
+
+ val mergeStrategy = if (mergeMode.equals(RecordMergeMode.EVENT_TIME_ORDERING.name)) {
+ HoodieRecordMerger.EVENT_TIME_BASED_MERGE_STRATEGY_UUID
+ } else {
+ HoodieRecordMerger.COMMIT_TIME_BASED_MERGE_STRATEGY_UUID
+ }
+ val mergeOpts: Map[String, String] = Map(
+ HoodieStorageConfig.LOGFILE_DATA_BLOCK_FORMAT.key -> "parquet",
+ HoodieWriteConfig.RECORD_MERGE_IMPL_CLASSES.key ->
+ (if (recordType.equals("SPARK")) sparkMergeClasses else avroMergerClasses),
+ HoodieWriteConfig.RECORD_MERGE_STRATEGY_ID.key -> mergeStrategy)
+ val fgReaderOpts: Map[String, String] = Map(
+ HoodieReaderConfig.FILE_GROUP_READER_ENABLED.key -> useFgReader,
+ HoodieReaderConfig.MERGE_USE_RECORD_POSITIONS.key -> positionUsed,
+ HoodieWriteConfig.RECORD_MERGE_MODE.key -> mergeMode
+ )
+ val deleteOpts: Map[String, String] = Map(
+ DELETE_KEY -> "op", DELETE_MARKER -> "d")
+ val opts = mergeOpts ++ fgReaderOpts ++ deleteOpts
+ val columns = Seq("ts", "key", "rider", "driver", "fare", "op")
+
+ val data = Seq(
+ (10, "1", "rider-A", "driver-A", 19.10, "i"),
+ (10, "2", "rider-B", "driver-B", 27.70, "i"),
+ (10, "3", "rider-C", "driver-C", 33.90, "i"),
+ (10, "4", "rider-D", "driver-D", 34.15, "i"),
+ (10, "5", "rider-E", "driver-E", 17.85, "i"))
+ val inserts = spark.createDataFrame(data).toDF(columns: _*)
+ inserts.write.format("hudi").
+ option(RECORDKEY_FIELD.key(), "key").
+ option(PRECOMBINE_FIELD.key(), "ts").
+ option(TABLE_TYPE.key(), tableType).
+ option(DataSourceWriteOptions.TABLE_NAME.key(), "test_table").
+ option(HoodieCompactionConfig.INLINE_COMPACT.key(), "false").
+ options(opts).
+ mode(SaveMode.Overwrite).
+ save(basePath)
+
+ // Delete using delete markers.
+ val updateData = Seq(
+ (11, "1", "rider-X", "driver-X", 19.10, "d"),
+ (9, "2", "rider-Y", "driver-Y", 27.70, "d"))
+ val updates = spark.createDataFrame(updateData).toDF(columns: _*)
+ updates.write.format("hudi").
+ option(OPERATION.key(), "upsert").
+ option(HoodieCompactionConfig.INLINE_COMPACT.key(), "false").
+ options(opts).
+ mode(SaveMode.Append).
+ save(basePath)
+
+ // Delete from operation.
+ val deletesData = Seq((-5, "4", "rider-D", "driver-D", 34.15, 6))
+ val deletes = spark.createDataFrame(deletesData).toDF(columns: _*)
+ deletes.write.format("hudi").
+ option(OPERATION.key(), "DELETE").
+ option(HoodieCompactionConfig.INLINE_COMPACT.key(), "false").
+ options(opts).
+ mode(SaveMode.Append).
+ save(basePath)
+
+ // Validate in the end.
+ val df = spark.read.format("hudi").options(opts).load(basePath)
+ val finalDf = df.select("ts", "key", "rider", "driver", "fare", "op").sort("key")
+ finalDf.show(false)
+ val expected = if (mergeMode == RecordMergeMode.COMMIT_TIME_ORDERING.name()) {
+ expectedCommitTimeBased
+ } else {
+ expectedEventTimeBased
+ }
+ val expectedDf = spark.createDataFrame(expected).toDF(columns: _*).sort("key")
+ TestCustomDeleteRecords.validate(expectedDf, finalDf)
+ }
+}
+
+object TestCustomDeleteRecords {
+ def provideParams(): java.util.List[Arguments] = {
+ java.util.Arrays.asList(
+ Arguments.of("true", "MERGE_ON_READ", "AVRO", "false", "EVENT_TIME_ORDERING"),
+ Arguments.of("true", "MERGE_ON_READ", "AVRO", "true", "EVENT_TIME_ORDERING"),
+ Arguments.of("true", "MERGE_ON_READ", "AVRO", "false", "COMMIT_TIME_ORDERING"),
+ Arguments.of("true", "MERGE_ON_READ", "AVRO", "true", "COMMIT_TIME_ORDERING"),
+ Arguments.of("true", "MERGE_ON_READ", "SPARK", "false", "EVENT_TIME_ORDERING"),
+ Arguments.of("true", "MERGE_ON_READ", "SPARK", "true", "EVENT_TIME_ORDERING"),
+ Arguments.of("true", "MERGE_ON_READ", "SPARK", "false", "COMMIT_TIME_ORDERING"),
+ Arguments.of("true", "MERGE_ON_READ", "SPARK", "true", "COMMIT_TIME_ORDERING"))
+ }
+
+ def validate(expectedDf: Dataset[Row], actualDf: Dataset[Row]): Unit = {
Review Comment:
Could you use existing util to avoid code duplication `SparkClientFunctionalTestHarness#areDataframesEqual`
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/common/table/read/TestCustomDeleteRecords.scala:
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read
+
+import org.apache.hudi.{DataSourceWriteOptions, DefaultSparkRecordMerger, OverwriteWithLatestSparkRecordMerger}
+import org.apache.hudi.DataSourceWriteOptions.{OPERATION, PRECOMBINE_FIELD, RECORDKEY_FIELD, TABLE_TYPE}
+import org.apache.hudi.common.config.{HoodieReaderConfig, HoodieStorageConfig, RecordMergeMode}
+import org.apache.hudi.common.model.{HoodieAvroRecordMerger, HoodieRecordMerger, OverwriteWithLatestMerger}
+import org.apache.hudi.common.model.DefaultHoodieRecordPayload.{DELETE_KEY, DELETE_MARKER}
+import org.apache.hudi.config.{HoodieCompactionConfig, HoodieWriteConfig}
+import org.apache.hudi.testutils.SparkClientFunctionalTestHarness
+
+import org.apache.spark.sql.{Dataset, Row, SaveMode}
+import org.junit.jupiter.api.Assertions.assertTrue
+import org.junit.jupiter.params.ParameterizedTest
+import org.junit.jupiter.params.provider.{Arguments, MethodSource}
+
+class TestCustomDeleteRecords extends SparkClientFunctionalTestHarness {
+ val expectedEventTimeBased: Seq[(Int, String, String, String, Double, String)] = Seq(
+ (10, "5", "rider-E", "driver-E", 17.85, "i"),
+ (10, "3", "rider-C", "driver-C", 33.9, "i"),
+ (10, "2", "rider-B", "driver-B", 27.7, "i"))
+ val expectedCommitTimeBased: Seq[(Int, String, String, String, Double, String)] = Seq(
+ (10, "5", "rider-E", "driver-E", 17.85, "i"),
+ (10, "3", "rider-C", "driver-C", 33.9, "i"))
+
+ @ParameterizedTest
+ @MethodSource(Array("provideParams"))
+ def testCustomDelete(useFgReader: String,
+ tableType: String,
+ recordType: String,
+ positionUsed: String,
+ mergeMode: String): Unit = {
+ val sparkMergeClasses = List(
+ classOf[DefaultSparkRecordMerger].getName,
+ classOf[OverwriteWithLatestSparkRecordMerger].getName).mkString(",")
+ val avroMergerClasses = List(
+ classOf[HoodieAvroRecordMerger].getName,
+ classOf[OverwriteWithLatestMerger].getName).mkString(",")
+
+ val mergeStrategy = if (mergeMode.equals(RecordMergeMode.EVENT_TIME_ORDERING.name)) {
+ HoodieRecordMerger.EVENT_TIME_BASED_MERGE_STRATEGY_UUID
+ } else {
+ HoodieRecordMerger.COMMIT_TIME_BASED_MERGE_STRATEGY_UUID
+ }
+ val mergeOpts: Map[String, String] = Map(
+ HoodieStorageConfig.LOGFILE_DATA_BLOCK_FORMAT.key -> "parquet",
Review Comment:
Test both parquet and avro?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2691523270
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
}, {
"hash" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974",
"triggerID" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* 2eacf85acc06405bb4a11f54050bcfb647e2691f Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646)
* 7a82eff1dfdbc4f4e6181b689218a7726a32f03b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2691610313
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
}, {
"hash" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974",
"triggerID" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* 7a82eff1dfdbc4f4e6181b689218a7726a32f03b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #12843:
URL: https://github.com/apache/hudi/pull/12843#discussion_r2003422629
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/KeyBasedFileGroupRecordBuffer.java:
##########
@@ -82,7 +86,12 @@ public void processDataBlock(HoodieDataBlock dataBlock, Option<KeySpec> keySpecO
Map<String, Object> metadata = readerContext.generateMetadataForRecord(
nextRecord, schema);
String recordKey = (String) metadata.get(HoodieReaderContext.INTERNAL_META_RECORD_KEY);
- processNextDataRecord(nextRecord, metadata, recordKey);
+
+ if (shouldCheckCustomDeleteMarker && isCustomDeleteRecord(nextRecord)) {
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #12843:
URL: https://github.com/apache/hudi/pull/12843#discussion_r2004401953
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/common/table/read/TestCustomDeleteRecords.scala:
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read
+
+import org.apache.hudi.{DataSourceWriteOptions, DefaultSparkRecordMerger, OverwriteWithLatestSparkRecordMerger}
+import org.apache.hudi.DataSourceWriteOptions.{OPERATION, PRECOMBINE_FIELD, RECORDKEY_FIELD, TABLE_TYPE}
+import org.apache.hudi.common.config.{HoodieReaderConfig, HoodieStorageConfig, RecordMergeMode}
+import org.apache.hudi.common.model.{HoodieAvroRecordMerger, HoodieRecordMerger, OverwriteWithLatestMerger}
+import org.apache.hudi.common.model.DefaultHoodieRecordPayload.{DELETE_KEY, DELETE_MARKER}
+import org.apache.hudi.config.{HoodieCompactionConfig, HoodieWriteConfig}
+import org.apache.hudi.testutils.SparkClientFunctionalTestHarness
+
+import org.apache.spark.sql.{Dataset, Row, SaveMode}
+import org.junit.jupiter.api.Assertions.assertTrue
+import org.junit.jupiter.params.ParameterizedTest
+import org.junit.jupiter.params.provider.{Arguments, MethodSource}
+
+class TestCustomDeleteRecords extends SparkClientFunctionalTestHarness {
+ val expectedEventTimeBased: Seq[(Int, String, String, String, Double, String)] = Seq(
+ (10, "5", "rider-E", "driver-E", 17.85, "i"),
+ (10, "3", "rider-C", "driver-C", 33.9, "i"),
+ (10, "2", "rider-B", "driver-B", 27.7, "i"))
+ val expectedCommitTimeBased: Seq[(Int, String, String, String, Double, String)] = Seq(
+ (10, "5", "rider-E", "driver-E", 17.85, "i"),
+ (10, "3", "rider-C", "driver-C", 33.9, "i"))
+
+ @ParameterizedTest
+ @MethodSource(Array("provideParams"))
+ def testCustomDelete(useFgReader: String,
+ tableType: String,
+ recordType: String,
+ positionUsed: String,
+ mergeMode: String): Unit = {
+ val sparkMergeClasses = List(
+ classOf[DefaultSparkRecordMerger].getName,
+ classOf[OverwriteWithLatestSparkRecordMerger].getName).mkString(",")
+ val avroMergerClasses = List(
+ classOf[HoodieAvroRecordMerger].getName,
+ classOf[OverwriteWithLatestMerger].getName).mkString(",")
+
+ val mergeStrategy = if (mergeMode.equals(RecordMergeMode.EVENT_TIME_ORDERING.name)) {
+ HoodieRecordMerger.EVENT_TIME_BASED_MERGE_STRATEGY_UUID
+ } else {
+ HoodieRecordMerger.COMMIT_TIME_BASED_MERGE_STRATEGY_UUID
+ }
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2741379408
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
}, {
"hash" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974",
"triggerID" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"triggerType" : "PUSH"
}, {
"hash" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3981",
"triggerID" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"triggerType" : "PUSH"
}, {
"hash" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3984",
"triggerID" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"triggerType" : "PUSH"
}, {
"hash" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"triggerType" : "PUSH"
}, {
"hash" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4100",
"triggerID" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"triggerType" : "PUSH"
}, {
"hash" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4181",
"triggerID" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"triggerType" : "PUSH"
}, {
"hash" : "16be88870989aa1e54cf03f15ff61654d590e736",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4274",
"triggerID" : "16be88870989aa1e54cf03f15ff61654d590e736",
"triggerType" : "PUSH"
}, {
"hash" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"triggerType" : "PUSH"
}, {
"hash" : "a5227479708c9669e82835056cc515043539114a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4279",
"triggerID" : "a5227479708c9669e82835056cc515043539114a",
"triggerType" : "PUSH"
}, {
"hash" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"triggerType" : "PUSH"
}, {
"hash" : "8ebbd8b0698af9d7dadc71007805e8b6fed1c464",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4280",
"triggerID" : "8ebbd8b0698af9d7dadc71007805e8b6fed1c464",
"triggerType" : "PUSH"
}, {
"hash" : "d711c309c285da81e8a214ebe670c23c01d01f27",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4281",
"triggerID" : "d711c309c285da81e8a214ebe670c23c01d01f27",
"triggerType" : "PUSH"
}, {
"hash" : "1f50e01a909603339fd95c7972af791625d8616d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4282",
"triggerID" : "1f50e01a909603339fd95c7972af791625d8616d",
"triggerType" : "PUSH"
}, {
"hash" : "16edc580ea76612ea2a8f0e6243f6ede5fdf9546",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4289",
"triggerID" : "16edc580ea76612ea2a8f0e6243f6ede5fdf9546",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* 3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616 UNKNOWN
* 49c310b28459d5414dce0cfdcba59ea0dac1eb6f UNKNOWN
* 4cf517aa15396f2766bbc35c6b6b59400f1fbebd UNKNOWN
* 16edc580ea76612ea2a8f0e6243f6ede5fdf9546 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4289)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2744155837
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
}, {
"hash" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974",
"triggerID" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"triggerType" : "PUSH"
}, {
"hash" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3981",
"triggerID" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"triggerType" : "PUSH"
}, {
"hash" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3984",
"triggerID" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"triggerType" : "PUSH"
}, {
"hash" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"triggerType" : "PUSH"
}, {
"hash" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4100",
"triggerID" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"triggerType" : "PUSH"
}, {
"hash" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4181",
"triggerID" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"triggerType" : "PUSH"
}, {
"hash" : "16be88870989aa1e54cf03f15ff61654d590e736",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4274",
"triggerID" : "16be88870989aa1e54cf03f15ff61654d590e736",
"triggerType" : "PUSH"
}, {
"hash" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"triggerType" : "PUSH"
}, {
"hash" : "a5227479708c9669e82835056cc515043539114a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4279",
"triggerID" : "a5227479708c9669e82835056cc515043539114a",
"triggerType" : "PUSH"
}, {
"hash" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"triggerType" : "PUSH"
}, {
"hash" : "8ebbd8b0698af9d7dadc71007805e8b6fed1c464",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4280",
"triggerID" : "8ebbd8b0698af9d7dadc71007805e8b6fed1c464",
"triggerType" : "PUSH"
}, {
"hash" : "d711c309c285da81e8a214ebe670c23c01d01f27",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4281",
"triggerID" : "d711c309c285da81e8a214ebe670c23c01d01f27",
"triggerType" : "PUSH"
}, {
"hash" : "1f50e01a909603339fd95c7972af791625d8616d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4282",
"triggerID" : "1f50e01a909603339fd95c7972af791625d8616d",
"triggerType" : "PUSH"
}, {
"hash" : "16edc580ea76612ea2a8f0e6243f6ede5fdf9546",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4289",
"triggerID" : "16edc580ea76612ea2a8f0e6243f6ede5fdf9546",
"triggerType" : "PUSH"
}, {
"hash" : "6c6788f3c699fc556ccd25c0f37518fc0cf7b0a5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4318",
"triggerID" : "6c6788f3c699fc556ccd25c0f37518fc0cf7b0a5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* 3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616 UNKNOWN
* 49c310b28459d5414dce0cfdcba59ea0dac1eb6f UNKNOWN
* 4cf517aa15396f2766bbc35c6b6b59400f1fbebd UNKNOWN
* 16edc580ea76612ea2a8f0e6243f6ede5fdf9546 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4289)
* 6c6788f3c699fc556ccd25c0f37518fc0cf7b0a5 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4318)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12843:
URL: https://github.com/apache/hudi/pull/12843#discussion_r2011181436
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -143,6 +148,29 @@ public HoodieBaseFileGroupRecordBuffer(HoodieReaderContext<T> readerContext,
} catch (IOException e) {
throw new HoodieIOException("IOException when creating ExternalSpillableMap at " + spillableMapBasePath, e);
}
+ this.shouldCheckCustomDeleteMarker = hasCustomDeleteConfigs(props, readerSchema);
+ if (shouldCheckCustomDeleteMarker) {
+ this.customDeleteMarkerKey = props.getProperty(DELETE_KEY);
+ this.customDeleteMarkerValue = props.getProperty(DELETE_MARKER);
+ }
+ }
+
+ protected boolean hasCustomDeleteConfigs(TypedProperties props, Schema schema) {
+ // DELETE_KEY and DELETE_MARKER both should be set.
+ if (StringUtils.isNullOrEmpty(props.getProperty(DELETE_KEY))
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2749895258
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
}, {
"hash" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974",
"triggerID" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"triggerType" : "PUSH"
}, {
"hash" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3981",
"triggerID" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"triggerType" : "PUSH"
}, {
"hash" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3984",
"triggerID" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"triggerType" : "PUSH"
}, {
"hash" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"triggerType" : "PUSH"
}, {
"hash" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4100",
"triggerID" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"triggerType" : "PUSH"
}, {
"hash" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4181",
"triggerID" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"triggerType" : "PUSH"
}, {
"hash" : "16be88870989aa1e54cf03f15ff61654d590e736",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4274",
"triggerID" : "16be88870989aa1e54cf03f15ff61654d590e736",
"triggerType" : "PUSH"
}, {
"hash" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"triggerType" : "PUSH"
}, {
"hash" : "a5227479708c9669e82835056cc515043539114a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4279",
"triggerID" : "a5227479708c9669e82835056cc515043539114a",
"triggerType" : "PUSH"
}, {
"hash" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"triggerType" : "PUSH"
}, {
"hash" : "8ebbd8b0698af9d7dadc71007805e8b6fed1c464",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4280",
"triggerID" : "8ebbd8b0698af9d7dadc71007805e8b6fed1c464",
"triggerType" : "PUSH"
}, {
"hash" : "d711c309c285da81e8a214ebe670c23c01d01f27",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4281",
"triggerID" : "d711c309c285da81e8a214ebe670c23c01d01f27",
"triggerType" : "PUSH"
}, {
"hash" : "1f50e01a909603339fd95c7972af791625d8616d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4282",
"triggerID" : "1f50e01a909603339fd95c7972af791625d8616d",
"triggerType" : "PUSH"
}, {
"hash" : "16edc580ea76612ea2a8f0e6243f6ede5fdf9546",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4289",
"triggerID" : "16edc580ea76612ea2a8f0e6243f6ede5fdf9546",
"triggerType" : "PUSH"
}, {
"hash" : "6c6788f3c699fc556ccd25c0f37518fc0cf7b0a5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4318",
"triggerID" : "6c6788f3c699fc556ccd25c0f37518fc0cf7b0a5",
"triggerType" : "PUSH"
}, {
"hash" : "dd0c492908ae259b4df9c537f2c518628756d6aa",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4369",
"triggerID" : "dd0c492908ae259b4df9c537f2c518628756d6aa",
"triggerType" : "PUSH"
}, {
"hash" : "19815dab31ee8c45968c2aa0d4b92fd6b3cc587f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "19815dab31ee8c45968c2aa0d4b92fd6b3cc587f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* 3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616 UNKNOWN
* 49c310b28459d5414dce0cfdcba59ea0dac1eb6f UNKNOWN
* 4cf517aa15396f2766bbc35c6b6b59400f1fbebd UNKNOWN
* 6c6788f3c699fc556ccd25c0f37518fc0cf7b0a5 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4318)
* dd0c492908ae259b4df9c537f2c518628756d6aa Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4369)
* 19815dab31ee8c45968c2aa0d4b92fd6b3cc587f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12843:
URL: https://github.com/apache/hudi/pull/12843#discussion_r2011113067
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReaderSchemaHandler.java:
##########
@@ -207,17 +213,22 @@ private static String[] getMandatoryFieldsForMerging(HoodieTableConfig cfg, Type
requiredFields.addAll(Arrays.asList(fields.get()));
}
}
-
+ // Add precombine field for event time ordering merge mode.
if (mergingConfigs.getLeft() == RecordMergeMode.EVENT_TIME_ORDERING) {
String preCombine = cfg.getPreCombineField();
if (!StringUtils.isNullOrEmpty(preCombine)) {
requiredFields.add(preCombine);
}
}
-
+ // Add `HOODIE_IS_DELETED_FIELD` field if exists.
if (dataSchema.getField(HoodieRecord.HOODIE_IS_DELETED_FIELD) != null) {
requiredFields.add(HoodieRecord.HOODIE_IS_DELETED_FIELD);
}
+ // Add custom delete key field if exists.
+ if (FileGroupRecordBuffer.hasCustomDeleteConfigs(props, dataSchema)) {
+ requiredFields.add(props.getProperty(DELETE_KEY));
Review Comment:
Fixed.
##########
hudi-common/src/main/java/org/apache/hudi/common/engine/HoodieReaderContext.java:
##########
@@ -215,6 +215,19 @@ public ClosableIterator<T> getFileRecordIterator(
*/
public abstract Object getValue(T record, Schema schema, String fieldName);
+ /**
+ * Cast to Java boolean value.
+ * If the object is not compatible with boolean type, throws.
+ */
+ public boolean castToBoolean(Object value) {
Review Comment:
nit: This is only used in one place. No need for a new API in reader context which can cause confusion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2750015853
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
}, {
"hash" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974",
"triggerID" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"triggerType" : "PUSH"
}, {
"hash" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3981",
"triggerID" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"triggerType" : "PUSH"
}, {
"hash" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3984",
"triggerID" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"triggerType" : "PUSH"
}, {
"hash" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"triggerType" : "PUSH"
}, {
"hash" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4100",
"triggerID" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"triggerType" : "PUSH"
}, {
"hash" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4181",
"triggerID" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"triggerType" : "PUSH"
}, {
"hash" : "16be88870989aa1e54cf03f15ff61654d590e736",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4274",
"triggerID" : "16be88870989aa1e54cf03f15ff61654d590e736",
"triggerType" : "PUSH"
}, {
"hash" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"triggerType" : "PUSH"
}, {
"hash" : "a5227479708c9669e82835056cc515043539114a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4279",
"triggerID" : "a5227479708c9669e82835056cc515043539114a",
"triggerType" : "PUSH"
}, {
"hash" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"triggerType" : "PUSH"
}, {
"hash" : "8ebbd8b0698af9d7dadc71007805e8b6fed1c464",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4280",
"triggerID" : "8ebbd8b0698af9d7dadc71007805e8b6fed1c464",
"triggerType" : "PUSH"
}, {
"hash" : "d711c309c285da81e8a214ebe670c23c01d01f27",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4281",
"triggerID" : "d711c309c285da81e8a214ebe670c23c01d01f27",
"triggerType" : "PUSH"
}, {
"hash" : "1f50e01a909603339fd95c7972af791625d8616d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4282",
"triggerID" : "1f50e01a909603339fd95c7972af791625d8616d",
"triggerType" : "PUSH"
}, {
"hash" : "16edc580ea76612ea2a8f0e6243f6ede5fdf9546",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4289",
"triggerID" : "16edc580ea76612ea2a8f0e6243f6ede5fdf9546",
"triggerType" : "PUSH"
}, {
"hash" : "6c6788f3c699fc556ccd25c0f37518fc0cf7b0a5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4318",
"triggerID" : "6c6788f3c699fc556ccd25c0f37518fc0cf7b0a5",
"triggerType" : "PUSH"
}, {
"hash" : "dd0c492908ae259b4df9c537f2c518628756d6aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4369",
"triggerID" : "dd0c492908ae259b4df9c537f2c518628756d6aa",
"triggerType" : "PUSH"
}, {
"hash" : "19815dab31ee8c45968c2aa0d4b92fd6b3cc587f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "19815dab31ee8c45968c2aa0d4b92fd6b3cc587f",
"triggerType" : "PUSH"
}, {
"hash" : "e486a5f141c8e78e1dc63184e95c1d960b067c5a",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4373",
"triggerID" : "e486a5f141c8e78e1dc63184e95c1d960b067c5a",
"triggerType" : "PUSH"
}, {
"hash" : "b3f0a89eda2a4bb6f6b21edd3133cc77acdf89bc",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4375",
"triggerID" : "b3f0a89eda2a4bb6f6b21edd3133cc77acdf89bc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* 3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616 UNKNOWN
* 49c310b28459d5414dce0cfdcba59ea0dac1eb6f UNKNOWN
* 4cf517aa15396f2766bbc35c6b6b59400f1fbebd UNKNOWN
* 19815dab31ee8c45968c2aa0d4b92fd6b3cc587f UNKNOWN
* e486a5f141c8e78e1dc63184e95c1d960b067c5a Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4373)
* b3f0a89eda2a4bb6f6b21edd3133cc77acdf89bc Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4375)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2739339689
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
}, {
"hash" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974",
"triggerID" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"triggerType" : "PUSH"
}, {
"hash" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3981",
"triggerID" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"triggerType" : "PUSH"
}, {
"hash" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3984",
"triggerID" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"triggerType" : "PUSH"
}, {
"hash" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"triggerType" : "PUSH"
}, {
"hash" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4100",
"triggerID" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"triggerType" : "PUSH"
}, {
"hash" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4181",
"triggerID" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"triggerType" : "PUSH"
}, {
"hash" : "16be88870989aa1e54cf03f15ff61654d590e736",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4274",
"triggerID" : "16be88870989aa1e54cf03f15ff61654d590e736",
"triggerType" : "PUSH"
}, {
"hash" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "49c310b28459d5414dce0cfdcba59ea0dac1eb6f",
"triggerType" : "PUSH"
}, {
"hash" : "a5227479708c9669e82835056cc515043539114a",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4279",
"triggerID" : "a5227479708c9669e82835056cc515043539114a",
"triggerType" : "PUSH"
}, {
"hash" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cf517aa15396f2766bbc35c6b6b59400f1fbebd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* 3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616 UNKNOWN
* 49c310b28459d5414dce0cfdcba59ea0dac1eb6f UNKNOWN
* a5227479708c9669e82835056cc515043539114a Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4279)
* 4cf517aa15396f2766bbc35c6b6b59400f1fbebd UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2738586698
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
}, {
"hash" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c54e6402544f2b5d3614469121927a6f824c1d5",
"triggerType" : "PUSH"
}, {
"hash" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0314fa60c38070be1e9a04da9591acaecf34d4f9",
"triggerType" : "PUSH"
}, {
"hash" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3646",
"triggerID" : "2eacf85acc06405bb4a11f54050bcfb647e2691f",
"triggerType" : "PUSH"
}, {
"hash" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3974",
"triggerID" : "7a82eff1dfdbc4f4e6181b689218a7726a32f03b",
"triggerType" : "PUSH"
}, {
"hash" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3981",
"triggerID" : "26f9abed586abb54ac4a404dac8a0b62aab1026e",
"triggerType" : "PUSH"
}, {
"hash" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3984",
"triggerID" : "fa3151a7387c16435a20c3f1d0a72a1b9f31e75d",
"triggerType" : "PUSH"
}, {
"hash" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616",
"triggerType" : "PUSH"
}, {
"hash" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4100",
"triggerID" : "e888cdfc0f2bf40a5422d9290b8dbeb9e643d564",
"triggerType" : "PUSH"
}, {
"hash" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4181",
"triggerID" : "a9e67270ca6cc81e56c3a752326f9d884620f1ed",
"triggerType" : "PUSH"
}, {
"hash" : "16be88870989aa1e54cf03f15ff61654d590e736",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "16be88870989aa1e54cf03f15ff61654d590e736",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c54e6402544f2b5d3614469121927a6f824c1d5 UNKNOWN
* 0314fa60c38070be1e9a04da9591acaecf34d4f9 UNKNOWN
* 3e9f3d89f2985c41a0c0bd84fcbd1ebc2b5ce616 UNKNOWN
* a9e67270ca6cc81e56c3a752326f9d884620f1ed Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4181)
* 16be88870989aa1e54cf03f15ff61654d590e736 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12843:
URL: https://github.com/apache/hudi/pull/12843#issuecomment-2660280229
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645",
"triggerID" : "c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c1c84dfbde2b735d9624cdbe0b1b114f227fd7ec Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3645)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org