You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "lokeshj1703 (via GitHub)" <gi...@apache.org> on 2025/07/07 16:58:29 UTC
[PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
lokeshj1703 opened a new pull request, #13526:
URL: https://github.com/apache/hudi/pull/13526
### Change Logs
Currently we only support one ordering field to be configured. The Jira aims to add support for multiple ordering fields such that records will be ordered by the first ordering field and then by the second ordering field (if first ordering field is same) and so on.
### Impact
_Describe any public API or user-facing feature change or any performance impact._
### Risk level (write none, low medium or high below)
_If medium or high, explain what verification was done to mitigate the risks._
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none"._
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3102467675
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
}, {
"hash" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"triggerType" : "PUSH"
}, {
"hash" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947",
"triggerID" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"triggerType" : "PUSH"
}, {
"hash" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6949",
"triggerID" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"triggerType" : "PUSH"
}, {
"hash" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e19abadae9dece1aa78868839605a3d23a8b29fd UNKNOWN
* a20a7af499d5e0f6dc415ff74459d91ac88bf3d0 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947)
* 9ce11db126c7a2124c41ee9c1b97e1820fd6af8e Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6949)
* 4ba45f607a07f3f036e164dd68532fbc8b4bc3b8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3102392951
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
}, {
"hash" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"triggerType" : "PUSH"
}, {
"hash" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947",
"triggerID" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"triggerType" : "PUSH"
}, {
"hash" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e19abadae9dece1aa78868839605a3d23a8b29fd UNKNOWN
* a20a7af499d5e0f6dc415ff74459d91ac88bf3d0 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947)
* 9ce11db126c7a2124c41ee9c1b97e1820fd6af8e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3102949310
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
}, {
"hash" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"triggerType" : "PUSH"
}, {
"hash" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947",
"triggerID" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"triggerType" : "PUSH"
}, {
"hash" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6949",
"triggerID" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"triggerType" : "PUSH"
}, {
"hash" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"triggerType" : "PUSH"
}, {
"hash" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6950",
"triggerID" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e19abadae9dece1aa78868839605a3d23a8b29fd UNKNOWN
* 4ba45f607a07f3f036e164dd68532fbc8b4bc3b8 UNKNOWN
* a1f62b31b47715ce88932618c3817fbd7dd2e96e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6950)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3105794467
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
}, {
"hash" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"triggerType" : "PUSH"
}, {
"hash" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947",
"triggerID" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"triggerType" : "PUSH"
}, {
"hash" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6949",
"triggerID" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"triggerType" : "PUSH"
}, {
"hash" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"triggerType" : "PUSH"
}, {
"hash" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6950",
"triggerID" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"triggerType" : "PUSH"
}, {
"hash" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6974",
"triggerID" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"triggerType" : "PUSH"
}, {
"hash" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e19abadae9dece1aa78868839605a3d23a8b29fd UNKNOWN
* 4ba45f607a07f3f036e164dd68532fbc8b4bc3b8 UNKNOWN
* 21043f2bc87787d692ec55c2e4150ab29a396264 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6974)
* 7c1ded9e7881ffea36adeceb34c89ea25353bc0d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3106001905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
}, {
"hash" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"triggerType" : "PUSH"
}, {
"hash" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947",
"triggerID" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"triggerType" : "PUSH"
}, {
"hash" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6949",
"triggerID" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"triggerType" : "PUSH"
}, {
"hash" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"triggerType" : "PUSH"
}, {
"hash" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6950",
"triggerID" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"triggerType" : "PUSH"
}, {
"hash" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6974",
"triggerID" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"triggerType" : "PUSH"
}, {
"hash" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6991",
"triggerID" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e19abadae9dece1aa78868839605a3d23a8b29fd UNKNOWN
* 4ba45f607a07f3f036e164dd68532fbc8b4bc3b8 UNKNOWN
* 7c1ded9e7881ffea36adeceb34c89ea25353bc0d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6991)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2225497977
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -814,7 +814,7 @@ public String getRecordMergeStrategyId() {
public static Triple<RecordMergeMode, String, String> inferCorrectMergingBehavior(RecordMergeMode recordMergeMode,
String payloadClassName,
String recordMergeStrategyId,
- String orderingFieldName,
+ String orderingFieldNames,
Review Comment:
We only check if the preCombine fields are set in this function. So do not require a list here. Renamed to orderingFieldNamesAsString to make it more clear.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -926,8 +926,12 @@ static RecordMergeMode inferRecordMergeModeFromMergeStrategyId(String recordMerg
}
}
- public String getPreCombineField() {
- return getString(PRECOMBINE_FIELD);
+ public Option<List<String>> getPreCombineFieldList() {
Review Comment:
Addressed. Also renamed `PRECOMBINE_FIELD` to `PRECOMBINE_FIELDS`
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java:
##########
@@ -1153,8 +1153,8 @@ public TableBuilder setBaseFileFormat(String baseFileFormat) {
return this;
}
- public TableBuilder setPreCombineField(String preCombineField) {
- this.preCombineField = preCombineField;
+ public TableBuilder setPreCombineFields(String preCombineField) {
Review Comment:
This string is a comma separated combination of fields. Added docs to make it more clear.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2227075992
##########
hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java:
##########
@@ -1659,6 +1666,11 @@ public static Comparable<?> unwrapAvroValueWrapper(Object avroValueWrapper) {
return ((BytesWrapper) avroValueWrapper).getValue();
} else if (avroValueWrapper instanceof StringWrapper) {
return ((StringWrapper) avroValueWrapper).getValue();
+ } else if (avroValueWrapper instanceof ArrayWrapper) {
Review Comment:
I will rename the `OrderingValue` into `ArrayComparable` to make it more generalized.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3121532157
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
}, {
"hash" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"triggerType" : "PUSH"
}, {
"hash" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947",
"triggerID" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"triggerType" : "PUSH"
}, {
"hash" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6949",
"triggerID" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"triggerType" : "PUSH"
}, {
"hash" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"triggerType" : "PUSH"
}, {
"hash" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6950",
"triggerID" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"triggerType" : "PUSH"
}, {
"hash" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6974",
"triggerID" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"triggerType" : "PUSH"
}, {
"hash" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6991",
"triggerID" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"triggerType" : "PUSH"
}, {
"hash" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6993",
"triggerID" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"triggerType" : "PUSH"
}, {
"hash" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6995",
"triggerID" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"triggerType" : "PUSH"
}, {
"hash" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6996",
"triggerID" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"triggerType" : "PUSH"
}, {
"hash" : "dc851f6226047629aec76f30cd9843ac635a66d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7001",
"triggerID" : "dc851f6226047629aec76f30cd9843ac635a66d9",
"triggerType" : "PUSH"
}, {
"hash" : "8c51a6c54679f5723053ae3d9efe343bdeea9445",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7041",
"triggerID" : "8c51a6c54679f5723053ae3d9efe343bdeea9445",
"triggerType" : "PUSH"
}, {
"hash" : "723b2bdc505e58d9a7525be996402ce55d2b9bff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7123",
"triggerID" : "723b2bdc505e58d9a7525be996402ce55d2b9bff",
"triggerType" : "PUSH"
}, {
"hash" : "c7902b06721cfb4e51d166b5799c20d58f079404",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7125",
"triggerID" : "c7902b06721cfb4e51d166b5799c20d58f079404",
"triggerType" : "PUSH"
}, {
"hash" : "5e3962074d7b4f7dceaedf9a391e763e054f9b9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7139",
"triggerID" : "5e3962074d7b4f7dceaedf9a391e763e054f9b9d",
"triggerType" : "PUSH"
}, {
"hash" : "371f9b89c48b3dc9a9b95b15266b9f2fd4201a56",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7165",
"triggerID" : "371f9b89c48b3dc9a9b95b15266b9f2fd4201a56",
"triggerType" : "PUSH"
}, {
"hash" : "2f9d6614a9ea6ed28bdc3654d37a7dc17563175b",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7169",
"triggerID" : "2f9d6614a9ea6ed28bdc3654d37a7dc17563175b",
"triggerType" : "PUSH"
}, {
"hash" : "c74a62c8f7f57fc91116e40fb3db163c60853a23",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c74a62c8f7f57fc91116e40fb3db163c60853a23",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e19abadae9dece1aa78868839605a3d23a8b29fd UNKNOWN
* 4ba45f607a07f3f036e164dd68532fbc8b4bc3b8 UNKNOWN
* 2f9d6614a9ea6ed28bdc3654d37a7dc17563175b Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7169)
* c74a62c8f7f57fc91116e40fb3db163c60853a23 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3121568234
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
}, {
"hash" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"triggerType" : "PUSH"
}, {
"hash" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947",
"triggerID" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"triggerType" : "PUSH"
}, {
"hash" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6949",
"triggerID" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"triggerType" : "PUSH"
}, {
"hash" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"triggerType" : "PUSH"
}, {
"hash" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6950",
"triggerID" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"triggerType" : "PUSH"
}, {
"hash" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6974",
"triggerID" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"triggerType" : "PUSH"
}, {
"hash" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6991",
"triggerID" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"triggerType" : "PUSH"
}, {
"hash" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6993",
"triggerID" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"triggerType" : "PUSH"
}, {
"hash" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6995",
"triggerID" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"triggerType" : "PUSH"
}, {
"hash" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6996",
"triggerID" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"triggerType" : "PUSH"
}, {
"hash" : "dc851f6226047629aec76f30cd9843ac635a66d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7001",
"triggerID" : "dc851f6226047629aec76f30cd9843ac635a66d9",
"triggerType" : "PUSH"
}, {
"hash" : "8c51a6c54679f5723053ae3d9efe343bdeea9445",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7041",
"triggerID" : "8c51a6c54679f5723053ae3d9efe343bdeea9445",
"triggerType" : "PUSH"
}, {
"hash" : "723b2bdc505e58d9a7525be996402ce55d2b9bff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7123",
"triggerID" : "723b2bdc505e58d9a7525be996402ce55d2b9bff",
"triggerType" : "PUSH"
}, {
"hash" : "c7902b06721cfb4e51d166b5799c20d58f079404",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7125",
"triggerID" : "c7902b06721cfb4e51d166b5799c20d58f079404",
"triggerType" : "PUSH"
}, {
"hash" : "5e3962074d7b4f7dceaedf9a391e763e054f9b9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7139",
"triggerID" : "5e3962074d7b4f7dceaedf9a391e763e054f9b9d",
"triggerType" : "PUSH"
}, {
"hash" : "371f9b89c48b3dc9a9b95b15266b9f2fd4201a56",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7165",
"triggerID" : "371f9b89c48b3dc9a9b95b15266b9f2fd4201a56",
"triggerType" : "PUSH"
}, {
"hash" : "2f9d6614a9ea6ed28bdc3654d37a7dc17563175b",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7169",
"triggerID" : "2f9d6614a9ea6ed28bdc3654d37a7dc17563175b",
"triggerType" : "PUSH"
}, {
"hash" : "c74a62c8f7f57fc91116e40fb3db163c60853a23",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7170",
"triggerID" : "c74a62c8f7f57fc91116e40fb3db163c60853a23",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e19abadae9dece1aa78868839605a3d23a8b29fd UNKNOWN
* 4ba45f607a07f3f036e164dd68532fbc8b4bc3b8 UNKNOWN
* 2f9d6614a9ea6ed28bdc3654d37a7dc17563175b Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7169)
* c74a62c8f7f57fc91116e40fb3db163c60853a23 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7170)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2193935936
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -169,8 +170,9 @@ public class HoodieWriteConfig extends HoodieConfig {
public static final ConfigProperty<String> PRECOMBINE_FIELD_NAME = ConfigProperty
.key("hoodie.datasource.write.precombine.field")
.noDefaultValue()
- .withDocumentation("Field used in preCombining before actual write. When two records have the same key value, "
- + "we will pick the one with the largest value for the precombine field, determined by Object.compareTo(..)");
+ .withDocumentation("Comma separated list of fields used in preCombining before actual write. When two records have the same key value, "
Review Comment:
wondering if we can rename the config to "hoodie.write.ordering.field"
and then add old config value as alternate key.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3121424152
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
}, {
"hash" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"triggerType" : "PUSH"
}, {
"hash" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947",
"triggerID" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"triggerType" : "PUSH"
}, {
"hash" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6949",
"triggerID" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"triggerType" : "PUSH"
}, {
"hash" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"triggerType" : "PUSH"
}, {
"hash" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6950",
"triggerID" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"triggerType" : "PUSH"
}, {
"hash" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6974",
"triggerID" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"triggerType" : "PUSH"
}, {
"hash" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6991",
"triggerID" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"triggerType" : "PUSH"
}, {
"hash" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6993",
"triggerID" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"triggerType" : "PUSH"
}, {
"hash" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6995",
"triggerID" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"triggerType" : "PUSH"
}, {
"hash" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6996",
"triggerID" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"triggerType" : "PUSH"
}, {
"hash" : "dc851f6226047629aec76f30cd9843ac635a66d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7001",
"triggerID" : "dc851f6226047629aec76f30cd9843ac635a66d9",
"triggerType" : "PUSH"
}, {
"hash" : "8c51a6c54679f5723053ae3d9efe343bdeea9445",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7041",
"triggerID" : "8c51a6c54679f5723053ae3d9efe343bdeea9445",
"triggerType" : "PUSH"
}, {
"hash" : "723b2bdc505e58d9a7525be996402ce55d2b9bff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7123",
"triggerID" : "723b2bdc505e58d9a7525be996402ce55d2b9bff",
"triggerType" : "PUSH"
}, {
"hash" : "c7902b06721cfb4e51d166b5799c20d58f079404",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7125",
"triggerID" : "c7902b06721cfb4e51d166b5799c20d58f079404",
"triggerType" : "PUSH"
}, {
"hash" : "5e3962074d7b4f7dceaedf9a391e763e054f9b9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7139",
"triggerID" : "5e3962074d7b4f7dceaedf9a391e763e054f9b9d",
"triggerType" : "PUSH"
}, {
"hash" : "371f9b89c48b3dc9a9b95b15266b9f2fd4201a56",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7165",
"triggerID" : "371f9b89c48b3dc9a9b95b15266b9f2fd4201a56",
"triggerType" : "PUSH"
}, {
"hash" : "2f9d6614a9ea6ed28bdc3654d37a7dc17563175b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2f9d6614a9ea6ed28bdc3654d37a7dc17563175b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e19abadae9dece1aa78868839605a3d23a8b29fd UNKNOWN
* 4ba45f607a07f3f036e164dd68532fbc8b4bc3b8 UNKNOWN
* 371f9b89c48b3dc9a9b95b15266b9f2fd4201a56 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7165)
* 2f9d6614a9ea6ed28bdc3654d37a7dc17563175b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3112181560
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
}, {
"hash" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"triggerType" : "PUSH"
}, {
"hash" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947",
"triggerID" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"triggerType" : "PUSH"
}, {
"hash" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6949",
"triggerID" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"triggerType" : "PUSH"
}, {
"hash" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"triggerType" : "PUSH"
}, {
"hash" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6950",
"triggerID" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"triggerType" : "PUSH"
}, {
"hash" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6974",
"triggerID" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"triggerType" : "PUSH"
}, {
"hash" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6991",
"triggerID" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"triggerType" : "PUSH"
}, {
"hash" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6993",
"triggerID" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"triggerType" : "PUSH"
}, {
"hash" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6995",
"triggerID" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"triggerType" : "PUSH"
}, {
"hash" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6996",
"triggerID" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"triggerType" : "PUSH"
}, {
"hash" : "dc851f6226047629aec76f30cd9843ac635a66d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7001",
"triggerID" : "dc851f6226047629aec76f30cd9843ac635a66d9",
"triggerType" : "PUSH"
}, {
"hash" : "8c51a6c54679f5723053ae3d9efe343bdeea9445",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7041",
"triggerID" : "8c51a6c54679f5723053ae3d9efe343bdeea9445",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e19abadae9dece1aa78868839605a3d23a8b29fd UNKNOWN
* 4ba45f607a07f3f036e164dd68532fbc8b4bc3b8 UNKNOWN
* 8c51a6c54679f5723053ae3d9efe343bdeea9445 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7041)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3045946840
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 26a8660da376934b6f864b04f96e160d342886ca Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3053628145
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 125b574e0176d7ec688acd05f9138f821f6dfe03 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657)
* c90d258b5f530a935ac34d36919454be963c1d5b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3058297678
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c90d258b5f530a935ac34d36919454be963c1d5b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662)
* f4b6ff8db1543238f986e2d604f4f71971313b95 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2230929901
##########
hudi-common/src/main/java/org/apache/hudi/common/engine/HoodieReaderContext.java:
##########
@@ -380,21 +380,23 @@ private BiFunction<T, Schema, String> virtualKeyExtractor(String[] recordKeyFiel
/**
* Gets the ordering value in particular type.
*
- * @param record An option of record.
- * @param schema The Avro schema of the record.
- * @param orderingFieldName name of the ordering field
+ * @param record An option of record.
+ * @param schema The Avro schema of the record.
+ * @param orderingFieldNames name of the ordering field
* @return The ordering value.
*/
public Comparable getOrderingValue(T record,
Schema schema,
- Option<String> orderingFieldName) {
- if (orderingFieldName.isEmpty()) {
- return DEFAULT_ORDERING_VALUE;
+ Option<List<String>> orderingFieldNames) {
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3104903545
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
}, {
"hash" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"triggerType" : "PUSH"
}, {
"hash" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947",
"triggerID" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"triggerType" : "PUSH"
}, {
"hash" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6949",
"triggerID" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"triggerType" : "PUSH"
}, {
"hash" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"triggerType" : "PUSH"
}, {
"hash" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6950",
"triggerID" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"triggerType" : "PUSH"
}, {
"hash" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6974",
"triggerID" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e19abadae9dece1aa78868839605a3d23a8b29fd UNKNOWN
* 4ba45f607a07f3f036e164dd68532fbc8b4bc3b8 UNKNOWN
* 21043f2bc87787d692ec55c2e4150ab29a396264 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6974)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2224119631
##########
hudi-common/src/main/avro/HoodieDeleteRecordList.avsc:
##########
@@ -54,17 +54,23 @@
"doc": "Ordering value determining the order of merging on the same key",
"type": [
"null",
- "org.apache.hudi.avro.model.BooleanWrapper",
- "org.apache.hudi.avro.model.IntWrapper",
- "org.apache.hudi.avro.model.LongWrapper",
- "org.apache.hudi.avro.model.FloatWrapper",
- "org.apache.hudi.avro.model.DoubleWrapper",
- "org.apache.hudi.avro.model.BytesWrapper",
- "org.apache.hudi.avro.model.StringWrapper",
- "org.apache.hudi.avro.model.DateWrapper",
- "org.apache.hudi.avro.model.DecimalWrapper",
- "org.apache.hudi.avro.model.TimeMicrosWrapper",
- "org.apache.hudi.avro.model.TimestampMicrosWrapper"
+ {
+ "type": "array",
Review Comment:
This is not a safe schema evolution. We need to be able to read the log files of records written by previous Hudi releases.
##########
hudi-common/src/main/java/org/apache/hudi/common/engine/HoodieReaderContext.java:
##########
@@ -380,21 +380,23 @@ private BiFunction<T, Schema, String> virtualKeyExtractor(String[] recordKeyFiel
/**
* Gets the ordering value in particular type.
*
- * @param record An option of record.
- * @param schema The Avro schema of the record.
- * @param orderingFieldName name of the ordering field
+ * @param record An option of record.
+ * @param schema The Avro schema of the record.
+ * @param orderingFieldNames name of the ordering field
* @return The ordering value.
*/
public Comparable getOrderingValue(T record,
Schema schema,
- Option<String> orderingFieldName) {
- if (orderingFieldName.isEmpty()) {
- return DEFAULT_ORDERING_VALUE;
+ Option<List<String>> orderingFieldNames) {
Review Comment:
Would it be simpler to just use an empty list instead of an empty option when there are are no ordering fields specified?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3095883109
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 98a828795959a9cfc01d6c04ea8739228fa6009c Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906)
* fc0eba477035f595d72c7f0ce133a2d023df0a53 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3101799521
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
}, {
"hash" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 64d9924c454d76f90eaa6f5089e5f36345bf8f6f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919)
* bf66a590541998a22aa83fe644c03ff5d7eb2b47 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944)
* e19abadae9dece1aa78868839605a3d23a8b29fd UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3101805602
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
}, {
"hash" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bf66a590541998a22aa83fe644c03ff5d7eb2b47 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944)
* e19abadae9dece1aa78868839605a3d23a8b29fd UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2220609107
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestDataSource.java:
##########
@@ -41,6 +41,7 @@ public class TestDataSource extends AbstractBaseTestSource {
private static final Logger LOG = LoggerFactory.getLogger(TestDataSource.class);
public static boolean returnEmptyBatch = false;
+ public static Option<String> recordInstantTime = Option.empty();
Review Comment:
we should be resetting this to `Option.empty()` in either BeforeEach or in the constructor of `TestDataSource`
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestHoodieFileGroupReaderBase.java:
##########
@@ -604,13 +651,30 @@ protected void validateOutputFromFileGroupReader(StorageConfiguration<?> storage
List<HoodieRecord> expectedHoodieUnmergedRecords) throws Exception {
HoodieTableMetaClient metaClient = HoodieTestUtils.createMetaClient(storageConf, tablePath);
Schema avroSchema = new TableSchemaResolver(metaClient).getTableAvroSchema();
+ expectedHoodieRecords = getExpectedHoodieRecordsWithOrderingValue(expectedHoodieRecords, metaClient, avroSchema);
+ expectedHoodieUnmergedRecords = getExpectedHoodieRecordsWithOrderingValue(expectedHoodieUnmergedRecords, metaClient, avroSchema);
List<HoodieTestDataGenerator.RecordIdentifier> expectedRecords = convertHoodieRecords(expectedHoodieRecords, avroSchema);
List<HoodieTestDataGenerator.RecordIdentifier> expectedUnmergedRecords = convertHoodieRecords(expectedHoodieUnmergedRecords, avroSchema);
validateOutputFromFileGroupReaderWithExistingRecords(
storageConf, tablePath, containsBaseFile, expectedLogFileNum, recordMergeMode,
expectedRecords, expectedUnmergedRecords);
}
+ private static List<HoodieRecord> getExpectedHoodieRecordsWithOrderingValue(List<HoodieRecord> expectedHoodieRecords, HoodieTableMetaClient metaClient, Schema avroSchema) {
+ return expectedHoodieRecords.stream().map(rec -> {
+ RawTripTestPayload oldPayload = (RawTripTestPayload) rec.getData();
+ try {
+ List<String> orderingFields = metaClient.getTableConfig().getPreCombineFieldList().get();
+ HoodieAvroRecord avroRecord = ((HoodieAvroRecord) rec);
+ Comparable orderingValue = Comparables.create(orderingFields, field -> (Comparable) avroRecord.getColumnValueAsJava(avroSchema, field, new TypedProperties()));
+ RawTripTestPayload newPayload = new RawTripTestPayload(Option.ofNullable(oldPayload.getJsonData()), oldPayload.getRowKey(), oldPayload.getPartitionPath(), null, false, orderingValue);
Review Comment:
Have you thought about or attempted fixing HoodieTestDataGenerator only to instantiate `RawTripTestPayload` w/ right ordering value.
then, we fix it for all test data generation right, and not just the `TestHoodieFileGroupReaderBase.java`
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/TestHoodieDeltaStreamer.java:
##########
@@ -1955,6 +1957,62 @@ public void testFilterDupesWithPrecombine(
UtilitiesTestBase.Helpers.deleteFileFromDfs(fs, tableBasePath);
}
+ @ParameterizedTest
+ @EnumSource(HoodieTableType.class)
+ public void testDeltaStreamerWithMultipleOrderingFields(HoodieTableType tableType) throws Exception {
+ String tableBasePath = basePath + "/test_with_multiple_ordering_fields";
+ HoodieDeltaStreamer.Config cfg =
+ TestHelpers.makeConfig(tableBasePath, WriteOperationType.BULK_INSERT);
+ cfg.tableType = tableType.name();
+ cfg.filterDupes = true;
+ cfg.sourceOrderingFields = "timestamp,rider";
+ cfg.recordMergeMode = RecordMergeMode.EVENT_TIME_ORDERING;
+ cfg.payloadClassName = DefaultHoodieRecordPayload.class.getName();
+ cfg.recordMergeStrategyId = HoodieRecordMerger.EVENT_TIME_BASED_MERGE_STRATEGY_UUID;
+ // Set record merge mode to event time ordering
+ cfg.configs.add(String.format("%s=%s", HoodieWriteConfig.RECORD_MERGE_MODE.key(), RecordMergeMode.EVENT_TIME_ORDERING.name()));
+ //addRecordMerger(HoodieRecordType.AVRO, cfg.configs);
Review Comment:
why commented out?
if not required, can we remove it
##########
hudi-common/src/main/java/org/apache/hudi/Comparables.java:
##########
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hudi.common.util.ValidationUtils;
+
+import java.io.Serializable;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+import static org.apache.hudi.common.model.HoodieRecord.DEFAULT_ORDERING_VALUE;
+
+public class Comparables implements Comparable, Serializable {
+ protected static final long serialVersionUID = 1L;
+ private static final Comparables DEFAULT_VALUE = new Comparables(DEFAULT_ORDERING_VALUE);
+
+ private final List<Comparable> comparables;
+
+ public Comparables(List<Comparable> comparables) {
+ this.comparables = comparables;
+ }
+
+ public Comparables(Comparable comparable) {
+ this.comparables = Collections.singletonList(comparable);
+ }
+
+ public static Comparables getDefault() {
+ return DEFAULT_VALUE;
+ }
+
+ public static boolean isDefault(Comparable orderingVal) {
+ if (orderingVal instanceof Comparables) {
+ return ((Comparables) orderingVal).comparables.size() == 1
+ && ((Comparables) orderingVal).comparables.get(0).equals(DEFAULT_ORDERING_VALUE);
+ } else {
+ return orderingVal.equals(DEFAULT_ORDERING_VALUE);
+ }
+ }
+
+ /**
+ * Returns whether the given two comparable values come from the same runtime class.
+ */
+ public static boolean isSameClass(Comparable<?> v, Comparable<?> o) {
+ if (v.getClass() != o.getClass()) {
+ // If class is not same return false
+ return false;
+ } else if (v instanceof Comparables) {
+ if (((Comparables) v).comparables.size() != ((Comparables) o).comparables.size()) {
+ // if comparables size is not same return false
+ return false;
+ } else {
+ // compare class of comparable list of both arguments
+ return IntStream.range(0, ((Comparables) v).comparables.size())
+ .mapToObj(i -> ((Comparables) v).comparables.get(i).getClass() == ((Comparables) o).comparables.get(i).getClass())
+ .reduce(Boolean::logicalAnd)
+ .orElse(true);
+ }
+ }
+ // return true if class is same and input objects are not instance of Comparables class
+ return true;
+ }
+
+ public static Comparable getDefaultOrderingValue() {
+ return DEFAULT_ORDERING_VALUE;
+ }
+
+ @Override
+ public int compareTo(Object o) {
+ ValidationUtils.checkArgument(o instanceof Comparables, "Comparables can only be compared with another Comparables");
+ Comparables otherComparables = (Comparables) o;
+ ValidationUtils.checkArgument(comparables.size() == otherComparables.comparables.size(), "Comparables should be of same size");
+ for (int i = 0; i < comparables.size(); i++) {
+ int comparingValue = comparables.get(i).compareTo(otherComparables.comparables.get(i));
Review Comment:
yes, lets ensure the ConfigProperty calls this out
##########
hudi-common/src/main/java/org/apache/hudi/Comparables.java:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hudi.common.util.ValidationUtils;
+
+import java.io.Serializable;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.model.HoodieRecord.DEFAULT_ORDERING_VALUE;
+
+public class Comparables implements Comparable, Serializable {
+ protected static final long serialVersionUID = 1L;
+
+ private final List<Comparable> comparables;
Review Comment:
hey @the-other-tim-brown : Whats your take on this? do you have any suggestion towards this.
@lokeshj1703 : can we do a mirco benchmarking. using master for a single ordering field vs this branch.
we can create a MOR file slice w/ N no of log files so that we do repeated compare calls. and then do snapshot read to see if we can spot any perf difference.
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestHoodieFileGroupReaderBase.java:
##########
@@ -185,6 +189,49 @@ public void testReadFileGroupInMergeOnReadTable(RecordMergeMode recordMergeMode,
}
}
+ @ParameterizedTest
+ @ValueSource(strings = {"avro", "parquet"})
+ public void testReadFileGroupWithMultipleOrderingFields(String logDataBlockFormat) throws Exception {
+ RecordMergeMode recordMergeMode = RecordMergeMode.EVENT_TIME_ORDERING;
+ Map<String, String> writeConfigs = new HashMap<>(getCommonConfigs(recordMergeMode, true));
+ writeConfigs.put(HoodieStorageConfig.LOGFILE_DATA_BLOCK_FORMAT.key(), logDataBlockFormat);
+ writeConfigs.put("hoodie.datasource.write.table.type", HoodieTableType.MERGE_ON_READ.name());
+ // Use two precombine values - combination of timestamp and rider
+ String orderingValues = "timestamp,rider";
+ writeConfigs.put("hoodie.datasource.write.precombine.field", orderingValues);
+ writeConfigs.put("hoodie.payload.ordering.field", orderingValues);
+
+ try (HoodieTestDataGenerator dataGen = new HoodieTestDataGenerator(0xDEEF)) {
+ // Initial commit. rider column gets value of rider-002
+ List<HoodieRecord> initialRecords = dataGen.generateInserts("002", 100);
+ commitToTable(initialRecords, INSERT.value(), true, writeConfigs);
+ validateOutputFromFileGroupReader(
+ getStorageConf(), getBasePath(), true, 0, recordMergeMode,
+ initialRecords, initialRecords);
+
+ // The updates have rider values as rider-001 and the existing records have rider values as rider-001
Review Comment:
guess there is a typo in this comment.
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/TestHoodieDeltaStreamer.java:
##########
@@ -1955,6 +1957,62 @@ public void testFilterDupesWithPrecombine(
UtilitiesTestBase.Helpers.deleteFileFromDfs(fs, tableBasePath);
}
+ @ParameterizedTest
+ @EnumSource(HoodieTableType.class)
+ public void testDeltaStreamerWithMultipleOrderingFields(HoodieTableType tableType) throws Exception {
+ String tableBasePath = basePath + "/test_with_multiple_ordering_fields";
+ HoodieDeltaStreamer.Config cfg =
+ TestHelpers.makeConfig(tableBasePath, WriteOperationType.BULK_INSERT);
+ cfg.tableType = tableType.name();
+ cfg.filterDupes = true;
+ cfg.sourceOrderingFields = "timestamp,rider";
+ cfg.recordMergeMode = RecordMergeMode.EVENT_TIME_ORDERING;
+ cfg.payloadClassName = DefaultHoodieRecordPayload.class.getName();
+ cfg.recordMergeStrategyId = HoodieRecordMerger.EVENT_TIME_BASED_MERGE_STRATEGY_UUID;
+ // Set record merge mode to event time ordering
+ cfg.configs.add(String.format("%s=%s", HoodieWriteConfig.RECORD_MERGE_MODE.key(), RecordMergeMode.EVENT_TIME_ORDERING.name()));
+ //addRecordMerger(HoodieRecordType.AVRO, cfg.configs);
+ TestDataSource.recordInstantTime = Option.of("002");
+ new HoodieStreamer(cfg, jsc).sync();
+
+ assertRecordCount(1000, tableBasePath, sqlContext);
+ TestHelpers.assertCommitMetadata("00000", tableBasePath, 1);
+
+ // Generate new updates with lower recordInstantTime so that updates are rejected
+ TestDataSource.recordInstantTime = Option.of("001");
+ runStreamSync(cfg, false, 50, WriteOperationType.UPSERT);
+ int numInserts = 25;
+ // TestDataSource generates 500 inserts, 450 updates and 50 deletes
+ assertRecordCount(1025, tableBasePath, sqlContext); // if filter dupes is not enabled, we should be expecting 3000 records here.
+ TestHelpers.assertCommitMetadata("00001", tableBasePath, 2);
+ // Filter records with rider-001 value and deduct the number of inserts
+ long numUpdates = sparkSession.read().format("hudi").load(tableBasePath).filter("rider = 'rider-001'").count()
+ - numInserts;
+ // There should be no updates since ordering value rider-001 is lower than existing record ordering value rider-002
+ assertEquals(0, numUpdates);
+
+ // Generate new updates with lower recordInstantTime so that updates are rejected
Review Comment:
again, lets fix the comments
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestHoodieFileGroupReaderBase.java:
##########
@@ -185,6 +189,49 @@ public void testReadFileGroupInMergeOnReadTable(RecordMergeMode recordMergeMode,
}
}
+ @ParameterizedTest
+ @ValueSource(strings = {"avro", "parquet"})
+ public void testReadFileGroupWithMultipleOrderingFields(String logDataBlockFormat) throws Exception {
Review Comment:
we may not need diff log block formats.
but lets add a test at spark ds layer for COW table. (I assume it does not make sense to add COW here w/ FG reader)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2227083864
##########
hudi-common/src/main/java/org/apache/hudi/common/util/OrderingValues.java:
##########
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util;
+
+import org.apache.hudi.common.model.EmptyOrderingValue;
+import org.apache.hudi.common.model.MultipleFieldsOrderingValue;
+import org.apache.hudi.common.model.OrderingValue;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.function.Function;
+
+/**
+ * Factory class for {@link org.apache.hudi.common.model.OrderingValue}.
+ */
+public class OrderingValues {
+ private static final Comparable<?> DEFAULT_VALUE = org.apache.hudi.common.model.HoodieRecord.DEFAULT_ORDERING_VALUE;
+
+ /**
+ * Creates an {@code OrderingValue} with given values.
+ *
+ * <p>CAUTION: if the values are fetched through ordering fields,
+ * please use {@link #create(List, Function)} or {@link #create(String[], Function)}
+ * as much as possible to avoid streaming on the ordering fields in row level.</p>
+ */
+ public static Comparable create(Comparable[] orderingValues) {
+ if (orderingValues.length == 1) {
+ return orderingValues[0];
Review Comment:
Thanks for the explanation, I am not strongly opinionated about either approach so was curious what you found when trying it out. I'm guessing we would have to make even more changes to plug that interface everywhere we use comparable today so makes sense to try to keep the change to a minimum here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3106069517
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
}, {
"hash" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"triggerType" : "PUSH"
}, {
"hash" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947",
"triggerID" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"triggerType" : "PUSH"
}, {
"hash" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6949",
"triggerID" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"triggerType" : "PUSH"
}, {
"hash" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"triggerType" : "PUSH"
}, {
"hash" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6950",
"triggerID" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"triggerType" : "PUSH"
}, {
"hash" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6974",
"triggerID" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"triggerType" : "PUSH"
}, {
"hash" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6991",
"triggerID" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"triggerType" : "PUSH"
}, {
"hash" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e19abadae9dece1aa78868839605a3d23a8b29fd UNKNOWN
* 4ba45f607a07f3f036e164dd68532fbc8b4bc3b8 UNKNOWN
* 7c1ded9e7881ffea36adeceb34c89ea25353bc0d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6991)
* a5ef18cb6ffcfb3a347018a6908bd3a8f0334712 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3106949495
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
}, {
"hash" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"triggerType" : "PUSH"
}, {
"hash" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947",
"triggerID" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"triggerType" : "PUSH"
}, {
"hash" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6949",
"triggerID" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"triggerType" : "PUSH"
}, {
"hash" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"triggerType" : "PUSH"
}, {
"hash" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6950",
"triggerID" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"triggerType" : "PUSH"
}, {
"hash" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6974",
"triggerID" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"triggerType" : "PUSH"
}, {
"hash" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6991",
"triggerID" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"triggerType" : "PUSH"
}, {
"hash" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6993",
"triggerID" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"triggerType" : "PUSH"
}, {
"hash" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6995",
"triggerID" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"triggerType" : "PUSH"
}, {
"hash" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e19abadae9dece1aa78868839605a3d23a8b29fd UNKNOWN
* 4ba45f607a07f3f036e164dd68532fbc8b4bc3b8 UNKNOWN
* a5ef18cb6ffcfb3a347018a6908bd3a8f0334712 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6993)
* 63af22477ed62d797f70568430ce6f142ffc3cc0 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6995)
* 2dba2b4f6c8d57fea9ff262e842f45153d56787f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2227162939
##########
hudi-common/src/main/java/org/apache/hudi/common/engine/HoodieReaderContext.java:
##########
@@ -380,21 +380,23 @@ private BiFunction<T, Schema, String> virtualKeyExtractor(String[] recordKeyFiel
/**
* Gets the ordering value in particular type.
*
- * @param record An option of record.
- * @param schema The Avro schema of the record.
- * @param orderingFieldName name of the ordering field
+ * @param record An option of record.
+ * @param schema The Avro schema of the record.
+ * @param orderingFieldNames name of the ordering field
* @return The ordering value.
*/
public Comparable getOrderingValue(T record,
Schema schema,
- Option<String> orderingFieldName) {
- if (orderingFieldName.isEmpty()) {
- return DEFAULT_ORDERING_VALUE;
+ Option<List<String>> orderingFieldNames) {
Review Comment:
+1 with @the-other-tim-brown , let's use empty list and array to represent the non ordering value use cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2225488297
##########
hudi-common/src/main/java/org/apache/hudi/common/engine/HoodieReaderContext.java:
##########
@@ -380,21 +380,23 @@ private BiFunction<T, Schema, String> virtualKeyExtractor(String[] recordKeyFiel
/**
* Gets the ordering value in particular type.
*
- * @param record An option of record.
- * @param schema The Avro schema of the record.
- * @param orderingFieldName name of the ordering field
+ * @param record An option of record.
+ * @param schema The Avro schema of the record.
+ * @param orderingFieldNames name of the ordering field
* @return The ordering value.
*/
public Comparable getOrderingValue(T record,
Schema schema,
- Option<String> orderingFieldName) {
- if (orderingFieldName.isEmpty()) {
- return DEFAULT_ORDERING_VALUE;
+ Option<List<String>> orderingFieldNames) {
Review Comment:
HoodieFileGroupReader uses Option for commit time ordering it seems.
```
this.orderingFieldNames = readerContext.getMergeMode() == RecordMergeMode.COMMIT_TIME_ORDERING
? Option.empty()
: ConfigUtils.getOrderingFields(props).map(Arrays::asList)
.or(() -> hoodieTableMetaClient.getTableConfig().getPreCombineFields());
```
The function is getting called from `BufferedRecord#forRecordWithContext` which is using orderingFieldNames from HoodieFileGroupReader. So did not change the functionality here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2226869880
##########
hudi-common/src/main/java/org/apache/hudi/common/engine/HoodieReaderContext.java:
##########
@@ -479,6 +483,15 @@ public final UnaryOperator<T> projectRecord(Schema from, Schema to) {
return projectRecord(from, to, Collections.emptyMap());
}
+ /**
+ * Converts the ordering values to the specific engine type.
+ */
+ public final Comparable convertOrderingValueToEngineType(Comparable value) {
+ return value instanceof OrderingValue
Review Comment:
Can we make it so all ordering values are an instance of OrderingValue so the casts can be avoided?
##########
hudi-common/src/main/java/org/apache/hudi/common/util/OrderingValues.java:
##########
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util;
+
+import org.apache.hudi.common.model.EmptyOrderingValue;
+import org.apache.hudi.common.model.MultipleFieldsOrderingValue;
+import org.apache.hudi.common.model.OrderingValue;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.function.Function;
+
+/**
+ * Factory class for {@link org.apache.hudi.common.model.OrderingValue}.
+ */
+public class OrderingValues {
+ private static final Comparable<?> DEFAULT_VALUE = org.apache.hudi.common.model.HoodieRecord.DEFAULT_ORDERING_VALUE;
+
+ /**
+ * Creates an {@code OrderingValue} with given values.
+ *
+ * <p>CAUTION: if the values are fetched through ordering fields,
+ * please use {@link #create(List, Function)} or {@link #create(String[], Function)}
+ * as much as possible to avoid streaming on the ordering fields in row level.</p>
+ */
+ public static Comparable create(Comparable[] orderingValues) {
+ if (orderingValues.length == 1) {
+ return orderingValues[0];
Review Comment:
@danny0405 I saw you originally had `SingleFieldOrderingValue` at one point. That could make it easier to deal with the ordering values in all the new code paths with FileGroupReader and BufferedRecord by using a return type of `OrderingValue` instead of comparable so it is more clear what the value represents. What was the reason for removing it?
##########
hudi-common/src/main/java/org/apache/hudi/common/model/MultipleFieldsOrderingValue.java:
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.hudi.common.util.ValidationUtils;
+
+import java.io.Serializable;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Objects;
+import java.util.function.Function;
+
+/**
+ * A wrapper class to manage multiple fields ordering values across Hudi.
+ */
+public class MultipleFieldsOrderingValue implements OrderingValue<MultipleFieldsOrderingValue>, Serializable {
Review Comment:
Can you add some unit tests for this class?
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -926,8 +927,12 @@ static RecordMergeMode inferRecordMergeModeFromMergeStrategyId(String recordMerg
}
}
- public String getPreCombineField() {
- return getString(PRECOMBINE_FIELD);
+ public Option<List<String>> getPreCombineFields() {
Review Comment:
Should we use an empty list here as well to represent a table without an ordering field?
##########
hudi-hadoop-common/src/test/java/org/apache/hudi/common/table/log/block/TestHoodieDeleteBlock.java:
##########
@@ -100,7 +100,7 @@ public static Stream<Arguments> orderingValueParams() {
{new String[] {"val1", "val2", "val3", null}},
{new Timestamp[] {new Timestamp(1690766971000L), new Timestamp(1672536571000L)}},
// {new LocalDate[] {LocalDate.of(2023, 1, 1), LocalDate.of(1980, 7, 1)}} // HUDI-8854
- {new BigDecimal[] {new BigDecimal("12345678901234.2948"), new BigDecimal("23456789012345.4856")}}
+ {new BigDecimal[] {new BigDecimal("12345678901234.294800000000000"), new BigDecimal("23456789012345.485600000000000")}}
Review Comment:
Why does this need to change?
##########
hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java:
##########
@@ -1659,6 +1666,11 @@ public static Comparable<?> unwrapAvroValueWrapper(Object avroValueWrapper) {
return ((BytesWrapper) avroValueWrapper).getValue();
} else if (avroValueWrapper instanceof StringWrapper) {
return ((StringWrapper) avroValueWrapper).getValue();
+ } else if (avroValueWrapper instanceof ArrayWrapper) {
Review Comment:
This is not generic, it assumes all usages of ArrayWrapper is confined to `OrderingValues`. If that is the case, should the model be named something to indicate that instead of using a more generic name of ArrayWrapper?
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/HoodieDatasetBulkInsertHelper.scala:
##########
@@ -255,6 +257,16 @@ object HoodieDatasetBulkInsertHelper
}
}
+ private def getOrderingValue(nestedFieldPaths: List[NestedFieldPath], r: InternalRow): Comparable[_] = {
+ if (nestedFieldPaths.isEmpty) {
+ EmptyOrderingValue.getInstance();
+ } else if (nestedFieldPaths.size == 1) {
+ getNestedInternalRowValue(r, nestedFieldPaths.head).asInstanceOf[Comparable[_]]
+ } else {
+ new MultipleFieldsOrderingValue(nestedFieldPaths.map(_ => getNestedInternalRowValue(r, nestedFieldPaths.head).asInstanceOf[Comparable[_]]).toArray)
Review Comment:
Instead of adding this logic here, can you just use `create` method which has similar checks?
`OrderingValues.create(nestedFieldPaths.map(f => getNestedInternalRowValue(r, f).asInstanceOf[Comparable[_]]).toArray)`
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java:
##########
@@ -1190,8 +1190,12 @@ public TableBuilder setBaseFileFormat(String baseFileFormat) {
return this;
}
- public TableBuilder setPreCombineField(String preCombineField) {
- this.preCombineField = preCombineField;
+ /**
+ * Sets preCombine fields to be used by the table
Review Comment:
Add note that it is a comma separate list
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestCOWDataSource.scala:
##########
@@ -142,6 +141,48 @@ class TestCOWDataSource extends HoodieSparkClientTestBase with ScalaAssertionSup
spark.read.format("org.apache.hudi").options(readOpts).load(basePath).count()
}
+ @Test
+ def testMultipleOrderingFields() {
+ val (writeOpts, readOpts) = getWriterReaderOpts(HoodieRecordType.AVRO)
+
+ // Insert Operation
+ var records = recordsToStrings(dataGen.generateInserts("002", 100)).asScala.toList
+ var inputDF = spark.read.json(spark.sparkContext.parallelize(records, 2))
+
+ val commonOptsWithMultipleOrderingFields = writeOpts ++ Map(
+ "hoodie.insert.shuffle.parallelism" -> "4",
+ "hoodie.upsert.shuffle.parallelism" -> "4",
+ DataSourceWriteOptions.RECORDKEY_FIELD.key -> "_row_key",
+ DataSourceWriteOptions.PARTITIONPATH_FIELD.key -> "partition",
+ DataSourceWriteOptions.PRECOMBINE_FIELD.key() -> "timestamp,rider",
Review Comment:
Does the test assert on a case like timestamp is larger but rider is smaller?
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/HoodieStreamerUtils.java:
##########
@@ -111,8 +112,11 @@ public static Option<JavaRDD<HoodieRecord>> createHoodieRecords(HoodieStreamer.C
try {
HoodieKey hoodieKey = new HoodieKey(builtinKeyGenerator.getRecordKey(genRec), builtinKeyGenerator.getPartitionPath(genRec));
GenericRecord gr = isDropPartitionColumns(props) ? HoodieAvroUtils.removeFields(genRec, partitionColumns) : genRec;
- HoodieRecordPayload payload = shouldUseOrderingField ? DataSourceUtils.createPayload(payloadClassName, gr,
- (Comparable) HoodieAvroUtils.getNestedFieldVal(gr, cfg.sourceOrderingField, false, useConsistentLogicalTimestamp))
+ Comparable orderingValue = shouldUseOrderingField
+ ? OrderingValues.create(cfg.sourceOrderingFields.split(","),
+ field -> (Comparable) HoodieAvroUtils.getNestedFieldVal(gr, field, false, useConsistentLogicalTimestamp))
+ : null;
+ HoodieRecordPayload payload = shouldUseOrderingField ? DataSourceUtils.createPayload(payloadClassName, gr, orderingValue)
Review Comment:
Can we swap to `HoodieRecordUtils.loadPayload` here while we're updating? It caches the constructor lookup and has significantly higher throughput as a result.
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/MergeOnReadSnapshotRelation.scala:
##########
@@ -92,7 +92,7 @@ abstract class BaseMergeOnReadSnapshotRelation(sqlContext: SQLContext,
* by the query), therefore saving on throughput
*/
protected lazy val mandatoryFieldsForMerging: Seq[String] =
- Seq(recordKeyField) ++ preCombineFieldOpt.map(Seq(_)).getOrElse(Seq())
+ Seq(recordKeyField) ++ preCombineFieldsOpt.orElse(List())
Review Comment:
Is there an equivalent of `Collections.emptyList` in scala? Seq.empty perhaps?
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/HoodieDatasetBulkInsertHelper.scala:
##########
@@ -255,6 +257,16 @@ object HoodieDatasetBulkInsertHelper
}
}
+ private def getOrderingValue(nestedFieldPaths: List[NestedFieldPath], r: InternalRow): Comparable[_] = {
+ if (nestedFieldPaths.isEmpty) {
+ EmptyOrderingValue.getInstance();
+ } else if (nestedFieldPaths.size == 1) {
+ getNestedInternalRowValue(r, nestedFieldPaths.head).asInstanceOf[Comparable[_]]
+ } else {
+ new MultipleFieldsOrderingValue(nestedFieldPaths.map(_ => getNestedInternalRowValue(r, nestedFieldPaths.head).asInstanceOf[Comparable[_]]).toArray)
Review Comment:
I think `nestedFieldPaths.head` will always return the first element instead of iterating throught the list
##########
hudi-common/src/main/java/org/apache/hudi/common/engine/HoodieReaderContext.java:
##########
@@ -380,21 +380,23 @@ private BiFunction<T, Schema, String> virtualKeyExtractor(String[] recordKeyFiel
/**
* Gets the ordering value in particular type.
*
- * @param record An option of record.
- * @param schema The Avro schema of the record.
- * @param orderingFieldName name of the ordering field
+ * @param record An option of record.
+ * @param schema The Avro schema of the record.
+ * @param orderingFieldNames name of the ordering field
* @return The ordering value.
*/
public Comparable getOrderingValue(T record,
Schema schema,
- Option<String> orderingFieldName) {
- if (orderingFieldName.isEmpty()) {
- return DEFAULT_ORDERING_VALUE;
+ Option<List<String>> orderingFieldNames) {
Review Comment:
It was Option because there was not concept of a list of values for ordering fields so we needed a way to indicate that there were no ordering fields set. Now that it is a list, an empty list can represent the same.
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/catalyst/catalog/HoodieCatalogTable.scala:
##########
@@ -127,9 +128,9 @@ class HoodieCatalogTable(val spark: SparkSession, var table: CatalogTable) exten
lazy val primaryKeys: Array[String] = tableConfig.getRecordKeyFields.orElse(Array.empty)
/**
- * PreCombine Field
+ * Comparables Field
*/
- lazy val preCombineKey: Option[String] = Option(tableConfig.getPreCombineField)
+ lazy val preCombineKeys: Option[String] = toScalaOption(tableConfig.getPreCombineFieldsAsString)
Review Comment:
Can we make this a list/seq of strings instead to ensure all users of this are properly handling the possibility of multiple ordering fields?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3117555964
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
}, {
"hash" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"triggerType" : "PUSH"
}, {
"hash" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947",
"triggerID" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"triggerType" : "PUSH"
}, {
"hash" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6949",
"triggerID" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"triggerType" : "PUSH"
}, {
"hash" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"triggerType" : "PUSH"
}, {
"hash" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6950",
"triggerID" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"triggerType" : "PUSH"
}, {
"hash" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6974",
"triggerID" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"triggerType" : "PUSH"
}, {
"hash" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6991",
"triggerID" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"triggerType" : "PUSH"
}, {
"hash" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6993",
"triggerID" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"triggerType" : "PUSH"
}, {
"hash" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6995",
"triggerID" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"triggerType" : "PUSH"
}, {
"hash" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6996",
"triggerID" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"triggerType" : "PUSH"
}, {
"hash" : "dc851f6226047629aec76f30cd9843ac635a66d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7001",
"triggerID" : "dc851f6226047629aec76f30cd9843ac635a66d9",
"triggerType" : "PUSH"
}, {
"hash" : "8c51a6c54679f5723053ae3d9efe343bdeea9445",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7041",
"triggerID" : "8c51a6c54679f5723053ae3d9efe343bdeea9445",
"triggerType" : "PUSH"
}, {
"hash" : "723b2bdc505e58d9a7525be996402ce55d2b9bff",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "723b2bdc505e58d9a7525be996402ce55d2b9bff",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e19abadae9dece1aa78868839605a3d23a8b29fd UNKNOWN
* 4ba45f607a07f3f036e164dd68532fbc8b4bc3b8 UNKNOWN
* 8c51a6c54679f5723053ae3d9efe343bdeea9445 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7041)
* 723b2bdc505e58d9a7525be996402ce55d2b9bff UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2218392949
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -900,8 +900,13 @@ static RecordMergeMode inferRecordMergeModeFromMergeStrategyId(String recordMerg
}
}
- public String getPreCombineField() {
- return getString(PRECOMBINE_FIELD);
+ public Option<List<String>> getPreCombineFieldList() {
Review Comment:
Addressed
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/catalyst/catalog/HoodieCatalogTable.scala:
##########
@@ -127,9 +128,9 @@ class HoodieCatalogTable(val spark: SparkSession, var table: CatalogTable) exten
lazy val primaryKeys: Array[String] = tableConfig.getRecordKeyFields.orElse(Array.empty)
/**
- * PreCombine Field
+ * Comparables Field
*/
- lazy val preCombineKey: Option[String] = Option(tableConfig.getPreCombineField)
+ lazy val preCombineKey: Option[String] = convertHudiOptionToScalaOption(tableConfig.getPreCombineFields)
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2218396954
##########
hudi-common/src/main/java/org/apache/hudi/Comparables.java:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hudi.common.util.ValidationUtils;
+
+import java.io.Serializable;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.model.HoodieRecord.DEFAULT_ORDERING_VALUE;
+
+public class Comparables implements Comparable, Serializable {
+ protected static final long serialVersionUID = 1L;
+
+ private final List<Comparable> comparables;
Review Comment:
I have added an optimisation to use singleton list when size is 1. This is kind of a mid way solution between list and maintaining a separate singleton element. It will be easier to maintain. Although the memory utilisation is a bit higher compared to a singleton element but much lower compared to maintaining a list. Please let me know what you think.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -1360,8 +1362,13 @@ public HoodieTableType getTableType() {
HoodieTableConfig.TYPE, HoodieTableConfig.TYPE.defaultValue().name()).toUpperCase());
}
- public String getPreCombineField() {
- return getString(PRECOMBINE_FIELD_NAME);
+ public List<String> getPreCombineField() {
Review Comment:
Addressed
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/HoodieDatasetBulkInsertHelper.scala:
##########
@@ -213,12 +213,12 @@ object HoodieDatasetBulkInsertHelper
}), SQLConf.get).toJavaRDD())
}
- private def dedupeRows(rdd: RDD[InternalRow], schema: StructType, preCombineFieldRef: String, isGlobalIndex: Boolean, targetParallelism: Int): RDD[InternalRow] = {
+ private def dedupeRows(rdd: RDD[InternalRow], schema: StructType, preCombineFields: List[String], isGlobalIndex: Boolean, targetParallelism: Int): RDD[InternalRow] = {
val recordKeyMetaFieldOrd = schema.fieldIndex(HoodieRecord.RECORD_KEY_METADATA_FIELD)
val partitionPathMetaFieldOrd = schema.fieldIndex(HoodieRecord.PARTITION_PATH_METADATA_FIELD)
// NOTE: Pre-combine field could be a nested field
- val preCombineFieldPath = composeNestedFieldPath(schema, preCombineFieldRef)
- .getOrElse(throw new HoodieException(s"Pre-combine field $preCombineFieldRef is missing in $schema"))
+ val preCombineFieldPaths = preCombineFields.map(preCombineField => composeNestedFieldPath(schema, preCombineField)
+ .getOrElse(throw new HoodieException(s"Pre-combine field $preCombineFields is missing in $schema")))
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2218397802
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/cdc/HoodieCDCRDD.scala:
##########
@@ -137,24 +139,14 @@ class HoodieCDCRDD(
keyFields.head
}
- private lazy val preCombineFieldOpt: Option[String] = Option(metaClient.getTableConfig.getPreCombineField)
+ private lazy val preCombineFieldsOpt: Option[List[String]] = convertHudiOptionToScalaOption(metaClient.getTableConfig.getPreCombineFieldList
+ .map[List[String]](list => list.asScala.toList))
private lazy val tableState = {
val metadataConfig = HoodieMetadataConfig.newBuilder()
.fromProperties(props)
.build()
- HoodieTableState(
- basePath.toUri.toString,
- Some(split.changes.last.getInstant),
- recordKeyField,
- preCombineFieldOpt,
- usesVirtualKeys = !populateMetaFields,
- metaClient.getTableConfig.getPayloadClass,
- metadataConfig,
- // TODO support CDC with spark record
- recordMergeImplClasses = List(classOf[HoodieAvroRecordMerger].getName),
- recordMergeStrategyId = HoodieRecordMerger.PAYLOAD_BASED_MERGE_STRATEGY_UUID
- )
+ HoodieTableState(basePath.toUri.toString, Some(split.changes.last.getInstant), recordKeyField, preCombineFieldsOpt, usesVirtualKeys = !populateMetaFields, metaClient.getTableConfig.getPayloadClass, metadataConfig, recordMergeImplClasses = List(classOf[HoodieAvroRecordMerger].getName), recordMergeStrategyId = HoodieRecordMerger.PAYLOAD_BASED_MERGE_STRATEGY_UUID)
Review Comment:
Unintentional change. Reverted it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3095718257
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c6a88f8bdbb390dbc6602bc55839fdec432d6b4e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904)
* 98a828795959a9cfc01d6c04ea8739228fa6009c Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906)
* fc0eba477035f595d72c7f0ce133a2d023df0a53 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2218397313
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/util/JavaScalaConverters.scala:
##########
@@ -69,4 +69,9 @@ object JavaScalaConverters {
def convertJavaPropertiesToScalaMap(javaProperties: java.util.Properties): Map[String, String] = {
javaProperties.asScala.toMap
}
+
+ def convertHudiOptionToScalaOption[A](option: org.apache.hudi.common.util.Option[A]): Option[A] = {
Review Comment:
Addressed
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/Iterators.scala:
##########
@@ -86,10 +86,10 @@ class LogFileIterator(logFiles: List[HoodieLogFile],
}
private val maxCompactionMemoryInBytes: Long = getMaxCompactionMemoryInBytes(new JobConf(config))
- protected val payloadProps: TypedProperties = tableState.preCombineFieldOpt
+ protected val payloadProps: TypedProperties = tableState.preCombineFieldsOpt
.map { preCombineField =>
HoodiePayloadConfig.newBuilder
- .withPayloadOrderingField(preCombineField)
+ .withPayloadOrderingField(String.join(",", preCombineField: _*))
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2227064773
##########
hudi-common/src/main/java/org/apache/hudi/common/util/OrderingValues.java:
##########
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util;
+
+import org.apache.hudi.common.model.EmptyOrderingValue;
+import org.apache.hudi.common.model.MultipleFieldsOrderingValue;
+import org.apache.hudi.common.model.OrderingValue;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.function.Function;
+
+/**
+ * Factory class for {@link org.apache.hudi.common.model.OrderingValue}.
+ */
+public class OrderingValues {
+ private static final Comparable<?> DEFAULT_VALUE = org.apache.hudi.common.model.HoodieRecord.DEFAULT_ORDERING_VALUE;
+
+ /**
+ * Creates an {@code OrderingValue} with given values.
+ *
+ * <p>CAUTION: if the values are fetched through ordering fields,
+ * please use {@link #create(List, Function)} or {@link #create(String[], Function)}
+ * as much as possible to avoid streaming on the ordering fields in row level.</p>
+ */
+ public static Comparable create(Comparable[] orderingValues) {
+ if (orderingValues.length == 1) {
+ return orderingValues[0];
Review Comment:
I'm inclined we continue to use `Comparable` as the high-level interface for ordering val, the `SingleFieldOrderingValue` is just a wrapper and totally redundant, removing it can also keep the storage and computation compatible with before, the disadvantages here is we need to do `instanceof` in a few places,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3106954593
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
}, {
"hash" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"triggerType" : "PUSH"
}, {
"hash" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947",
"triggerID" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"triggerType" : "PUSH"
}, {
"hash" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6949",
"triggerID" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"triggerType" : "PUSH"
}, {
"hash" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"triggerType" : "PUSH"
}, {
"hash" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6950",
"triggerID" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"triggerType" : "PUSH"
}, {
"hash" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6974",
"triggerID" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"triggerType" : "PUSH"
}, {
"hash" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6991",
"triggerID" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"triggerType" : "PUSH"
}, {
"hash" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6993",
"triggerID" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"triggerType" : "PUSH"
}, {
"hash" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6995",
"triggerID" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"triggerType" : "PUSH"
}, {
"hash" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e19abadae9dece1aa78868839605a3d23a8b29fd UNKNOWN
* 4ba45f607a07f3f036e164dd68532fbc8b4bc3b8 UNKNOWN
* 63af22477ed62d797f70568430ce6f142ffc3cc0 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6995)
* 2dba2b4f6c8d57fea9ff262e842f45153d56787f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3107306951
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
}, {
"hash" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"triggerType" : "PUSH"
}, {
"hash" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947",
"triggerID" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"triggerType" : "PUSH"
}, {
"hash" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6949",
"triggerID" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"triggerType" : "PUSH"
}, {
"hash" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"triggerType" : "PUSH"
}, {
"hash" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6950",
"triggerID" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"triggerType" : "PUSH"
}, {
"hash" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6974",
"triggerID" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"triggerType" : "PUSH"
}, {
"hash" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6991",
"triggerID" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"triggerType" : "PUSH"
}, {
"hash" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6993",
"triggerID" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"triggerType" : "PUSH"
}, {
"hash" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6995",
"triggerID" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"triggerType" : "PUSH"
}, {
"hash" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6996",
"triggerID" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e19abadae9dece1aa78868839605a3d23a8b29fd UNKNOWN
* 4ba45f607a07f3f036e164dd68532fbc8b4bc3b8 UNKNOWN
* 63af22477ed62d797f70568430ce6f142ffc3cc0 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6995)
* 2dba2b4f6c8d57fea9ff262e842f45153d56787f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6996)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3111728653
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
}, {
"hash" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"triggerType" : "PUSH"
}, {
"hash" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947",
"triggerID" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"triggerType" : "PUSH"
}, {
"hash" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6949",
"triggerID" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"triggerType" : "PUSH"
}, {
"hash" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"triggerType" : "PUSH"
}, {
"hash" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6950",
"triggerID" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"triggerType" : "PUSH"
}, {
"hash" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6974",
"triggerID" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"triggerType" : "PUSH"
}, {
"hash" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6991",
"triggerID" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"triggerType" : "PUSH"
}, {
"hash" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6993",
"triggerID" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"triggerType" : "PUSH"
}, {
"hash" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6995",
"triggerID" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"triggerType" : "PUSH"
}, {
"hash" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6996",
"triggerID" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"triggerType" : "PUSH"
}, {
"hash" : "dc851f6226047629aec76f30cd9843ac635a66d9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7001",
"triggerID" : "dc851f6226047629aec76f30cd9843ac635a66d9",
"triggerType" : "PUSH"
}, {
"hash" : "8c51a6c54679f5723053ae3d9efe343bdeea9445",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8c51a6c54679f5723053ae3d9efe343bdeea9445",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e19abadae9dece1aa78868839605a3d23a8b29fd UNKNOWN
* 4ba45f607a07f3f036e164dd68532fbc8b4bc3b8 UNKNOWN
* dc851f6226047629aec76f30cd9843ac635a66d9 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7001)
* 8c51a6c54679f5723053ae3d9efe343bdeea9445 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2222179712
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/block/HoodieDeleteBlock.java:
##########
@@ -142,11 +142,18 @@ private byte[] serializeV3() throws IOException {
HoodieDeleteRecordList.Builder recordListBuilder = HOODIE_DELETE_RECORD_LIST_BUILDER_STUB.get();
HoodieDeleteRecord.Builder recordBuilder = HOODIE_DELETE_RECORD_BUILDER_STUB.get();
List<HoodieDeleteRecord> deleteRecordList = Arrays.stream(getRecordsToDelete())
- .map(record -> HoodieDeleteRecord.newBuilder(recordBuilder)
+ .map(record -> {
+ List<Comparable> orderingVals = record.getOrderingValue() instanceof Comparables
Review Comment:
not a fan of `instanceof` per record, after this patch, the `record.getOrderingValue()` should always be `Comparables` right? If this is true, could we switch it's return type to just `Comparables`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2222196084
##########
hudi-common/src/main/java/org/apache/hudi/Comparables.java:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hudi.common.util.ValidationUtils;
+
+import java.io.Serializable;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.model.HoodieRecord.DEFAULT_ORDERING_VALUE;
+
+public class Comparables implements Comparable, Serializable {
+ protected static final long serialVersionUID = 1L;
+
+ private final List<Comparable> comparables;
Review Comment:
This is my concern too, my idea is we rename the class into just `OrderingValue` and introduce two sub-classes here: `SingleFieldOrderingValue` and `MultipleFieldsOrderingValue`. We should override the core methods: `#compareTo`, `#isSameClass` and `#isDefault` to get rid of the reduadant list loop.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2222196084
##########
hudi-common/src/main/java/org/apache/hudi/Comparables.java:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hudi.common.util.ValidationUtils;
+
+import java.io.Serializable;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.model.HoodieRecord.DEFAULT_ORDERING_VALUE;
+
+public class Comparables implements Comparable, Serializable {
+ protected static final long serialVersionUID = 1L;
+
+ private final List<Comparable> comparables;
Review Comment:
This is my concern too, my idea is we rename the class into just `OrderingValue` and introduce two sub-classes here: `SingleFieldOrderingValue` and `MultipleFieldsOrderingValue`. We should override the core methods: `#compareTo` and `#isSameClass` to get rid of the reduadant list loop.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2221872516
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/HoodieDatasetBulkInsertHelper.scala:
##########
@@ -233,13 +233,14 @@ object HoodieDatasetBulkInsertHelper
(rowKey, row.copy())
}
.reduceByKey ((oneRow, otherRow) => {
- val onePreCombineVal = getNestedInternalRowValue(oneRow, preCombineFieldPath).asInstanceOf[Comparable[AnyRef]]
- val otherPreCombineVal = getNestedInternalRowValue(otherRow, preCombineFieldPath).asInstanceOf[Comparable[AnyRef]]
- if (onePreCombineVal.compareTo(otherPreCombineVal.asInstanceOf[AnyRef]) >= 0) {
+ val onePreCombineVals = new Comparables(JavaConverters.seqAsJavaList(preCombineFieldPaths.map(preCombineFieldPath => getNestedInternalRowValue(oneRow, preCombineFieldPath).asInstanceOf[Comparable[AnyRef]])))
Review Comment:
I have reverted the fix I made for this. It does not work for scala-2.13. Current change works for both scala versions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3121531082
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
}, {
"hash" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"triggerType" : "PUSH"
}, {
"hash" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947",
"triggerID" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"triggerType" : "PUSH"
}, {
"hash" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6949",
"triggerID" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"triggerType" : "PUSH"
}, {
"hash" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"triggerType" : "PUSH"
}, {
"hash" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6950",
"triggerID" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"triggerType" : "PUSH"
}, {
"hash" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6974",
"triggerID" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"triggerType" : "PUSH"
}, {
"hash" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6991",
"triggerID" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"triggerType" : "PUSH"
}, {
"hash" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6993",
"triggerID" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"triggerType" : "PUSH"
}, {
"hash" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6995",
"triggerID" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"triggerType" : "PUSH"
}, {
"hash" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6996",
"triggerID" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"triggerType" : "PUSH"
}, {
"hash" : "dc851f6226047629aec76f30cd9843ac635a66d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7001",
"triggerID" : "dc851f6226047629aec76f30cd9843ac635a66d9",
"triggerType" : "PUSH"
}, {
"hash" : "8c51a6c54679f5723053ae3d9efe343bdeea9445",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7041",
"triggerID" : "8c51a6c54679f5723053ae3d9efe343bdeea9445",
"triggerType" : "PUSH"
}, {
"hash" : "723b2bdc505e58d9a7525be996402ce55d2b9bff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7123",
"triggerID" : "723b2bdc505e58d9a7525be996402ce55d2b9bff",
"triggerType" : "PUSH"
}, {
"hash" : "c7902b06721cfb4e51d166b5799c20d58f079404",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7125",
"triggerID" : "c7902b06721cfb4e51d166b5799c20d58f079404",
"triggerType" : "PUSH"
}, {
"hash" : "5e3962074d7b4f7dceaedf9a391e763e054f9b9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7139",
"triggerID" : "5e3962074d7b4f7dceaedf9a391e763e054f9b9d",
"triggerType" : "PUSH"
}, {
"hash" : "371f9b89c48b3dc9a9b95b15266b9f2fd4201a56",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7165",
"triggerID" : "371f9b89c48b3dc9a9b95b15266b9f2fd4201a56",
"triggerType" : "PUSH"
}, {
"hash" : "2f9d6614a9ea6ed28bdc3654d37a7dc17563175b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7169",
"triggerID" : "2f9d6614a9ea6ed28bdc3654d37a7dc17563175b",
"triggerType" : "PUSH"
}, {
"hash" : "c74a62c8f7f57fc91116e40fb3db163c60853a23",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c74a62c8f7f57fc91116e40fb3db163c60853a23",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e19abadae9dece1aa78868839605a3d23a8b29fd UNKNOWN
* 4ba45f607a07f3f036e164dd68532fbc8b4bc3b8 UNKNOWN
* 371f9b89c48b3dc9a9b95b15266b9f2fd4201a56 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7165)
* 2f9d6614a9ea6ed28bdc3654d37a7dc17563175b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7169)
* c74a62c8f7f57fc91116e40fb3db163c60853a23 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3045942036
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 26a8660da376934b6f864b04f96e160d342886ca Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3047918918
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 26155cf4cb689ae854bc81e5c2c9c4ae8134ec85 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2193964054
##########
hudi-common/src/main/java/org/apache/hudi/Comparables.java:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hudi.common.util.ValidationUtils;
+
+import java.io.Serializable;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.model.HoodieRecord.DEFAULT_ORDERING_VALUE;
+
+public class Comparables implements Comparable, Serializable {
+ protected static final long serialVersionUID = 1L;
+
+ private final List<Comparable> comparables;
Review Comment:
Instead of maintaining everything as a list even for single element, can we maintain two instance variables.
one for single element and the other as list.
and a private boolean flag to denote whether this is single element or multiple elements.
lets avoid maintaining a list for single element ordering fields which will be most common use-case.
##########
hudi-common/src/main/java/org/apache/hudi/Comparables.java:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hudi.common.util.ValidationUtils;
+
+import java.io.Serializable;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.model.HoodieRecord.DEFAULT_ORDERING_VALUE;
+
+public class Comparables implements Comparable, Serializable {
+ protected static final long serialVersionUID = 1L;
+
+ private final List<Comparable> comparables;
Review Comment:
we can introduce two diff static methods to assist w/ creating Comparables either w/ single Comparable or a list of Comparable
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -900,8 +900,13 @@ static RecordMergeMode inferRecordMergeModeFromMergeStrategyId(String recordMerg
}
}
- public String getPreCombineField() {
- return getString(PRECOMBINE_FIELD);
+ public Option<List<String>> getPreCombineFieldList() {
Review Comment:
ok, I see there are diff set of users.
one of them just tries to intercept the config value and sets it in some other props to determines the merge mode/payload etc.
while the other set of callers are actually using it for merging which needs a list.
we can keep both methods then.
but lets ensure we call `getPreCombineFields` from within `getPreCombineFieldList` and then convert to list.
Also, we can trim for empty string.
`filter(StringUtils::nonEmpty)` before returning.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -900,8 +900,13 @@ static RecordMergeMode inferRecordMergeModeFromMergeStrategyId(String recordMerg
}
}
- public String getPreCombineField() {
- return getString(PRECOMBINE_FIELD);
+ public Option<List<String>> getPreCombineFieldList() {
Review Comment:
why do we need two methods(getPreCombineFieldList and getPreCombineFields). lets try to have just 1 method and fix callers to use the same.
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/catalyst/catalog/HoodieCatalogTable.scala:
##########
@@ -127,9 +128,9 @@ class HoodieCatalogTable(val spark: SparkSession, var table: CatalogTable) exten
lazy val primaryKeys: Array[String] = tableConfig.getRecordKeyFields.orElse(Array.empty)
/**
- * PreCombine Field
+ * Comparables Field
*/
- lazy val preCombineKey: Option[String] = Option(tableConfig.getPreCombineField)
+ lazy val preCombineKey: Option[String] = convertHudiOptionToScalaOption(tableConfig.getPreCombineFields)
Review Comment:
`preCombineKeys`
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/SevenToEightUpgradeHandler.java:
##########
@@ -175,7 +175,7 @@ static void upgradePartitionFields(HoodieWriteConfig config, HoodieTableConfig t
static void upgradeMergeMode(HoodieTableConfig tableConfig, Map<ConfigProperty, String> tablePropsToAdd) {
String payloadClass = tableConfig.getPayloadClass();
- String preCombineField = tableConfig.getPreCombineField();
+ String preCombineField = tableConfig.getPreCombineFields().orElse(null);
Review Comment:
`preCombineFields`
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -169,8 +170,9 @@ public class HoodieWriteConfig extends HoodieConfig {
public static final ConfigProperty<String> PRECOMBINE_FIELD_NAME = ConfigProperty
.key("hoodie.datasource.write.precombine.field")
.noDefaultValue()
- .withDocumentation("Field used in preCombining before actual write. When two records have the same key value, "
- + "we will pick the one with the largest value for the precombine field, determined by Object.compareTo(..)");
+ .withDocumentation("Comma separated list of fields used in preCombining before actual write. When two records have the same key value, "
Review Comment:
can we enhance the docs to call out this is not just for combining records w/n same batch.
but also used w/ merge mode like EventTime based merge mode.
##########
hudi-common/src/main/java/org/apache/hudi/Comparables.java:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hudi.common.util.ValidationUtils;
+
+import java.io.Serializable;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.model.HoodieRecord.DEFAULT_ORDERING_VALUE;
+
+public class Comparables implements Comparable, Serializable {
+ protected static final long serialVersionUID = 1L;
+
+ private final List<Comparable> comparables;
+
+ public Comparables(List<Comparable> comparables) {
+ this.comparables = comparables;
+ }
+
+ public Comparables(Comparable comparable) {
+ this.comparables = Collections.singletonList(comparable);
+ }
+
+ public static boolean isDefault(Comparable orderingVal) {
Review Comment:
Can we move static methods to the end of the class
##########
hudi-common/src/main/java/org/apache/hudi/Comparables.java:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hudi.common.util.ValidationUtils;
+
+import java.io.Serializable;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.model.HoodieRecord.DEFAULT_ORDERING_VALUE;
Review Comment:
java doc please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2199372741
##########
hudi-common/src/main/java/org/apache/hudi/common/engine/HoodieReaderContext.java:
##########
@@ -385,21 +386,24 @@ private BiFunction<T, Schema, String> virtualKeyExtractor(String[] recordKeyFiel
/**
* Gets the ordering value in particular type.
*
- * @param record An option of record.
- * @param schema The Avro schema of the record.
- * @param orderingFieldName name of the ordering field
+ * @param record An option of record.
+ * @param schema The Avro schema of the record.
+ * @param orderingFieldNames name of the ordering field
* @return The ordering value.
*/
public Comparable getOrderingValue(T record,
Schema schema,
- Option<String> orderingFieldName) {
- if (orderingFieldName.isEmpty()) {
- return DEFAULT_ORDERING_VALUE;
+ Option<List<String>> orderingFieldNames) {
+ if (orderingFieldNames.isEmpty()) {
+ return Comparables.getDefault();
}
- Object value = getValue(record, schema, orderingFieldName.get());
- Comparable finalOrderingVal = value != null ? convertValueToEngineType((Comparable) value) : DEFAULT_ORDERING_VALUE;
- return finalOrderingVal;
+ return new Comparables(
+ orderingFieldNames.get().stream().map(field -> {
+ Object value = getValue(record, schema, field);
+ // API getDefaultOrderingValue is only used inside Comparables constructor
Review Comment:
Let's pay attention to the performance issue in this method that every record passes through. How about using a fixed-size list with `ArrayList(orderingFieldNames.size())` instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2201270328
##########
hudi-common/src/main/java/org/apache/hudi/Comparables.java:
##########
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hudi.common.util.ValidationUtils;
+
+import java.io.Serializable;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+import static org.apache.hudi.common.model.HoodieRecord.DEFAULT_ORDERING_VALUE;
+
+public class Comparables implements Comparable, Serializable {
+ protected static final long serialVersionUID = 1L;
+ private static final Comparables DEFAULT_VALUE = new Comparables(DEFAULT_ORDERING_VALUE);
+
+ private final List<Comparable> comparables;
+
+ public Comparables(List<Comparable> comparables) {
+ this.comparables = comparables;
+ }
+
+ public Comparables(Comparable comparable) {
+ this.comparables = Collections.singletonList(comparable);
+ }
+
+ public static Comparables getDefault() {
+ return DEFAULT_VALUE;
+ }
+
+ public static boolean isDefault(Comparable orderingVal) {
+ if (orderingVal instanceof Comparables) {
+ return ((Comparables) orderingVal).comparables.size() == 1
+ && ((Comparables) orderingVal).comparables.get(0).equals(DEFAULT_ORDERING_VALUE);
+ } else {
+ return orderingVal.equals(DEFAULT_ORDERING_VALUE);
+ }
+ }
+
+ /**
+ * Returns whether the given two comparable values come from the same runtime class.
+ */
+ public static boolean isSameClass(Comparable<?> v, Comparable<?> o) {
+ if (v.getClass() != o.getClass()) {
+ // If class is not same return false
+ return false;
+ } else if (v instanceof Comparables) {
+ if (((Comparables) v).comparables.size() != ((Comparables) o).comparables.size()) {
+ // if comparables size is not same return false
+ return false;
+ } else {
+ // compare class of comparable list of both arguments
+ return IntStream.range(0, ((Comparables) v).comparables.size())
+ .mapToObj(i -> ((Comparables) v).comparables.get(i).getClass() == ((Comparables) o).comparables.get(i).getClass())
+ .reduce(Boolean::logicalAnd)
+ .orElse(true);
+ }
+ }
+ // return true if class is same and input objects are not instance of Comparables class
+ return true;
+ }
+
+ public static Comparable getDefaultOrderingValue() {
+ return DEFAULT_ORDERING_VALUE;
+ }
+
+ @Override
+ public int compareTo(Object o) {
+ ValidationUtils.checkArgument(o instanceof Comparables, "Comparables can only be compared with another Comparables");
+ Comparables otherComparables = (Comparables) o;
+ ValidationUtils.checkArgument(comparables.size() == otherComparables.comparables.size(), "Comparables should be of same size");
+ for (int i = 0; i < comparables.size(); i++) {
+ int comparingValue = comparables.get(i).compareTo(otherComparables.comparables.get(i));
Review Comment:
The comparison logic depends on the order of the field, which should be the generic case, but we may need to clearly describe this to the users.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2218363553
##########
hudi-common/src/main/java/org/apache/hudi/Comparables.java:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hudi.common.util.ValidationUtils;
+
+import java.io.Serializable;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.model.HoodieRecord.DEFAULT_ORDERING_VALUE;
+
+public class Comparables implements Comparable, Serializable {
+ protected static final long serialVersionUID = 1L;
+
+ private final List<Comparable> comparables;
Review Comment:
I have added optimisation so that whenever the constructor is called and size of comparables is 1, we use singletonList instead. This is more efficient than a list. Also it does not add any complexity related to managing both a singleton element and a list. Please take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3058448379
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f4b6ff8db1543238f986e2d604f4f71971313b95 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3121302552
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
}, {
"hash" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"triggerType" : "PUSH"
}, {
"hash" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947",
"triggerID" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"triggerType" : "PUSH"
}, {
"hash" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6949",
"triggerID" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"triggerType" : "PUSH"
}, {
"hash" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"triggerType" : "PUSH"
}, {
"hash" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6950",
"triggerID" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"triggerType" : "PUSH"
}, {
"hash" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6974",
"triggerID" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"triggerType" : "PUSH"
}, {
"hash" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6991",
"triggerID" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"triggerType" : "PUSH"
}, {
"hash" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6993",
"triggerID" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"triggerType" : "PUSH"
}, {
"hash" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6995",
"triggerID" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"triggerType" : "PUSH"
}, {
"hash" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6996",
"triggerID" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"triggerType" : "PUSH"
}, {
"hash" : "dc851f6226047629aec76f30cd9843ac635a66d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7001",
"triggerID" : "dc851f6226047629aec76f30cd9843ac635a66d9",
"triggerType" : "PUSH"
}, {
"hash" : "8c51a6c54679f5723053ae3d9efe343bdeea9445",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7041",
"triggerID" : "8c51a6c54679f5723053ae3d9efe343bdeea9445",
"triggerType" : "PUSH"
}, {
"hash" : "723b2bdc505e58d9a7525be996402ce55d2b9bff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7123",
"triggerID" : "723b2bdc505e58d9a7525be996402ce55d2b9bff",
"triggerType" : "PUSH"
}, {
"hash" : "c7902b06721cfb4e51d166b5799c20d58f079404",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7125",
"triggerID" : "c7902b06721cfb4e51d166b5799c20d58f079404",
"triggerType" : "PUSH"
}, {
"hash" : "5e3962074d7b4f7dceaedf9a391e763e054f9b9d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7139",
"triggerID" : "5e3962074d7b4f7dceaedf9a391e763e054f9b9d",
"triggerType" : "PUSH"
}, {
"hash" : "371f9b89c48b3dc9a9b95b15266b9f2fd4201a56",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "371f9b89c48b3dc9a9b95b15266b9f2fd4201a56",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e19abadae9dece1aa78868839605a3d23a8b29fd UNKNOWN
* 4ba45f607a07f3f036e164dd68532fbc8b4bc3b8 UNKNOWN
* 5e3962074d7b4f7dceaedf9a391e763e054f9b9d Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7139)
* 371f9b89c48b3dc9a9b95b15266b9f2fd4201a56 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3121339232
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
}, {
"hash" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"triggerType" : "PUSH"
}, {
"hash" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947",
"triggerID" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"triggerType" : "PUSH"
}, {
"hash" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6949",
"triggerID" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"triggerType" : "PUSH"
}, {
"hash" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"triggerType" : "PUSH"
}, {
"hash" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6950",
"triggerID" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"triggerType" : "PUSH"
}, {
"hash" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6974",
"triggerID" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"triggerType" : "PUSH"
}, {
"hash" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6991",
"triggerID" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"triggerType" : "PUSH"
}, {
"hash" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6993",
"triggerID" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"triggerType" : "PUSH"
}, {
"hash" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6995",
"triggerID" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"triggerType" : "PUSH"
}, {
"hash" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6996",
"triggerID" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"triggerType" : "PUSH"
}, {
"hash" : "dc851f6226047629aec76f30cd9843ac635a66d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7001",
"triggerID" : "dc851f6226047629aec76f30cd9843ac635a66d9",
"triggerType" : "PUSH"
}, {
"hash" : "8c51a6c54679f5723053ae3d9efe343bdeea9445",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7041",
"triggerID" : "8c51a6c54679f5723053ae3d9efe343bdeea9445",
"triggerType" : "PUSH"
}, {
"hash" : "723b2bdc505e58d9a7525be996402ce55d2b9bff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7123",
"triggerID" : "723b2bdc505e58d9a7525be996402ce55d2b9bff",
"triggerType" : "PUSH"
}, {
"hash" : "c7902b06721cfb4e51d166b5799c20d58f079404",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7125",
"triggerID" : "c7902b06721cfb4e51d166b5799c20d58f079404",
"triggerType" : "PUSH"
}, {
"hash" : "5e3962074d7b4f7dceaedf9a391e763e054f9b9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7139",
"triggerID" : "5e3962074d7b4f7dceaedf9a391e763e054f9b9d",
"triggerType" : "PUSH"
}, {
"hash" : "371f9b89c48b3dc9a9b95b15266b9f2fd4201a56",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7165",
"triggerID" : "371f9b89c48b3dc9a9b95b15266b9f2fd4201a56",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e19abadae9dece1aa78868839605a3d23a8b29fd UNKNOWN
* 4ba45f607a07f3f036e164dd68532fbc8b4bc3b8 UNKNOWN
* 371f9b89c48b3dc9a9b95b15266b9f2fd4201a56 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7165)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3104507065
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
}, {
"hash" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"triggerType" : "PUSH"
}, {
"hash" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947",
"triggerID" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"triggerType" : "PUSH"
}, {
"hash" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6949",
"triggerID" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"triggerType" : "PUSH"
}, {
"hash" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"triggerType" : "PUSH"
}, {
"hash" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6950",
"triggerID" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"triggerType" : "PUSH"
}, {
"hash" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6974",
"triggerID" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e19abadae9dece1aa78868839605a3d23a8b29fd UNKNOWN
* 4ba45f607a07f3f036e164dd68532fbc8b4bc3b8 UNKNOWN
* a1f62b31b47715ce88932618c3817fbd7dd2e96e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6950)
* 21043f2bc87787d692ec55c2e4150ab29a396264 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6974)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3102259303
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
}, {
"hash" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"triggerType" : "PUSH"
}, {
"hash" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947",
"triggerID" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e19abadae9dece1aa78868839605a3d23a8b29fd UNKNOWN
* a20a7af499d5e0f6dc415ff74459d91ac88bf3d0 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3109067519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
}, {
"hash" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"triggerType" : "PUSH"
}, {
"hash" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947",
"triggerID" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"triggerType" : "PUSH"
}, {
"hash" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6949",
"triggerID" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"triggerType" : "PUSH"
}, {
"hash" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"triggerType" : "PUSH"
}, {
"hash" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6950",
"triggerID" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"triggerType" : "PUSH"
}, {
"hash" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6974",
"triggerID" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"triggerType" : "PUSH"
}, {
"hash" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6991",
"triggerID" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"triggerType" : "PUSH"
}, {
"hash" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6993",
"triggerID" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"triggerType" : "PUSH"
}, {
"hash" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6995",
"triggerID" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"triggerType" : "PUSH"
}, {
"hash" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6996",
"triggerID" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"triggerType" : "PUSH"
}, {
"hash" : "dc851f6226047629aec76f30cd9843ac635a66d9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7001",
"triggerID" : "dc851f6226047629aec76f30cd9843ac635a66d9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e19abadae9dece1aa78868839605a3d23a8b29fd UNKNOWN
* 4ba45f607a07f3f036e164dd68532fbc8b4bc3b8 UNKNOWN
* dc851f6226047629aec76f30cd9843ac635a66d9 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7001)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2222153009
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -169,8 +170,9 @@ public class HoodieWriteConfig extends HoodieConfig {
public static final ConfigProperty<String> PRECOMBINE_FIELD_NAME = ConfigProperty
.key("hoodie.datasource.write.precombine.field")
.noDefaultValue()
- .withDocumentation("Field used in preCombining before actual write. When two records have the same key value, "
- + "we will pick the one with the largest value for the precombine field, determined by Object.compareTo(..)");
+ .withDocumentation("Comma separated list of fields used in preCombining before actual write. When two records have the same key value, "
Review Comment:
The field is not just used for write, it is also a read config I think, actually it should be a table config instead of a write config to avoid ambiguties, maybe "hoodie.table.ordefing.fields".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2223719842
##########
hudi-common/src/main/java/org/apache/hudi/Comparables.java:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hudi.common.util.ValidationUtils;
+
+import java.io.Serializable;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.model.HoodieRecord.DEFAULT_ORDERING_VALUE;
+
+public class Comparables implements Comparable, Serializable {
+ protected static final long serialVersionUID = 1L;
+
+ private final List<Comparable> comparables;
Review Comment:
The overhead of a SingletonList is minimal, it is just wrapping a record similar to what we will be doing by introducing any wrapper class.
I agree with @danny0405 though that it makes most sense to have some interface and two sub-classes here so the common case of a single ordering field is as straightforward as possible. My preference is also to have the name reflect the use case here instead of being overly general at this point. If we find a second use case, we can update names at that point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3051290708
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8e74196c55e365e52dd5114c59ba96b2dfffb2ab Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652)
* 125b574e0176d7ec688acd05f9138f821f6dfe03 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3117795722
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
}, {
"hash" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"triggerType" : "PUSH"
}, {
"hash" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947",
"triggerID" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"triggerType" : "PUSH"
}, {
"hash" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6949",
"triggerID" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"triggerType" : "PUSH"
}, {
"hash" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"triggerType" : "PUSH"
}, {
"hash" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6950",
"triggerID" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"triggerType" : "PUSH"
}, {
"hash" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6974",
"triggerID" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"triggerType" : "PUSH"
}, {
"hash" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6991",
"triggerID" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"triggerType" : "PUSH"
}, {
"hash" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6993",
"triggerID" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"triggerType" : "PUSH"
}, {
"hash" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6995",
"triggerID" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"triggerType" : "PUSH"
}, {
"hash" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6996",
"triggerID" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"triggerType" : "PUSH"
}, {
"hash" : "dc851f6226047629aec76f30cd9843ac635a66d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7001",
"triggerID" : "dc851f6226047629aec76f30cd9843ac635a66d9",
"triggerType" : "PUSH"
}, {
"hash" : "8c51a6c54679f5723053ae3d9efe343bdeea9445",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7041",
"triggerID" : "8c51a6c54679f5723053ae3d9efe343bdeea9445",
"triggerType" : "PUSH"
}, {
"hash" : "723b2bdc505e58d9a7525be996402ce55d2b9bff",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7123",
"triggerID" : "723b2bdc505e58d9a7525be996402ce55d2b9bff",
"triggerType" : "PUSH"
}, {
"hash" : "c7902b06721cfb4e51d166b5799c20d58f079404",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7125",
"triggerID" : "c7902b06721cfb4e51d166b5799c20d58f079404",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e19abadae9dece1aa78868839605a3d23a8b29fd UNKNOWN
* 4ba45f607a07f3f036e164dd68532fbc8b4bc3b8 UNKNOWN
* 723b2bdc505e58d9a7525be996402ce55d2b9bff Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7123)
* c7902b06721cfb4e51d166b5799c20d58f079404 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7125)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3111767972
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
}, {
"hash" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"triggerType" : "PUSH"
}, {
"hash" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947",
"triggerID" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"triggerType" : "PUSH"
}, {
"hash" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6949",
"triggerID" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"triggerType" : "PUSH"
}, {
"hash" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"triggerType" : "PUSH"
}, {
"hash" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6950",
"triggerID" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"triggerType" : "PUSH"
}, {
"hash" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6974",
"triggerID" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"triggerType" : "PUSH"
}, {
"hash" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6991",
"triggerID" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"triggerType" : "PUSH"
}, {
"hash" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6993",
"triggerID" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"triggerType" : "PUSH"
}, {
"hash" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6995",
"triggerID" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"triggerType" : "PUSH"
}, {
"hash" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6996",
"triggerID" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"triggerType" : "PUSH"
}, {
"hash" : "dc851f6226047629aec76f30cd9843ac635a66d9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7001",
"triggerID" : "dc851f6226047629aec76f30cd9843ac635a66d9",
"triggerType" : "PUSH"
}, {
"hash" : "8c51a6c54679f5723053ae3d9efe343bdeea9445",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7041",
"triggerID" : "8c51a6c54679f5723053ae3d9efe343bdeea9445",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e19abadae9dece1aa78868839605a3d23a8b29fd UNKNOWN
* 4ba45f607a07f3f036e164dd68532fbc8b4bc3b8 UNKNOWN
* dc851f6226047629aec76f30cd9843ac635a66d9 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7001)
* 8c51a6c54679f5723053ae3d9efe343bdeea9445 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7041)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3118241795
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
}, {
"hash" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"triggerType" : "PUSH"
}, {
"hash" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947",
"triggerID" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"triggerType" : "PUSH"
}, {
"hash" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6949",
"triggerID" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"triggerType" : "PUSH"
}, {
"hash" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"triggerType" : "PUSH"
}, {
"hash" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6950",
"triggerID" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"triggerType" : "PUSH"
}, {
"hash" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6974",
"triggerID" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"triggerType" : "PUSH"
}, {
"hash" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6991",
"triggerID" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"triggerType" : "PUSH"
}, {
"hash" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6993",
"triggerID" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"triggerType" : "PUSH"
}, {
"hash" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6995",
"triggerID" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"triggerType" : "PUSH"
}, {
"hash" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6996",
"triggerID" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"triggerType" : "PUSH"
}, {
"hash" : "dc851f6226047629aec76f30cd9843ac635a66d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7001",
"triggerID" : "dc851f6226047629aec76f30cd9843ac635a66d9",
"triggerType" : "PUSH"
}, {
"hash" : "8c51a6c54679f5723053ae3d9efe343bdeea9445",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7041",
"triggerID" : "8c51a6c54679f5723053ae3d9efe343bdeea9445",
"triggerType" : "PUSH"
}, {
"hash" : "723b2bdc505e58d9a7525be996402ce55d2b9bff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7123",
"triggerID" : "723b2bdc505e58d9a7525be996402ce55d2b9bff",
"triggerType" : "PUSH"
}, {
"hash" : "c7902b06721cfb4e51d166b5799c20d58f079404",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7125",
"triggerID" : "c7902b06721cfb4e51d166b5799c20d58f079404",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e19abadae9dece1aa78868839605a3d23a8b29fd UNKNOWN
* 4ba45f607a07f3f036e164dd68532fbc8b4bc3b8 UNKNOWN
* c7902b06721cfb4e51d166b5799c20d58f079404 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7125)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3108487114
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
}, {
"hash" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"triggerType" : "PUSH"
}, {
"hash" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947",
"triggerID" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"triggerType" : "PUSH"
}, {
"hash" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6949",
"triggerID" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"triggerType" : "PUSH"
}, {
"hash" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"triggerType" : "PUSH"
}, {
"hash" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6950",
"triggerID" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"triggerType" : "PUSH"
}, {
"hash" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6974",
"triggerID" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"triggerType" : "PUSH"
}, {
"hash" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6991",
"triggerID" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"triggerType" : "PUSH"
}, {
"hash" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6993",
"triggerID" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"triggerType" : "PUSH"
}, {
"hash" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6995",
"triggerID" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"triggerType" : "PUSH"
}, {
"hash" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6996",
"triggerID" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"triggerType" : "PUSH"
}, {
"hash" : "dc851f6226047629aec76f30cd9843ac635a66d9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7001",
"triggerID" : "dc851f6226047629aec76f30cd9843ac635a66d9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e19abadae9dece1aa78868839605a3d23a8b29fd UNKNOWN
* 4ba45f607a07f3f036e164dd68532fbc8b4bc3b8 UNKNOWN
* 2dba2b4f6c8d57fea9ff262e842f45153d56787f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6996)
* dc851f6226047629aec76f30cd9843ac635a66d9 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7001)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3117789746
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
}, {
"hash" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"triggerType" : "PUSH"
}, {
"hash" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947",
"triggerID" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"triggerType" : "PUSH"
}, {
"hash" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6949",
"triggerID" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"triggerType" : "PUSH"
}, {
"hash" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"triggerType" : "PUSH"
}, {
"hash" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6950",
"triggerID" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"triggerType" : "PUSH"
}, {
"hash" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6974",
"triggerID" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"triggerType" : "PUSH"
}, {
"hash" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6991",
"triggerID" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"triggerType" : "PUSH"
}, {
"hash" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6993",
"triggerID" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"triggerType" : "PUSH"
}, {
"hash" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6995",
"triggerID" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"triggerType" : "PUSH"
}, {
"hash" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6996",
"triggerID" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"triggerType" : "PUSH"
}, {
"hash" : "dc851f6226047629aec76f30cd9843ac635a66d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7001",
"triggerID" : "dc851f6226047629aec76f30cd9843ac635a66d9",
"triggerType" : "PUSH"
}, {
"hash" : "8c51a6c54679f5723053ae3d9efe343bdeea9445",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7041",
"triggerID" : "8c51a6c54679f5723053ae3d9efe343bdeea9445",
"triggerType" : "PUSH"
}, {
"hash" : "723b2bdc505e58d9a7525be996402ce55d2b9bff",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7123",
"triggerID" : "723b2bdc505e58d9a7525be996402ce55d2b9bff",
"triggerType" : "PUSH"
}, {
"hash" : "c7902b06721cfb4e51d166b5799c20d58f079404",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c7902b06721cfb4e51d166b5799c20d58f079404",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e19abadae9dece1aa78868839605a3d23a8b29fd UNKNOWN
* 4ba45f607a07f3f036e164dd68532fbc8b4bc3b8 UNKNOWN
* 8c51a6c54679f5723053ae3d9efe343bdeea9445 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7041)
* 723b2bdc505e58d9a7525be996402ce55d2b9bff Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7123)
* c7902b06721cfb4e51d166b5799c20d58f079404 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3049834581
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 26155cf4cb689ae854bc81e5c2c9c4ae8134ec85 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647)
* 8e74196c55e365e52dd5114c59ba96b2dfffb2ab Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3045937174
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 26a8660da376934b6f864b04f96e160d342886ca UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3047706634
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 26a8660da376934b6f864b04f96e160d342886ca Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628)
* 26155cf4cb689ae854bc81e5c2c9c4ae8134ec85 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3098031596
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fc0eba477035f595d72c7f0ce133a2d023df0a53 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907)
* 64d9924c454d76f90eaa6f5089e5f36345bf8f6f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2218363553
##########
hudi-common/src/main/java/org/apache/hudi/Comparables.java:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hudi.common.util.ValidationUtils;
+
+import java.io.Serializable;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.model.HoodieRecord.DEFAULT_ORDERING_VALUE;
+
+public class Comparables implements Comparable, Serializable {
+ protected static final long serialVersionUID = 1L;
+
+ private final List<Comparable> comparables;
Review Comment:
I have added optimisation so that whenever the constructor is called and size of comparables is 1, we use singletonList instead. This is more efficient than a list. Also it does not add any complexity related to managing both a singleton element and a list. Please take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3096075850
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fc0eba477035f595d72c7f0ce133a2d023df0a53 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2220566934
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -169,8 +170,9 @@ public class HoodieWriteConfig extends HoodieConfig {
public static final ConfigProperty<String> PRECOMBINE_FIELD_NAME = ConfigProperty
.key("hoodie.datasource.write.precombine.field")
.noDefaultValue()
- .withDocumentation("Field used in preCombining before actual write. When two records have the same key value, "
- + "we will pick the one with the largest value for the precombine field, determined by Object.compareTo(..)");
+ .withDocumentation("Comma separated list of fields used in preCombining before actual write. When two records have the same key value, "
Review Comment:
Please file a separate jira and put out a separate for this config name change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3102472643
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
}, {
"hash" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"triggerType" : "PUSH"
}, {
"hash" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947",
"triggerID" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"triggerType" : "PUSH"
}, {
"hash" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6949",
"triggerID" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"triggerType" : "PUSH"
}, {
"hash" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e19abadae9dece1aa78868839605a3d23a8b29fd UNKNOWN
* 9ce11db126c7a2124c41ee9c1b97e1820fd6af8e Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6949)
* 4ba45f607a07f3f036e164dd68532fbc8b4bc3b8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3105997836
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
}, {
"hash" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"triggerType" : "PUSH"
}, {
"hash" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947",
"triggerID" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"triggerType" : "PUSH"
}, {
"hash" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6949",
"triggerID" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"triggerType" : "PUSH"
}, {
"hash" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"triggerType" : "PUSH"
}, {
"hash" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6950",
"triggerID" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"triggerType" : "PUSH"
}, {
"hash" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6974",
"triggerID" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"triggerType" : "PUSH"
}, {
"hash" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6991",
"triggerID" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e19abadae9dece1aa78868839605a3d23a8b29fd UNKNOWN
* 4ba45f607a07f3f036e164dd68532fbc8b4bc3b8 UNKNOWN
* 21043f2bc87787d692ec55c2e4150ab29a396264 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6974)
* 7c1ded9e7881ffea36adeceb34c89ea25353bc0d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6991)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3049844504
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8e74196c55e365e52dd5114c59ba96b2dfffb2ab Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3053784032
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c90d258b5f530a935ac34d36919454be963c1d5b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2231696725
##########
hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java:
##########
@@ -1499,21 +1503,21 @@ private static Schema getActualSchemaFromUnion(Schema schema, Object data) {
public static HoodieRecord createHoodieRecordFromAvro(
IndexedRecord data,
String payloadClass,
- String preCombineField,
+ Option<String[]> preCombineFields,
Review Comment:
We are setting it to default ordering value if no preCombine fields are available.
`org.apache.hudi.common.util.SpillableMapUtils#getPreCombineVal`
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestHoodieFileGroupReaderBase.java:
##########
@@ -183,6 +187,48 @@ public void testReadFileGroupInMergeOnReadTable(RecordMergeMode recordMergeMode,
}
}
+ @Test
+ public void testReadFileGroupWithMultipleOrderingFields() throws Exception {
+ RecordMergeMode recordMergeMode = RecordMergeMode.EVENT_TIME_ORDERING;
+ Map<String, String> writeConfigs = new HashMap<>(getCommonConfigs(recordMergeMode, true));
+ writeConfigs.put(HoodieStorageConfig.LOGFILE_DATA_BLOCK_FORMAT.key(), "avro");
+ writeConfigs.put("hoodie.datasource.write.table.type", HoodieTableType.MERGE_ON_READ.name());
+ // Use two precombine values - combination of timestamp and rider
+ String orderingValues = "timestamp,rider";
+ writeConfigs.put("hoodie.datasource.write.precombine.field", orderingValues);
+ writeConfigs.put("hoodie.payload.ordering.field", orderingValues);
+
+ try (HoodieTestDataGenerator dataGen = new HoodieTestDataGenerator(0xDEEF)) {
+ // Initial commit. rider column gets value of rider-002
+ List<HoodieRecord> initialRecords = dataGen.generateInserts("002", 100);
+ commitToTable(initialRecords, INSERT.value(), true, writeConfigs);
+ validateOutputFromFileGroupReader(
+ getStorageConf(), getBasePath(), true, 0, recordMergeMode,
+ initialRecords, initialRecords);
+
+ // The updates have rider values as rider-001 and the existing records have rider values as rider-002
Review Comment:
Yes, this is already covered in test I added in TestMORDataSource
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java:
##########
@@ -115,10 +115,8 @@ void addProjectionToJobConf(final RealtimeSplit realtimeSplit, final JobConf job
List<String> fieldsToAdd = new ArrayList<>();
if (!realtimeSplit.getDeltaLogPaths().isEmpty()) {
HoodieRealtimeInputFormatUtils.addVirtualKeysProjection(jobConf, realtimeSplit.getVirtualKeyInfo());
- String preCombineKey = tableConfig.getPreCombineField();
- if (!StringUtils.isNullOrEmpty(preCombineKey)) {
- fieldsToAdd.add(preCombineKey);
- }
+ List<String> preCombineKeys = tableConfig.getPreCombineFields();
+ preCombineKeys.stream().filter(StringUtils::nonEmpty).forEach(fieldsToAdd::add);
Review Comment:
Fixed it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3121455029
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
}, {
"hash" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"triggerType" : "PUSH"
}, {
"hash" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947",
"triggerID" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"triggerType" : "PUSH"
}, {
"hash" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6949",
"triggerID" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"triggerType" : "PUSH"
}, {
"hash" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"triggerType" : "PUSH"
}, {
"hash" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6950",
"triggerID" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"triggerType" : "PUSH"
}, {
"hash" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6974",
"triggerID" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"triggerType" : "PUSH"
}, {
"hash" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6991",
"triggerID" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"triggerType" : "PUSH"
}, {
"hash" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6993",
"triggerID" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"triggerType" : "PUSH"
}, {
"hash" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6995",
"triggerID" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"triggerType" : "PUSH"
}, {
"hash" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6996",
"triggerID" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"triggerType" : "PUSH"
}, {
"hash" : "dc851f6226047629aec76f30cd9843ac635a66d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7001",
"triggerID" : "dc851f6226047629aec76f30cd9843ac635a66d9",
"triggerType" : "PUSH"
}, {
"hash" : "8c51a6c54679f5723053ae3d9efe343bdeea9445",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7041",
"triggerID" : "8c51a6c54679f5723053ae3d9efe343bdeea9445",
"triggerType" : "PUSH"
}, {
"hash" : "723b2bdc505e58d9a7525be996402ce55d2b9bff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7123",
"triggerID" : "723b2bdc505e58d9a7525be996402ce55d2b9bff",
"triggerType" : "PUSH"
}, {
"hash" : "c7902b06721cfb4e51d166b5799c20d58f079404",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7125",
"triggerID" : "c7902b06721cfb4e51d166b5799c20d58f079404",
"triggerType" : "PUSH"
}, {
"hash" : "5e3962074d7b4f7dceaedf9a391e763e054f9b9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7139",
"triggerID" : "5e3962074d7b4f7dceaedf9a391e763e054f9b9d",
"triggerType" : "PUSH"
}, {
"hash" : "371f9b89c48b3dc9a9b95b15266b9f2fd4201a56",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7165",
"triggerID" : "371f9b89c48b3dc9a9b95b15266b9f2fd4201a56",
"triggerType" : "PUSH"
}, {
"hash" : "2f9d6614a9ea6ed28bdc3654d37a7dc17563175b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7169",
"triggerID" : "2f9d6614a9ea6ed28bdc3654d37a7dc17563175b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e19abadae9dece1aa78868839605a3d23a8b29fd UNKNOWN
* 4ba45f607a07f3f036e164dd68532fbc8b4bc3b8 UNKNOWN
* 371f9b89c48b3dc9a9b95b15266b9f2fd4201a56 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7165)
* 2f9d6614a9ea6ed28bdc3654d37a7dc17563175b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7169)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3119905631
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
}, {
"hash" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"triggerType" : "PUSH"
}, {
"hash" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947",
"triggerID" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"triggerType" : "PUSH"
}, {
"hash" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6949",
"triggerID" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"triggerType" : "PUSH"
}, {
"hash" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"triggerType" : "PUSH"
}, {
"hash" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6950",
"triggerID" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"triggerType" : "PUSH"
}, {
"hash" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6974",
"triggerID" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"triggerType" : "PUSH"
}, {
"hash" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6991",
"triggerID" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"triggerType" : "PUSH"
}, {
"hash" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6993",
"triggerID" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"triggerType" : "PUSH"
}, {
"hash" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6995",
"triggerID" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"triggerType" : "PUSH"
}, {
"hash" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6996",
"triggerID" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"triggerType" : "PUSH"
}, {
"hash" : "dc851f6226047629aec76f30cd9843ac635a66d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7001",
"triggerID" : "dc851f6226047629aec76f30cd9843ac635a66d9",
"triggerType" : "PUSH"
}, {
"hash" : "8c51a6c54679f5723053ae3d9efe343bdeea9445",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7041",
"triggerID" : "8c51a6c54679f5723053ae3d9efe343bdeea9445",
"triggerType" : "PUSH"
}, {
"hash" : "723b2bdc505e58d9a7525be996402ce55d2b9bff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7123",
"triggerID" : "723b2bdc505e58d9a7525be996402ce55d2b9bff",
"triggerType" : "PUSH"
}, {
"hash" : "c7902b06721cfb4e51d166b5799c20d58f079404",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7125",
"triggerID" : "c7902b06721cfb4e51d166b5799c20d58f079404",
"triggerType" : "PUSH"
}, {
"hash" : "5e3962074d7b4f7dceaedf9a391e763e054f9b9d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7139",
"triggerID" : "5e3962074d7b4f7dceaedf9a391e763e054f9b9d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e19abadae9dece1aa78868839605a3d23a8b29fd UNKNOWN
* 4ba45f607a07f3f036e164dd68532fbc8b4bc3b8 UNKNOWN
* c7902b06721cfb4e51d166b5799c20d58f079404 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7125)
* 5e3962074d7b4f7dceaedf9a391e763e054f9b9d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7139)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3120160448
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
}, {
"hash" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"triggerType" : "PUSH"
}, {
"hash" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947",
"triggerID" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"triggerType" : "PUSH"
}, {
"hash" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6949",
"triggerID" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"triggerType" : "PUSH"
}, {
"hash" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"triggerType" : "PUSH"
}, {
"hash" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6950",
"triggerID" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"triggerType" : "PUSH"
}, {
"hash" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6974",
"triggerID" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"triggerType" : "PUSH"
}, {
"hash" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6991",
"triggerID" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"triggerType" : "PUSH"
}, {
"hash" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6993",
"triggerID" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"triggerType" : "PUSH"
}, {
"hash" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6995",
"triggerID" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"triggerType" : "PUSH"
}, {
"hash" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6996",
"triggerID" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"triggerType" : "PUSH"
}, {
"hash" : "dc851f6226047629aec76f30cd9843ac635a66d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7001",
"triggerID" : "dc851f6226047629aec76f30cd9843ac635a66d9",
"triggerType" : "PUSH"
}, {
"hash" : "8c51a6c54679f5723053ae3d9efe343bdeea9445",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7041",
"triggerID" : "8c51a6c54679f5723053ae3d9efe343bdeea9445",
"triggerType" : "PUSH"
}, {
"hash" : "723b2bdc505e58d9a7525be996402ce55d2b9bff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7123",
"triggerID" : "723b2bdc505e58d9a7525be996402ce55d2b9bff",
"triggerType" : "PUSH"
}, {
"hash" : "c7902b06721cfb4e51d166b5799c20d58f079404",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7125",
"triggerID" : "c7902b06721cfb4e51d166b5799c20d58f079404",
"triggerType" : "PUSH"
}, {
"hash" : "5e3962074d7b4f7dceaedf9a391e763e054f9b9d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7139",
"triggerID" : "5e3962074d7b4f7dceaedf9a391e763e054f9b9d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e19abadae9dece1aa78868839605a3d23a8b29fd UNKNOWN
* 4ba45f607a07f3f036e164dd68532fbc8b4bc3b8 UNKNOWN
* 5e3962074d7b4f7dceaedf9a391e763e054f9b9d Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7139)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2241597592
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -169,8 +170,9 @@ public class HoodieWriteConfig extends HoodieConfig {
public static final ConfigProperty<String> PRECOMBINE_FIELD_NAME = ConfigProperty
.key("hoodie.datasource.write.precombine.field")
.noDefaultValue()
- .withDocumentation("Field used in preCombining before actual write. When two records have the same key value, "
- + "we will pick the one with the largest value for the precombine field, determined by Object.compareTo(..)");
+ .withDocumentation("Comma separated list of fields used in preCombining before actual write. When two records have the same key value, "
Review Comment:
I will be fixing it as part of https://issues.apache.org/jira/browse/HUDI-9619
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2218393316
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/SevenToEightUpgradeHandler.java:
##########
@@ -175,7 +175,7 @@ static void upgradePartitionFields(HoodieWriteConfig config, HoodieTableConfig t
static void upgradeMergeMode(HoodieTableConfig tableConfig, Map<ConfigProperty, String> tablePropsToAdd) {
String payloadClass = tableConfig.getPayloadClass();
- String preCombineField = tableConfig.getPreCombineField();
+ String preCombineField = tableConfig.getPreCombineFields().orElse(null);
Review Comment:
Addressed
##########
hudi-common/src/main/java/org/apache/hudi/Comparables.java:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hudi.common.util.ValidationUtils;
+
+import java.io.Serializable;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.model.HoodieRecord.DEFAULT_ORDERING_VALUE;
Review Comment:
Addressed
##########
hudi-common/src/main/java/org/apache/hudi/Comparables.java:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hudi.common.util.ValidationUtils;
+
+import java.io.Serializable;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.model.HoodieRecord.DEFAULT_ORDERING_VALUE;
+
+public class Comparables implements Comparable, Serializable {
+ protected static final long serialVersionUID = 1L;
+
+ private final List<Comparable> comparables;
+
+ public Comparables(List<Comparable> comparables) {
+ this.comparables = comparables;
+ }
+
+ public Comparables(Comparable comparable) {
+ this.comparables = Collections.singletonList(comparable);
+ }
+
+ public static boolean isDefault(Comparable orderingVal) {
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3098035774
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fc0eba477035f595d72c7f0ce133a2d023df0a53 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907)
* 64d9924c454d76f90eaa6f5089e5f36345bf8f6f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3095687837
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c6a88f8bdbb390dbc6602bc55839fdec432d6b4e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2225404302
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/block/HoodieDeleteBlock.java:
##########
@@ -142,11 +142,18 @@ private byte[] serializeV3() throws IOException {
HoodieDeleteRecordList.Builder recordListBuilder = HOODIE_DELETE_RECORD_LIST_BUILDER_STUB.get();
HoodieDeleteRecord.Builder recordBuilder = HOODIE_DELETE_RECORD_BUILDER_STUB.get();
List<HoodieDeleteRecord> deleteRecordList = Arrays.stream(getRecordsToDelete())
- .map(record -> HoodieDeleteRecord.newBuilder(recordBuilder)
+ .map(record -> {
+ List<Comparable> orderingVals = record.getOrderingValue() instanceof Comparables
Review Comment:
This is taken care of in recent commits made by Danny
##########
hudi-common/src/main/java/org/apache/hudi/common/engine/HoodieReaderContext.java:
##########
@@ -479,6 +481,15 @@ public final UnaryOperator<T> projectRecord(Schema from, Schema to) {
return projectRecord(from, to, Collections.emptyMap());
}
+ /**
+ * Converts the comparable values in Comparables to the specific engine type
+ */
+ public final Comparable convertValueToEngineType(Comparable value) {
+ return value instanceof Comparables
Review Comment:
This is taken care of in recent commits made by Danny
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3102591718
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
}, {
"hash" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"triggerType" : "PUSH"
}, {
"hash" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947",
"triggerID" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"triggerType" : "PUSH"
}, {
"hash" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6949",
"triggerID" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"triggerType" : "PUSH"
}, {
"hash" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"triggerType" : "PUSH"
}, {
"hash" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e19abadae9dece1aa78868839605a3d23a8b29fd UNKNOWN
* 9ce11db126c7a2124c41ee9c1b97e1820fd6af8e Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6949)
* 4ba45f607a07f3f036e164dd68532fbc8b4bc3b8 UNKNOWN
* a1f62b31b47715ce88932618c3817fbd7dd2e96e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3107921003
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
}, {
"hash" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"triggerType" : "PUSH"
}, {
"hash" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947",
"triggerID" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"triggerType" : "PUSH"
}, {
"hash" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6949",
"triggerID" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"triggerType" : "PUSH"
}, {
"hash" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"triggerType" : "PUSH"
}, {
"hash" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6950",
"triggerID" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"triggerType" : "PUSH"
}, {
"hash" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6974",
"triggerID" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"triggerType" : "PUSH"
}, {
"hash" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6991",
"triggerID" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"triggerType" : "PUSH"
}, {
"hash" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6993",
"triggerID" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"triggerType" : "PUSH"
}, {
"hash" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6995",
"triggerID" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"triggerType" : "PUSH"
}, {
"hash" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6996",
"triggerID" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e19abadae9dece1aa78868839605a3d23a8b29fd UNKNOWN
* 4ba45f607a07f3f036e164dd68532fbc8b4bc3b8 UNKNOWN
* 2dba2b4f6c8d57fea9ff262e842f45153d56787f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6996)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3058291333
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c90d258b5f530a935ac34d36919454be963c1d5b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662)
* f4b6ff8db1543238f986e2d604f4f71971313b95 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3095547360
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f4b6ff8db1543238f986e2d604f4f71971313b95 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671)
* c6a88f8bdbb390dbc6602bc55839fdec432d6b4e Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3095544059
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f4b6ff8db1543238f986e2d604f4f71971313b95 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671)
* c6a88f8bdbb390dbc6602bc55839fdec432d6b4e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3095706930
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c6a88f8bdbb390dbc6602bc55839fdec432d6b4e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904)
* 98a828795959a9cfc01d6c04ea8739228fa6009c Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2222203052
##########
hudi-common/src/main/java/org/apache/hudi/common/engine/HoodieReaderContext.java:
##########
@@ -479,6 +481,15 @@ public final UnaryOperator<T> projectRecord(Schema from, Schema to) {
return projectRecord(from, to, Collections.emptyMap());
}
+ /**
+ * Converts the comparable values in Comparables to the specific engine type
+ */
+ public final Comparable convertValueToEngineType(Comparable value) {
+ return value instanceof Comparables
Review Comment:
Maybe we just add a new method `convertOrderingValueToEngineType(OrderingValue)` so that we can get rid of the `instanceof` per record.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2222196084
##########
hudi-common/src/main/java/org/apache/hudi/Comparables.java:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hudi.common.util.ValidationUtils;
+
+import java.io.Serializable;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.model.HoodieRecord.DEFAULT_ORDERING_VALUE;
+
+public class Comparables implements Comparable, Serializable {
+ protected static final long serialVersionUID = 1L;
+
+ private final List<Comparable> comparables;
Review Comment:
This is my concern too, my idea is we rename the class into just `OrderingValue`(or keep the name if we want to generalize it in the future) and introduce two sub-classes here: `SingleFieldOrderingValue` and `MultipleFieldsOrderingValue`. We should override the core methods: `#compareTo`, `#isSameClass` and `#isDefault` to get rid of the reduadant list loop.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2221841788
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestHoodieFileGroupReaderBase.java:
##########
@@ -185,6 +189,49 @@ public void testReadFileGroupInMergeOnReadTable(RecordMergeMode recordMergeMode,
}
}
+ @ParameterizedTest
+ @ValueSource(strings = {"avro", "parquet"})
+ public void testReadFileGroupWithMultipleOrderingFields(String logDataBlockFormat) throws Exception {
+ RecordMergeMode recordMergeMode = RecordMergeMode.EVENT_TIME_ORDERING;
+ Map<String, String> writeConfigs = new HashMap<>(getCommonConfigs(recordMergeMode, true));
+ writeConfigs.put(HoodieStorageConfig.LOGFILE_DATA_BLOCK_FORMAT.key(), logDataBlockFormat);
+ writeConfigs.put("hoodie.datasource.write.table.type", HoodieTableType.MERGE_ON_READ.name());
+ // Use two precombine values - combination of timestamp and rider
+ String orderingValues = "timestamp,rider";
+ writeConfigs.put("hoodie.datasource.write.precombine.field", orderingValues);
+ writeConfigs.put("hoodie.payload.ordering.field", orderingValues);
+
+ try (HoodieTestDataGenerator dataGen = new HoodieTestDataGenerator(0xDEEF)) {
+ // Initial commit. rider column gets value of rider-002
+ List<HoodieRecord> initialRecords = dataGen.generateInserts("002", 100);
+ commitToTable(initialRecords, INSERT.value(), true, writeConfigs);
+ validateOutputFromFileGroupReader(
+ getStorageConf(), getBasePath(), true, 0, recordMergeMode,
+ initialRecords, initialRecords);
+
+ // The updates have rider values as rider-001 and the existing records have rider values as rider-001
Review Comment:
Addressed
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/TestHoodieDeltaStreamer.java:
##########
@@ -1955,6 +1957,62 @@ public void testFilterDupesWithPrecombine(
UtilitiesTestBase.Helpers.deleteFileFromDfs(fs, tableBasePath);
}
+ @ParameterizedTest
+ @EnumSource(HoodieTableType.class)
+ public void testDeltaStreamerWithMultipleOrderingFields(HoodieTableType tableType) throws Exception {
+ String tableBasePath = basePath + "/test_with_multiple_ordering_fields";
+ HoodieDeltaStreamer.Config cfg =
+ TestHelpers.makeConfig(tableBasePath, WriteOperationType.BULK_INSERT);
+ cfg.tableType = tableType.name();
+ cfg.filterDupes = true;
+ cfg.sourceOrderingFields = "timestamp,rider";
+ cfg.recordMergeMode = RecordMergeMode.EVENT_TIME_ORDERING;
+ cfg.payloadClassName = DefaultHoodieRecordPayload.class.getName();
+ cfg.recordMergeStrategyId = HoodieRecordMerger.EVENT_TIME_BASED_MERGE_STRATEGY_UUID;
+ // Set record merge mode to event time ordering
+ cfg.configs.add(String.format("%s=%s", HoodieWriteConfig.RECORD_MERGE_MODE.key(), RecordMergeMode.EVENT_TIME_ORDERING.name()));
+ //addRecordMerger(HoodieRecordType.AVRO, cfg.configs);
Review Comment:
Addressed
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestDataSource.java:
##########
@@ -41,6 +41,7 @@ public class TestDataSource extends AbstractBaseTestSource {
private static final Logger LOG = LoggerFactory.getLogger(TestDataSource.class);
public static boolean returnEmptyBatch = false;
+ public static Option<String> recordInstantTime = Option.empty();
Review Comment:
Addressed
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/TestHoodieDeltaStreamer.java:
##########
@@ -1955,6 +1957,62 @@ public void testFilterDupesWithPrecombine(
UtilitiesTestBase.Helpers.deleteFileFromDfs(fs, tableBasePath);
}
+ @ParameterizedTest
+ @EnumSource(HoodieTableType.class)
+ public void testDeltaStreamerWithMultipleOrderingFields(HoodieTableType tableType) throws Exception {
+ String tableBasePath = basePath + "/test_with_multiple_ordering_fields";
+ HoodieDeltaStreamer.Config cfg =
+ TestHelpers.makeConfig(tableBasePath, WriteOperationType.BULK_INSERT);
+ cfg.tableType = tableType.name();
+ cfg.filterDupes = true;
+ cfg.sourceOrderingFields = "timestamp,rider";
+ cfg.recordMergeMode = RecordMergeMode.EVENT_TIME_ORDERING;
+ cfg.payloadClassName = DefaultHoodieRecordPayload.class.getName();
+ cfg.recordMergeStrategyId = HoodieRecordMerger.EVENT_TIME_BASED_MERGE_STRATEGY_UUID;
+ // Set record merge mode to event time ordering
+ cfg.configs.add(String.format("%s=%s", HoodieWriteConfig.RECORD_MERGE_MODE.key(), RecordMergeMode.EVENT_TIME_ORDERING.name()));
+ //addRecordMerger(HoodieRecordType.AVRO, cfg.configs);
+ TestDataSource.recordInstantTime = Option.of("002");
+ new HoodieStreamer(cfg, jsc).sync();
+
+ assertRecordCount(1000, tableBasePath, sqlContext);
+ TestHelpers.assertCommitMetadata("00000", tableBasePath, 1);
+
+ // Generate new updates with lower recordInstantTime so that updates are rejected
+ TestDataSource.recordInstantTime = Option.of("001");
+ runStreamSync(cfg, false, 50, WriteOperationType.UPSERT);
+ int numInserts = 25;
+ // TestDataSource generates 500 inserts, 450 updates and 50 deletes
+ assertRecordCount(1025, tableBasePath, sqlContext); // if filter dupes is not enabled, we should be expecting 3000 records here.
+ TestHelpers.assertCommitMetadata("00001", tableBasePath, 2);
+ // Filter records with rider-001 value and deduct the number of inserts
+ long numUpdates = sparkSession.read().format("hudi").load(tableBasePath).filter("rider = 'rider-001'").count()
+ - numInserts;
+ // There should be no updates since ordering value rider-001 is lower than existing record ordering value rider-002
+ assertEquals(0, numUpdates);
+
+ // Generate new updates with lower recordInstantTime so that updates are rejected
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3104372127
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
}, {
"hash" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"triggerType" : "PUSH"
}, {
"hash" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947",
"triggerID" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"triggerType" : "PUSH"
}, {
"hash" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6949",
"triggerID" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"triggerType" : "PUSH"
}, {
"hash" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"triggerType" : "PUSH"
}, {
"hash" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6950",
"triggerID" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"triggerType" : "PUSH"
}, {
"hash" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e19abadae9dece1aa78868839605a3d23a8b29fd UNKNOWN
* 4ba45f607a07f3f036e164dd68532fbc8b4bc3b8 UNKNOWN
* a1f62b31b47715ce88932618c3817fbd7dd2e96e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6950)
* 21043f2bc87787d692ec55c2e4150ab29a396264 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3119757729
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
}, {
"hash" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"triggerType" : "PUSH"
}, {
"hash" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947",
"triggerID" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"triggerType" : "PUSH"
}, {
"hash" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6949",
"triggerID" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"triggerType" : "PUSH"
}, {
"hash" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"triggerType" : "PUSH"
}, {
"hash" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6950",
"triggerID" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"triggerType" : "PUSH"
}, {
"hash" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6974",
"triggerID" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"triggerType" : "PUSH"
}, {
"hash" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6991",
"triggerID" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"triggerType" : "PUSH"
}, {
"hash" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6993",
"triggerID" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"triggerType" : "PUSH"
}, {
"hash" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6995",
"triggerID" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"triggerType" : "PUSH"
}, {
"hash" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6996",
"triggerID" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"triggerType" : "PUSH"
}, {
"hash" : "dc851f6226047629aec76f30cd9843ac635a66d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7001",
"triggerID" : "dc851f6226047629aec76f30cd9843ac635a66d9",
"triggerType" : "PUSH"
}, {
"hash" : "8c51a6c54679f5723053ae3d9efe343bdeea9445",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7041",
"triggerID" : "8c51a6c54679f5723053ae3d9efe343bdeea9445",
"triggerType" : "PUSH"
}, {
"hash" : "723b2bdc505e58d9a7525be996402ce55d2b9bff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7123",
"triggerID" : "723b2bdc505e58d9a7525be996402ce55d2b9bff",
"triggerType" : "PUSH"
}, {
"hash" : "c7902b06721cfb4e51d166b5799c20d58f079404",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7125",
"triggerID" : "c7902b06721cfb4e51d166b5799c20d58f079404",
"triggerType" : "PUSH"
}, {
"hash" : "5e3962074d7b4f7dceaedf9a391e763e054f9b9d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5e3962074d7b4f7dceaedf9a391e763e054f9b9d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e19abadae9dece1aa78868839605a3d23a8b29fd UNKNOWN
* 4ba45f607a07f3f036e164dd68532fbc8b4bc3b8 UNKNOWN
* c7902b06721cfb4e51d166b5799c20d58f079404 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7125)
* 5e3962074d7b4f7dceaedf9a391e763e054f9b9d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #13526:
URL: https://github.com/apache/hudi/pull/13526
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3051284834
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8e74196c55e365e52dd5114c59ba96b2dfffb2ab Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652)
* 125b574e0176d7ec688acd05f9138f821f6dfe03 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2194111259
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/HoodieDatasetBulkInsertHelper.scala:
##########
@@ -213,12 +213,12 @@ object HoodieDatasetBulkInsertHelper
}), SQLConf.get).toJavaRDD())
}
- private def dedupeRows(rdd: RDD[InternalRow], schema: StructType, preCombineFieldRef: String, isGlobalIndex: Boolean, targetParallelism: Int): RDD[InternalRow] = {
+ private def dedupeRows(rdd: RDD[InternalRow], schema: StructType, preCombineFields: List[String], isGlobalIndex: Boolean, targetParallelism: Int): RDD[InternalRow] = {
val recordKeyMetaFieldOrd = schema.fieldIndex(HoodieRecord.RECORD_KEY_METADATA_FIELD)
val partitionPathMetaFieldOrd = schema.fieldIndex(HoodieRecord.PARTITION_PATH_METADATA_FIELD)
// NOTE: Pre-combine field could be a nested field
- val preCombineFieldPath = composeNestedFieldPath(schema, preCombineFieldRef)
- .getOrElse(throw new HoodieException(s"Pre-combine field $preCombineFieldRef is missing in $schema"))
+ val preCombineFieldPaths = preCombineFields.map(preCombineField => composeNestedFieldPath(schema, preCombineField)
+ .getOrElse(throw new HoodieException(s"Pre-combine field $preCombineFields is missing in $schema")))
Review Comment:
minor.
within HoodieException, we should be referring to `preCombineField` only and not `preCombineFields`
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -1360,8 +1362,13 @@ public HoodieTableType getTableType() {
HoodieTableConfig.TYPE, HoodieTableConfig.TYPE.defaultValue().name()).toUpperCase());
}
- public String getPreCombineField() {
- return getString(PRECOMBINE_FIELD_NAME);
+ public List<String> getPreCombineField() {
Review Comment:
getPreCombineFields()
"s" in the end
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/HoodieDatasetBulkInsertHelper.scala:
##########
@@ -233,13 +233,14 @@ object HoodieDatasetBulkInsertHelper
(rowKey, row.copy())
}
.reduceByKey ((oneRow, otherRow) => {
- val onePreCombineVal = getNestedInternalRowValue(oneRow, preCombineFieldPath).asInstanceOf[Comparable[AnyRef]]
- val otherPreCombineVal = getNestedInternalRowValue(otherRow, preCombineFieldPath).asInstanceOf[Comparable[AnyRef]]
- if (onePreCombineVal.compareTo(otherPreCombineVal.asInstanceOf[AnyRef]) >= 0) {
+ val onePreCombineVals = new Comparables(JavaConverters.seqAsJavaList(preCombineFieldPaths.map(preCombineFieldPath => getNestedInternalRowValue(oneRow, preCombineFieldPath).asInstanceOf[Comparable[AnyRef]])))
Review Comment:
`new Comparables(preCombineFieldPaths.map(preCombineFieldPath => getNestedInternalRowValue(oneRow, preCombineFieldPath).asInstanceOf[Comparable[AnyRef]]).toList.asJava)`
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/Iterators.scala:
##########
@@ -86,10 +86,10 @@ class LogFileIterator(logFiles: List[HoodieLogFile],
}
private val maxCompactionMemoryInBytes: Long = getMaxCompactionMemoryInBytes(new JobConf(config))
- protected val payloadProps: TypedProperties = tableState.preCombineFieldOpt
+ protected val payloadProps: TypedProperties = tableState.preCombineFieldsOpt
.map { preCombineField =>
HoodiePayloadConfig.newBuilder
- .withPayloadOrderingField(preCombineField)
+ .withPayloadOrderingField(String.join(",", preCombineField: _*))
Review Comment:
`withPayloadOrderingFields`
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/util/JavaScalaConverters.scala:
##########
@@ -69,4 +69,9 @@ object JavaScalaConverters {
def convertJavaPropertiesToScalaMap(javaProperties: java.util.Properties): Map[String, String] = {
javaProperties.asScala.toMap
}
+
+ def convertHudiOptionToScalaOption[A](option: org.apache.hudi.common.util.Option[A]): Option[A] = {
Review Comment:
don't we already have one for this
https://github.com/apache/hudi/blob/7c6edde1f6a1004df75bd82d206d1b665e80e883/hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/HoodieConversionUtils.scala#L40
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/cdc/CDCFileGroupIterator.scala:
##########
@@ -74,13 +76,13 @@ class CDCFileGroupIterator(split: HoodieCDCFileGroupSplit,
extends Iterator[InternalRow]
with SparkAdapterSupport with AvroDeserializerSupport with Closeable {
- protected val payloadProps: Properties = Option(metaClient.getTableConfig.getPreCombineField)
- .map { preCombineField =>
+ protected val payloadProps: Properties = metaClient.getTableConfig.getPreCombineFields
+ .map[TypedProperties](JFunction.toJavaFunction(preCombineFields =>
HoodiePayloadConfig.newBuilder
- .withPayloadOrderingField(preCombineField)
+ .withPayloadOrderingField(preCombineFields)
.build
.getProps
- }.getOrElse(new Properties())
+ )).orElse(new TypedProperties())
Review Comment:
why changed from new Properties to new TypedProperties
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/cdc/HoodieCDCRDD.scala:
##########
@@ -91,13 +93,13 @@ class HoodieCDCRDD(
private val props = HoodieFileIndex.getConfigProperties(spark, Map.empty, metaClient.getTableConfig)
- protected val payloadProps: Properties = Option(metaClient.getTableConfig.getPreCombineField)
- .map { preCombineField =>
+ protected val payloadProps: Properties = metaClient.getTableConfig.getPreCombineFields
+ .map[TypedProperties](JFunction.toJavaFunction(preCombineFields =>
HoodiePayloadConfig.newBuilder
- .withPayloadOrderingField(preCombineField)
+ .withPayloadOrderingField(preCombineFields)
.build
.getProps
- }.getOrElse(new Properties())
+ )).orElse(new TypedProperties())
Review Comment:
same question as above
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/cdc/HoodieCDCRDD.scala:
##########
@@ -137,24 +139,14 @@ class HoodieCDCRDD(
keyFields.head
}
- private lazy val preCombineFieldOpt: Option[String] = Option(metaClient.getTableConfig.getPreCombineField)
+ private lazy val preCombineFieldsOpt: Option[List[String]] = convertHudiOptionToScalaOption(metaClient.getTableConfig.getPreCombineFieldList
+ .map[List[String]](list => list.asScala.toList))
private lazy val tableState = {
val metadataConfig = HoodieMetadataConfig.newBuilder()
.fromProperties(props)
.build()
- HoodieTableState(
- basePath.toUri.toString,
- Some(split.changes.last.getInstant),
- recordKeyField,
- preCombineFieldOpt,
- usesVirtualKeys = !populateMetaFields,
- metaClient.getTableConfig.getPayloadClass,
- metadataConfig,
- // TODO support CDC with spark record
- recordMergeImplClasses = List(classOf[HoodieAvroRecordMerger].getName),
- recordMergeStrategyId = HoodieRecordMerger.PAYLOAD_BASED_MERGE_STRATEGY_UUID
- )
+ HoodieTableState(basePath.toUri.toString, Some(split.changes.last.getInstant), recordKeyField, preCombineFieldsOpt, usesVirtualKeys = !populateMetaFields, metaClient.getTableConfig.getPayloadClass, metadataConfig, recordMergeImplClasses = List(classOf[HoodieAvroRecordMerger].getName), recordMergeStrategyId = HoodieRecordMerger.PAYLOAD_BASED_MERGE_STRATEGY_UUID)
Review Comment:
what happened here. why format changes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3121617890
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
}, {
"hash" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"triggerType" : "PUSH"
}, {
"hash" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947",
"triggerID" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"triggerType" : "PUSH"
}, {
"hash" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6949",
"triggerID" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"triggerType" : "PUSH"
}, {
"hash" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"triggerType" : "PUSH"
}, {
"hash" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6950",
"triggerID" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"triggerType" : "PUSH"
}, {
"hash" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6974",
"triggerID" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"triggerType" : "PUSH"
}, {
"hash" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6991",
"triggerID" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"triggerType" : "PUSH"
}, {
"hash" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6993",
"triggerID" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"triggerType" : "PUSH"
}, {
"hash" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6995",
"triggerID" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"triggerType" : "PUSH"
}, {
"hash" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6996",
"triggerID" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"triggerType" : "PUSH"
}, {
"hash" : "dc851f6226047629aec76f30cd9843ac635a66d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7001",
"triggerID" : "dc851f6226047629aec76f30cd9843ac635a66d9",
"triggerType" : "PUSH"
}, {
"hash" : "8c51a6c54679f5723053ae3d9efe343bdeea9445",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7041",
"triggerID" : "8c51a6c54679f5723053ae3d9efe343bdeea9445",
"triggerType" : "PUSH"
}, {
"hash" : "723b2bdc505e58d9a7525be996402ce55d2b9bff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7123",
"triggerID" : "723b2bdc505e58d9a7525be996402ce55d2b9bff",
"triggerType" : "PUSH"
}, {
"hash" : "c7902b06721cfb4e51d166b5799c20d58f079404",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7125",
"triggerID" : "c7902b06721cfb4e51d166b5799c20d58f079404",
"triggerType" : "PUSH"
}, {
"hash" : "5e3962074d7b4f7dceaedf9a391e763e054f9b9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7139",
"triggerID" : "5e3962074d7b4f7dceaedf9a391e763e054f9b9d",
"triggerType" : "PUSH"
}, {
"hash" : "371f9b89c48b3dc9a9b95b15266b9f2fd4201a56",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7165",
"triggerID" : "371f9b89c48b3dc9a9b95b15266b9f2fd4201a56",
"triggerType" : "PUSH"
}, {
"hash" : "2f9d6614a9ea6ed28bdc3654d37a7dc17563175b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7169",
"triggerID" : "2f9d6614a9ea6ed28bdc3654d37a7dc17563175b",
"triggerType" : "PUSH"
}, {
"hash" : "c74a62c8f7f57fc91116e40fb3db163c60853a23",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7170",
"triggerID" : "c74a62c8f7f57fc91116e40fb3db163c60853a23",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e19abadae9dece1aa78868839605a3d23a8b29fd UNKNOWN
* 4ba45f607a07f3f036e164dd68532fbc8b4bc3b8 UNKNOWN
* c74a62c8f7f57fc91116e40fb3db163c60853a23 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7170)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2227075274
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/HoodieDatasetBulkInsertHelper.scala:
##########
@@ -255,6 +257,16 @@ object HoodieDatasetBulkInsertHelper
}
}
+ private def getOrderingValue(nestedFieldPaths: List[NestedFieldPath], r: InternalRow): Comparable[_] = {
+ if (nestedFieldPaths.isEmpty) {
+ EmptyOrderingValue.getInstance();
+ } else if (nestedFieldPaths.size == 1) {
+ getNestedInternalRowValue(r, nestedFieldPaths.head).asInstanceOf[Comparable[_]]
+ } else {
+ new MultipleFieldsOrderingValue(nestedFieldPaths.map(_ => getNestedInternalRowValue(r, nestedFieldPaths.head).asInstanceOf[Comparable[_]]).toArray)
Review Comment:
I tried, but got some scala typeinference issues, not very familiar with scala, it's great if you can help here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2231170119
##########
hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java:
##########
@@ -1499,21 +1503,21 @@ private static Schema getActualSchemaFromUnion(Schema schema, Object data) {
public static HoodieRecord createHoodieRecordFromAvro(
IndexedRecord data,
String payloadClass,
- String preCombineField,
+ Option<String[]> preCombineFields,
Review Comment:
Should we have this be an empty array if there are no ordering fields?
##########
hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java:
##########
@@ -1659,6 +1666,11 @@ public static Comparable<?> unwrapAvroValueWrapper(Object avroValueWrapper) {
return ((BytesWrapper) avroValueWrapper).getValue();
} else if (avroValueWrapper instanceof StringWrapper) {
return ((StringWrapper) avroValueWrapper).getValue();
+ } else if (avroValueWrapper instanceof ArrayWrapper) {
Review Comment:
The code still assumes that the caller will always want the array wrapper to be converted to OrderingValues, is that safe? This is a generic util class so it can be used from anywhere meaning it is not confined to ordering fields usecases.
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java:
##########
@@ -115,10 +115,8 @@ void addProjectionToJobConf(final RealtimeSplit realtimeSplit, final JobConf job
List<String> fieldsToAdd = new ArrayList<>();
if (!realtimeSplit.getDeltaLogPaths().isEmpty()) {
HoodieRealtimeInputFormatUtils.addVirtualKeysProjection(jobConf, realtimeSplit.getVirtualKeyInfo());
- String preCombineKey = tableConfig.getPreCombineField();
- if (!StringUtils.isNullOrEmpty(preCombineKey)) {
- fieldsToAdd.add(preCombineKey);
- }
+ List<String> preCombineKeys = tableConfig.getPreCombineFields();
+ preCombineKeys.stream().filter(StringUtils::nonEmpty).forEach(fieldsToAdd::add);
Review Comment:
The table config should already be filtering out empty strings, right? I think we can remove this check here
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestHoodieFileGroupReaderBase.java:
##########
@@ -183,6 +187,48 @@ public void testReadFileGroupInMergeOnReadTable(RecordMergeMode recordMergeMode,
}
}
+ @Test
+ public void testReadFileGroupWithMultipleOrderingFields() throws Exception {
+ RecordMergeMode recordMergeMode = RecordMergeMode.EVENT_TIME_ORDERING;
+ Map<String, String> writeConfigs = new HashMap<>(getCommonConfigs(recordMergeMode, true));
+ writeConfigs.put(HoodieStorageConfig.LOGFILE_DATA_BLOCK_FORMAT.key(), "avro");
+ writeConfigs.put("hoodie.datasource.write.table.type", HoodieTableType.MERGE_ON_READ.name());
+ // Use two precombine values - combination of timestamp and rider
+ String orderingValues = "timestamp,rider";
+ writeConfigs.put("hoodie.datasource.write.precombine.field", orderingValues);
+ writeConfigs.put("hoodie.payload.ordering.field", orderingValues);
+
+ try (HoodieTestDataGenerator dataGen = new HoodieTestDataGenerator(0xDEEF)) {
+ // Initial commit. rider column gets value of rider-002
+ List<HoodieRecord> initialRecords = dataGen.generateInserts("002", 100);
+ commitToTable(initialRecords, INSERT.value(), true, writeConfigs);
+ validateOutputFromFileGroupReader(
+ getStorageConf(), getBasePath(), true, 0, recordMergeMode,
+ initialRecords, initialRecords);
+
+ // The updates have rider values as rider-001 and the existing records have rider values as rider-002
Review Comment:
Can you add a batch of timestamp 1 and rider-001 to show that the first value ordering is working properly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3053623899
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 125b574e0176d7ec688acd05f9138f821f6dfe03 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657)
* c90d258b5f530a935ac34d36919454be963c1d5b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2199372741
##########
hudi-common/src/main/java/org/apache/hudi/common/engine/HoodieReaderContext.java:
##########
@@ -385,21 +386,24 @@ private BiFunction<T, Schema, String> virtualKeyExtractor(String[] recordKeyFiel
/**
* Gets the ordering value in particular type.
*
- * @param record An option of record.
- * @param schema The Avro schema of the record.
- * @param orderingFieldName name of the ordering field
+ * @param record An option of record.
+ * @param schema The Avro schema of the record.
+ * @param orderingFieldNames name of the ordering field
* @return The ordering value.
*/
public Comparable getOrderingValue(T record,
Schema schema,
- Option<String> orderingFieldName) {
- if (orderingFieldName.isEmpty()) {
- return DEFAULT_ORDERING_VALUE;
+ Option<List<String>> orderingFieldNames) {
+ if (orderingFieldNames.isEmpty()) {
+ return Comparables.getDefault();
}
- Object value = getValue(record, schema, orderingFieldName.get());
- Comparable finalOrderingVal = value != null ? convertValueToEngineType((Comparable) value) : DEFAULT_ORDERING_VALUE;
- return finalOrderingVal;
+ return new Comparables(
+ orderingFieldNames.get().stream().map(field -> {
+ Object value = getValue(record, schema, field);
+ // API getDefaultOrderingValue is only used inside Comparables constructor
Review Comment:
Let's pay attention to the performance issue in this method that every record passes through. How about using a fixed-size list with `ArrayList(orderingFieldNames.size())` instead? The operation of `Stream` will have inevitable additional overhead
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3098539681
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 64d9924c454d76f90eaa6f5089e5f36345bf8f6f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2221320005
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -169,8 +170,9 @@ public class HoodieWriteConfig extends HoodieConfig {
public static final ConfigProperty<String> PRECOMBINE_FIELD_NAME = ConfigProperty
.key("hoodie.datasource.write.precombine.field")
.noDefaultValue()
- .withDocumentation("Field used in preCombining before actual write. When two records have the same key value, "
- + "we will pick the one with the largest value for the precombine field, determined by Object.compareTo(..)");
+ .withDocumentation("Comma separated list of fields used in preCombining before actual write. When two records have the same key value, "
Review Comment:
Created https://issues.apache.org/jira/browse/HUDI-9619
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3101793468
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 64d9924c454d76f90eaa6f5089e5f36345bf8f6f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919)
* bf66a590541998a22aa83fe644c03ff5d7eb2b47 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3121322057
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
}, {
"hash" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"triggerType" : "PUSH"
}, {
"hash" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947",
"triggerID" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"triggerType" : "PUSH"
}, {
"hash" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6949",
"triggerID" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"triggerType" : "PUSH"
}, {
"hash" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"triggerType" : "PUSH"
}, {
"hash" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6950",
"triggerID" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"triggerType" : "PUSH"
}, {
"hash" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6974",
"triggerID" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"triggerType" : "PUSH"
}, {
"hash" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6991",
"triggerID" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"triggerType" : "PUSH"
}, {
"hash" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6993",
"triggerID" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"triggerType" : "PUSH"
}, {
"hash" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6995",
"triggerID" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"triggerType" : "PUSH"
}, {
"hash" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6996",
"triggerID" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"triggerType" : "PUSH"
}, {
"hash" : "dc851f6226047629aec76f30cd9843ac635a66d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7001",
"triggerID" : "dc851f6226047629aec76f30cd9843ac635a66d9",
"triggerType" : "PUSH"
}, {
"hash" : "8c51a6c54679f5723053ae3d9efe343bdeea9445",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7041",
"triggerID" : "8c51a6c54679f5723053ae3d9efe343bdeea9445",
"triggerType" : "PUSH"
}, {
"hash" : "723b2bdc505e58d9a7525be996402ce55d2b9bff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7123",
"triggerID" : "723b2bdc505e58d9a7525be996402ce55d2b9bff",
"triggerType" : "PUSH"
}, {
"hash" : "c7902b06721cfb4e51d166b5799c20d58f079404",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7125",
"triggerID" : "c7902b06721cfb4e51d166b5799c20d58f079404",
"triggerType" : "PUSH"
}, {
"hash" : "5e3962074d7b4f7dceaedf9a391e763e054f9b9d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7139",
"triggerID" : "5e3962074d7b4f7dceaedf9a391e763e054f9b9d",
"triggerType" : "PUSH"
}, {
"hash" : "371f9b89c48b3dc9a9b95b15266b9f2fd4201a56",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7165",
"triggerID" : "371f9b89c48b3dc9a9b95b15266b9f2fd4201a56",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e19abadae9dece1aa78868839605a3d23a8b29fd UNKNOWN
* 4ba45f607a07f3f036e164dd68532fbc8b4bc3b8 UNKNOWN
* 5e3962074d7b4f7dceaedf9a391e763e054f9b9d Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7139)
* 371f9b89c48b3dc9a9b95b15266b9f2fd4201a56 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7165)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3117576868
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
}, {
"hash" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"triggerType" : "PUSH"
}, {
"hash" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947",
"triggerID" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"triggerType" : "PUSH"
}, {
"hash" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6949",
"triggerID" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"triggerType" : "PUSH"
}, {
"hash" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"triggerType" : "PUSH"
}, {
"hash" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6950",
"triggerID" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"triggerType" : "PUSH"
}, {
"hash" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6974",
"triggerID" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"triggerType" : "PUSH"
}, {
"hash" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6991",
"triggerID" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"triggerType" : "PUSH"
}, {
"hash" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6993",
"triggerID" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"triggerType" : "PUSH"
}, {
"hash" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6995",
"triggerID" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"triggerType" : "PUSH"
}, {
"hash" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6996",
"triggerID" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"triggerType" : "PUSH"
}, {
"hash" : "dc851f6226047629aec76f30cd9843ac635a66d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7001",
"triggerID" : "dc851f6226047629aec76f30cd9843ac635a66d9",
"triggerType" : "PUSH"
}, {
"hash" : "8c51a6c54679f5723053ae3d9efe343bdeea9445",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7041",
"triggerID" : "8c51a6c54679f5723053ae3d9efe343bdeea9445",
"triggerType" : "PUSH"
}, {
"hash" : "723b2bdc505e58d9a7525be996402ce55d2b9bff",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7123",
"triggerID" : "723b2bdc505e58d9a7525be996402ce55d2b9bff",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e19abadae9dece1aa78868839605a3d23a8b29fd UNKNOWN
* 4ba45f607a07f3f036e164dd68532fbc8b4bc3b8 UNKNOWN
* 8c51a6c54679f5723053ae3d9efe343bdeea9445 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7041)
* 723b2bdc505e58d9a7525be996402ce55d2b9bff Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7123)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2230929092
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestCOWDataSource.scala:
##########
@@ -142,6 +141,48 @@ class TestCOWDataSource extends HoodieSparkClientTestBase with ScalaAssertionSup
spark.read.format("org.apache.hudi").options(readOpts).load(basePath).count()
}
+ @Test
+ def testMultipleOrderingFields() {
+ val (writeOpts, readOpts) = getWriterReaderOpts(HoodieRecordType.AVRO)
+
+ // Insert Operation
+ var records = recordsToStrings(dataGen.generateInserts("002", 100)).asScala.toList
+ var inputDF = spark.read.json(spark.sparkContext.parallelize(records, 2))
+
+ val commonOptsWithMultipleOrderingFields = writeOpts ++ Map(
+ "hoodie.insert.shuffle.parallelism" -> "4",
+ "hoodie.upsert.shuffle.parallelism" -> "4",
+ DataSourceWriteOptions.RECORDKEY_FIELD.key -> "_row_key",
+ DataSourceWriteOptions.PARTITIONPATH_FIELD.key -> "partition",
+ DataSourceWriteOptions.PRECOMBINE_FIELD.key() -> "timestamp,rider",
Review Comment:
Added a test case for it.
##########
hudi-hadoop-common/src/test/java/org/apache/hudi/common/table/log/block/TestHoodieDeleteBlock.java:
##########
@@ -100,7 +100,7 @@ public static Stream<Arguments> orderingValueParams() {
{new String[] {"val1", "val2", "val3", null}},
{new Timestamp[] {new Timestamp(1690766971000L), new Timestamp(1672536571000L)}},
// {new LocalDate[] {LocalDate.of(2023, 1, 1), LocalDate.of(1980, 7, 1)}} // HUDI-8854
- {new BigDecimal[] {new BigDecimal("12345678901234.2948"), new BigDecimal("23456789012345.4856")}}
+ {new BigDecimal[] {new BigDecimal("12345678901234.294800000000000"), new BigDecimal("23456789012345.485600000000000")}}
Review Comment:
These were probably required due to avro schema changes. Not required anymore. Removed them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3106866630
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
}, {
"hash" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"triggerType" : "PUSH"
}, {
"hash" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947",
"triggerID" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"triggerType" : "PUSH"
}, {
"hash" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6949",
"triggerID" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"triggerType" : "PUSH"
}, {
"hash" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"triggerType" : "PUSH"
}, {
"hash" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6950",
"triggerID" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"triggerType" : "PUSH"
}, {
"hash" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6974",
"triggerID" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"triggerType" : "PUSH"
}, {
"hash" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6991",
"triggerID" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"triggerType" : "PUSH"
}, {
"hash" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6993",
"triggerID" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"triggerType" : "PUSH"
}, {
"hash" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e19abadae9dece1aa78868839605a3d23a8b29fd UNKNOWN
* 4ba45f607a07f3f036e164dd68532fbc8b4bc3b8 UNKNOWN
* a5ef18cb6ffcfb3a347018a6908bd3a8f0334712 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6993)
* 63af22477ed62d797f70568430ce6f142ffc3cc0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3107971912
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
}, {
"hash" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"triggerType" : "PUSH"
}, {
"hash" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947",
"triggerID" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"triggerType" : "PUSH"
}, {
"hash" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6949",
"triggerID" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"triggerType" : "PUSH"
}, {
"hash" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"triggerType" : "PUSH"
}, {
"hash" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6950",
"triggerID" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"triggerType" : "PUSH"
}, {
"hash" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6974",
"triggerID" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"triggerType" : "PUSH"
}, {
"hash" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6991",
"triggerID" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"triggerType" : "PUSH"
}, {
"hash" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6993",
"triggerID" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"triggerType" : "PUSH"
}, {
"hash" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6995",
"triggerID" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"triggerType" : "PUSH"
}, {
"hash" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6996",
"triggerID" : "2dba2b4f6c8d57fea9ff262e842f45153d56787f",
"triggerType" : "PUSH"
}, {
"hash" : "dc851f6226047629aec76f30cd9843ac635a66d9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "dc851f6226047629aec76f30cd9843ac635a66d9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e19abadae9dece1aa78868839605a3d23a8b29fd UNKNOWN
* 4ba45f607a07f3f036e164dd68532fbc8b4bc3b8 UNKNOWN
* 2dba2b4f6c8d57fea9ff262e842f45153d56787f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6996)
* dc851f6226047629aec76f30cd9843ac635a66d9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2225487839
##########
hudi-common/src/main/avro/HoodieDeleteRecordList.avsc:
##########
@@ -54,17 +54,23 @@
"doc": "Ordering value determining the order of merging on the same key",
"type": [
"null",
- "org.apache.hudi.avro.model.BooleanWrapper",
- "org.apache.hudi.avro.model.IntWrapper",
- "org.apache.hudi.avro.model.LongWrapper",
- "org.apache.hudi.avro.model.FloatWrapper",
- "org.apache.hudi.avro.model.DoubleWrapper",
- "org.apache.hudi.avro.model.BytesWrapper",
- "org.apache.hudi.avro.model.StringWrapper",
- "org.apache.hudi.avro.model.DateWrapper",
- "org.apache.hudi.avro.model.DecimalWrapper",
- "org.apache.hudi.avro.model.TimeMicrosWrapper",
- "org.apache.hudi.avro.model.TimestampMicrosWrapper"
+ {
+ "type": "array",
Review Comment:
Addressed
##########
hudi-common/src/main/java/org/apache/hudi/common/engine/HoodieReaderContext.java:
##########
@@ -380,21 +380,23 @@ private BiFunction<T, Schema, String> virtualKeyExtractor(String[] recordKeyFiel
/**
* Gets the ordering value in particular type.
*
- * @param record An option of record.
- * @param schema The Avro schema of the record.
- * @param orderingFieldName name of the ordering field
+ * @param record An option of record.
+ * @param schema The Avro schema of the record.
+ * @param orderingFieldNames name of the ordering field
* @return The ordering value.
*/
public Comparable getOrderingValue(T record,
Schema schema,
- Option<String> orderingFieldName) {
- if (orderingFieldName.isEmpty()) {
- return DEFAULT_ORDERING_VALUE;
+ Option<List<String>> orderingFieldNames) {
Review Comment:
HoodieFileGroupReader uses Option for commit time ordering it seems.
```
this.orderingFieldNames = readerContext.getMergeMode() == RecordMergeMode.COMMIT_TIME_ORDERING
? Option.empty()
: ConfigUtils.getOrderingFields(props).map(Arrays::asList)
.or(() -> hoodieTableMetaClient.getTableConfig().getPreCombineFields());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3102048217
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
}, {
"hash" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"triggerType" : "PUSH"
}, {
"hash" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947",
"triggerID" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bf66a590541998a22aa83fe644c03ff5d7eb2b47 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944)
* e19abadae9dece1aa78868839605a3d23a8b29fd UNKNOWN
* a20a7af499d5e0f6dc415ff74459d91ac88bf3d0 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2223825596
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -926,8 +926,12 @@ static RecordMergeMode inferRecordMergeModeFromMergeStrategyId(String recordMerg
}
}
- public String getPreCombineField() {
- return getString(PRECOMBINE_FIELD);
+ public Option<List<String>> getPreCombineFieldList() {
Review Comment:
I think we should swap the naming here. We want developers to be aware that there are potentially multiple strings as ordering fields so we should make this `getPrecombineFields` and the other method `getPrecombineFieldsAsString`
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -814,7 +814,7 @@ public String getRecordMergeStrategyId() {
public static Triple<RecordMergeMode, String, String> inferCorrectMergingBehavior(RecordMergeMode recordMergeMode,
String payloadClassName,
String recordMergeStrategyId,
- String orderingFieldName,
+ String orderingFieldNames,
Review Comment:
Should be a list?
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java:
##########
@@ -1153,8 +1153,8 @@ public TableBuilder setBaseFileFormat(String baseFileFormat) {
return this;
}
- public TableBuilder setPreCombineField(String preCombineField) {
- this.preCombineField = preCombineField;
+ public TableBuilder setPreCombineFields(String preCombineField) {
Review Comment:
Can this be a varargs input so the user can specify more than 1 field?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3102452201
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
}, {
"hash" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"triggerType" : "PUSH"
}, {
"hash" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947",
"triggerID" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"triggerType" : "PUSH"
}, {
"hash" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6949",
"triggerID" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e19abadae9dece1aa78868839605a3d23a8b29fd UNKNOWN
* a20a7af499d5e0f6dc415ff74459d91ac88bf3d0 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947)
* 9ce11db126c7a2124c41ee9c1b97e1820fd6af8e Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6949)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3102665806
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
}, {
"hash" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"triggerType" : "PUSH"
}, {
"hash" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947",
"triggerID" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"triggerType" : "PUSH"
}, {
"hash" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6949",
"triggerID" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"triggerType" : "PUSH"
}, {
"hash" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"triggerType" : "PUSH"
}, {
"hash" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6950",
"triggerID" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e19abadae9dece1aa78868839605a3d23a8b29fd UNKNOWN
* 9ce11db126c7a2124c41ee9c1b97e1820fd6af8e Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6949)
* 4ba45f607a07f3f036e164dd68532fbc8b4bc3b8 UNKNOWN
* a1f62b31b47715ce88932618c3817fbd7dd2e96e Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6950)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2241061275
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -169,8 +170,9 @@ public class HoodieWriteConfig extends HoodieConfig {
public static final ConfigProperty<String> PRECOMBINE_FIELD_NAME = ConfigProperty
.key("hoodie.datasource.write.precombine.field")
.noDefaultValue()
- .withDocumentation("Field used in preCombining before actual write. When two records have the same key value, "
- + "we will pick the one with the largest value for the precombine field, determined by Object.compareTo(..)");
+ .withDocumentation("Comma separated list of fields used in preCombining before actual write. When two records have the same key value, "
Review Comment:
@lokeshj1703 the config name should def. become plural now as per this PR. Also agree with @danny0405 to make it a table config (this will be consistent with record key fields and partition fields)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3049829990
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 26155cf4cb689ae854bc81e5c2c9c4ae8134ec85 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647)
* 8e74196c55e365e52dd5114c59ba96b2dfffb2ab UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3047712358
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 26a8660da376934b6f864b04f96e160d342886ca Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628)
* 26155cf4cb689ae854bc81e5c2c9c4ae8134ec85 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3051504628
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 125b574e0176d7ec688acd05f9138f821f6dfe03 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2231220444
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestHoodieFileGroupReaderBase.java:
##########
@@ -183,6 +187,48 @@ public void testReadFileGroupInMergeOnReadTable(RecordMergeMode recordMergeMode,
}
}
+ @Test
+ public void testReadFileGroupWithMultipleOrderingFields() throws Exception {
+ RecordMergeMode recordMergeMode = RecordMergeMode.EVENT_TIME_ORDERING;
+ Map<String, String> writeConfigs = new HashMap<>(getCommonConfigs(recordMergeMode, true));
+ writeConfigs.put(HoodieStorageConfig.LOGFILE_DATA_BLOCK_FORMAT.key(), "avro");
+ writeConfigs.put("hoodie.datasource.write.table.type", HoodieTableType.MERGE_ON_READ.name());
+ // Use two precombine values - combination of timestamp and rider
+ String orderingValues = "timestamp,rider";
+ writeConfigs.put("hoodie.datasource.write.precombine.field", orderingValues);
+ writeConfigs.put("hoodie.payload.ordering.field", orderingValues);
+
+ try (HoodieTestDataGenerator dataGen = new HoodieTestDataGenerator(0xDEEF)) {
+ // Initial commit. rider column gets value of rider-002
+ List<HoodieRecord> initialRecords = dataGen.generateInserts("002", 100);
+ commitToTable(initialRecords, INSERT.value(), true, writeConfigs);
+ validateOutputFromFileGroupReader(
+ getStorageConf(), getBasePath(), true, 0, recordMergeMode,
+ initialRecords, initialRecords);
+
+ // The updates have rider values as rider-001 and the existing records have rider values as rider-002
Review Comment:
If this case is already covered in the DataSource tests you added, please ignore this. I see there are unit tests on the comparison as well so I'm just looking for 1 functional test case for sanity that everything wires through correctly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2230928254
##########
hudi-common/src/main/java/org/apache/hudi/common/model/MultipleFieldsOrderingValue.java:
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.hudi.common.util.ValidationUtils;
+
+import java.io.Serializable;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Objects;
+import java.util.function.Function;
+
+/**
+ * A wrapper class to manage multiple fields ordering values across Hudi.
+ */
+public class MultipleFieldsOrderingValue implements OrderingValue<MultipleFieldsOrderingValue>, Serializable {
Review Comment:
Addressed
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/catalyst/catalog/HoodieCatalogTable.scala:
##########
@@ -127,9 +128,9 @@ class HoodieCatalogTable(val spark: SparkSession, var table: CatalogTable) exten
lazy val primaryKeys: Array[String] = tableConfig.getRecordKeyFields.orElse(Array.empty)
/**
- * PreCombine Field
+ * Comparables Field
*/
- lazy val preCombineKey: Option[String] = Option(tableConfig.getPreCombineField)
+ lazy val preCombineKeys: Option[String] = toScalaOption(tableConfig.getPreCombineFieldsAsString)
Review Comment:
Addressed
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/MergeOnReadSnapshotRelation.scala:
##########
@@ -92,7 +92,7 @@ abstract class BaseMergeOnReadSnapshotRelation(sqlContext: SQLContext,
* by the query), therefore saving on throughput
*/
protected lazy val mandatoryFieldsForMerging: Seq[String] =
- Seq(recordKeyField) ++ preCombineFieldOpt.map(Seq(_)).getOrElse(Seq())
+ Seq(recordKeyField) ++ preCombineFieldsOpt.orElse(List())
Review Comment:
Fixed as part of refactoring
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/HoodieStreamerUtils.java:
##########
@@ -111,8 +112,11 @@ public static Option<JavaRDD<HoodieRecord>> createHoodieRecords(HoodieStreamer.C
try {
HoodieKey hoodieKey = new HoodieKey(builtinKeyGenerator.getRecordKey(genRec), builtinKeyGenerator.getPartitionPath(genRec));
GenericRecord gr = isDropPartitionColumns(props) ? HoodieAvroUtils.removeFields(genRec, partitionColumns) : genRec;
- HoodieRecordPayload payload = shouldUseOrderingField ? DataSourceUtils.createPayload(payloadClassName, gr,
- (Comparable) HoodieAvroUtils.getNestedFieldVal(gr, cfg.sourceOrderingField, false, useConsistentLogicalTimestamp))
+ Comparable orderingValue = shouldUseOrderingField
+ ? OrderingValues.create(cfg.sourceOrderingFields.split(","),
+ field -> (Comparable) HoodieAvroUtils.getNestedFieldVal(gr, field, false, useConsistentLogicalTimestamp))
+ : null;
+ HoodieRecordPayload payload = shouldUseOrderingField ? DataSourceUtils.createPayload(payloadClassName, gr, orderingValue)
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2230929536
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java:
##########
@@ -1190,8 +1190,12 @@ public TableBuilder setBaseFileFormat(String baseFileFormat) {
return this;
}
- public TableBuilder setPreCombineField(String preCombineField) {
- this.preCombineField = preCombineField;
+ /**
+ * Sets preCombine fields to be used by the table
Review Comment:
Addressed
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -926,8 +927,12 @@ static RecordMergeMode inferRecordMergeModeFromMergeStrategyId(String recordMerg
}
}
- public String getPreCombineField() {
- return getString(PRECOMBINE_FIELD);
+ public Option<List<String>> getPreCombineFields() {
Review Comment:
Addressed
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/HoodieDatasetBulkInsertHelper.scala:
##########
@@ -255,6 +257,16 @@ object HoodieDatasetBulkInsertHelper
}
}
+ private def getOrderingValue(nestedFieldPaths: List[NestedFieldPath], r: InternalRow): Comparable[_] = {
+ if (nestedFieldPaths.isEmpty) {
+ EmptyOrderingValue.getInstance();
+ } else if (nestedFieldPaths.size == 1) {
+ getNestedInternalRowValue(r, nestedFieldPaths.head).asInstanceOf[Comparable[_]]
+ } else {
+ new MultipleFieldsOrderingValue(nestedFieldPaths.map(_ => getNestedInternalRowValue(r, nestedFieldPaths.head).asInstanceOf[Comparable[_]]).toArray)
Review Comment:
Addressed
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/HoodieDatasetBulkInsertHelper.scala:
##########
@@ -255,6 +257,16 @@ object HoodieDatasetBulkInsertHelper
}
}
+ private def getOrderingValue(nestedFieldPaths: List[NestedFieldPath], r: InternalRow): Comparable[_] = {
+ if (nestedFieldPaths.isEmpty) {
+ EmptyOrderingValue.getInstance();
+ } else if (nestedFieldPaths.size == 1) {
+ getNestedInternalRowValue(r, nestedFieldPaths.head).asInstanceOf[Comparable[_]]
+ } else {
+ new MultipleFieldsOrderingValue(nestedFieldPaths.map(_ => getNestedInternalRowValue(r, nestedFieldPaths.head).asInstanceOf[Comparable[_]]).toArray)
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3095722207
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 98a828795959a9cfc01d6c04ea8739228fa6009c Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906)
* fc0eba477035f595d72c7f0ce133a2d023df0a53 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3095698279
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c6a88f8bdbb390dbc6602bc55839fdec432d6b4e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904)
* 98a828795959a9cfc01d6c04ea8739228fa6009c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2218499070
##########
hudi-common/src/main/java/org/apache/hudi/common/engine/HoodieReaderContext.java:
##########
@@ -385,21 +386,24 @@ private BiFunction<T, Schema, String> virtualKeyExtractor(String[] recordKeyFiel
/**
* Gets the ordering value in particular type.
*
- * @param record An option of record.
- * @param schema The Avro schema of the record.
- * @param orderingFieldName name of the ordering field
+ * @param record An option of record.
+ * @param schema The Avro schema of the record.
+ * @param orderingFieldNames name of the ordering field
* @return The ordering value.
*/
public Comparable getOrderingValue(T record,
Schema schema,
- Option<String> orderingFieldName) {
- if (orderingFieldName.isEmpty()) {
- return DEFAULT_ORDERING_VALUE;
+ Option<List<String>> orderingFieldNames) {
+ if (orderingFieldNames.isEmpty()) {
+ return Comparables.getDefault();
}
- Object value = getValue(record, schema, orderingFieldName.get());
- Comparable finalOrderingVal = value != null ? convertValueToEngineType((Comparable) value) : DEFAULT_ORDERING_VALUE;
- return finalOrderingVal;
+ return new Comparables(
+ orderingFieldNames.get().stream().map(field -> {
+ Object value = getValue(record, schema, field);
+ // API getDefaultOrderingValue is only used inside Comparables constructor
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3101831848
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
}, {
"hash" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"triggerType" : "PUSH"
}, {
"hash" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bf66a590541998a22aa83fe644c03ff5d7eb2b47 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944)
* e19abadae9dece1aa78868839605a3d23a8b29fd UNKNOWN
* a20a7af499d5e0f6dc415ff74459d91ac88bf3d0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2221841060
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/HoodieDatasetBulkInsertHelper.scala:
##########
@@ -233,13 +233,14 @@ object HoodieDatasetBulkInsertHelper
(rowKey, row.copy())
}
.reduceByKey ((oneRow, otherRow) => {
- val onePreCombineVal = getNestedInternalRowValue(oneRow, preCombineFieldPath).asInstanceOf[Comparable[AnyRef]]
- val otherPreCombineVal = getNestedInternalRowValue(otherRow, preCombineFieldPath).asInstanceOf[Comparable[AnyRef]]
- if (onePreCombineVal.compareTo(otherPreCombineVal.asInstanceOf[AnyRef]) >= 0) {
+ val onePreCombineVals = new Comparables(JavaConverters.seqAsJavaList(preCombineFieldPaths.map(preCombineFieldPath => getNestedInternalRowValue(oneRow, preCombineFieldPath).asInstanceOf[Comparable[AnyRef]])))
Review Comment:
Addressed
##########
hudi-common/src/main/java/org/apache/hudi/Comparables.java:
##########
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi;
+
+import org.apache.hudi.common.util.ValidationUtils;
+
+import java.io.Serializable;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+import static org.apache.hudi.common.model.HoodieRecord.DEFAULT_ORDERING_VALUE;
+
+public class Comparables implements Comparable, Serializable {
+ protected static final long serialVersionUID = 1L;
+ private static final Comparables DEFAULT_VALUE = new Comparables(DEFAULT_ORDERING_VALUE);
+
+ private final List<Comparable> comparables;
+
+ public Comparables(List<Comparable> comparables) {
+ this.comparables = comparables;
+ }
+
+ public Comparables(Comparable comparable) {
+ this.comparables = Collections.singletonList(comparable);
+ }
+
+ public static Comparables getDefault() {
+ return DEFAULT_VALUE;
+ }
+
+ public static boolean isDefault(Comparable orderingVal) {
+ if (orderingVal instanceof Comparables) {
+ return ((Comparables) orderingVal).comparables.size() == 1
+ && ((Comparables) orderingVal).comparables.get(0).equals(DEFAULT_ORDERING_VALUE);
+ } else {
+ return orderingVal.equals(DEFAULT_ORDERING_VALUE);
+ }
+ }
+
+ /**
+ * Returns whether the given two comparable values come from the same runtime class.
+ */
+ public static boolean isSameClass(Comparable<?> v, Comparable<?> o) {
+ if (v.getClass() != o.getClass()) {
+ // If class is not same return false
+ return false;
+ } else if (v instanceof Comparables) {
+ if (((Comparables) v).comparables.size() != ((Comparables) o).comparables.size()) {
+ // if comparables size is not same return false
+ return false;
+ } else {
+ // compare class of comparable list of both arguments
+ return IntStream.range(0, ((Comparables) v).comparables.size())
+ .mapToObj(i -> ((Comparables) v).comparables.get(i).getClass() == ((Comparables) o).comparables.get(i).getClass())
+ .reduce(Boolean::logicalAnd)
+ .orElse(true);
+ }
+ }
+ // return true if class is same and input objects are not instance of Comparables class
+ return true;
+ }
+
+ public static Comparable getDefaultOrderingValue() {
+ return DEFAULT_ORDERING_VALUE;
+ }
+
+ @Override
+ public int compareTo(Object o) {
+ ValidationUtils.checkArgument(o instanceof Comparables, "Comparables can only be compared with another Comparables");
+ Comparables otherComparables = (Comparables) o;
+ ValidationUtils.checkArgument(comparables.size() == otherComparables.comparables.size(), "Comparables should be of same size");
+ for (int i = 0; i < comparables.size(); i++) {
+ int comparingValue = comparables.get(i).compareTo(otherComparables.comparables.get(i));
Review Comment:
Yes, the property `org.apache.hudi.config.HoodieWriteConfig#PRECOMBINE_FIELD_NAME` already calls it out
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2221842316
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestHoodieFileGroupReaderBase.java:
##########
@@ -604,13 +651,30 @@ protected void validateOutputFromFileGroupReader(StorageConfiguration<?> storage
List<HoodieRecord> expectedHoodieUnmergedRecords) throws Exception {
HoodieTableMetaClient metaClient = HoodieTestUtils.createMetaClient(storageConf, tablePath);
Schema avroSchema = new TableSchemaResolver(metaClient).getTableAvroSchema();
+ expectedHoodieRecords = getExpectedHoodieRecordsWithOrderingValue(expectedHoodieRecords, metaClient, avroSchema);
+ expectedHoodieUnmergedRecords = getExpectedHoodieRecordsWithOrderingValue(expectedHoodieUnmergedRecords, metaClient, avroSchema);
List<HoodieTestDataGenerator.RecordIdentifier> expectedRecords = convertHoodieRecords(expectedHoodieRecords, avroSchema);
List<HoodieTestDataGenerator.RecordIdentifier> expectedUnmergedRecords = convertHoodieRecords(expectedHoodieUnmergedRecords, avroSchema);
validateOutputFromFileGroupReaderWithExistingRecords(
storageConf, tablePath, containsBaseFile, expectedLogFileNum, recordMergeMode,
expectedRecords, expectedUnmergedRecords);
}
+ private static List<HoodieRecord> getExpectedHoodieRecordsWithOrderingValue(List<HoodieRecord> expectedHoodieRecords, HoodieTableMetaClient metaClient, Schema avroSchema) {
+ return expectedHoodieRecords.stream().map(rec -> {
+ RawTripTestPayload oldPayload = (RawTripTestPayload) rec.getData();
+ try {
+ List<String> orderingFields = metaClient.getTableConfig().getPreCombineFieldList().get();
+ HoodieAvroRecord avroRecord = ((HoodieAvroRecord) rec);
+ Comparable orderingValue = Comparables.create(orderingFields, field -> (Comparable) avroRecord.getColumnValueAsJava(avroSchema, field, new TypedProperties()));
+ RawTripTestPayload newPayload = new RawTripTestPayload(Option.ofNullable(oldPayload.getJsonData()), oldPayload.getRowKey(), oldPayload.getPartitionPath(), null, false, orderingValue);
Review Comment:
We can fix it in a separate jira. Multiple tests might fail with this kind of a fix.
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestHoodieFileGroupReaderBase.java:
##########
@@ -185,6 +189,49 @@ public void testReadFileGroupInMergeOnReadTable(RecordMergeMode recordMergeMode,
}
}
+ @ParameterizedTest
+ @ValueSource(strings = {"avro", "parquet"})
+ public void testReadFileGroupWithMultipleOrderingFields(String logDataBlockFormat) throws Exception {
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3106912131
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
}, {
"hash" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"triggerType" : "PUSH"
}, {
"hash" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947",
"triggerID" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"triggerType" : "PUSH"
}, {
"hash" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6949",
"triggerID" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"triggerType" : "PUSH"
}, {
"hash" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"triggerType" : "PUSH"
}, {
"hash" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6950",
"triggerID" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"triggerType" : "PUSH"
}, {
"hash" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6974",
"triggerID" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"triggerType" : "PUSH"
}, {
"hash" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6991",
"triggerID" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"triggerType" : "PUSH"
}, {
"hash" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6993",
"triggerID" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"triggerType" : "PUSH"
}, {
"hash" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6995",
"triggerID" : "63af22477ed62d797f70568430ce6f142ffc3cc0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e19abadae9dece1aa78868839605a3d23a8b29fd UNKNOWN
* 4ba45f607a07f3f036e164dd68532fbc8b4bc3b8 UNKNOWN
* a5ef18cb6ffcfb3a347018a6908bd3a8f0334712 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6993)
* 63af22477ed62d797f70568430ce6f142ffc3cc0 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6995)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2227070876
##########
hudi-common/src/main/java/org/apache/hudi/common/engine/HoodieReaderContext.java:
##########
@@ -479,6 +483,15 @@ public final UnaryOperator<T> projectRecord(Schema from, Schema to) {
return projectRecord(from, to, Collections.emptyMap());
}
+ /**
+ * Converts the ordering values to the specific engine type.
+ */
+ public final Comparable convertOrderingValueToEngineType(Comparable value) {
+ return value instanceof OrderingValue
Review Comment:
This is by design, the single field ordering value cover 90% use cases and I want to keep it simple and efficient.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3106275357
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
}, {
"hash" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"triggerType" : "PUSH"
}, {
"hash" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947",
"triggerID" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"triggerType" : "PUSH"
}, {
"hash" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6949",
"triggerID" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"triggerType" : "PUSH"
}, {
"hash" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"triggerType" : "PUSH"
}, {
"hash" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6950",
"triggerID" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"triggerType" : "PUSH"
}, {
"hash" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6974",
"triggerID" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"triggerType" : "PUSH"
}, {
"hash" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6991",
"triggerID" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"triggerType" : "PUSH"
}, {
"hash" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6993",
"triggerID" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e19abadae9dece1aa78868839605a3d23a8b29fd UNKNOWN
* 4ba45f607a07f3f036e164dd68532fbc8b4bc3b8 UNKNOWN
* a5ef18cb6ffcfb3a347018a6908bd3a8f0334712 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6993)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13526:
URL: https://github.com/apache/hudi/pull/13526#issuecomment-3106269857
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "26a8660da376934b6f864b04f96e160d342886ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6628",
"triggerID" : "26a8660da376934b6f864b04f96e160d342886ca",
"triggerType" : "PUSH"
}, {
"hash" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6647",
"triggerID" : "26155cf4cb689ae854bc81e5c2c9c4ae8134ec85",
"triggerType" : "PUSH"
}, {
"hash" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6652",
"triggerID" : "8e74196c55e365e52dd5114c59ba96b2dfffb2ab",
"triggerType" : "PUSH"
}, {
"hash" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6657",
"triggerID" : "125b574e0176d7ec688acd05f9138f821f6dfe03",
"triggerType" : "PUSH"
}, {
"hash" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6662",
"triggerID" : "c90d258b5f530a935ac34d36919454be963c1d5b",
"triggerType" : "PUSH"
}, {
"hash" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6671",
"triggerID" : "f4b6ff8db1543238f986e2d604f4f71971313b95",
"triggerType" : "PUSH"
}, {
"hash" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6904",
"triggerID" : "c6a88f8bdbb390dbc6602bc55839fdec432d6b4e",
"triggerType" : "PUSH"
}, {
"hash" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6906",
"triggerID" : "98a828795959a9cfc01d6c04ea8739228fa6009c",
"triggerType" : "PUSH"
}, {
"hash" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6907",
"triggerID" : "fc0eba477035f595d72c7f0ce133a2d023df0a53",
"triggerType" : "PUSH"
}, {
"hash" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6919",
"triggerID" : "64d9924c454d76f90eaa6f5089e5f36345bf8f6f",
"triggerType" : "PUSH"
}, {
"hash" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6944",
"triggerID" : "bf66a590541998a22aa83fe644c03ff5d7eb2b47",
"triggerType" : "PUSH"
}, {
"hash" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e19abadae9dece1aa78868839605a3d23a8b29fd",
"triggerType" : "PUSH"
}, {
"hash" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6947",
"triggerID" : "a20a7af499d5e0f6dc415ff74459d91ac88bf3d0",
"triggerType" : "PUSH"
}, {
"hash" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6949",
"triggerID" : "9ce11db126c7a2124c41ee9c1b97e1820fd6af8e",
"triggerType" : "PUSH"
}, {
"hash" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ba45f607a07f3f036e164dd68532fbc8b4bc3b8",
"triggerType" : "PUSH"
}, {
"hash" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6950",
"triggerID" : "a1f62b31b47715ce88932618c3817fbd7dd2e96e",
"triggerType" : "PUSH"
}, {
"hash" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6974",
"triggerID" : "21043f2bc87787d692ec55c2e4150ab29a396264",
"triggerType" : "PUSH"
}, {
"hash" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6991",
"triggerID" : "7c1ded9e7881ffea36adeceb34c89ea25353bc0d",
"triggerType" : "PUSH"
}, {
"hash" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6993",
"triggerID" : "a5ef18cb6ffcfb3a347018a6908bd3a8f0334712",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e19abadae9dece1aa78868839605a3d23a8b29fd UNKNOWN
* 4ba45f607a07f3f036e164dd68532fbc8b4bc3b8 UNKNOWN
* 7c1ded9e7881ffea36adeceb34c89ea25353bc0d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6991)
* a5ef18cb6ffcfb3a347018a6908bd3a8f0334712 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6993)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9569] Add support for multiple ordering fields [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13526:
URL: https://github.com/apache/hudi/pull/13526#discussion_r2225133216
##########
hudi-common/src/main/avro/HoodieDeleteRecordList.avsc:
##########
@@ -54,17 +54,23 @@
"doc": "Ordering value determining the order of merging on the same key",
"type": [
"null",
- "org.apache.hudi.avro.model.BooleanWrapper",
- "org.apache.hudi.avro.model.IntWrapper",
- "org.apache.hudi.avro.model.LongWrapper",
- "org.apache.hudi.avro.model.FloatWrapper",
- "org.apache.hudi.avro.model.DoubleWrapper",
- "org.apache.hudi.avro.model.BytesWrapper",
- "org.apache.hudi.avro.model.StringWrapper",
- "org.apache.hudi.avro.model.DateWrapper",
- "org.apache.hudi.avro.model.DecimalWrapper",
- "org.apache.hudi.avro.model.TimeMicrosWrapper",
- "org.apache.hudi.avro.model.TimestampMicrosWrapper"
+ {
+ "type": "array",
Review Comment:
Can we introduce a new wrapper class here: `org.apache.hudi.avro.model.ArrayWrapper` to hold an array of avro type records, it is more efficient and also making the deserailization compatible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org