You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "codope (via GitHub)" <gi...@apache.org> on 2025/03/24 07:38:06 UTC
[PR] [HUDI-9213] Handle record index field position while create RLI payload [hudi]
codope opened a new pull request, #13020:
URL: https://github.com/apache/hudi/pull/13020
### Change Logs
In RLI payload, the field `position` was added in 1.0 and its parsing fails when running compaction while downgrading or upgrading (with table version 6). This PR parses only when the field is present.
### Impact
Smooth upgrade/downgrade with RLI
### Risk level (write none, low medium or high below)
low
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none"._
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9213] Handle record index field position while create RLI payload [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on PR #13020:
URL: https://github.com/apache/hudi/pull/13020#issuecomment-2748812811
Closing in favor of #13022
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9213] Handle record index field position while create RLI payload [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13020:
URL: https://github.com/apache/hudi/pull/13020#issuecomment-2748702673
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "622b9578751762cb1c0df92887518aaf55e8fe0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4345",
"triggerID" : "622b9578751762cb1c0df92887518aaf55e8fe0d",
"triggerType" : "PUSH"
}, {
"hash" : "1bf87d407d33d26510a8f51dd5fb0271d04e8057",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4359",
"triggerID" : "1bf87d407d33d26510a8f51dd5fb0271d04e8057",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1bf87d407d33d26510a8f51dd5fb0271d04e8057 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4359)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9213] Handle record index field position while create RLI payload [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #13020:
URL: https://github.com/apache/hudi/pull/13020#discussion_r2010471353
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestMORDataSource.scala:
##########
@@ -426,6 +427,130 @@ class TestMORDataSource extends HoodieSparkClientTestBase with SparkDatasetMixin
}
}
+ @Test
+ def testUpgradeDowngradeWithRecordIndex() : Unit = {
Review Comment:
Moved the test and the fix to #13022
We can close this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9213] Handle record index field position while create RLI payload [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #13020:
URL: https://github.com/apache/hudi/pull/13020#discussion_r2010404507
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/SevenToEightUpgradeHandler.java:
##########
@@ -192,6 +192,14 @@ static void upgradeMergeMode(HoodieTableConfig tableConfig, Map<ConfigProperty,
HoodieTableConfig.RECORD_MERGE_MODE,
RecordMergeMode.EVENT_TIME_ORDERING.name());
}
+ } else if (tableConfig.getPayloadClass() != null
+ && tableConfig.getPayloadClass().equals(DefaultHoodieRecordPayload.class.getName())) {
Review Comment:
noted, i'll wait for 13009 to land first. I don't want to enforce a payload in the test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9213] Handle record index field position while create RLI payload [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13020:
URL: https://github.com/apache/hudi/pull/13020#issuecomment-2748360578
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "622b9578751762cb1c0df92887518aaf55e8fe0d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4345",
"triggerID" : "622b9578751762cb1c0df92887518aaf55e8fe0d",
"triggerType" : "PUSH"
}, {
"hash" : "1bf87d407d33d26510a8f51dd5fb0271d04e8057",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4359",
"triggerID" : "1bf87d407d33d26510a8f51dd5fb0271d04e8057",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 622b9578751762cb1c0df92887518aaf55e8fe0d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4345)
* 1bf87d407d33d26510a8f51dd5fb0271d04e8057 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4359)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9213] Handle record index field position while create RLI payload [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #13020:
URL: https://github.com/apache/hudi/pull/13020#discussion_r2010403125
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestMORDataSource.scala:
##########
@@ -426,6 +427,130 @@ class TestMORDataSource extends HoodieSparkClientTestBase with SparkDatasetMixin
}
}
+ @Test
+ def testUpgradeDowngradeWithRecordIndex() : Unit = {
Review Comment:
makes sense.. i was planning to do that after either of the patch lands first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9213] Handle record index field position while create RLI payload [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13020:
URL: https://github.com/apache/hudi/pull/13020#issuecomment-2747381730
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "622b9578751762cb1c0df92887518aaf55e8fe0d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4345",
"triggerID" : "622b9578751762cb1c0df92887518aaf55e8fe0d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 622b9578751762cb1c0df92887518aaf55e8fe0d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4345)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9213] Handle record index field position while create RLI payload [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13020:
URL: https://github.com/apache/hudi/pull/13020#issuecomment-2747163511
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "622b9578751762cb1c0df92887518aaf55e8fe0d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "622b9578751762cb1c0df92887518aaf55e8fe0d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 622b9578751762cb1c0df92887518aaf55e8fe0d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9213] Handle record index field position while create RLI payload [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13020:
URL: https://github.com/apache/hudi/pull/13020#issuecomment-2748352033
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "622b9578751762cb1c0df92887518aaf55e8fe0d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4345",
"triggerID" : "622b9578751762cb1c0df92887518aaf55e8fe0d",
"triggerType" : "PUSH"
}, {
"hash" : "1bf87d407d33d26510a8f51dd5fb0271d04e8057",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1bf87d407d33d26510a8f51dd5fb0271d04e8057",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 622b9578751762cb1c0df92887518aaf55e8fe0d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4345)
* 1bf87d407d33d26510a8f51dd5fb0271d04e8057 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9213] Handle record index field position while create RLI payload [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope closed pull request #13020: [HUDI-9213] Handle record index field position while create RLI payload
URL: https://github.com/apache/hudi/pull/13020
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9213] Handle record index field position while create RLI payload [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13020:
URL: https://github.com/apache/hudi/pull/13020#discussion_r2010392608
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/SevenToEightUpgradeHandler.java:
##########
@@ -192,6 +192,14 @@ static void upgradeMergeMode(HoodieTableConfig tableConfig, Map<ConfigProperty,
HoodieTableConfig.RECORD_MERGE_MODE,
RecordMergeMode.EVENT_TIME_ORDERING.name());
}
+ } else if (tableConfig.getPayloadClass() != null
+ && tableConfig.getPayloadClass().equals(DefaultHoodieRecordPayload.class.getName())) {
Review Comment:
@linliu-code already has a fix towards this. https://github.com/apache/hudi/pull/13009
can we remove those changes in this patch.
in tests, we can use DefaultHoodieRecordPayload for now to avoid hitting this
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestMORDataSource.scala:
##########
@@ -426,6 +427,130 @@ class TestMORDataSource extends HoodieSparkClientTestBase with SparkDatasetMixin
}
}
+ @Test
+ def testUpgradeDowngradeWithRecordIndex() : Unit = {
Review Comment:
we can just have 1 patch for this and mdt compaction fix. and have just 1 test for both cases. looking to avoid adding more tests, if we could achieve in a single test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9213] Handle record index field position while create RLI payload [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13020:
URL: https://github.com/apache/hudi/pull/13020#issuecomment-2747168954
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "622b9578751762cb1c0df92887518aaf55e8fe0d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4345",
"triggerID" : "622b9578751762cb1c0df92887518aaf55e8fe0d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 622b9578751762cb1c0df92887518aaf55e8fe0d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4345)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org