You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "rahil-c (via GitHub)" <gi...@apache.org> on 2025/07/29 07:06:29 UTC
[PR] Handle autoUpgrade is disabled case in Upgrade/Downgrade command [hudi]
rahil-c opened a new pull request, #13644:
URL: https://github.com/apache/hudi/pull/13644
### Change Logs
This pr aims to address the case for certain upgrade handlers when the following config `AUTO_UPGRADE_VERSION` is disabled. Based on my understanding we can avoid running the upgrade process all together if we do this check earlier within the `run` command.
### Risk level (write none, low medium or high below)
low
### Documentation Update
No doc update needed for this change.
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Handle autoUpgrade is disabled case in Upgrade/Downgrade command [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13644:
URL: https://github.com/apache/hudi/pull/13644#issuecomment-3131495331
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "796091f35ce2d3324f005a33e1e6b55ef7c336db",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7292",
"triggerID" : "796091f35ce2d3324f005a33e1e6b55ef7c336db",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 796091f35ce2d3324f005a33e1e6b55ef7c336db Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7292)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Handle autoUpgrade is disabled case in Upgrade/Downgrade command [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13644:
URL: https://github.com/apache/hudi/pull/13644#issuecomment-3130998090
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "796091f35ce2d3324f005a33e1e6b55ef7c336db",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "796091f35ce2d3324f005a33e1e6b55ef7c336db",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 796091f35ce2d3324f005a33e1e6b55ef7c336db UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Handle autoUpgrade is disabled case in Upgrade/Downgrade command [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #13644:
URL: https://github.com/apache/hudi/pull/13644#discussion_r2243200730
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/UpgradeDowngrade.java:
##########
@@ -130,6 +130,17 @@ public boolean needsUpgrade(HoodieTableVersion toWriteVersion) {
* @param instantTime current instant time that should not be touched.
*/
public void run(HoodieTableVersion toVersion, String instantTime) {
Review Comment:
Let me audit the callers and get back to you on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Handle autoUpgrade is disabled case in Upgrade/Downgrade command [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on PR #13644:
URL: https://github.com/apache/hudi/pull/13644#issuecomment-3138251153
Closing this PR, as will consolidate the changes of this pr onto the existing: https://github.com/apache/hudi/pull/13642
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Handle autoUpgrade is disabled case in Upgrade/Downgrade command [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13644:
URL: https://github.com/apache/hudi/pull/13644#discussion_r2241033576
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/UpgradeDowngrade.java:
##########
@@ -275,4 +280,25 @@ protected Pair<Map<ConfigProperty, String>, List<ConfigProperty>> downgrade(Hood
throw new HoodieUpgradeDowngradeException(fromVersion.versionCode(), toVersion.versionCode(), false);
}
}
+
+ /**
+ * Handles the case when auto-upgrade is disabled for upgrade operations.
+ * Sets compatible writer version and determines if upgrade logic should be skipped.
+ *
+ * @param fromVersion current table version
+ * @param toVersion target table version
+ * @return true if upgrade logic should be skipped, false otherwise
+ */
+ private boolean handleAutoUpgradeDisabled(HoodieTableVersion fromVersion, HoodieTableVersion toVersion) {
+ if (!config.autoUpgrade() && fromVersion.versionCode() < toVersion.versionCode()) {
+ // Set appropriate writer version based on upgrade transition to maintain compatibility
+ if (fromVersion == HoodieTableVersion.SEVEN && toVersion == HoodieTableVersion.EIGHT) {
Review Comment:
`fromVersion` and `toVersion` can be two versions apart. So should this consider more cases, not only the cases where `fromVersion` and `toVersion` are one version apart?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/UpgradeDowngrade.java:
##########
@@ -130,6 +130,17 @@ public boolean needsUpgrade(HoodieTableVersion toWriteVersion) {
* @param instantTime current instant time that should not be touched.
*/
public void run(HoodieTableVersion toVersion, String instantTime) {
Review Comment:
Could you also audit the callers of the method and upgrade/downgrade process? I'm wondering if the `UpgradeDowngrade` should only be called if upgrade or downgrade is needed; and the caller should check auto upgrade flag. From what I read from the code:
- `BaseHoodieWriteClient#tryUpgrade` checks whether the table needs to be upgraded using `hoodie.write.table.version`. Also note that this method takes care of rollbacks already. So there is no need to do rollback within upgrade and downgrade handlers?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Handle autoUpgrade is disabled case in Upgrade/Downgrade command [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13644:
URL: https://github.com/apache/hudi/pull/13644#issuecomment-3131004570
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "796091f35ce2d3324f005a33e1e6b55ef7c336db",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7292",
"triggerID" : "796091f35ce2d3324f005a33e1e6b55ef7c336db",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 796091f35ce2d3324f005a33e1e6b55ef7c336db Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7292)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Handle autoUpgrade is disabled case in Upgrade/Downgrade command [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c closed pull request #13644: Handle autoUpgrade is disabled case in Upgrade/Downgrade command
URL: https://github.com/apache/hudi/pull/13644
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Handle autoUpgrade is disabled case in Upgrade/Downgrade command [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #13644:
URL: https://github.com/apache/hudi/pull/13644#discussion_r2243199176
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/UpgradeDowngrade.java:
##########
@@ -275,4 +280,25 @@ protected Pair<Map<ConfigProperty, String>, List<ConfigProperty>> downgrade(Hood
throw new HoodieUpgradeDowngradeException(fromVersion.versionCode(), toVersion.versionCode(), false);
}
}
+
+ /**
+ * Handles the case when auto-upgrade is disabled for upgrade operations.
+ * Sets compatible writer version and determines if upgrade logic should be skipped.
+ *
+ * @param fromVersion current table version
+ * @param toVersion target table version
+ * @return true if upgrade logic should be skipped, false otherwise
+ */
+ private boolean handleAutoUpgradeDisabled(HoodieTableVersion fromVersion, HoodieTableVersion toVersion) {
+ if (!config.autoUpgrade() && fromVersion.versionCode() < toVersion.versionCode()) {
+ // Set appropriate writer version based on upgrade transition to maintain compatibility
+ if (fromVersion == HoodieTableVersion.SEVEN && toVersion == HoodieTableVersion.EIGHT) {
Review Comment:
I think you might be correct, so you are saying the case the fromVersion is six or 7 and toVersion is 9. We would need to account for those differences here as well, if autoUpgrade is disabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Handle autoUpgrade is disabled case in Upgrade/Downgrade command [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on PR #13644:
URL: https://github.com/apache/hudi/pull/13644#issuecomment-3130995772
@nsivabalan @yihua if you can take a look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org