You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 11:45:29 UTC
[I] Auto upgrade and stale metaclient issue [hudi]
hudi-bot opened a new issue, #16870:
URL: https://github.com/apache/hudi/issues/16870
for SIX to EIGHT upgrade, today we must do via write client, so either DS ingest/SQL will be responsible for that.
During test we found the corner case:
* Writer do the write and upgrade, table version is 8 with timeline layout 2
* writer continue to do something else, but the metaclient it holds is still a stale one using version 6 and layout 1
* So the subsequent logic will fail when parsing instants. Here the upgrade succeeds not gracefully.
* For DS it will auto retry and meta client is recreated with the new versions. For SQL writers / data frame writers they need to restart the session to refresh metaclient.
In general, the impact is limited. Given the impact, it might not be desirable to tangle auto upgrade and write operations unless we compensate with test coverage and extra design. I know this decision is made out of usability - user set auto upgrade flag and things will flow with no further action required.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-9066
- Type: Bug
- Fix version(s):
- 1.1.0
---
## Comments
21/Feb/25 22:59;yihua;The meta client must be re-instantiated after the table version upgrade. This issue is critical. Let's fix this.;;;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Auto upgrade and stale metaclient issue [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #16870:
URL: https://github.com/apache/hudi/issues/16870#issuecomment-4795644760
Triage: this looks **fixed on master**.
The stale-metaclient-after-upgrade scenario is addressed by two complementary changes:
1. **HUDI-8395 / [PR #12224](https://github.com/apache/hudi/pull/12224)** (commit `71079958a7bc`): inside `BaseHoodieWriteClient.tryUpgrade()`, after upgrade the write client's internal metaClient is refreshed via both `reloadTableConfig()` and `reloadActiveTimeline()` — see `BaseHoodieWriteClient.java:1583-1584`.
2. **Commit `928d603f8fa98`** (Sep 2025, in PR #13987): in `HoodieSparkSqlWriter.scala:522-524`, immediately after `client.startCommit(...)` (which triggers the upgrade), the externally-held `tableMetaClient` is now reloaded with `tableMetaClient.reloadTableConfig()` — comment in code reads *"if table has undergone upgrade, we need to reload table config"*. This directly covers the SQL / DataFrame writer path called out here.
Note that `reloadTableConfig()` also calls `reloadTimelineLayoutAndPath()` (`HoodieTableMetaClient.java:599`), which refreshes `timelineLayoutVersion`, `tableFormat`, and `timelinePath` — so the layout-1 → layout-2 transition you described is handled, not just the version code.
If you can confirm on a recent build, this issue can likely be closed. If you still hit a stale-metaclient case (e.g. structured-streaming or a long-lived client reusing a metaClient *across* a version transition outside `HoodieSparkSqlWriter`), please post a repro and we can re-open.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Auto upgrade and stale metaclient issue [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed issue #16870: Auto upgrade and stale metaclient issue
URL: https://github.com/apache/hudi/issues/16870
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org