[jira] [Assigned] (HUDI-9066) Auto upgrade and stale metaclient issue

["Davis Zhang (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HUDI-9066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Davis Zhang reassigned HUDI-9066:
---------------------------------

    Assignee: Y Ethan Guo

> Auto upgrade and stale metaclient issue
> ---------------------------------------
>
>                 Key: HUDI-9066
>                 URL: https://issues.apache.org/jira/browse/HUDI-9066
>             Project: Apache Hudi
>          Issue Type: Bug
>            Reporter: Davis Zhang
>            Assignee: Y Ethan Guo
>            Priority: Major
>
> for SIX to EIGHT upgrade, today we must do via write client, so either DS ingest/SQL will be responsible for that.
>  
> During test we found the corner case:
>  * Writer do the write and upgrade, table version is 8 with timeline layout 2
>  * writer continue to do something else, but the metaclient it holds is still a stale one using version 6 and layout 1
>  * So the subsequent logic will fail when parsing instants. Here the upgrade succeeds not gracefully.
>  * For DS it will auto retry and meta client is recreated with the new versions. For SQL writers / data frame writers they need to restart the session to refresh metaclient.
>  
> In general, the impact is limited. Given the impact, it might not be desirable to tangle auto upgrade and write operations unless we compensate with test coverage and extra design. I know this decision is made out of usability - user set auto upgrade flag and things will flow with no further action required.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)