You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ratis.apache.org by "Tsz-wo Sze (Jira)" <ji...@apache.org> on 2021/03/15 13:39:00 UTC

[jira] [Updated] (RATIS-1332) Ratis server couln't be recovered from failed initialization state

     [ https://issues.apache.org/jira/browse/RATIS-1332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz-wo Sze updated RATIS-1332:
------------------------------
    Fix Version/s: 2.0.0

> Ratis server couln't be recovered from failed initialization state
> ------------------------------------------------------------------
>
>                 Key: RATIS-1332
>                 URL: https://issues.apache.org/jira/browse/RATIS-1332
>             Project: Ratis
>          Issue Type: Bug
>          Components: server
>            Reporter: Marton Elek
>            Assignee: Janus Chow
>            Priority: Blocker
>             Fix For: 2.0.0
>
>          Time Spent: 1h
>  Remaining Estimate: 0h
>
> I found this problem during the test of ratis 2.0.0-rc3 and earlier.
> I noticed that in some cases the Ozone Manager (with ratis enabled true) couldn't be started any more (see HDDS-4703 for details).
> After some investigation I found the following problem:
>  1. Ratis server initialized BEFORE om RPC (OzoneManager.startRpcServer)
>  2. If the RPC server is failed (due to missing DNS for example) the Ratis server is stopped during the initialization
>  3. AtomicOutputStream can leave some tmp files behind (like raft-meta.tmp, if it's not yet renamed)
>  4. After DNS problem is fixed the OM couldn't be started anymore as RaftStorageImpl.analyzeAndRecoverStorage requires FORMATTED or empty (!!!) directory. Directory with leftover tmp file is not empty.
> {code}
>   private StorageState analyzeAndRecoverStorage(boolean toLock) throws IOException {
>     StorageState storageState = storageDir.analyzeStorage(toLock);
>     if (storageState == StorageState.NORMAL) {
>         // ...
>     } else if (storageState == StorageState.NOT_FORMATTED &&
>         storageDir.isCurrentEmpty()) {
>      //never called this if one .tmp file exists from the previous attempts
>       format();
>       return StorageState.NORMAL;
>     } else {
>       return storageState;
>     }
>   }
> {code}
> The problem is that `cleanMetaTmpFile();` is called only in the first branch, but before checking if the directory is empty or not...



--
This message was sent by Atlassian Jira
(v8.3.4#803005)