[jira] [Commented] (KUDU-2293) tserver crashes with 'Found tablet in TABLET_DATA_COPYING state during StartTabletCopy()' message

["Mike Percy (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KUDU-2293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16357872#comment-16357872 ] 

Mike Percy commented on KUDU-2293:
----------------------------------

I think there are two problems that caused this issue:
 # In TabletCopyClient::Start(), we do not set the started_ flag until the end of the function, even though we may have started making changes on disk. That means a failure in the middle of Start() may never get cleaned up.
 # In TabletCopyClient::Abort(), we WARN_NOT_OK on several kinds of cleanup, including the cleanup to change from COPYING to TOMBSTONED, not to mention the fact that we may exit early due to a RETURN_NOT_OK in Abort()

Before we made disk errors non-fatal, this wasn't much of a big deal because generally we would simply crash if any of these conditions occurred. Now things are more complex and we need to be more careful about how we handle the in-memory tablet data state.

An easy fix would be to go through these two codepaths and crash whenever we hit any of these problems.

An ideal fix would involve discriminating between data-path errors and metadata-path errors, and only crashing when we have a metadata problem, while ensuring we unwind and tombstone ourselves if we run out of space on the data drive. This latter approach is much better because tombstoning ourselves actually helps address the disk-full problem.

> tserver crashes with 'Found tablet in TABLET_DATA_COPYING state during StartTabletCopy()' message
> -------------------------------------------------------------------------------------------------
>
>                 Key: KUDU-2293
>                 URL: https://issues.apache.org/jira/browse/KUDU-2293
>             Project: Kudu
>          Issue Type: Bug
>          Components: tserver
>    Affects Versions: 1.5.0, 1.6.0, 1.7.0
>            Reporter: Alexey Serbin
>            Priority: Major
>         Attachments: crash-at-tablet-copy-session-start.log
>
>
> When running out of disc space, tablet server can crash while trying to start tablet copying over already tombstoned replica.
> In essence, if {{DataDirManager::CreateDataDirGroup()}} returns an error due to the out-of-disc space condition while running {{TabletCopyClient::Start()}}, tablet server crashes with an error message like below.  The relevant part of the log is attached.
> {noformat}
> F0208 05:35:22.152496  2721 ts_tablet_manager.cc:563] T 5384471d823e46929029f9ff6ce212a3 P c713ac498df040caa897d3229214baa3: Tablet Copy: Found tablet in TABLET_DATA_COPYING state during StartTabletCopy(){noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)