You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@geode.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2020/09/29 16:05:00 UTC
[jira] [Commented] (GEODE-8200) Rebalance operations stuck in
"IN_PROGRESS" state forever
[ https://issues.apache.org/jira/browse/GEODE-8200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17204059#comment-17204059 ]
ASF subversion and git services commented on GEODE-8200:
--------------------------------------------------------
Commit 4721b164501bba82309041522172276ed3042f4a in geode's branch refs/heads/support/1.13 from Jianxia Chen
[ https://gitbox.apache.org/repos/asf?p=geode.git;h=4721b16 ]
GEODE-8200: Rebalance operations stuck in "IN_PROGRESS" state forever (#5350)
Record the locator that issues the original Rest API request. If the locator is offline afterwards, report an error.
Co-authored-by: Jianxia Chen <jc...@apache.org>
Co-authored-by: Jinmei Liao <ji...@pivotal.io>
(cherry picked from commit 426b9de66ae9adadde8f1994b2340fc9de809d81)
> Rebalance operations stuck in "IN_PROGRESS" state forever
> ---------------------------------------------------------
>
> Key: GEODE-8200
> URL: https://issues.apache.org/jira/browse/GEODE-8200
> Project: Geode
> Issue Type: Bug
> Components: management
> Reporter: Aaron Lindsey
> Assignee: Jianxia Chen
> Priority: Major
> Labels: GeodeOperationAPI
> Fix For: 1.14.0
>
> Attachments: GEODE-8200-exportedLogs.zip
>
>
> We use the management REST API to call rebalance immediately before stopping a server to limit the possibility of data loss. In a cluster with 3 locators, 3 servers, and no regions, we noticed that sometimes the rebalance operation never ends if one of the locators is restarting concurrently with the rebalance operation.
> More specifically, the scenario where we see this issue crop up is during an automated "rolling restart" operation in a Kubernetes environment which proceeds as follows:
> * At most one locator and one server are restarting at any point in time
> * Each locator/server waits until the previous locator/server is fully online before restarting
> * Immediately before stopping a server, a rebalance operation is performed and the server is not stopped until the rebalance operation is completed
> The impact of this issue is that the "rolling restart" operation will never complete, because it cannot proceed with stopping a server until the rebalance operation is completed. A human is then required to intervene and manually trigger a rebalance and stop the server. This type of "rolling restart" operation is triggered fairly often in Kubernetes — any time part of the configuration of the locators or servers changes.
> The following JSON is a sample response from the management REST API that shows the rebalance operation stuck in "IN_PROGRESS".
> {code}
> {
> "statusCode": "IN_PROGRESS",
> "links": {
> "self": "http://geodecluster-sample-locator.default/management/v1/operations/rebalances/a47f23c8-02b3-443c-a367-636fd6921ea7",
> "list": "http://geodecluster-sample-locator.default/management/v1/operations/rebalances"
> },
> "operationStart": "2020-05-27T22:38:30.619Z",
> "operationId": "a47f23c8-02b3-443c-a367-636fd6921ea7",
> "operation": {
> "simulate": false
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)