[jira] [Updated] (KUDU-1568) kudu-tserver cannot change config manually in some case.

["Todd Lipcon (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KUDU-1568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated KUDU-1568:
------------------------------
    Component/s: consensus

> kudu-tserver cannot change config manually in some case.
> --------------------------------------------------------
>
>                 Key: KUDU-1568
>                 URL: https://issues.apache.org/jira/browse/KUDU-1568
>             Project: Kudu
>          Issue Type: Bug
>          Components: consensus
>            Reporter: zhangsong
>
> Today, my cluster experience 4 node crash. 
> After restart kudu-server on this, i found a table is unable to accept write.
> After some search , i found  on both of followers node there are  messages :
> " T ed0d8b3a835e4c27afe695252ad0b8f5 P 3a77ef5039f447d29db5a44c92279a7a -> Peer 515ab1adcbd64081b646a86133f5f60d (follower_ip:7052): Couldn't send request to peer 515ab1adcbd64081b646a86133f5f60d for tablet ed0d8b3a835e4c27afe695252ad0b8f5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: FAILED: Corruption: Could not open LogReader. Reason: Unable to initialize log reader: Segment sequence numbers are not consecutive. Previous segment: seqno 0, path /export/servers/kudu/tserver_wal_data_7052/wals/ed0d8b3a835e4c27afe695252ad0b8f5.recovery/wal-000000018; Current segment: seqno 17, path /export/servers/kudu/tserver_wal_data_7052/wals/ed0d8b3a835e4c27afe695252ad0b8f5.recovery/wal-000000017. Retrying in the next heartbeat period. Already tried 26398 times."
> I want to use kudu-admin to add change the raft-config manually, but it didn't succeed :
> "Pending config: local: false peers { permanent_uuid: "515ab1adcbd64081b646a86133f5f60d" member_type: VOTER last_known_addr { host: "follower_ip" port: 7052 } } peers { permanent_uuid: "3a77ef5039f447d29db5a44c92279a7a" member_type: VOTER last_known_addr { host: "leader_ip" port: 7052 } }" 
> So i delete the tablet on this two followers but on master ui (using kudu-ts-cli), the raft config does not change ... and i could not add new voter to raft_config using kudu-admin .
> now i am facing a dangerous situation where only one leader is service the tablet...  (i create a new table and load the old table into the new one as work around)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)