You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kudu.apache.org by "Will Berkeley (JIRA)" <ji...@apache.org> on 2018/11/06 21:08:00 UTC
[jira] [Created] (KUDU-2621) Correctly deal with replicas with
missing cmeta
Will Berkeley created KUDU-2621:
-----------------------------------
Summary: Correctly deal with replicas with missing cmeta
Key: KUDU-2621
URL: https://issues.apache.org/jira/browse/KUDU-2621
Project: Kudu
Issue Type: Bug
Affects Versions: 1.9.0
Reporter: Will Berkeley
Attachments: alter_table-randomized-test.0.txt
I saw a failure of alter_table-randomized-test where the following sequence of events occurred:
# The master creates a tablet X with 3 replicas A, B, C (in a 3 node cluster).
# The master sends CreateTablet RPCs to A, B, and C
# The tablet replicas are created successfully on A and B. C is delayed for an unknown reason.
# A becomes leader.
# C is restarted. The CreateTablet RPC to C is still in flight so the master logs an error that the CreateTablet failed[1].
# C restarts. While opening its tablets, it finds that the replica of X has valid tablet meta, but no consensus metadata[2]. Its consensus is shut down (and not really ever started) and the tablet is failed unrecoverably. The data state of the replica is TABLET_DATA_READY.
# The leader A sees that C has failed. It attempts to copy to C. The copies fail because C is in TABLET_DATA_READY state but has no running consensus[3].
# The situation stays stuck this way for 60 seconds, after which the test fails because ExternalMiniTabletServer::Restart has a 60 second timeout waiting for all tablets on C to enter RUNNING state.
[1]: {noformat}
I1102 21:39:21.178781 7768 alter_table-randomized-test.cc:120] Restarting TS 0
W1102 21:39:21.236687 9058 catalog_manager.cc:2922] TS c36abe6b2bdb4e778576473f60926b7b (127.7.150.1:46291): Create Tablet RPC failed for tablet 291265fb155c467da5ce61ad8116ce67: Network error: failed to read from TLS socket (remote: 127.7.150.1:46291): Connection reset by peer (error 104)
I1102 21:39:21.236747 9058 catalog_manager.cc:2958] Scheduling retry of CreateTablet RPC for tablet 291265fb155c467da5ce61ad8116ce67 on TS c36abe6b2bdb4e778576473f60926b7b with a delay of 55 ms (attempt = 1)
{noformat}
[2]: {noformat}
E1102 21:39:21.471261 10071 ts_tablet_manager.cc:1040] T 291265fb155c467da5ce61ad8116ce67 P c36abe6b2bdb4e778576473f60926b7b: Failed to load consensus metadata: Not found: Unable to load consensus metadata for tablet 291265fb155c467da5ce61ad8116ce67: /tmp/dist-test-taskoKLkQa/test-tmp/alter_table-randomized-test.0.HmsConfigurations_AlterTableRandomized.TestRandomSequence_0.1541194740983682-7768/minicluster-data/ts-0/wal/consensus-meta/291265fb155c467da5ce61ad8116ce67: No such file or directory (error 2)
{noformat}
[3]: {noformat}
W1102 21:39:21.878939 9204 consensus_peers.cc:435] T 291265fb155c467da5ce61ad8116ce67 P e92a2036de454238809663ac303c6ade -> Peer c36abe6b2bdb4e778576473f60926b7b (127.7.150.1:46291): Unable to start Tablet Copy on peer: error { code: UNKNOWN_ERROR status { code: ILLEGAL_STATE message: "consensus unavailable: tablet not running: 291265fb155c467da5ce61ad8116ce67" } }
{noformat}
See also https://github.com/apache/kudu/blob/master/src/kudu/tserver/ts_tablet_manager.cc#L615.
I also tested a slightly different circumstance where I started a 3-tserver cluster, created a table, shut down the cluster, deleted the cmeta for one replica, then restarted the cluster. In that case, I saw the master in a tight retry loop adding the failed replica as a non-voter, attempting and failing to tombstone it, and the evicting it:
{noformat}
...
I1106 11:14:39.171367 57196544 catalog_manager.cc:3354] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6333ca8d0f91457c823ebb7baacd8dd5 with cas_config_opid_index 249040: aborting the task: latest config opid_index 249041; task opid_index 249040
I1106 11:14:39.174340 66318336 catalog_manager.cc:3180] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 6333ca8d0f91457c823ebb7baacd8dd5 on 7266e9b18a6d409492b71266864a23a6 (127.0.0.1:7050) (TS 7266e9b18a6d409492b71266864a23a6 not found in new config with opid_index 249042)
I1106 11:14:39.174396 66318336 catalog_manager.cc:3406] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 6333ca8d0f91457c823ebb7baacd8dd5 (attempt 1)
W1106 11:14:39.174880 56659968 catalog_manager.cc:3150] TS 7266e9b18a6d409492b71266864a23a6 (127.0.0.1:7050): delete failed for tablet 6333ca8d0f91457c823ebb7baacd8dd5 with error code TABLET_NOT_RUNNING: Illegal state: Raft Consensus not available. Tablet shutting down
I1106 11:14:39.174914 56659968 catalog_manager.cc:2958] Scheduling retry of 6333ca8d0f91457c823ebb7baacd8dd5 Delete Tablet RPC for TS=7266e9b18a6d409492b71266864a23a6 with a delay of 49 ms (attempt = 1)
I1106 11:14:39.177963 57196544 catalog_manager.cc:3321] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6333ca8d0f91457c823ebb7baacd8dd5 with cas_config_opid_index 249042: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I1106 11:14:39.178491 66318336 catalog_manager.cc:3778] T 6333ca8d0f91457c823ebb7baacd8dd5 P 9b0e0c0f181e407095f5cbaaec6bbc1b reported cstate change: config changed from index 249042 to 249043, NON_VOTER 7266e9b18a6d409492b71266864a23a6 (127.0.0.1) added. New cstate: current_term: 20 leader_uuid: "7f75d089dd05446791f499bcdb9aa305" committed_config { opid_index: 249043 OBSOLETE_local: false peers { permanent_uuid: "9b0e0c0f181e407095f5cbaaec6bbc1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 7150 } } peers { permanent_uuid: "7f75d089dd05446791f499bcdb9aa305" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 7250 } } peers { permanent_uuid: "7266e9b18a6d409492b71266864a23a6" member_type: NON_VOTER last_known_addr { host: "127.0.0.1" port: 7050 } attrs { promote: true } } }
I1106 11:14:39.182490 56659968 catalog_manager.cc:3180] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 6333ca8d0f91457c823ebb7baacd8dd5 on 7266e9b18a6d409492b71266864a23a6 (127.0.0.1:7050) (TS 7266e9b18a6d409492b71266864a23a6 not found in new config with opid_index 249040)
W1106 11:14:39.182994 56659968 catalog_manager.cc:3150] TS 7266e9b18a6d409492b71266864a23a6 (127.0.0.1:7050): delete failed for tablet 6333ca8d0f91457c823ebb7baacd8dd5 with error code TABLET_NOT_RUNNING: Illegal state: Raft Consensus not available. Tablet shutting down
I1106 11:14:39.183027 56659968 catalog_manager.cc:2958] Scheduling retry of 6333ca8d0f91457c823ebb7baacd8dd5 Delete Tablet RPC for TS=7266e9b18a6d409492b71266864a23a6 with a delay of 77 ms (attempt = 2)
I1106 11:14:39.184643 65781760 catalog_manager.cc:3529] Sending ChangeConfig:REMOVE_PEER on tablet 6333ca8d0f91457c823ebb7baacd8dd5 (attempt 1)
I1106 11:14:39.187868 57196544 catalog_manager.cc:3321] ChangeConfig:REMOVE_PEER RPC for tablet 6333ca8d0f91457c823ebb7baacd8dd5 with cas_config_opid_index 249043: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I1106 11:14:39.188225 65781760 catalog_manager.cc:3778] T 6333ca8d0f91457c823ebb7baacd8dd5 P 7f75d089dd05446791f499bcdb9aa305 reported cstate change: config changed from index 249043 to 249044, NON_VOTER 7266e9b18a6d409492b71266864a23a6 (127.0.0.1) evicted. New cstate: current_term: 20 leader_uuid: "7f75d089dd05446791f499bcdb9aa305" committed_config { opid_index: 249044 OBSOLETE_local: false peers { permanent_uuid: "9b0e0c0f181e407095f5cbaaec6bbc1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 7150 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f75d089dd05446791f499bcdb9aa305" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 7250 } health_report { overall_health: HEALTHY } } }
I1106 11:14:39.193640 65781760 catalog_manager.cc:3180] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 6333ca8d0f91457c823ebb7baacd8dd5 on 7266e9b18a6d409492b71266864a23a6 (127.0.0.1:7050) (TS 7266e9b18a6d409492b71266864a23a6 not found in new config with opid_index 249044)
I1106 11:14:39.193725 65781760 catalog_manager.cc:3406] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 6333ca8d0f91457c823ebb7baacd8dd5 (attempt 1)
W1106 11:14:39.194171 56659968 catalog_manager.cc:3150] TS 7266e9b18a6d409492b71266864a23a6 (127.0.0.1:7050): delete failed for tablet 6333ca8d0f91457c823ebb7baacd8dd5 with error code TABLET_NOT_RUNNING: Illegal state: Raft Consensus not available. Tablet shutting down
I1106 11:14:39.194200 56659968 catalog_manager.cc:2958] Scheduling retry of 6333ca8d0f91457c823ebb7baacd8dd5 Delete Tablet RPC for TS=7266e9b18a6d409492b71266864a23a6 with a delay of 55 ms (attempt = 1)
I1106 11:14:39.194442 65781760 catalog_manager.cc:3406] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 6333ca8d0f91457c823ebb7baacd8dd5 (attempt 1)
I1106 11:14:39.194947 57196544 catalog_manager.cc:3338] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6333ca8d0f91457c823ebb7baacd8dd5 with cas_config_opid_index 249044: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 7f75d089dd05446791f499bcdb9aa305 (127.0.0.1:7250) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 249044 OBSOLETE_local: false peers { permanent_uuid: "9b0e0c0f181e407095f5cbaaec6bbc1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 7150 } } peers { permanent_uuid: "7f75d089dd05446791f499bcdb9aa305" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 7250 } }.
Pending config: opid_index: 249045 OBSOLETE_local: false peers { permanent_uuid: "9b0e0c0f181e407095f5cbaaec6bbc1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 7150 } } peers { permanent_uuid: "7f75d089dd05446791f499bcdb9aa305" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 7250 } } peers { permanent_uuid: "7266e9b18a6d409492b71266864a23a6" member_type: NON_VOTER last_known_addr { host: "127.0.0.1" port: 7050 } attrs { promote: true } }
I1106 11:14:39.195006 57196544 catalog_manager.cc:2958] Scheduling retry of ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6333ca8d0f91457c823ebb7baacd8dd5 with cas_config_opid_index 249044 with a delay of 17 ms (attempt = 1)
I1106 11:14:39.195271 57196544 catalog_manager.cc:3354] ChangeConfig:REMOVE_PEER RPC for tablet 6333ca8d0f91457c823ebb7baacd8dd5 with cas_config_opid_index 249039: aborting the task: latest config opid_index 249044; task opid_index 249039
I1106 11:14:39.197533 57196544 catalog_manager.cc:3321] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6333ca8d0f91457c823ebb7baacd8dd5 with cas_config_opid_index 249044: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
...
{noformat}
I'm not sure how the former case would've been different in a 4+ node cluster. The latter case would've been better: the tablet would've recovered by placing a replica on the 4th server. However, I think in both cases an uncopy-over-able, un-tombstone-able replica would have been left on the third server.
Perhaps the fix is to allow the creation of a dummy cmeta if copying over a TABLET_DATA_READY tablet, just like we do when copying over a TABLET_DATA_TOMBSTONED tablet?
Logs from the alter_table-randomized-test are attached.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)