You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Daniel (Jira)" <ji...@apache.org> on 2022/03/14 07:31:00 UTC
[jira] [Created] (HDDS-6443) Datanode on K8s service is abnormal: UNAVAILABLE: io exception
Daniel created HDDS-6443:
----------------------------
Summary: Datanode on K8s service is abnormal: UNAVAILABLE: io exception
Key: HDDS-6443
URL: https://issues.apache.org/jira/browse/HDDS-6443
Project: Apache Ozone
Issue Type: Bug
Components: Ozone Datanode
Affects Versions: 1.2.1
Environment: The environment information is as follows:
1. Kubernetes: RK2 deployment K8s environment
Rke2 version v1.21.9 + rke2r1
2. Ozone version: 1.2.1
Reporter: Daniel
h2. 1. Test demo
Kubernetes Demo is provided in the following path:
Ozone - 1.2.1 / kubernetes/examples/ozone
h2. 2. Process
Use the following command to deploy Ozone:
kubectl apply-f .
Error messages are displayed on three DataNode nodes:
{code:java}
2022-03-14 07:07:46 INFO RaftServerConfigKeys:44 - raft.server.leaderelection.pre-vote = false (custom)
2022-03-14 07:07:46 INFO RoleInfo:130 - f99fd814-46cc-4192-b81d-1c98b5861cb7: start f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-LeaderElection29
2022-03-14 07:07:46 INFO LeaderElection:306 - f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-LeaderElection29 ELECTION round 0: submit vote requests at term 23 for 0: [e234f526-1428-46aa-b0d8-38c691c3be9b|rpc:10.42.10.15:9856|admin:10.42.10.15:9857|client:10.42.10.15:9858|dataStream:|priority:0, 8b4da263-b3d6-413e-8365-50f05ed24042|rpc:10.42.9.21:9856|admin:10.42.9.21:9857|client:10.42.9.21:9858|dataStream:|priority:1, f99fd814-46cc-4192-b81d-1c98b5861cb7|rpc:10.42.7.23:9856|admin:10.42.7.23:9857|client:10.42.7.23:9858|dataStream:|priority:0], old=null
2022-03-14 07:07:46 INFO LeaderElection:137 - f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-LeaderElection29 got exception when requesting votes: java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
2022-03-14 07:07:47 INFO LeaderElection:137 - f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-LeaderElection29 got exception when requesting votes: java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
2022-03-14 07:07:47 INFO LeaderElection:137 - f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC-LeaderElection28 got exception when requesting votes: java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
2022-03-14 07:07:47 INFO LeaderElection:89 - f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-LeaderElection29: ELECTION REJECTED received 0 response(s) and 2 exception(s):
2022-03-14 07:07:47 INFO LeaderElection:89 - f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC-LeaderElection28: ELECTION REJECTED received 0 response(s) and 2 exception(s):
2022-03-14 07:07:47 INFO LeaderElection:137 - Exception 0: java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
2022-03-14 07:07:47 INFO LeaderElection:137 - Exception 1: java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
2022-03-14 07:07:47 INFO LeaderElection:137 - Exception 0: java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
2022-03-14 07:07:47 INFO LeaderElection:137 - Exception 1: java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
2022-03-14 07:07:47 INFO LeaderElection:308 - f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-LeaderElection29 ELECTION round 0: result REJECTED
2022-03-14 07:07:47 INFO LeaderElection:308 - f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC-LeaderElection28 ELECTION round 0: result REJECTED
2022-03-14 07:07:47 INFO RaftServer$Division:290 - f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3: changes role from CANDIDATE to FOLLOWER at term 23 for REJECTED
2022-03-14 07:07:47 INFO RaftServer$Division:290 - f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC: changes role from CANDIDATE to FOLLOWER at term 20 for REJECTED
2022-03-14 07:07:47 INFO RoleInfo:121 - f99fd814-46cc-4192-b81d-1c98b5861cb7: shutdown f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-LeaderElection29
2022-03-14 07:07:47 INFO RoleInfo:121 - f99fd814-46cc-4192-b81d-1c98b5861cb7: shutdown f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC-LeaderElection28
2022-03-14 07:07:47 INFO RoleInfo:130 - f99fd814-46cc-4192-b81d-1c98b5861cb7: start f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-FollowerState
2022-03-14 07:07:47 INFO RoleInfo:130 - f99fd814-46cc-4192-b81d-1c98b5861cb7: start f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC-FollowerState
2022-03-14 07:07:52 INFO FollowerState:134 - f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC-FollowerState: change to CANDIDATE, lastRpcElapsedTime:5083479945ns, electionTimeout:5083ms
2022-03-14 07:07:52 INFO RoleInfo:108 - f99fd814-46cc-4192-b81d-1c98b5861cb7: shutdown f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC-FollowerState
2022-03-14 07:07:52 INFO RaftServer$Division:290 - f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC: changes role from FOLLOWER to CANDIDATE at term 20 for changeToCandidate
2022-03-14 07:07:52 INFO RaftServerConfigKeys:44 - raft.server.leaderelection.pre-vote = false (custom)
2022-03-14 07:07:52 INFO RoleInfo:130 - f99fd814-46cc-4192-b81d-1c98b5861cb7: start f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC-LeaderElection30
2022-03-14 07:07:52 INFO LeaderElection:306 - f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC-LeaderElection30 ELECTION round 0: submit vote requests at term 21 for 0: [e234f526-1428-46aa-b0d8-38c691c3be9b|rpc:10.42.10.15:9856|admin:10.42.10.15:9857|client:10.42.10.15:9858|dataStream:|priority:0, 8b4da263-b3d6-413e-8365-50f05ed24042|rpc:10.42.9.21:9856|admin:10.42.9.21:9857|client:10.42.9.21:9858|dataStream:|priority:0, f99fd814-46cc-4192-b81d-1c98b5861cb7|rpc:10.42.7.23:9856|admin:10.42.7.23:9857|client:10.42.7.23:9858|dataStream:|priority:1], old=null
2022-03-14 07:07:52 INFO LeaderElection:137 - f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC-LeaderElection30 got exception when requesting votes: java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
2022-03-14 07:07:52 INFO LeaderElection:137 - f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC-LeaderElection30 got exception when requesting votes: java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
2022-03-14 07:07:52 INFO LeaderElection:89 - f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC-LeaderElection30: ELECTION REJECTED received 0 response(s) and 2 exception(s):
2022-03-14 07:07:52 INFO LeaderElection:137 - Exception 0: java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
2022-03-14 07:07:52 INFO LeaderElection:137 - Exception 1: java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
2022-03-14 07:07:52 INFO LeaderElection:308 - f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC-LeaderElection30 ELECTION round 0: result REJECTED
2022-03-14 07:07:52 INFO RaftServer$Division:290 - f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC: changes role from CANDIDATE to FOLLOWER at term 21 for REJECTED
2022-03-14 07:07:52 INFO RoleInfo:121 - f99fd814-46cc-4192-b81d-1c98b5861cb7: shutdown f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC-LeaderElection30
2022-03-14 07:07:52 INFO RoleInfo:130 - f99fd814-46cc-4192-b81d-1c98b5861cb7: start f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC-FollowerState
2022-03-14 07:07:52 INFO FollowerState:134 - f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-FollowerState: change to CANDIDATE, lastRpcElapsedTime:5155386314ns, electionTimeout:5155ms
2022-03-14 07:07:52 INFO RoleInfo:108 - f99fd814-46cc-4192-b81d-1c98b5861cb7: shutdown f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-FollowerState
2022-03-14 07:07:52 INFO RaftServer$Division:290 - f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3: changes role from FOLLOWER to CANDIDATE at term 23 for changeToCandidate
2022-03-14 07:07:52 INFO RaftServerConfigKeys:44 - raft.server.leaderelection.pre-vote = false (custom)
2022-03-14 07:07:52 INFO RoleInfo:130 - f99fd814-46cc-4192-b81d-1c98b5861cb7: start f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-LeaderElection31
2022-03-14 07:07:52 INFO LeaderElection:306 - f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-LeaderElection31 ELECTION round 0: submit vote requests at term 24 for 0: [e234f526-1428-46aa-b0d8-38c691c3be9b|rpc:10.42.10.15:9856|admin:10.42.10.15:9857|client:10.42.10.15:9858|dataStream:|priority:0, 8b4da263-b3d6-413e-8365-50f05ed24042|rpc:10.42.9.21:9856|admin:10.42.9.21:9857|client:10.42.9.21:9858|dataStream:|priority:1, f99fd814-46cc-4192-b81d-1c98b5861cb7|rpc:10.42.7.23:9856|admin:10.42.7.23:9857|client:10.42.7.23:9858|dataStream:|priority:0], old=null
2022-03-14 07:07:52 INFO LeaderElection:137 - f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-LeaderElection31 got exception when requesting votes: java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
2022-03-14 07:07:52 INFO LeaderElection:137 - f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-LeaderElection31 got exception when requesting votes: java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
2022-03-14 07:07:52 INFO LeaderElection:89 - f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-LeaderElection31: ELECTION REJECTED received 0 response(s) and 2 exception(s):
2022-03-14 07:07:52 INFO LeaderElection:137 - Exception 0: java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
2022-03-14 07:07:52 INFO LeaderElection:137 - Exception 1: java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
2022-03-14 07:07:52 INFO LeaderElection:308 - f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-LeaderElection31 ELECTION round 0: result REJECTED
2022-03-14 07:07:52 INFO RaftServer$Division:290 - f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3: changes role from CANDIDATE to FOLLOWER at term 24 for REJECTED
2022-03-14 07:07:52 INFO RoleInfo:121 - f99fd814-46cc-4192-b81d-1c98b5861cb7: shutdown f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-LeaderElection31
2022-03-14 07:07:52 INFO RoleInfo:130 - f99fd814-46cc-4192-b81d-1c98b5861cb7: start f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-FollowerState
2022-03-14 07:07:57 INFO FollowerState:134 - f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC-FollowerState: change to CANDIDATE, lastRpcElapsedTime:5070286545ns, electionTimeout:5070ms
2022-03-14 07:07:57 INFO RoleInfo:108 - f99fd814-46cc-4192-b81d-1c98b5861cb7: shutdown f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC-FollowerState
2022-03-14 07:07:57 INFO RaftServer$Division:290 - f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC: changes role from FOLLOWER to CANDIDATE at term 21 for changeToCandidate
2022-03-14 07:07:57 INFO RaftServerConfigKeys:44 - raft.server.leaderelection.pre-vote = false (custom)
2022-03-14 07:07:57 INFO RoleInfo:130 - f99fd814-46cc-4192-b81d-1c98b5861cb7: start f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC-LeaderElection32
2022-03-14 07:07:57 INFO LeaderElection:306 - f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC-LeaderElection32 ELECTION round 0: submit vote requests at term 22 for 0: [e234f526-1428-46aa-b0d8-38c691c3be9b|rpc:10.42.10.15:9856|admin:10.42.10.15:9857|client:10.42.10.15:9858|dataStream:|priority:0, 8b4da263-b3d6-413e-8365-50f05ed24042|rpc:10.42.9.21:9856|admin:10.42.9.21:9857|client:10.42.9.21:9858|dataStream:|priority:0, f99fd814-46cc-4192-b81d-1c98b5861cb7|rpc:10.42.7.23:9856|admin:10.42.7.23:9857|client:10.42.7.23:9858|dataStream:|priority:1], old=null
2022-03-14 07:07:57 INFO LeaderElection:137 - f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC-LeaderElection32 got exception when requesting votes: java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
2022-03-14 07:07:57 INFO LeaderElection:137 - f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC-LeaderElection32 got exception when requesting votes: java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
2022-03-14 07:07:57 INFO LeaderElection:89 - f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC-LeaderElection32: ELECTION REJECTED received 0 response(s) and 2 exception(s):
2022-03-14 07:07:57 INFO LeaderElection:137 - Exception 0: java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
2022-03-14 07:07:57 INFO LeaderElection:137 - Exception 1: java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
2022-03-14 07:07:57 INFO LeaderElection:308 - f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC-LeaderElection32 ELECTION round 0: result REJECTED
2022-03-14 07:07:57 INFO RaftServer$Division:290 - f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC: changes role from CANDIDATE to FOLLOWER at term 22 for REJECTED
2022-03-14 07:07:57 INFO RoleInfo:121 - f99fd814-46cc-4192-b81d-1c98b5861cb7: shutdown f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC-LeaderElection32
2022-03-14 07:07:57 INFO RoleInfo:130 - f99fd814-46cc-4192-b81d-1c98b5861cb7: start f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC-FollowerState
2022-03-14 07:07:57 INFO FollowerState:134 - f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-FollowerState: change to CANDIDATE, lastRpcElapsedTime:5088311189ns, electionTimeout:5088ms
2022-03-14 07:07:57 INFO RoleInfo:108 - f99fd814-46cc-4192-b81d-1c98b5861cb7: shutdown f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-FollowerState
2022-03-14 07:07:57 INFO RaftServer$Division:290 - f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3: changes role from FOLLOWER to CANDIDATE at term 24 for changeToCandidate
2022-03-14 07:07:57 INFO RaftServerConfigKeys:44 - raft.server.leaderelection.pre-vote = false (custom)
2022-03-14 07:07:57 INFO RoleInfo:130 - f99fd814-46cc-4192-b81d-1c98b5861cb7: start f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-LeaderElection33
2022-03-14 07:07:57 INFO LeaderElection:306 - f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-LeaderElection33 ELECTION round 0: submit vote requests at term 25 for 0: [e234f526-1428-46aa-b0d8-38c691c3be9b|rpc:10.42.10.15:9856|admin:10.42.10.15:9857|client:10.42.10.15:9858|dataStream:|priority:0, 8b4da263-b3d6-413e-8365-50f05ed24042|rpc:10.42.9.21:9856|admin:10.42.9.21:9857|client:10.42.9.21:9858|dataStream:|priority:1, f99fd814-46cc-4192-b81d-1c98b5861cb7|rpc:10.42.7.23:9856|admin:10.42.7.23:9857|client:10.42.7.23:9858|dataStream:|priority:0], old=null
2022-03-14 07:07:57 INFO LeaderElection:137 - f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-LeaderElection33 got exception when requesting votes: java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
2022-03-14 07:07:57 INFO LeaderElection:137 - f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-LeaderElection33 got exception when requesting votes: java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
2022-03-14 07:07:57 INFO LeaderElection:89 - f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-LeaderElection33: ELECTION REJECTED received 0 response(s) and 2 exception(s):
2022-03-14 07:07:57 INFO LeaderElection:137 - Exception 0: java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
2022-03-14 07:07:57 INFO LeaderElection:137 - Exception 1: java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
2022-03-14 07:07:57 INFO LeaderElection:308 - f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-LeaderElection33 ELECTION round 0: result REJECTED
2022-03-14 07:07:57 INFO RaftServer$Division:290 - f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3: changes role from CANDIDATE to FOLLOWER at term 25 for REJECTED
2022-03-14 07:07:57 INFO RoleInfo:121 - f99fd814-46cc-4192-b81d-1c98b5861cb7: shutdown f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-LeaderElection33
2022-03-14 07:07:57 INFO RoleInfo:130 - f99fd814-46cc-4192-b81d-1c98b5861cb7: start f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-FollowerState
{code}
The logs for all three DataNode Pods are similar, and the logs are constantly refreshed.
h2. Validation functions
kubectl exec --stdin -n ozone --tty datanode-0 -- /bin/bash
bash-4.2$ ozone sh volume create volume1
bash-4.2$ ozone sh bucket create /volume1/bucket1
INTERNAL_ERROR Allocated 0 blocks. Requested 1 blocks
According to the error log information，we can see that the DataNode is abnormal.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org