You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ratis.apache.org by GitBox <gi...@apache.org> on 2021/12/03 04:47:58 UTC
[GitHub] [ratis] ChenSammi commented on pull request #521: RATIS-1418. Leader step down for not sending out heartbeat promptly
ChenSammi commented on pull request #521:
URL: https://github.com/apache/ratis/pull/521#issuecomment-985213693
@szetszwo , I'm doing some stress test on a three DN Ozone cluster.
One test case is 2048 concurrent writes. It finished smoothly with this patch.
The second test case is 2048 concurrent writes(whole cluster) + 8192 reads(per DN). 8192 reads succeed with accpetable performance. 2048 writes all failed. The root cause is still leader step down. It seems this patch is not enough when cluster has a heavy read&write load. And extending election timeout from 5s to 10s doesn't have much effect.
FYI, following are some LOGs. I will do further investigation and try to find a better solution. Welcome any idea or suggestion.
2021-12-02 13:58:18,914 [java.util.concurrent.ThreadPoolExecutor$Worker@43627a29[State = -1, empty queue]] WARN org.apache.ratis.grpc.server.GrpcLogAppender: 98e5b27a-c3e9-4f86-ab85-b2caf84f012b@group-151DB3A92008->5a4a8be1-c921-4ca7-af7c-62a37a55cab7-GrpcLogAppender: HEARTBEAT appendEntries Timeout, request=AppendEntriesRequest:cid=1182148,entriesCount=0,lastEntry=null
2021-12-02 13:58:18,914 [java.util.concurrent.ThreadPoolExecutor$Worker@43627a29[State = -1, empty queue]] WARN org.apache.ratis.grpc.server.GrpcLogAppender: 98e5b27a-c3e9-4f86-ab85-b2caf84f012b@group-151DB3A92008->efdf0ed2-f836-4f4b-9dc8-981416d8a68d-GrpcLogAppender: appendEntries Timeout, request=AppendEntriesRequest:cid=872615,entriesCount=1,lastEntry=(t:1, i:139211)
2021-12-02 13:58:18,914 [java.util.concurrent.ThreadPoolExecutor$Worker@43627a29[State = -1, empty queue]] WARN org.apache.ratis.grpc.server.GrpcLogAppender: 98e5b27a-c3e9-4f86-ab85-b2caf84f012b@group-151DB3A92008->5a4a8be1-c921-4ca7-af7c-62a37a55cab7-GrpcLogAppender: HEARTBEAT appendEntries Timeout, request=AppendEntriesRequest:cid=1182149,entriesCount=0,lastEntry=null
2021-12-02 13:58:18,915 [java.util.concurrent.ThreadPoolExecutor$Worker@43627a29[State = -1, empty queue]] WARN org.apache.ratis.grpc.server.GrpcLogAppender: 98e5b27a-c3e9-4f86-ab85-b2caf84f012b@group-151DB3A92008->efdf0ed2-f836-4f4b-9dc8-981416d8a68d-GrpcLogAppender: appendEntries Timeout, request=AppendEntriesRequest:cid=872617,entriesCount=1,lastEntry=(t:1, i:139212)
2021-12-02 13:58:18,915 [java.util.concurrent.ThreadPoolExecutor$Worker@43627a29[State = -1, empty queue]] WARN org.apache.ratis.grpc.server.GrpcLogAppender: 98e5b27a-c3e9-4f86-ab85-b2caf84f012b@group-151DB3A92008->5a4a8be1-c921-4ca7-af7c-62a37a55cab7-GrpcLogAppender: HEARTBEAT appendEntries Timeout, request=AppendEntriesRequest:cid=1182150,entriesCount=0,lastEntry=null
2021-12-02 13:58:18,915 [java.util.concurrent.ThreadPoolExecutor$Worker@43627a29[State = -1, empty queue]] WARN org.apache.ratis.grpc.server.GrpcLogAppender: 98e5b27a-c3e9-4f86-ab85-b2caf84f012b@group-151DB3A92008->efdf0ed2-f836-4f4b-9dc8-981416d8a68d-GrpcLogAppender: appendEntries Timeout, request=AppendEntriesRequest:cid=872619,entriesCount=1,lastEntry=(t:1, i:139213)
2021-12-02 13:58:18,915 [java.util.concurrent.ThreadPoolExecutor$Worker@43627a29[State = -1, empty queue]] WARN org.apache.ratis.grpc.server.GrpcLogAppender: 98e5b27a-c3e9-4f86-ab85-b2caf84f012b@group-151DB3A92008->5a4a8be1-c921-4ca7-af7c-62a37a55cab7-GrpcLogAppender: HEARTBEAT appendEntries Timeout, request=AppendEntriesRequest:cid=1182151,entriesCount=0,lastEntry=null
2021-12-02 13:58:18,916 [java.util.concurrent.ThreadPoolExecutor$Worker@43627a29[State = -1, empty queue]] WARN org.apache.ratis.grpc.server.GrpcLogAppender: 98e5b27a-c3e9-4f86-ab85-b2caf84f012b@group-151DB3A92008->5a4a8be1-c921-4ca7-af7c-62a37a55cab7-GrpcLogAppender: HEARTBEAT appendEntries Timeout, request=AppendEntriesRequest:cid=1182152,entriesCount=0,lastEntry=null
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org