You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@zookeeper.apache.org by "mutu (Jira)" <ji...@apache.org> on 2024/07/11 02:13:00 UTC
[jira] [Updated] (ZOOKEEPER-4844) Fail-slow disk while executing writeLongToFile can cause the follower to hang
[ https://issues.apache.org/jira/browse/ZOOKEEPER-4844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
mutu updated ZOOKEEPER-4844:
----------------------------
Component/s: server
Affects Version/s: 3.10.0
Description:
{*}Symptom:{*}If a thread is doing a file write and stuck in writeLongToFile, this thread will hang. This blocking shoud be handled by the zookeeper via PING. However, if the QuorumPeer executes the writeLongToFile and encounters a fail-slow disk, the entire follower can be stuck. The leader will abandon this follower, but the follower believes that it is a follower.
Callstack is as following:
{code:java}
at org.apache.zookeeper.common.AtomicFileOutputStream.write(AtomicFileOutputStream.java:72) at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221) at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291) at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:295) at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141) at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229) at java.io.BufferedWriter.flush(BufferedWriter.java:254) at org.apache.zookeeper.common.AtomicFileWritingIdiom.<init>(AtomicFileWritingIdiom.java:72) at org.apache.zookeeper.common.AtomicFileWritingIdiom.<init>(AtomicFileWritingIdiom.java:54) at org.apache.zookeeper.server.quorum.QuorumPeer.writeLongToFile(QuorumPeer.java:2233) at org.apache.zookeeper.server.quorum.QuorumPeer.setAcceptedEpoch(QuorumPeer.java:2262) at org.apache.zookeeper.server.quorum.Learner.registerWithLeader(Learner.java:510) at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:91) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1556) {code}
*Root cause:* The Quorum is blocked in writeLongToFile and can not execute readPacket, so no timeout exception is arised to trigger the error handler.
Summary: Fail-slow disk while executing writeLongToFile can cause the follower to hang (was: Fail-slow disk while Learner is executing writeLongToFile can cause the follower to hang)
> Fail-slow disk while executing writeLongToFile can cause the follower to hang
> -----------------------------------------------------------------------------
>
> Key: ZOOKEEPER-4844
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-4844
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
> Affects Versions: 3.10.0
> Reporter: mutu
> Priority: Major
>
> {*}Symptom:{*}If a thread is doing a file write and stuck in writeLongToFile, this thread will hang. This blocking shoud be handled by the zookeeper via PING. However, if the QuorumPeer executes the writeLongToFile and encounters a fail-slow disk, the entire follower can be stuck. The leader will abandon this follower, but the follower believes that it is a follower.
> Callstack is as following:
> {code:java}
> at org.apache.zookeeper.common.AtomicFileOutputStream.write(AtomicFileOutputStream.java:72) at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221) at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291) at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:295) at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141) at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229) at java.io.BufferedWriter.flush(BufferedWriter.java:254) at org.apache.zookeeper.common.AtomicFileWritingIdiom.<init>(AtomicFileWritingIdiom.java:72) at org.apache.zookeeper.common.AtomicFileWritingIdiom.<init>(AtomicFileWritingIdiom.java:54) at org.apache.zookeeper.server.quorum.QuorumPeer.writeLongToFile(QuorumPeer.java:2233) at org.apache.zookeeper.server.quorum.QuorumPeer.setAcceptedEpoch(QuorumPeer.java:2262) at org.apache.zookeeper.server.quorum.Learner.registerWithLeader(Learner.java:510) at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:91) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1556) {code}
> *Root cause:* The Quorum is blocked in writeLongToFile and can not execute readPacket, so no timeout exception is arised to trigger the error handler.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)