You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Jordan Zimmerman <jz...@netflix.com> on 2012/05/24 22:22:51 UTC
New Instance can't sync
I'm trying to add a new instance to the ensemble and it is throwing while trying to sync. Any ideas?
2012-05-24 20:21:08,751 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@639] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2012-05-24 20:21:08,752 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1435] - Closed socket connection for client /**********:52931 (no session established for client)
2012-05-24 20:21:09,338 - WARN [QuorumPeer:/0.0.0.0:2181:Follower@82] - Exception when following the leader
java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at org.apache.zookeeper.server.quorum.Learner.writePacket(Learner.java:134)
at org.apache.zookeeper.server.quorum.Learner.ping(Learner.java:418)
at org.apache.zookeeper.server.quorum.Follower.processPacket(Follower.java:108)
at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:79)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:645)
2012-05-24 20:21:09,339 - INFO [QuorumPeer:/0.0.0.0:2181:Follower@165] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:165)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:649)
2012-05-24 20:21:09,340 - INFO [QuorumPeer:/0.0.0.0:2181:FinalRequestProcessor@378] - shutdown of request processor complete
2012-05-24 20:21:09,340 - INFO [FollowerRequestProcessor:7:FollowerRequestProcessor@93] - FollowerRequestProcessor exited loop!
2012-05-24 20:21:09,340 - INFO [CommitProcessor:7:CommitProcessor@148] - CommitProcessor exited loop!
2012-05-24 20:21:09,341 - INFO [SyncThread:7:SyncRequestProcessor@151] - SyncRequestProcessor exited!
2012-05-24 20:21:09,341 - INFO [QuorumPeer:/0.0.0.0:2181:QuorumPeer@621] - LOOKING
2012-05-24 20:21:09,343 - INFO [QuorumPeer:/0.0.0.0:2181:FileSnap@82] - Reading snapshot /mnt/data/zookeeper/version-2/snapshot.d0f357962
20
Re: New Instance can't sync
Posted by Jordan Zimmerman <jz...@netflix.com>.
I found the problem. I needed to increase the values for initLimit and
syncLimit.
-JZ
On 5/24/12 1:23 PM, "Jordan Zimmerman" <jz...@netflix.com> wrote:
>Later on, I get this exception:
>
>2012-05-24 20:23:03,800 - WARN [QuorumPeer:/0.0.0.0:2181:QuorumPeer@497]
>- Unable to load database
>java.io.IOException: Transaction log:
>/mnt/data/zookeeper/version-2/log.d0f357a0f has invalid magic number 0 !=
>1514884167
> at
>org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.inStrea
>mCreated(FileTxnLog.java:510)
> at
>org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.createI
>nputArchive(FileTxnLog.java:527)
> at
>org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.goToNex
>tLog(FileTxnLog.java:493)
> at
>org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.init(Fi
>leTxnLog.java:475)
> at
>org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.<init>(
>FileTxnLog.java:454)
> at
>org.apache.zookeeper.server.persistence.FileTxnLog.read(FileTxnLog.java:32
>5)
> at
>org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnap
>Log.java:126)
> at
>org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:222)
> at
>org.apache.zookeeper.server.quorum.QuorumPeer.getLastLoggedZxid(QuorumPeer
>.java:493)
> at
>org.apache.zookeeper.server.quorum.Learner.registerWithLeader(Learner.java
>:255)
> at
>org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:66)
> at
>org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:645)
>2012-05-24 20:23:03,800 - WARN [QuorumPeer:/0.0.0.0:2181:Follower@82] -
>Exception when following the leader
>java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:392)
> at
>org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
> at
>org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.j
>ava:84)
> at
>org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:108)
> at
>org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:148)
> at
>org.apache.zookeeper.server.quorum.Learner.registerWithLeader(Learner.java
>:267)
> at
>org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:66)
> at
>org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:645)
>2012-05-24 20:23:03,801 - INFO [QuorumPeer:/0.0.0.0:2181:Follower@165] -
>shutdown called
>java.lang.Exception: shutdown Follower
> at
>org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:165)
> at
>org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:649)
>2012-05-24 20:23:03,801 - INFO [QuorumPeer:/0.0.0.0:2181:QuorumPeer@621]
>- LOOKING
>2012-05-24 20:23:03,803 - INFO [QuorumPeer:/0.0.0.0:2181:FileSnap@82] -
>Reading snapshot /mnt/data/zookeeper/version-2/snapshot.d0f357a0e
>
>
>From: Netflix <jz...@netflix.com>>
>Date: Thu, 24 May 2012 13:22:51 -0700
>To: "user@zookeeper.apache.org<ma...@zookeeper.apache.org>"
><us...@zookeeper.apache.org>>
>Subject: New Instance can't sync
>
>I'm trying to add a new instance to the ensemble and it is throwing while
>trying to sync. Any ideas?
>
>2012-05-24 20:21:08,751 - WARN
>[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@639] - Exception
>causing close of session 0x0 due to java.io.IOException: ZooKeeperServer
>not running
>2012-05-24 20:21:08,752 - INFO
>[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1435] - Closed
>socket connection for client /**********:52931 (no session established
>for client)
>2012-05-24 20:21:09,338 - WARN [QuorumPeer:/0.0.0.0:2181:Follower@82] -
>Exception when following the leader
>java.net.SocketException: Broken pipe
> at java.net.SocketOutputStream.socketWrite0(Native Method)
> at
>java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
> at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
> at
>java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> at
>java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
> at
>org.apache.zookeeper.server.quorum.Learner.writePacket(Learner.java:134)
> at
>org.apache.zookeeper.server.quorum.Learner.ping(Learner.java:418)
> at
>org.apache.zookeeper.server.quorum.Follower.processPacket(Follower.java:10
>8)
> at
>org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:79)
> at
>org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:645)
>2012-05-24 20:21:09,339 - INFO [QuorumPeer:/0.0.0.0:2181:Follower@165] -
>shutdown called
>java.lang.Exception: shutdown Follower
> at
>org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:165)
> at
>org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:649)
>2012-05-24 20:21:09,340 - INFO
>[QuorumPeer:/0.0.0.0:2181:FinalRequestProcessor@378] - shutdown of
>request processor complete
>2012-05-24 20:21:09,340 - INFO
>[FollowerRequestProcessor:7:FollowerRequestProcessor@93] -
>FollowerRequestProcessor exited loop!
>2012-05-24 20:21:09,340 - INFO [CommitProcessor:7:CommitProcessor@148] -
>CommitProcessor exited loop!
>2012-05-24 20:21:09,341 - INFO [SyncThread:7:SyncRequestProcessor@151] -
>SyncRequestProcessor exited!
>2012-05-24 20:21:09,341 - INFO [QuorumPeer:/0.0.0.0:2181:QuorumPeer@621]
>- LOOKING
>2012-05-24 20:21:09,343 - INFO [QuorumPeer:/0.0.0.0:2181:FileSnap@82] -
>Reading snapshot /mnt/data/zookeeper/version-2/snapshot.d0f357962
>20
Re: New Instance can't sync
Posted by Jordan Zimmerman <jz...@netflix.com>.
Later on, I get this exception:
2012-05-24 20:23:03,800 - WARN [QuorumPeer:/0.0.0.0:2181:QuorumPeer@497] - Unable to load database
java.io.IOException: Transaction log: /mnt/data/zookeeper/version-2/log.d0f357a0f has invalid magic number 0 != 1514884167
at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.inStreamCreated(FileTxnLog.java:510)
at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.createInputArchive(FileTxnLog.java:527)
at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.goToNextLog(FileTxnLog.java:493)
at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.init(FileTxnLog.java:475)
at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.<init>(FileTxnLog.java:454)
at org.apache.zookeeper.server.persistence.FileTxnLog.read(FileTxnLog.java:325)
at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:126)
at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:222)
at org.apache.zookeeper.server.quorum.QuorumPeer.getLastLoggedZxid(QuorumPeer.java:493)
at org.apache.zookeeper.server.quorum.Learner.registerWithLeader(Learner.java:255)
at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:66)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:645)
2012-05-24 20:23:03,800 - WARN [QuorumPeer:/0.0.0.0:2181:Follower@82] - Exception when following the leader
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:84)
at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:108)
at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:148)
at org.apache.zookeeper.server.quorum.Learner.registerWithLeader(Learner.java:267)
at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:66)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:645)
2012-05-24 20:23:03,801 - INFO [QuorumPeer:/0.0.0.0:2181:Follower@165] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:165)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:649)
2012-05-24 20:23:03,801 - INFO [QuorumPeer:/0.0.0.0:2181:QuorumPeer@621] - LOOKING
2012-05-24 20:23:03,803 - INFO [QuorumPeer:/0.0.0.0:2181:FileSnap@82] - Reading snapshot /mnt/data/zookeeper/version-2/snapshot.d0f357a0e
From: Netflix <jz...@netflix.com>>
Date: Thu, 24 May 2012 13:22:51 -0700
To: "user@zookeeper.apache.org<ma...@zookeeper.apache.org>" <us...@zookeeper.apache.org>>
Subject: New Instance can't sync
I'm trying to add a new instance to the ensemble and it is throwing while trying to sync. Any ideas?
2012-05-24 20:21:08,751 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@639] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2012-05-24 20:21:08,752 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1435] - Closed socket connection for client /**********:52931 (no session established for client)
2012-05-24 20:21:09,338 - WARN [QuorumPeer:/0.0.0.0:2181:Follower@82] - Exception when following the leader
java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at org.apache.zookeeper.server.quorum.Learner.writePacket(Learner.java:134)
at org.apache.zookeeper.server.quorum.Learner.ping(Learner.java:418)
at org.apache.zookeeper.server.quorum.Follower.processPacket(Follower.java:108)
at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:79)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:645)
2012-05-24 20:21:09,339 - INFO [QuorumPeer:/0.0.0.0:2181:Follower@165] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:165)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:649)
2012-05-24 20:21:09,340 - INFO [QuorumPeer:/0.0.0.0:2181:FinalRequestProcessor@378] - shutdown of request processor complete
2012-05-24 20:21:09,340 - INFO [FollowerRequestProcessor:7:FollowerRequestProcessor@93] - FollowerRequestProcessor exited loop!
2012-05-24 20:21:09,340 - INFO [CommitProcessor:7:CommitProcessor@148] - CommitProcessor exited loop!
2012-05-24 20:21:09,341 - INFO [SyncThread:7:SyncRequestProcessor@151] - SyncRequestProcessor exited!
2012-05-24 20:21:09,341 - INFO [QuorumPeer:/0.0.0.0:2181:QuorumPeer@621] - LOOKING
2012-05-24 20:21:09,343 - INFO [QuorumPeer:/0.0.0.0:2181:FileSnap@82] - Reading snapshot /mnt/data/zookeeper/version-2/snapshot.d0f357962
20