You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zookeeper.apache.org by Patrick Hunt <ph...@apache.org> on 2015/06/10 17:16:09 UTC
Interesting (very serious) server error reported on go-zookeeper
The report is pretty serious. Anyone can confirm? (and create a jira please)
https://github.com/samuel/go-zookeeper/issues/62
On Wed, Jun 10, 2015 at 7:59 AM, geo-turcsanyi <no...@github.com> wrote:
>
> When you try setting data on an empty path, it will crash and burn your zookeeper server. And by this I mean data corruption, you need to empty your data directory and then restart it, otherwise nobody will be able to connect to it any more and will get a connection refused.
>
> It would be nice to have an error if the path is empty, rather than letting the set happen and watching your zookeeper go down in style.
>
Re: Interesting (very serious) server error reported on go-zookeeper
Posted by Patrick Hunt <ph...@apache.org>.
Thanks Brian!
Patrick
On Wed, Jun 10, 2015 at 8:30 AM, Brian Brazil <br...@boxever.com> wrote:
> On 10 June 2015 at 16:27, Patrick Hunt <ph...@apache.org> wrote:
>
>> Yea, that's bad. We should cut a fix release for this. Can you create a
>> jira?
>>
>
> Already on it, https://issues.apache.org/jira/browse/ZOOKEEPER-2213
>
> Brian
>
>>
>> Likely our client (the ones we ship in zk itself) always fill in some
>> value rather than null (etc...) and the go client was doing things
>> differently.
>>
>> Patrick
>>
>> On Wed, Jun 10, 2015 at 8:23 AM, Brian Brazil <br...@boxever.com>
>> wrote:
>> > On 10 June 2015 at 16:16, Patrick Hunt <ph...@apache.org> wrote:
>> >
>> >> The report is pretty serious. Anyone can confirm? (and create a jira
>> >> please)
>> >>
>> >> https://github.com/samuel/go-zookeeper/issues/62
>> >
>> >
>> > I happen to be playing with that library right now, so I've attempted to
>> > reproduce it. The bug looks to be as stated on 3.4.5.
>> >
>> > The code:
>> > c, _, _ := zk.Connect([]string{"127.0.0.1"}, time.Second)
>> > c.Set("", []byte{}, 0)
>> >
>> > Causes on the zookeeper server:
>> >
>> > 2015-06-10 16:21:10,862 [myid:] - ERROR
>> > [SyncThread:0:SyncRequestProcessor@151] - Severe unrecoverable error,
>> > exiting
>> >
>> > java.lang.IllegalArgumentException: Invalid path
>> >
>> > at
>> > org.apache.zookeeper.common.PathTrie.findMaxPrefix(PathTrie.java:259)
>> >
>> > at
>> >
>> org.apache.zookeeper.server.DataTree.getMaxPrefixWithQuota(DataTree.java:634)
>> >
>> > at
>> org.apache.zookeeper.server.DataTree.setData(DataTree.java:616)
>> >
>> > at
>> > org.apache.zookeeper.server.DataTree.processTxn(DataTree.java:807)
>> >
>> > at
>> > org.apache.zookeeper.server.ZKDatabase.processTxn(ZKDatabase.java:329)
>> >
>> > at
>> >
>> org.apache.zookeeper.server.ZooKeeperServer.processTxn(ZooKeeperServer.java:965)
>> >
>> > at
>> >
>> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:116)
>> >
>> > at
>> >
>> org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:167)
>> >
>> > at
>> >
>> org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:101)
>> >
>> >
>> >
>> > Restarting the server produces:
>> >
>> > 2015-06-10 16:22:21,271 [myid:] - INFO [main:FileSnap@83] - Reading
>> > snapshot /tmp/zookeeper/version-2/snapshot.6
>> >
>> > 2015-06-10 16:22:21,352 [myid:] - ERROR [main:ZooKeeperServerMain@54] -
>> > Invalid arguments, exiting abnormally
>> >
>> > java.lang.IllegalArgumentException: Invalid path
>> >
>> > at
>> > org.apache.zookeeper.common.PathTrie.findMaxPrefix(PathTrie.java:259)
>> >
>> > at
>> >
>> org.apache.zookeeper.server.DataTree.getMaxPrefixWithQuota(DataTree.java:634)
>> >
>> > at
>> org.apache.zookeeper.server.DataTree.setData(DataTree.java:616)
>> >
>> > at
>> > org.apache.zookeeper.server.DataTree.processTxn(DataTree.java:807)
>> >
>> > at
>> >
>> org.apache.zookeeper.server.persistence.FileTxnSnapLog.processTransaction(FileTxnSnapLog.java:198)
>> >
>> > at
>> >
>> org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:151)
>> >
>> > at
>> > org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)
>> >
>> > at
>> >
>> org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:250)
>> >
>> > at
>> >
>> org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:377)
>> >
>> > at
>> >
>> org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:122)
>> >
>> > at
>> >
>> org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:112)
>> >
>> > at
>> >
>> org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:86)
>> >
>> > at
>> >
>> org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:52)
>> >
>> > at
>> >
>> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)
>> >
>> > at
>> >
>> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
>> >
>> >
>> > Brian
>> >
>> >
>> >>
>> >>
>> >> On Wed, Jun 10, 2015 at 7:59 AM, geo-turcsanyi <
>> notifications@github.com>
>> >> wrote:
>> >> >
>> >> > When you try setting data on an empty path, it will crash and burn
>> your
>> >> zookeeper server. And by this I mean data corruption, you need to empty
>> >> your data directory and then restart it, otherwise nobody will be able
>> to
>> >> connect to it any more and will get a connection refused.
>> >> >
>> >> > It would be nice to have an error if the path is empty, rather than
>> >> letting the set happen and watching your zookeeper go down in style.
>> >> >
>> >>
>>
Re: Interesting (very serious) server error reported on go-zookeeper
Posted by Brian Brazil <br...@boxever.com>.
On 10 June 2015 at 16:27, Patrick Hunt <ph...@apache.org> wrote:
> Yea, that's bad. We should cut a fix release for this. Can you create a
> jira?
>
Already on it, https://issues.apache.org/jira/browse/ZOOKEEPER-2213
Brian
>
> Likely our client (the ones we ship in zk itself) always fill in some
> value rather than null (etc...) and the go client was doing things
> differently.
>
> Patrick
>
> On Wed, Jun 10, 2015 at 8:23 AM, Brian Brazil <br...@boxever.com>
> wrote:
> > On 10 June 2015 at 16:16, Patrick Hunt <ph...@apache.org> wrote:
> >
> >> The report is pretty serious. Anyone can confirm? (and create a jira
> >> please)
> >>
> >> https://github.com/samuel/go-zookeeper/issues/62
> >
> >
> > I happen to be playing with that library right now, so I've attempted to
> > reproduce it. The bug looks to be as stated on 3.4.5.
> >
> > The code:
> > c, _, _ := zk.Connect([]string{"127.0.0.1"}, time.Second)
> > c.Set("", []byte{}, 0)
> >
> > Causes on the zookeeper server:
> >
> > 2015-06-10 16:21:10,862 [myid:] - ERROR
> > [SyncThread:0:SyncRequestProcessor@151] - Severe unrecoverable error,
> > exiting
> >
> > java.lang.IllegalArgumentException: Invalid path
> >
> > at
> > org.apache.zookeeper.common.PathTrie.findMaxPrefix(PathTrie.java:259)
> >
> > at
> >
> org.apache.zookeeper.server.DataTree.getMaxPrefixWithQuota(DataTree.java:634)
> >
> > at
> org.apache.zookeeper.server.DataTree.setData(DataTree.java:616)
> >
> > at
> > org.apache.zookeeper.server.DataTree.processTxn(DataTree.java:807)
> >
> > at
> > org.apache.zookeeper.server.ZKDatabase.processTxn(ZKDatabase.java:329)
> >
> > at
> >
> org.apache.zookeeper.server.ZooKeeperServer.processTxn(ZooKeeperServer.java:965)
> >
> > at
> >
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:116)
> >
> > at
> >
> org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:167)
> >
> > at
> >
> org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:101)
> >
> >
> >
> > Restarting the server produces:
> >
> > 2015-06-10 16:22:21,271 [myid:] - INFO [main:FileSnap@83] - Reading
> > snapshot /tmp/zookeeper/version-2/snapshot.6
> >
> > 2015-06-10 16:22:21,352 [myid:] - ERROR [main:ZooKeeperServerMain@54] -
> > Invalid arguments, exiting abnormally
> >
> > java.lang.IllegalArgumentException: Invalid path
> >
> > at
> > org.apache.zookeeper.common.PathTrie.findMaxPrefix(PathTrie.java:259)
> >
> > at
> >
> org.apache.zookeeper.server.DataTree.getMaxPrefixWithQuota(DataTree.java:634)
> >
> > at
> org.apache.zookeeper.server.DataTree.setData(DataTree.java:616)
> >
> > at
> > org.apache.zookeeper.server.DataTree.processTxn(DataTree.java:807)
> >
> > at
> >
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.processTransaction(FileTxnSnapLog.java:198)
> >
> > at
> >
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:151)
> >
> > at
> > org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)
> >
> > at
> >
> org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:250)
> >
> > at
> >
> org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:377)
> >
> > at
> >
> org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:122)
> >
> > at
> >
> org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:112)
> >
> > at
> >
> org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:86)
> >
> > at
> >
> org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:52)
> >
> > at
> >
> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)
> >
> > at
> >
> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
> >
> >
> > Brian
> >
> >
> >>
> >>
> >> On Wed, Jun 10, 2015 at 7:59 AM, geo-turcsanyi <
> notifications@github.com>
> >> wrote:
> >> >
> >> > When you try setting data on an empty path, it will crash and burn
> your
> >> zookeeper server. And by this I mean data corruption, you need to empty
> >> your data directory and then restart it, otherwise nobody will be able
> to
> >> connect to it any more and will get a connection refused.
> >> >
> >> > It would be nice to have an error if the path is empty, rather than
> >> letting the set happen and watching your zookeeper go down in style.
> >> >
> >>
>
Re: Interesting (very serious) server error reported on go-zookeeper
Posted by Patrick Hunt <ph...@apache.org>.
Yea, that's bad. We should cut a fix release for this. Can you create a jira?
Likely our client (the ones we ship in zk itself) always fill in some
value rather than null (etc...) and the go client was doing things
differently.
Patrick
On Wed, Jun 10, 2015 at 8:23 AM, Brian Brazil <br...@boxever.com> wrote:
> On 10 June 2015 at 16:16, Patrick Hunt <ph...@apache.org> wrote:
>
>> The report is pretty serious. Anyone can confirm? (and create a jira
>> please)
>>
>> https://github.com/samuel/go-zookeeper/issues/62
>
>
> I happen to be playing with that library right now, so I've attempted to
> reproduce it. The bug looks to be as stated on 3.4.5.
>
> The code:
> c, _, _ := zk.Connect([]string{"127.0.0.1"}, time.Second)
> c.Set("", []byte{}, 0)
>
> Causes on the zookeeper server:
>
> 2015-06-10 16:21:10,862 [myid:] - ERROR
> [SyncThread:0:SyncRequestProcessor@151] - Severe unrecoverable error,
> exiting
>
> java.lang.IllegalArgumentException: Invalid path
>
> at
> org.apache.zookeeper.common.PathTrie.findMaxPrefix(PathTrie.java:259)
>
> at
> org.apache.zookeeper.server.DataTree.getMaxPrefixWithQuota(DataTree.java:634)
>
> at org.apache.zookeeper.server.DataTree.setData(DataTree.java:616)
>
> at
> org.apache.zookeeper.server.DataTree.processTxn(DataTree.java:807)
>
> at
> org.apache.zookeeper.server.ZKDatabase.processTxn(ZKDatabase.java:329)
>
> at
> org.apache.zookeeper.server.ZooKeeperServer.processTxn(ZooKeeperServer.java:965)
>
> at
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:116)
>
> at
> org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:167)
>
> at
> org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:101)
>
>
>
> Restarting the server produces:
>
> 2015-06-10 16:22:21,271 [myid:] - INFO [main:FileSnap@83] - Reading
> snapshot /tmp/zookeeper/version-2/snapshot.6
>
> 2015-06-10 16:22:21,352 [myid:] - ERROR [main:ZooKeeperServerMain@54] -
> Invalid arguments, exiting abnormally
>
> java.lang.IllegalArgumentException: Invalid path
>
> at
> org.apache.zookeeper.common.PathTrie.findMaxPrefix(PathTrie.java:259)
>
> at
> org.apache.zookeeper.server.DataTree.getMaxPrefixWithQuota(DataTree.java:634)
>
> at org.apache.zookeeper.server.DataTree.setData(DataTree.java:616)
>
> at
> org.apache.zookeeper.server.DataTree.processTxn(DataTree.java:807)
>
> at
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.processTransaction(FileTxnSnapLog.java:198)
>
> at
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:151)
>
> at
> org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)
>
> at
> org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:250)
>
> at
> org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:377)
>
> at
> org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:122)
>
> at
> org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:112)
>
> at
> org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:86)
>
> at
> org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:52)
>
> at
> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)
>
> at
> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
>
>
> Brian
>
>
>>
>>
>> On Wed, Jun 10, 2015 at 7:59 AM, geo-turcsanyi <no...@github.com>
>> wrote:
>> >
>> > When you try setting data on an empty path, it will crash and burn your
>> zookeeper server. And by this I mean data corruption, you need to empty
>> your data directory and then restart it, otherwise nobody will be able to
>> connect to it any more and will get a connection refused.
>> >
>> > It would be nice to have an error if the path is empty, rather than
>> letting the set happen and watching your zookeeper go down in style.
>> >
>>
Re: Interesting (very serious) server error reported on go-zookeeper
Posted by Brian Brazil <br...@boxever.com>.
On 10 June 2015 at 16:16, Patrick Hunt <ph...@apache.org> wrote:
> The report is pretty serious. Anyone can confirm? (and create a jira
> please)
>
> https://github.com/samuel/go-zookeeper/issues/62
I happen to be playing with that library right now, so I've attempted to
reproduce it. The bug looks to be as stated on 3.4.5.
The code:
c, _, _ := zk.Connect([]string{"127.0.0.1"}, time.Second)
c.Set("", []byte{}, 0)
Causes on the zookeeper server:
2015-06-10 16:21:10,862 [myid:] - ERROR
[SyncThread:0:SyncRequestProcessor@151] - Severe unrecoverable error,
exiting
java.lang.IllegalArgumentException: Invalid path
at
org.apache.zookeeper.common.PathTrie.findMaxPrefix(PathTrie.java:259)
at
org.apache.zookeeper.server.DataTree.getMaxPrefixWithQuota(DataTree.java:634)
at org.apache.zookeeper.server.DataTree.setData(DataTree.java:616)
at
org.apache.zookeeper.server.DataTree.processTxn(DataTree.java:807)
at
org.apache.zookeeper.server.ZKDatabase.processTxn(ZKDatabase.java:329)
at
org.apache.zookeeper.server.ZooKeeperServer.processTxn(ZooKeeperServer.java:965)
at
org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:116)
at
org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:167)
at
org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:101)
Restarting the server produces:
2015-06-10 16:22:21,271 [myid:] - INFO [main:FileSnap@83] - Reading
snapshot /tmp/zookeeper/version-2/snapshot.6
2015-06-10 16:22:21,352 [myid:] - ERROR [main:ZooKeeperServerMain@54] -
Invalid arguments, exiting abnormally
java.lang.IllegalArgumentException: Invalid path
at
org.apache.zookeeper.common.PathTrie.findMaxPrefix(PathTrie.java:259)
at
org.apache.zookeeper.server.DataTree.getMaxPrefixWithQuota(DataTree.java:634)
at org.apache.zookeeper.server.DataTree.setData(DataTree.java:616)
at
org.apache.zookeeper.server.DataTree.processTxn(DataTree.java:807)
at
org.apache.zookeeper.server.persistence.FileTxnSnapLog.processTransaction(FileTxnSnapLog.java:198)
at
org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:151)
at
org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)
at
org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:250)
at
org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:377)
at
org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:122)
at
org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:112)
at
org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:86)
at
org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:52)
at
org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)
at
org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
Brian
>
>
> On Wed, Jun 10, 2015 at 7:59 AM, geo-turcsanyi <no...@github.com>
> wrote:
> >
> > When you try setting data on an empty path, it will crash and burn your
> zookeeper server. And by this I mean data corruption, you need to empty
> your data directory and then restart it, otherwise nobody will be able to
> connect to it any more and will get a connection refused.
> >
> > It would be nice to have an error if the path is empty, rather than
> letting the set happen and watching your zookeeper go down in style.
> >
>