You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Li Wang <li...@gmail.com> on 2021/07/14 05:43:27 UTC
Server crashed due to unreasonable length error
Hi,
When load testing the write operation against Zookeeper 3.7.0, I observed a
couple of times that the server crashed because the txn log was too large
and it was not able to load it. However the data size of write is only 4
bytes in the load test and the *jute.maxbuffer *was set to default (i.e.
1M). The error doesn't always happen.
I wonder if anyone has also seen this error or has any idea on what may
cause the issue?
StackTrace
=========
2021-07-01 16:02:00,837 [myid:3] - ERROR [main:QuorumPeerMain@114] -
Unexpected exception, exiting abnormally
java.lang.RuntimeException: Unable to run quorum server
at
org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:1200)
at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:1131)
at
org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:229)
at
org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:137)
at
org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:91)
Caused by: java.io.IOException: Unreasonable length = 3175014
at
org.apache.jute.BinaryInputArchive.checkLength(BinaryInputArchive.java:166)
at
org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:127)
at org.apache.zookeeper.server.persistence.Util.readTxnBytes(Util.java:159)
at
org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:749)
at
org.apache.zookeeper.server.persistence.FileTxnSnapLog.fastForwardFromEdits(FileTxnSnapLog.java:361)
at
org.apache.zookeeper.server.persistence.FileTxnSnapLog.lambda$restore$0(FileTxnSnapLog.java:267)
at
org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:312)
at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:287)
at
org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:1145)
Thanks,
Li
Re: Server crashed due to unreasonable length error
Posted by Li Wang <li...@gmail.com>.
Hi Damien,
I digged out the stacktrace when it happened last time. Hope it helps. The
size of txn logs from the client requests should be very small, as the data
size is only 4 bytes in our write load test. The "overflowed"
closeSessionTxn seems to be the issue.
2021-07-01 16:02:00,837 [myid:3] - ERROR [main:QuorumPeerMain@114] -
Unexpected exception, exiting abnormally
java.lang.RuntimeException: Unable to run quorum server
at
org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:1200)
at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:1131)
at
org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:229)
at
org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:137)
at
org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:91)
Caused by: java.io.IOException: Unreasonable length = 3175014
at
org.apache.jute.BinaryInputArchive.checkLength(BinaryInputArchive.java:166)
at
org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:127)
at org.apache.zookeeper.server.persistence.Util.readTxnBytes(Util.java:159)
at
org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:749)
at
org.apache.zookeeper.server.persistence.FileTxnSnapLog.fastForwardFromEdits(FileTxnSnapLog.java:361)
at
org.apache.zookeeper.server.persistence.FileTxnSnapLog.lambda$restore$0(FileTxnSnapLog.java:267)
at
org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:312)
at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:287)
at
org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:1145)
Cheers,
Li
On Wed, Jul 28, 2021 at 6:55 PM Li Wang <li...@gmail.com> wrote:
>
> On Wed, Jul 28, 2021 at 5:20 AM Damien Diederen <dd...@apache.org>
> wrote:
>
>>
>> Hi Li, all,
>>
>> > When load testing the write operation against Zookeeper 3.7.0, I
>> observed a
>> > couple of times that the server crashed because the txn log was too
>> large
>> > and it was not able to load it.
>>
>> Difficult to say without more details, but I suspect ZOOKEEPER-4306 to
>> be the culprit:
>>
>> https://issues.apache.org/jira/browse/ZOOKEEPER-4306
>
>
> Yes, ZOOKEEPER-4306 could be the culprit. In my write operation test, all
> the nodes were created as ephemeral.
>
>>
>> Would it be possible for you to share the transaction log ZooKeeper
>> fails to load?
>>
>
> I observed the issues a couple of times about one month ago. I tried to
> investigate this issue more recently, but was not able to reproduce it.
> I remembered I read the txn log file using zkTxnLogToolkit.sh and it was
> similar to what's mentioned in the
> https://issues.apache.org/jira/browse/ZOOKEEPER-4306.
> Unfortunately, I didn't save the txn log.
>
> I will save the txn log if I can reproduce it or it happens again.
>
> Thanks,
>
> Li
>
>
>> HTH, -D
>>
>>
>>
>> --8<---------------original message------------->8---
>>
>> Li Wang <li...@gmail.com> writes:
>> > Hi,
>> >
>> >
>> >
>> > When load testing the write operation against Zookeeper 3.7.0, I
>> observed a
>> > couple of times that the server crashed because the txn log was too
>> large
>> > and it was not able to load it. However the data size of write is only 4
>> > bytes in the load test and the *jute.maxbuffer *was set to default (i.e.
>> > 1M). The error doesn't always happen.
>> >
>> >
>> > I wonder if anyone has also seen this error or has any idea on what may
>> > cause the issue?
>> >
>> >
>> > StackTrace
>> >
>> > =========
>> >
>> >
>> > 2021-07-01 16:02:00,837 [myid:3] - ERROR [main:QuorumPeerMain@114] -
>> > Unexpected exception, exiting abnormally
>> >
>> > java.lang.RuntimeException: Unable to run quorum server
>> >
>> > at
>> >
>> org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:1200)
>> >
>> > at
>> org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:1131)
>> >
>> > at
>> >
>> org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:229)
>> >
>> > at
>> >
>> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:137)
>> >
>> > at
>> >
>> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:91)
>> >
>> > Caused by: java.io.IOException: Unreasonable length = 3175014
>> >
>> > at
>> >
>> org.apache.jute.BinaryInputArchive.checkLength(BinaryInputArchive.java:166)
>> >
>> > at
>> >
>> org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:127)
>> >
>> > at
>> org.apache.zookeeper.server.persistence.Util.readTxnBytes(Util.java:159)
>> >
>> > at
>> >
>> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:749)
>> >
>> > at
>> >
>> org.apache.zookeeper.server.persistence.FileTxnSnapLog.fastForwardFromEdits(FileTxnSnapLog.java:361)
>> >
>> > at
>> >
>> org.apache.zookeeper.server.persistence.FileTxnSnapLog.lambda$restore$0(FileTxnSnapLog.java:267)
>> >
>> > at
>> >
>> org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:312)
>> >
>> > at
>> org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:287)
>> >
>> > at
>> >
>> org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:1145)
>> >
>> >
>> > Thanks,
>> >
>> >
>> > Li
>>
>
Re: Server crashed due to unreasonable length error
Posted by Li Wang <li...@gmail.com>.
Hi Damien,
I digged out the stacktrace when it happened last time. Hope it helps. The
size of txn logs from the client requests should be very small, as the data
size is only 4 bytes in our write load test. The "overflowed"
closeSessionTxn seems to be the issue.
2021-07-01 16:02:00,837 [myid:3] - ERROR [main:QuorumPeerMain@114] -
Unexpected exception, exiting abnormally
java.lang.RuntimeException: Unable to run quorum server
at
org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:1200)
at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:1131)
at
org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:229)
at
org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:137)
at
org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:91)
Caused by: java.io.IOException: Unreasonable length = 3175014
at
org.apache.jute.BinaryInputArchive.checkLength(BinaryInputArchive.java:166)
at
org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:127)
at org.apache.zookeeper.server.persistence.Util.readTxnBytes(Util.java:159)
at
org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:749)
at
org.apache.zookeeper.server.persistence.FileTxnSnapLog.fastForwardFromEdits(FileTxnSnapLog.java:361)
at
org.apache.zookeeper.server.persistence.FileTxnSnapLog.lambda$restore$0(FileTxnSnapLog.java:267)
at
org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:312)
at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:287)
at
org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:1145)
Cheers,
Li
On Wed, Jul 28, 2021 at 6:55 PM Li Wang <li...@gmail.com> wrote:
>
> On Wed, Jul 28, 2021 at 5:20 AM Damien Diederen <dd...@apache.org>
> wrote:
>
>>
>> Hi Li, all,
>>
>> > When load testing the write operation against Zookeeper 3.7.0, I
>> observed a
>> > couple of times that the server crashed because the txn log was too
>> large
>> > and it was not able to load it.
>>
>> Difficult to say without more details, but I suspect ZOOKEEPER-4306 to
>> be the culprit:
>>
>> https://issues.apache.org/jira/browse/ZOOKEEPER-4306
>
>
> Yes, ZOOKEEPER-4306 could be the culprit. In my write operation test, all
> the nodes were created as ephemeral.
>
>>
>> Would it be possible for you to share the transaction log ZooKeeper
>> fails to load?
>>
>
> I observed the issues a couple of times about one month ago. I tried to
> investigate this issue more recently, but was not able to reproduce it.
> I remembered I read the txn log file using zkTxnLogToolkit.sh and it was
> similar to what's mentioned in the
> https://issues.apache.org/jira/browse/ZOOKEEPER-4306.
> Unfortunately, I didn't save the txn log.
>
> I will save the txn log if I can reproduce it or it happens again.
>
> Thanks,
>
> Li
>
>
>> HTH, -D
>>
>>
>>
>> --8<---------------original message------------->8---
>>
>> Li Wang <li...@gmail.com> writes:
>> > Hi,
>> >
>> >
>> >
>> > When load testing the write operation against Zookeeper 3.7.0, I
>> observed a
>> > couple of times that the server crashed because the txn log was too
>> large
>> > and it was not able to load it. However the data size of write is only 4
>> > bytes in the load test and the *jute.maxbuffer *was set to default (i.e.
>> > 1M). The error doesn't always happen.
>> >
>> >
>> > I wonder if anyone has also seen this error or has any idea on what may
>> > cause the issue?
>> >
>> >
>> > StackTrace
>> >
>> > =========
>> >
>> >
>> > 2021-07-01 16:02:00,837 [myid:3] - ERROR [main:QuorumPeerMain@114] -
>> > Unexpected exception, exiting abnormally
>> >
>> > java.lang.RuntimeException: Unable to run quorum server
>> >
>> > at
>> >
>> org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:1200)
>> >
>> > at
>> org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:1131)
>> >
>> > at
>> >
>> org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:229)
>> >
>> > at
>> >
>> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:137)
>> >
>> > at
>> >
>> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:91)
>> >
>> > Caused by: java.io.IOException: Unreasonable length = 3175014
>> >
>> > at
>> >
>> org.apache.jute.BinaryInputArchive.checkLength(BinaryInputArchive.java:166)
>> >
>> > at
>> >
>> org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:127)
>> >
>> > at
>> org.apache.zookeeper.server.persistence.Util.readTxnBytes(Util.java:159)
>> >
>> > at
>> >
>> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:749)
>> >
>> > at
>> >
>> org.apache.zookeeper.server.persistence.FileTxnSnapLog.fastForwardFromEdits(FileTxnSnapLog.java:361)
>> >
>> > at
>> >
>> org.apache.zookeeper.server.persistence.FileTxnSnapLog.lambda$restore$0(FileTxnSnapLog.java:267)
>> >
>> > at
>> >
>> org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:312)
>> >
>> > at
>> org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:287)
>> >
>> > at
>> >
>> org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:1145)
>> >
>> >
>> > Thanks,
>> >
>> >
>> > Li
>>
>
Re: Server crashed due to unreasonable length error
Posted by Li Wang <li...@gmail.com>.
On Wed, Jul 28, 2021 at 5:20 AM Damien Diederen <dd...@apache.org>
wrote:
>
> Hi Li, all,
>
> > When load testing the write operation against Zookeeper 3.7.0, I
> observed a
> > couple of times that the server crashed because the txn log was too large
> > and it was not able to load it.
>
> Difficult to say without more details, but I suspect ZOOKEEPER-4306 to
> be the culprit:
>
> https://issues.apache.org/jira/browse/ZOOKEEPER-4306
Yes, ZOOKEEPER-4306 could be the culprit. In my write operation test, all
the nodes were created as ephemeral.
>
> Would it be possible for you to share the transaction log ZooKeeper
> fails to load?
>
I observed the issues a couple of times about one month ago. I tried to
investigate this issue more recently, but was not able to reproduce it.
I remembered I read the txn log file using zkTxnLogToolkit.sh and it was
similar to what's mentioned in the
https://issues.apache.org/jira/browse/ZOOKEEPER-4306.
Unfortunately, I didn't save the txn log.
I will save the txn log if I can reproduce it or it happens again.
Thanks,
Li
> HTH, -D
>
>
>
> --8<---------------original message------------->8---
>
> Li Wang <li...@gmail.com> writes:
> > Hi,
> >
> >
> >
> > When load testing the write operation against Zookeeper 3.7.0, I
> observed a
> > couple of times that the server crashed because the txn log was too large
> > and it was not able to load it. However the data size of write is only 4
> > bytes in the load test and the *jute.maxbuffer *was set to default (i.e.
> > 1M). The error doesn't always happen.
> >
> >
> > I wonder if anyone has also seen this error or has any idea on what may
> > cause the issue?
> >
> >
> > StackTrace
> >
> > =========
> >
> >
> > 2021-07-01 16:02:00,837 [myid:3] - ERROR [main:QuorumPeerMain@114] -
> > Unexpected exception, exiting abnormally
> >
> > java.lang.RuntimeException: Unable to run quorum server
> >
> > at
> >
> org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:1200)
> >
> > at
> org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:1131)
> >
> > at
> >
> org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:229)
> >
> > at
> >
> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:137)
> >
> > at
> >
> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:91)
> >
> > Caused by: java.io.IOException: Unreasonable length = 3175014
> >
> > at
> >
> org.apache.jute.BinaryInputArchive.checkLength(BinaryInputArchive.java:166)
> >
> > at
> >
> org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:127)
> >
> > at
> org.apache.zookeeper.server.persistence.Util.readTxnBytes(Util.java:159)
> >
> > at
> >
> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:749)
> >
> > at
> >
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.fastForwardFromEdits(FileTxnSnapLog.java:361)
> >
> > at
> >
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.lambda$restore$0(FileTxnSnapLog.java:267)
> >
> > at
> >
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:312)
> >
> > at
> org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:287)
> >
> > at
> >
> org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:1145)
> >
> >
> > Thanks,
> >
> >
> > Li
>
Re: Server crashed due to unreasonable length error
Posted by Li Wang <li...@gmail.com>.
On Wed, Jul 28, 2021 at 5:20 AM Damien Diederen <dd...@apache.org>
wrote:
>
> Hi Li, all,
>
> > When load testing the write operation against Zookeeper 3.7.0, I
> observed a
> > couple of times that the server crashed because the txn log was too large
> > and it was not able to load it.
>
> Difficult to say without more details, but I suspect ZOOKEEPER-4306 to
> be the culprit:
>
> https://issues.apache.org/jira/browse/ZOOKEEPER-4306
Yes, ZOOKEEPER-4306 could be the culprit. In my write operation test, all
the nodes were created as ephemeral.
>
> Would it be possible for you to share the transaction log ZooKeeper
> fails to load?
>
I observed the issues a couple of times about one month ago. I tried to
investigate this issue more recently, but was not able to reproduce it.
I remembered I read the txn log file using zkTxnLogToolkit.sh and it was
similar to what's mentioned in the
https://issues.apache.org/jira/browse/ZOOKEEPER-4306.
Unfortunately, I didn't save the txn log.
I will save the txn log if I can reproduce it or it happens again.
Thanks,
Li
> HTH, -D
>
>
>
> --8<---------------original message------------->8---
>
> Li Wang <li...@gmail.com> writes:
> > Hi,
> >
> >
> >
> > When load testing the write operation against Zookeeper 3.7.0, I
> observed a
> > couple of times that the server crashed because the txn log was too large
> > and it was not able to load it. However the data size of write is only 4
> > bytes in the load test and the *jute.maxbuffer *was set to default (i.e.
> > 1M). The error doesn't always happen.
> >
> >
> > I wonder if anyone has also seen this error or has any idea on what may
> > cause the issue?
> >
> >
> > StackTrace
> >
> > =========
> >
> >
> > 2021-07-01 16:02:00,837 [myid:3] - ERROR [main:QuorumPeerMain@114] -
> > Unexpected exception, exiting abnormally
> >
> > java.lang.RuntimeException: Unable to run quorum server
> >
> > at
> >
> org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:1200)
> >
> > at
> org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:1131)
> >
> > at
> >
> org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:229)
> >
> > at
> >
> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:137)
> >
> > at
> >
> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:91)
> >
> > Caused by: java.io.IOException: Unreasonable length = 3175014
> >
> > at
> >
> org.apache.jute.BinaryInputArchive.checkLength(BinaryInputArchive.java:166)
> >
> > at
> >
> org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:127)
> >
> > at
> org.apache.zookeeper.server.persistence.Util.readTxnBytes(Util.java:159)
> >
> > at
> >
> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:749)
> >
> > at
> >
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.fastForwardFromEdits(FileTxnSnapLog.java:361)
> >
> > at
> >
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.lambda$restore$0(FileTxnSnapLog.java:267)
> >
> > at
> >
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:312)
> >
> > at
> org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:287)
> >
> > at
> >
> org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:1145)
> >
> >
> > Thanks,
> >
> >
> > Li
>
Re: Server crashed due to unreasonable length error
Posted by Damien Diederen <dd...@apache.org>.
Hi Li, all,
> When load testing the write operation against Zookeeper 3.7.0, I observed a
> couple of times that the server crashed because the txn log was too large
> and it was not able to load it.
Difficult to say without more details, but I suspect ZOOKEEPER-4306 to
be the culprit:
https://issues.apache.org/jira/browse/ZOOKEEPER-4306
Would it be possible for you to share the transaction log ZooKeeper
fails to load?
HTH, -D
--8<---------------original message------------->8---
Li Wang <li...@gmail.com> writes:
> Hi,
>
>
>
> When load testing the write operation against Zookeeper 3.7.0, I observed a
> couple of times that the server crashed because the txn log was too large
> and it was not able to load it. However the data size of write is only 4
> bytes in the load test and the *jute.maxbuffer *was set to default (i.e.
> 1M). The error doesn't always happen.
>
>
> I wonder if anyone has also seen this error or has any idea on what may
> cause the issue?
>
>
> StackTrace
>
> =========
>
>
> 2021-07-01 16:02:00,837 [myid:3] - ERROR [main:QuorumPeerMain@114] -
> Unexpected exception, exiting abnormally
>
> java.lang.RuntimeException: Unable to run quorum server
>
> at
> org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:1200)
>
> at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:1131)
>
> at
> org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:229)
>
> at
> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:137)
>
> at
> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:91)
>
> Caused by: java.io.IOException: Unreasonable length = 3175014
>
> at
> org.apache.jute.BinaryInputArchive.checkLength(BinaryInputArchive.java:166)
>
> at
> org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:127)
>
> at org.apache.zookeeper.server.persistence.Util.readTxnBytes(Util.java:159)
>
> at
> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:749)
>
> at
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.fastForwardFromEdits(FileTxnSnapLog.java:361)
>
> at
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.lambda$restore$0(FileTxnSnapLog.java:267)
>
> at
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:312)
>
> at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:287)
>
> at
> org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:1145)
>
>
> Thanks,
>
>
> Li
Re: Server crashed due to unreasonable length error
Posted by Damien Diederen <dd...@apache.org>.
Hi Li, all,
> When load testing the write operation against Zookeeper 3.7.0, I observed a
> couple of times that the server crashed because the txn log was too large
> and it was not able to load it.
Difficult to say without more details, but I suspect ZOOKEEPER-4306 to
be the culprit:
https://issues.apache.org/jira/browse/ZOOKEEPER-4306
Would it be possible for you to share the transaction log ZooKeeper
fails to load?
HTH, -D
--8<---------------original message------------->8---
Li Wang <li...@gmail.com> writes:
> Hi,
>
>
>
> When load testing the write operation against Zookeeper 3.7.0, I observed a
> couple of times that the server crashed because the txn log was too large
> and it was not able to load it. However the data size of write is only 4
> bytes in the load test and the *jute.maxbuffer *was set to default (i.e.
> 1M). The error doesn't always happen.
>
>
> I wonder if anyone has also seen this error or has any idea on what may
> cause the issue?
>
>
> StackTrace
>
> =========
>
>
> 2021-07-01 16:02:00,837 [myid:3] - ERROR [main:QuorumPeerMain@114] -
> Unexpected exception, exiting abnormally
>
> java.lang.RuntimeException: Unable to run quorum server
>
> at
> org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:1200)
>
> at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:1131)
>
> at
> org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:229)
>
> at
> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:137)
>
> at
> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:91)
>
> Caused by: java.io.IOException: Unreasonable length = 3175014
>
> at
> org.apache.jute.BinaryInputArchive.checkLength(BinaryInputArchive.java:166)
>
> at
> org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:127)
>
> at org.apache.zookeeper.server.persistence.Util.readTxnBytes(Util.java:159)
>
> at
> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:749)
>
> at
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.fastForwardFromEdits(FileTxnSnapLog.java:361)
>
> at
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.lambda$restore$0(FileTxnSnapLog.java:267)
>
> at
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:312)
>
> at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:287)
>
> at
> org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:1145)
>
>
> Thanks,
>
>
> Li