You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Chris W <ch...@gmail.com> on 2014/03/19 00:46:51 UTC
Zookeeper exceptions - SEVERE
I am running a 3 node zookeeper 3.4.5 Quorum. I am running into issues
with Zookeeper transaction logs
[myid:2] - ERROR [main:QuorumPeer@453] - Unable to load database on disk
java.io.IOException: Unreasonable length = 1048587
at
org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:100)
at org.apache.zookeeper.server.persistence.Util.readTxnBytes(Util.java:233)
at
org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:602)
at
org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:157)
at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)
To unblock temporarily, i deleted the most recent txn log. How do i tell
zookeeper to not grow the transaction log beyond x MegaBytes?
How often does the transaction log get updated? Does zk transactions log
grow everytime we index data into a new collection?
--
Best
--
C
Re: Zookeeper exceptions - SEVERE
Posted by Chris W <ch...@gmail.com>.
Thanks. Temporarily got over the problem by specifying custom limits
through jute.maxbuffer=<customlimit>
On Tue, Mar 18, 2014 at 9:45 PM, Shalin Shekhar Mangar <
shalinmangar@gmail.com> wrote:
> Sorry guys I spoke too fast. I looked at the code again. No it doesn't
> correlate with commits at all. I was mistaken.
>
> On Wed, Mar 19, 2014 at 10:06 AM, Chris W <ch...@gmail.com> wrote:
> > Thanks, Shawn and Shalin
> >
> > How does the frequency of commit affect zookeeper?
> >
> >
> > Thanks
> >
> >
> > On Tue, Mar 18, 2014 at 9:12 PM, Shalin Shekhar Mangar <
> > shalinmangar@gmail.com> wrote:
> >
> >> SolrCloud will update Zookeeper on state changes (node goes to
> >> recovery, comes back up etc) or for leader election and during
> >> collection API commands. It doesn't correlate directly with indexing
> >> but is correlated with how frequently you call commit.
> >>
> >> On Wed, Mar 19, 2014 at 5:46 AM, Shawn Heisey <so...@elyograg.org>
> wrote:
> >> > On 3/18/2014 5:46 PM, Chris W wrote:
> >> >>
> >> >> I am running a 3 node zookeeper 3.4.5 Quorum. I am running into
> issues
> >> >> with Zookeeper transaction logs
> >> >>
> >> >> [myid:2] - ERROR [main:QuorumPeer@453] - Unable to load database
> on
> >> disk
> >> >> java.io.IOException: Unreasonable length = 1048587
> >> >> at
> >> >>
> >>
> org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:100)
> >> >> at
> >> >>
> org.apache.zookeeper.server.persistence.Util.readTxnBytes(Util.java:233)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:602)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:157)
> >> >> at
> >> >>
> org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)
> >> >>
> >> >> To unblock temporarily, i deleted the most recent txn log. How do i
> tell
> >> >> zookeeper to not grow the transaction log beyond x MegaBytes?
> >> >>
> >> >> How often does the transaction log get updated? Does zk transactions
> log
> >> >> grow everytime we index data into a new collection?
> >> >
> >> >
> >> > Zookeeper is a separate project at Apache. ZK file management is
> >> discussed
> >> > in the ZK documentation.
> >> >
> >> >
> >>
> http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
> >> >
> >> > There is a bug filed on Zookeeper for the issue you are seeing, with a
> >> > fairly simple workaround. It is fixed in the 3.4.6 version, which was
> >> > released last week. I will see whether we can get ZK upgraded to
> 3.4.6
> >> in
> >> > the Solr 4.8.0 release. I don't think we want to risk doing that
> >> upgrade in
> >> > 4.7.1, but I could be wrong.
> >> >
> >> > https://issues.apache.org/jira/browse/ZOOKEEPER-1513
> >> > http://zookeeper.apache.org/releases.html
> >> >
> >> > I am actually not sure how often SolrCloud updates Zookeeper. It
> happens
> >> > whenever the collections API is called for sure, and it may happen
> >> anytime
> >> > you index data as well.
> >> >
> >> > Thanks,
> >> > Shawn
> >> >
> >>
> >>
> >>
> >> --
> >> Regards,
> >> Shalin Shekhar Mangar.
> >>
> >
> >
> >
> > --
> > Best
> > --
> > C
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
--
Best
--
C
Re: Zookeeper exceptions - SEVERE
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
Sorry guys I spoke too fast. I looked at the code again. No it doesn't
correlate with commits at all. I was mistaken.
On Wed, Mar 19, 2014 at 10:06 AM, Chris W <ch...@gmail.com> wrote:
> Thanks, Shawn and Shalin
>
> How does the frequency of commit affect zookeeper?
>
>
> Thanks
>
>
> On Tue, Mar 18, 2014 at 9:12 PM, Shalin Shekhar Mangar <
> shalinmangar@gmail.com> wrote:
>
>> SolrCloud will update Zookeeper on state changes (node goes to
>> recovery, comes back up etc) or for leader election and during
>> collection API commands. It doesn't correlate directly with indexing
>> but is correlated with how frequently you call commit.
>>
>> On Wed, Mar 19, 2014 at 5:46 AM, Shawn Heisey <so...@elyograg.org> wrote:
>> > On 3/18/2014 5:46 PM, Chris W wrote:
>> >>
>> >> I am running a 3 node zookeeper 3.4.5 Quorum. I am running into issues
>> >> with Zookeeper transaction logs
>> >>
>> >> [myid:2] - ERROR [main:QuorumPeer@453] - Unable to load database on
>> disk
>> >> java.io.IOException: Unreasonable length = 1048587
>> >> at
>> >>
>> org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:100)
>> >> at
>> >> org.apache.zookeeper.server.persistence.Util.readTxnBytes(Util.java:233)
>> >> at
>> >>
>> >>
>> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:602)
>> >> at
>> >>
>> >>
>> org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:157)
>> >> at
>> >> org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)
>> >>
>> >> To unblock temporarily, i deleted the most recent txn log. How do i tell
>> >> zookeeper to not grow the transaction log beyond x MegaBytes?
>> >>
>> >> How often does the transaction log get updated? Does zk transactions log
>> >> grow everytime we index data into a new collection?
>> >
>> >
>> > Zookeeper is a separate project at Apache. ZK file management is
>> discussed
>> > in the ZK documentation.
>> >
>> >
>> http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
>> >
>> > There is a bug filed on Zookeeper for the issue you are seeing, with a
>> > fairly simple workaround. It is fixed in the 3.4.6 version, which was
>> > released last week. I will see whether we can get ZK upgraded to 3.4.6
>> in
>> > the Solr 4.8.0 release. I don't think we want to risk doing that
>> upgrade in
>> > 4.7.1, but I could be wrong.
>> >
>> > https://issues.apache.org/jira/browse/ZOOKEEPER-1513
>> > http://zookeeper.apache.org/releases.html
>> >
>> > I am actually not sure how often SolrCloud updates Zookeeper. It happens
>> > whenever the collections API is called for sure, and it may happen
>> anytime
>> > you index data as well.
>> >
>> > Thanks,
>> > Shawn
>> >
>>
>>
>>
>> --
>> Regards,
>> Shalin Shekhar Mangar.
>>
>
>
>
> --
> Best
> --
> C
--
Regards,
Shalin Shekhar Mangar.
Re: Zookeeper exceptions - SEVERE
Posted by Chris W <ch...@gmail.com>.
Thanks, Shawn and Shalin
How does the frequency of commit affect zookeeper?
Thanks
On Tue, Mar 18, 2014 at 9:12 PM, Shalin Shekhar Mangar <
shalinmangar@gmail.com> wrote:
> SolrCloud will update Zookeeper on state changes (node goes to
> recovery, comes back up etc) or for leader election and during
> collection API commands. It doesn't correlate directly with indexing
> but is correlated with how frequently you call commit.
>
> On Wed, Mar 19, 2014 at 5:46 AM, Shawn Heisey <so...@elyograg.org> wrote:
> > On 3/18/2014 5:46 PM, Chris W wrote:
> >>
> >> I am running a 3 node zookeeper 3.4.5 Quorum. I am running into issues
> >> with Zookeeper transaction logs
> >>
> >> [myid:2] - ERROR [main:QuorumPeer@453] - Unable to load database on
> disk
> >> java.io.IOException: Unreasonable length = 1048587
> >> at
> >>
> org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:100)
> >> at
> >> org.apache.zookeeper.server.persistence.Util.readTxnBytes(Util.java:233)
> >> at
> >>
> >>
> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:602)
> >> at
> >>
> >>
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:157)
> >> at
> >> org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)
> >>
> >> To unblock temporarily, i deleted the most recent txn log. How do i tell
> >> zookeeper to not grow the transaction log beyond x MegaBytes?
> >>
> >> How often does the transaction log get updated? Does zk transactions log
> >> grow everytime we index data into a new collection?
> >
> >
> > Zookeeper is a separate project at Apache. ZK file management is
> discussed
> > in the ZK documentation.
> >
> >
> http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
> >
> > There is a bug filed on Zookeeper for the issue you are seeing, with a
> > fairly simple workaround. It is fixed in the 3.4.6 version, which was
> > released last week. I will see whether we can get ZK upgraded to 3.4.6
> in
> > the Solr 4.8.0 release. I don't think we want to risk doing that
> upgrade in
> > 4.7.1, but I could be wrong.
> >
> > https://issues.apache.org/jira/browse/ZOOKEEPER-1513
> > http://zookeeper.apache.org/releases.html
> >
> > I am actually not sure how often SolrCloud updates Zookeeper. It happens
> > whenever the collections API is called for sure, and it may happen
> anytime
> > you index data as well.
> >
> > Thanks,
> > Shawn
> >
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
--
Best
--
C
Re: Zookeeper exceptions - SEVERE
Posted by Gopal Patwa <go...@gmail.com>.
Shalin, "correlated with how frequently you call commit" is it soft commit
or hard commit? , I guess it should be later one.
just curious what data it update to zookeeper during commit
On Tue, Mar 18, 2014 at 9:12 PM, Shalin Shekhar Mangar <
shalinmangar@gmail.com> wrote:
> SolrCloud will update Zookeeper on state changes (node goes to
> recovery, comes back up etc) or for leader election and during
> collection API commands. It doesn't correlate directly with indexing
> but is correlated with how frequently you call commit.
>
> On Wed, Mar 19, 2014 at 5:46 AM, Shawn Heisey <so...@elyograg.org> wrote:
> > On 3/18/2014 5:46 PM, Chris W wrote:
> >>
> >> I am running a 3 node zookeeper 3.4.5 Quorum. I am running into issues
> >> with Zookeeper transaction logs
> >>
> >> [myid:2] - ERROR [main:QuorumPeer@453] - Unable to load database on
> disk
> >> java.io.IOException: Unreasonable length = 1048587
> >> at
> >>
> org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:100)
> >> at
> >> org.apache.zookeeper.server.persistence.Util.readTxnBytes(Util.java:233)
> >> at
> >>
> >>
> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:602)
> >> at
> >>
> >>
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:157)
> >> at
> >> org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)
> >>
> >> To unblock temporarily, i deleted the most recent txn log. How do i tell
> >> zookeeper to not grow the transaction log beyond x MegaBytes?
> >>
> >> How often does the transaction log get updated? Does zk transactions log
> >> grow everytime we index data into a new collection?
> >
> >
> > Zookeeper is a separate project at Apache. ZK file management is
> discussed
> > in the ZK documentation.
> >
> >
> http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
> >
> > There is a bug filed on Zookeeper for the issue you are seeing, with a
> > fairly simple workaround. It is fixed in the 3.4.6 version, which was
> > released last week. I will see whether we can get ZK upgraded to 3.4.6
> in
> > the Solr 4.8.0 release. I don't think we want to risk doing that
> upgrade in
> > 4.7.1, but I could be wrong.
> >
> > https://issues.apache.org/jira/browse/ZOOKEEPER-1513
> > http://zookeeper.apache.org/releases.html
> >
> > I am actually not sure how often SolrCloud updates Zookeeper. It happens
> > whenever the collections API is called for sure, and it may happen
> anytime
> > you index data as well.
> >
> > Thanks,
> > Shawn
> >
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
Re: Zookeeper exceptions - SEVERE
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
SolrCloud will update Zookeeper on state changes (node goes to
recovery, comes back up etc) or for leader election and during
collection API commands. It doesn't correlate directly with indexing
but is correlated with how frequently you call commit.
On Wed, Mar 19, 2014 at 5:46 AM, Shawn Heisey <so...@elyograg.org> wrote:
> On 3/18/2014 5:46 PM, Chris W wrote:
>>
>> I am running a 3 node zookeeper 3.4.5 Quorum. I am running into issues
>> with Zookeeper transaction logs
>>
>> [myid:2] - ERROR [main:QuorumPeer@453] - Unable to load database on disk
>> java.io.IOException: Unreasonable length = 1048587
>> at
>> org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:100)
>> at
>> org.apache.zookeeper.server.persistence.Util.readTxnBytes(Util.java:233)
>> at
>>
>> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:602)
>> at
>>
>> org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:157)
>> at
>> org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)
>>
>> To unblock temporarily, i deleted the most recent txn log. How do i tell
>> zookeeper to not grow the transaction log beyond x MegaBytes?
>>
>> How often does the transaction log get updated? Does zk transactions log
>> grow everytime we index data into a new collection?
>
>
> Zookeeper is a separate project at Apache. ZK file management is discussed
> in the ZK documentation.
>
> http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
>
> There is a bug filed on Zookeeper for the issue you are seeing, with a
> fairly simple workaround. It is fixed in the 3.4.6 version, which was
> released last week. I will see whether we can get ZK upgraded to 3.4.6 in
> the Solr 4.8.0 release. I don't think we want to risk doing that upgrade in
> 4.7.1, but I could be wrong.
>
> https://issues.apache.org/jira/browse/ZOOKEEPER-1513
> http://zookeeper.apache.org/releases.html
>
> I am actually not sure how often SolrCloud updates Zookeeper. It happens
> whenever the collections API is called for sure, and it may happen anytime
> you index data as well.
>
> Thanks,
> Shawn
>
--
Regards,
Shalin Shekhar Mangar.
Re: Zookeeper exceptions - SEVERE
Posted by Shawn Heisey <so...@elyograg.org>.
On 3/18/2014 5:46 PM, Chris W wrote:
> I am running a 3 node zookeeper 3.4.5 Quorum. I am running into issues
> with Zookeeper transaction logs
>
> [myid:2] - ERROR [main:QuorumPeer@453] - Unable to load database on disk
> java.io.IOException: Unreasonable length = 1048587
> at
> org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:100)
> at org.apache.zookeeper.server.persistence.Util.readTxnBytes(Util.java:233)
> at
> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:602)
> at
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:157)
> at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)
>
> To unblock temporarily, i deleted the most recent txn log. How do i tell
> zookeeper to not grow the transaction log beyond x MegaBytes?
>
> How often does the transaction log get updated? Does zk transactions log
> grow everytime we index data into a new collection?
Zookeeper is a separate project at Apache. ZK file management is
discussed in the ZK documentation.
http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
There is a bug filed on Zookeeper for the issue you are seeing, with a
fairly simple workaround. It is fixed in the 3.4.6 version, which was
released last week. I will see whether we can get ZK upgraded to 3.4.6
in the Solr 4.8.0 release. I don't think we want to risk doing that
upgrade in 4.7.1, but I could be wrong.
https://issues.apache.org/jira/browse/ZOOKEEPER-1513
http://zookeeper.apache.org/releases.html
I am actually not sure how often SolrCloud updates Zookeeper. It
happens whenever the collections API is called for sure, and it may
happen anytime you index data as well.
Thanks,
Shawn