You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@accumulo.apache.org by "Dickson, Matt MR" <ma...@defence.gov.au> on 2014/05/27 01:48:34 UTC
200k fate locks in zookeeper [SEC=UNOFFICIAL]
UNOFFICIAL
I'm seeing an issue caused by there being almost 200k fate locks in zookeeper. The problem is that zookeeper is configured to return no more than 1mb of data for each query. When the Accumulo master queries the fate directory on startup the response exceeds the 1mb limit and zookeeper disconnects the session. Accumulo then retries the query, and it all happens again. We have managed to increase the query limit for the zkcli client using the jute.maxbuffer setting, but are not clear how to setup Accumulo to use the larger query size when calling the zookeeper jar. We tried setting this through the zoo.cfg, but it doesn't seem to make a difference.
For completeness, the reason there are so many fate locks is because the Accumulo binaries were inadvertantly deleted while ingest and queries were running :( which has left things in this state.
Any ideas would be great.
Thanks in advance,
Matt
RE: 200k fate locks in zookeeper [SEC=UNOFFICIAL]
Posted by "Dickson, Matt MR" <ma...@defence.gov.au>.
UNOFFICIAL
Setting this via the accumulo-env.sh fixed the problem.
Thanks again for the swift response to my questions.
________________________________
From: Keith Turner [mailto:keith@deenlo.com]
Sent: Wednesday, 28 May 2014 01:48
To: user@accumulo.apache.org
Subject: Re: 200k fate locks in zookeeper [SEC=UNOFFICIAL]
The zookeeper admin manual say that prop is a java system property. So one way to set it is via the java command line. Could modify accumulo-env.sh and modify the servers java options to add -Djute.maxbuffer=XX I am not sure what formats that property accepts.
On Mon, May 26, 2014 at 7:48 PM, Dickson, Matt MR <ma...@defence.gov.au>> wrote:
UNOFFICIAL
I'm seeing an issue caused by there being almost 200k fate locks in zookeeper. The problem is that zookeeper is configured to return no more than 1mb of data for each query. When the Accumulo master queries the fate directory on startup the response exceeds the 1mb limit and zookeeper disconnects the session. Accumulo then retries the query, and it all happens again. We have managed to increase the query limit for the zkcli client using the jute.maxbuffer setting, but are not clear how to setup Accumulo to use the larger query size when calling the zookeeper jar. We tried setting this through the zoo.cfg, but it doesn't seem to make a difference.
For completeness, the reason there are so many fate locks is because the Accumulo binaries were inadvertantly deleted while ingest and queries were running :( which has left things in this state.
Any ideas would be great.
Thanks in advance,
Matt
Re: 200k fate locks in zookeeper [SEC=UNOFFICIAL]
Posted by Keith Turner <ke...@deenlo.com>.
The zookeeper admin manual say that prop is a java system property. So one
way to set it is via the java command line. Could modify accumulo-env.sh
and modify the servers java options to add -Djute.maxbuffer=XX I am not
sure what formats that property accepts.
On Mon, May 26, 2014 at 7:48 PM, Dickson, Matt MR <
matt.dickson@defence.gov.au> wrote:
> *UNOFFICIAL*
> I'm seeing an issue caused by there being almost 200k fate locks in
> zookeeper. The problem is that zookeeper is configured to return no more
> than 1mb of data for each query. When the Accumulo master queries the fate
> directory on startup the response exceeds the 1mb limit and zookeeper
> disconnects the session. Accumulo then retries the query, and it all
> happens again. We have managed to increase the query limit for the zkcli
> client using the jute.maxbuffer setting, but are not clear how to setup
> Accumulo to use the larger query size when calling the zookeeper jar. We
> tried setting this through the zoo.cfg, but it doesn't seem to make a
> difference.
>
> For completeness, the reason there are so many fate locks is because the
> Accumulo binaries were inadvertantly deleted while ingest and queries were
> running :( which has left things in this state.
>
> Any ideas would be great.
>
> Thanks in advance,
> Matt
>