You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@accumulo.apache.org by Mario Pastorelli <ma...@teralytics.ch> on 2016/11/11 10:11:53 UTC
Running low on memory and Zookeeper Session expired / disconnected
Hi all,
I have two recurring errors with Accumulo in my cluster and I would like to
know more about them. The first, usually happening at ingestion time when I
write with the batch writers many records, is the "Running low on memory".
We keep adding memory to Accumulo but this is a blind guess and I was
wondering if there is a way to understand how much memory Accumulo would
need considering the amount of data that will be written. Should we write
slowly to Accumulo to avoid this? What is filling all the memory at
ingestion time?
Secondly, we have these zookeeper session expired and other zookeeper
timeouts. Zookeeper on our cluster works quite well, we have many systems
using it. How can I debug a "zookeeper session expired" in Accumulo?
Thanks,
Mario
--
Mario Pastorelli | TERALYTICS
*software engineer*
Teralytics AG | Zollstrasse 62 | 8005 Zurich | Switzerland
phone: +41794381682
email: mario.pastorelli@teralytics.ch
www.teralytics.net
Company registration number: CH-020.3.037.709-7 | Trade register Canton
Zurich
Board of directors: Georg Polzer, Luciano Franceschina, Mark Schmitz, Yann
de Vries
This e-mail message contains confidential information which is for the sole
attention and use of the intended recipient. Please notify us at once if
you think that it may not be intended for you and delete it immediately.
Re: Running low on memory and Zookeeper Session expired / disconnected
Posted by Dylan Hutchison <dh...@cs.washington.edu>.
Oh no, do keep using the native maps. I meant to say that the memory and
Zookeeper problems during ingest can be a result of *not *using native
maps.
The Zookeeper expire results from the tservers having to spend so much time
garbage collecting that the tservers can't respond to the Zookeeper
heartbeat, so Zookeeper expires their session and the tservers subsequently
kill themselves.
If you are using native maps and having these problems, then something else
may be the source.
On Fri, Nov 11, 2016 at 2:24 AM, Mario Pastorelli <
mario.pastorelli@teralytics.ch> wrote:
> Yes we are as they are supposed to be faster. Good to know they could be
> the problem but what would be the solution, disabling them? Also, I guess
> there isn't an easy way to see that they are the problem, right?
>
> On Fri, Nov 11, 2016 at 11:21 AM, Dylan Hutchison <
> dhutchis@cs.washington.edu> wrote:
>
>> Mario, are you using native in-memory maps? I've seen these problems run
>> rampant under Java memory maps.
>>
>> On Fri, Nov 11, 2016 at 2:11 AM, Mario Pastorelli <
>> mario.pastorelli@teralytics.ch> wrote:
>>
>>> Hi all,
>>>
>>> I have two recurring errors with Accumulo in my cluster and I would like
>>> to know more about them. The first, usually happening at ingestion time
>>> when I write with the batch writers many records, is the "Running low on
>>> memory". We keep adding memory to Accumulo but this is a blind guess and I
>>> was wondering if there is a way to understand how much memory Accumulo
>>> would need considering the amount of data that will be written. Should we
>>> write slowly to Accumulo to avoid this? What is filling all the memory at
>>> ingestion time?
>>> Secondly, we have these zookeeper session expired and other zookeeper
>>> timeouts. Zookeeper on our cluster works quite well, we have many systems
>>> using it. How can I debug a "zookeeper session expired" in Accumulo?
>>>
>>> Thanks,
>>> Mario
>>>
>>> --
>>> Mario Pastorelli | TERALYTICS
>>>
>>> *software engineer*
>>>
>>> Teralytics AG | Zollstrasse 62 | 8005 Zurich | Switzerland
>>> phone: +41794381682
>>> email: mario.pastorelli@teralytics.ch
>>> www.teralytics.net
>>>
>>> Company registration number: CH-020.3.037.709-7 | Trade register Canton
>>> Zurich
>>> Board of directors: Georg Polzer, Luciano Franceschina, Mark Schmitz,
>>> Yann de Vries
>>>
>>> This e-mail message contains confidential information which is for the
>>> sole attention and use of the intended recipient. Please notify us at once
>>> if you think that it may not be intended for you and delete it immediately.
>>>
>>
>>
>
>
> --
> Mario Pastorelli | TERALYTICS
>
> *software engineer*
>
> Teralytics AG | Zollstrasse 62 | 8005 Zurich | Switzerland
> phone: +41794381682
> email: mario.pastorelli@teralytics.ch
> www.teralytics.net
>
> Company registration number: CH-020.3.037.709-7 | Trade register Canton
> Zurich
> Board of directors: Georg Polzer, Luciano Franceschina, Mark Schmitz, Yann
> de Vries
>
> This e-mail message contains confidential information which is for the
> sole attention and use of the intended recipient. Please notify us at once
> if you think that it may not be intended for you and delete it immediately.
>
Re: Running low on memory and Zookeeper Session expired / disconnected
Posted by Mario Pastorelli <ma...@teralytics.ch>.
Yes we are as they are supposed to be faster. Good to know they could be
the problem but what would be the solution, disabling them? Also, I guess
there isn't an easy way to see that they are the problem, right?
On Fri, Nov 11, 2016 at 11:21 AM, Dylan Hutchison <
dhutchis@cs.washington.edu> wrote:
> Mario, are you using native in-memory maps? I've seen these problems run
> rampant under Java memory maps.
>
> On Fri, Nov 11, 2016 at 2:11 AM, Mario Pastorelli <
> mario.pastorelli@teralytics.ch> wrote:
>
>> Hi all,
>>
>> I have two recurring errors with Accumulo in my cluster and I would like
>> to know more about them. The first, usually happening at ingestion time
>> when I write with the batch writers many records, is the "Running low on
>> memory". We keep adding memory to Accumulo but this is a blind guess and I
>> was wondering if there is a way to understand how much memory Accumulo
>> would need considering the amount of data that will be written. Should we
>> write slowly to Accumulo to avoid this? What is filling all the memory at
>> ingestion time?
>> Secondly, we have these zookeeper session expired and other zookeeper
>> timeouts. Zookeeper on our cluster works quite well, we have many systems
>> using it. How can I debug a "zookeeper session expired" in Accumulo?
>>
>> Thanks,
>> Mario
>>
>> --
>> Mario Pastorelli | TERALYTICS
>>
>> *software engineer*
>>
>> Teralytics AG | Zollstrasse 62 | 8005 Zurich | Switzerland
>> phone: +41794381682
>> email: mario.pastorelli@teralytics.ch
>> www.teralytics.net
>>
>> Company registration number: CH-020.3.037.709-7 | Trade register Canton
>> Zurich
>> Board of directors: Georg Polzer, Luciano Franceschina, Mark Schmitz,
>> Yann de Vries
>>
>> This e-mail message contains confidential information which is for the
>> sole attention and use of the intended recipient. Please notify us at once
>> if you think that it may not be intended for you and delete it immediately.
>>
>
>
--
Mario Pastorelli | TERALYTICS
*software engineer*
Teralytics AG | Zollstrasse 62 | 8005 Zurich | Switzerland
phone: +41794381682
email: mario.pastorelli@teralytics.ch
www.teralytics.net
Company registration number: CH-020.3.037.709-7 | Trade register Canton
Zurich
Board of directors: Georg Polzer, Luciano Franceschina, Mark Schmitz, Yann
de Vries
This e-mail message contains confidential information which is for the sole
attention and use of the intended recipient. Please notify us at once if
you think that it may not be intended for you and delete it immediately.
Re: Running low on memory and Zookeeper Session expired / disconnected
Posted by Dylan Hutchison <dh...@cs.washington.edu>.
Mario, are you using native in-memory maps? I've seen these problems run
rampant under Java memory maps.
On Fri, Nov 11, 2016 at 2:11 AM, Mario Pastorelli <
mario.pastorelli@teralytics.ch> wrote:
> Hi all,
>
> I have two recurring errors with Accumulo in my cluster and I would like
> to know more about them. The first, usually happening at ingestion time
> when I write with the batch writers many records, is the "Running low on
> memory". We keep adding memory to Accumulo but this is a blind guess and I
> was wondering if there is a way to understand how much memory Accumulo
> would need considering the amount of data that will be written. Should we
> write slowly to Accumulo to avoid this? What is filling all the memory at
> ingestion time?
> Secondly, we have these zookeeper session expired and other zookeeper
> timeouts. Zookeeper on our cluster works quite well, we have many systems
> using it. How can I debug a "zookeeper session expired" in Accumulo?
>
> Thanks,
> Mario
>
> --
> Mario Pastorelli | TERALYTICS
>
> *software engineer*
>
> Teralytics AG | Zollstrasse 62 | 8005 Zurich | Switzerland
> phone: +41794381682
> email: mario.pastorelli@teralytics.ch
> www.teralytics.net
>
> Company registration number: CH-020.3.037.709-7 | Trade register Canton
> Zurich
> Board of directors: Georg Polzer, Luciano Franceschina, Mark Schmitz, Yann
> de Vries
>
> This e-mail message contains confidential information which is for the
> sole attention and use of the intended recipient. Please notify us at once
> if you think that it may not be intended for you and delete it immediately.
>
Re: Running low on memory and Zookeeper Session expired / disconnected
Posted by Josh Elser <jo...@gmail.com>.
Hi Mario,
First off, look for any JVM GC pause warnings in the TabletServer logs.
There's a callback which should fire at a given interval and will warn
when it does not. Typically, this is because of a stop-the-world GC
cycle. These cycles also block the ZK heartbeat action as Dylan
described. They're the most common culprit.
Other related lower-lever configurations may be relevant:
* swappiness -- the OS swaps memory pages of the tserver to disk which
drastically increase the latency of the process. D
* I/O saturation - Dramatic pressure on the i/o subsystem can break the
system to its knees without ample memory headroom
* CPU saturation by other processes on the same physical host.
Other stuff is also covered
http://accumulo.apache.org/1.8/accumulo_user_manual.html#_keeping_the_tablet_server_lock
Also, if there are many services running on the same hosts that talk to
ZooKeeper, you might bump into ZooKeeper's rate-limiting via maxClientCnxns.
https://community.hortonworks.com/articles/51191/understanding-apache-zookeeper-connection-rate-lim.html
Regarding your first point, unless you're falling behind on minor
compactions, I wouldn't be too worried about the "running low on
memory". In sustained ingest situations, you *want* memory to be
saturated :). It's a hint as to the system's state if other things are
going wrong, but it's not directly a problem by itself.
Mario Pastorelli wrote:
> Hi all,
>
> I have two recurring errors with Accumulo in my cluster and I would like
> to know more about them. The first, usually happening at ingestion time
> when I write with the batch writers many records, is the "Running low on
> memory". We keep adding memory to Accumulo but this is a blind guess and
> I was wondering if there is a way to understand how much memory Accumulo
> would need considering the amount of data that will be written. Should
> we write slowly to Accumulo to avoid this? What is filling all the
> memory at ingestion time?
> Secondly, we have these zookeeper session expired and other zookeeper
> timeouts. Zookeeper on our cluster works quite well, we have many
> systems using it. How can I debug a "zookeeper session expired" in Accumulo?
>
> Thanks,
> Mario
>
> --
> Mario Pastorelli| TERALYTICS
>
> *software engineer*
>
> Teralytics AG | Zollstrasse 62 | 8005 Zurich | Switzerland
> phone:+41794381682
> email: mario.pastorelli@teralytics.ch
> <ma...@teralytics.ch>
> www.teralytics.net <http://www.teralytics.net/>
>
> Company registration number: CH-020.3.037.709-7 | Trade register Canton
> Zurich
> Board of directors: Georg Polzer, Luciano Franceschina, Mark Schmitz,
> Yann de Vries
>
> This e-mail message contains confidential information which is for the
> sole attention and use of the intended recipient. Please notify us at
> once if you think that it may not be intended for you and delete it
> immediately.
>