You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@accumulo.apache.org by "Tran, An - 0553 - MITLL" <at...@ll.mit.edu> on 2013/11/20 01:10:57 UTC
TabletServerBatchReaderIterator Java heap space error
Hi,
I am currently getting TabletServerBatchReaderIterator Java heap space errors. Below are the log outputs. Do any of you know what is going on and what is the proper solution to fix this issue?
A little detail on the use case is I have an application which uses a thread pool that does concurrent queries on Accumulo. Any insight on this issue is greatly appreciated .
Thanks.
19 Nov 2013 14:13:06,845 52142 [batch scanner 7-7 looking up 4 ranges at 192.168.1.66:9997] WARN TabletServerBatchReaderIterator run
Java heap space
java.lang.OutOfMemoryError: Java heap space
19 Nov 2013 14:13:06,845 52142 [batch scanner 7-6 looking up 5 ranges at 192.168.1.68:9997] WARN TabletServerBatchReaderIterator run
Java heap space
java.lang.OutOfMemoryError: Java heap space
19 Nov 2013 14:13:06,844 52141 [batch scanner 7-3 looking up 3 ranges at 192.168.1.69:9997] WARN TabletServerBatchReaderIterator run
Java heap space
java.lang.OutOfMemoryError: Java heap space
19 Nov 2013 14:13:06,844 52141 [batch scanner 7-2 looking up 3 ranges at 192.168.1.63:9997] WARN TabletServerBatchReaderIterator run
Java heap space
java.lang.OutOfMemoryError: Java heap space
19 Nov 2013 14:13:06,843 52140 [batch scanner 6-1 looking up 5 ranges at 192.168.1.67:9997] WARN TabletServerBatchReaderIterator run
Java heap space
java.lang.OutOfMemoryError: Java heap space
19 Nov 2013 14:13:06,846 52143 [batch scanner 7-4 looking up 7 ranges at 192.168.1.67:9997] WARN TabletServerBatchReaderIterator run
Java heap space
java.lang.OutOfMemoryError: Java heap space
19 Nov 2013 14:13:06,846 52143 [batch scanner 6-3 looking up 5 ranges at 192.168.1.64:9997] WARN TabletServerBatchReaderIterator run
Java heap space
ava.lang.OutOfMemoryError: Java heap space
at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.doLookup(TabletServerBatchReaderIterator.java:564)
at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator$QueryTask.run(TabletServerBatchReaderIterator.java:338)
at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Re: TabletServerBatchReaderIterator Java heap space error
Posted by Josh Elser <jo...@gmail.com>.
No, the shared buffer on the BatchScanner is on the instance, not a
static member across all instances.
On 11/19/2013 07:41 PM, Tran, An - 0553 - MITLL wrote:
> Hi Josh,
>
> That was very helpful. If I have each of my thread in my thread pool
> create an Accumulo connector prior to creating a batch scanner will each
> of those thread still share the same buffer?
>
> Thanks,
>
> -An
>
> On 11/19/13 7:37 PM, "Josh Elser" <jo...@gmail.com> wrote:
>
>> An,
>>
>> The BatchScanner will query multiple servers at once, up to the
>> numThreads argument you specified when invoking createBatchScanner from
>> a Connector.
>>
>> Each of these connections will write data into a shared buffer in
>> TabletServerBatchReaderIterator. Each server the BatchScanner is talking
>> to will return up to the configured table property
>> `table.scan.max.memory` (default 512k) amount of Key-Value pairs.
>> However, you'll probably not see each thread always returning the full
>> amount as the tabletserver will time-slice your query to ensure that
>> your one query thread does not consume all resources.
>>
>> In short, make you have sufficient JVM heap overhead for the value of
>> `table.scan.max.memory` of the table you're querying (`config -t
>> 'yourTableName'` in the Accumulo shell) multiplied by the number of
>> threads you're using with the BatchScanner multiplied by two. Remember
>> while you are processing the Key-Value pairs that you queried from
>> Accumulo, it will already be fetching the next batch.
>>
>> - Josh
>>
>> On 11/19/2013 07:10 PM, Tran, An - 0553 - MITLL wrote:
>>> Hi,
>>>
>>> I am currently getting TabletServerBatchReaderIterator Java heap space
>>> errors. Below are the log outputs. Do any of you know what is going on
>>> and what is the proper solution to fix this issue?
>>> A little detail on the use case is I have an application which uses a
>>> thread pool that does concurrent queries on Accumulo. Any insight on
>>> this issue is greatly appreciated .
>>>
>>> Thanks.
>>>
>>> 19 Nov 2013 14:13:06,845 52142 [batch scanner 7-7 looking up 4 ranges
>>> at 192.168.1.66:9997] WARN TabletServerBatchReaderIterator run
>>> Java heap space
>>> java.lang.OutOfMemoryError: Java heap space
>>> 19 Nov 2013 14:13:06,845 52142 [batch scanner 7-6 looking up 5 ranges
>>> at 192.168.1.68:9997] WARN TabletServerBatchReaderIterator run
>>> Java heap space
>>> java.lang.OutOfMemoryError: Java heap space
>>> 19 Nov 2013 14:13:06,844 52141 [batch scanner 7-3 looking up 3 ranges
>>> at 192.168.1.69:9997] WARN TabletServerBatchReaderIterator run
>>> Java heap space
>>> java.lang.OutOfMemoryError: Java heap space
>>> 19 Nov 2013 14:13:06,844 52141 [batch scanner 7-2 looking up 3 ranges
>>> at 192.168.1.63:9997] WARN TabletServerBatchReaderIterator run
>>> Java heap space
>>> java.lang.OutOfMemoryError: Java heap space
>>> 19 Nov 2013 14:13:06,843 52140 [batch scanner 6-1 looking up 5 ranges
>>> at 192.168.1.67:9997] WARN TabletServerBatchReaderIterator run
>>> Java heap space
>>> java.lang.OutOfMemoryError: Java heap space
>>> 19 Nov 2013 14:13:06,846 52143 [batch scanner 7-4 looking up 7 ranges
>>> at 192.168.1.67:9997] WARN TabletServerBatchReaderIterator run
>>> Java heap space
>>> java.lang.OutOfMemoryError: Java heap space
>>> 19 Nov 2013 14:13:06,846 52143 [batch scanner 6-3 looking up 5 ranges
>>> at 192.168.1.64:9997] WARN TabletServerBatchReaderIterator run
>>> Java heap space
>>> ava.lang.OutOfMemoryError: Java heap space
>>> at
>>> org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.doLo
>>> okup(TabletServerBatchReaderIterator.java:564)
>>> at
>>> org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator$Quer
>>> yTask.run(TabletServerBatchReaderIterator.java:338)
>>> at
>>> org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable
>>> .java:47)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java
>>> :1145)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav
>>> a:615)
>>> at java.lang.Thread.run(Thread.java:724)
>>>
>>>
Re: TabletServerBatchReaderIterator Java heap space error
Posted by "Tran, An - 0553 - MITLL" <at...@ll.mit.edu>.
Hi Josh,
That was very helpful. If I have each of my thread in my thread pool
create an Accumulo connector prior to creating a batch scanner will each
of those thread still share the same buffer?
Thanks,
-An
On 11/19/13 7:37 PM, "Josh Elser" <jo...@gmail.com> wrote:
>An,
>
>The BatchScanner will query multiple servers at once, up to the
>numThreads argument you specified when invoking createBatchScanner from
>a Connector.
>
>Each of these connections will write data into a shared buffer in
>TabletServerBatchReaderIterator. Each server the BatchScanner is talking
>to will return up to the configured table property
>`table.scan.max.memory` (default 512k) amount of Key-Value pairs.
>However, you'll probably not see each thread always returning the full
>amount as the tabletserver will time-slice your query to ensure that
>your one query thread does not consume all resources.
>
>In short, make you have sufficient JVM heap overhead for the value of
>`table.scan.max.memory` of the table you're querying (`config -t
>'yourTableName'` in the Accumulo shell) multiplied by the number of
>threads you're using with the BatchScanner multiplied by two. Remember
>while you are processing the Key-Value pairs that you queried from
>Accumulo, it will already be fetching the next batch.
>
>- Josh
>
>On 11/19/2013 07:10 PM, Tran, An - 0553 - MITLL wrote:
>> Hi,
>>
>> I am currently getting TabletServerBatchReaderIterator Java heap space
>>errors. Below are the log outputs. Do any of you know what is going on
>>and what is the proper solution to fix this issue?
>> A little detail on the use case is I have an application which uses a
>>thread pool that does concurrent queries on Accumulo. Any insight on
>>this issue is greatly appreciated .
>>
>> Thanks.
>>
>> 19 Nov 2013 14:13:06,845 52142 [batch scanner 7-7 looking up 4 ranges
>>at 192.168.1.66:9997] WARN TabletServerBatchReaderIterator run
>> Java heap space
>> java.lang.OutOfMemoryError: Java heap space
>> 19 Nov 2013 14:13:06,845 52142 [batch scanner 7-6 looking up 5 ranges
>>at 192.168.1.68:9997] WARN TabletServerBatchReaderIterator run
>> Java heap space
>> java.lang.OutOfMemoryError: Java heap space
>> 19 Nov 2013 14:13:06,844 52141 [batch scanner 7-3 looking up 3 ranges
>>at 192.168.1.69:9997] WARN TabletServerBatchReaderIterator run
>> Java heap space
>> java.lang.OutOfMemoryError: Java heap space
>> 19 Nov 2013 14:13:06,844 52141 [batch scanner 7-2 looking up 3 ranges
>>at 192.168.1.63:9997] WARN TabletServerBatchReaderIterator run
>> Java heap space
>> java.lang.OutOfMemoryError: Java heap space
>> 19 Nov 2013 14:13:06,843 52140 [batch scanner 6-1 looking up 5 ranges
>>at 192.168.1.67:9997] WARN TabletServerBatchReaderIterator run
>> Java heap space
>> java.lang.OutOfMemoryError: Java heap space
>> 19 Nov 2013 14:13:06,846 52143 [batch scanner 7-4 looking up 7 ranges
>>at 192.168.1.67:9997] WARN TabletServerBatchReaderIterator run
>> Java heap space
>> java.lang.OutOfMemoryError: Java heap space
>> 19 Nov 2013 14:13:06,846 52143 [batch scanner 6-3 looking up 5 ranges
>>at 192.168.1.64:9997] WARN TabletServerBatchReaderIterator run
>> Java heap space
>> ava.lang.OutOfMemoryError: Java heap space
>> at
>>org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.doLo
>>okup(TabletServerBatchReaderIterator.java:564)
>> at
>>org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator$Quer
>>yTask.run(TabletServerBatchReaderIterator.java:338)
>> at
>>org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable
>>.java:47)
>> at
>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java
>>:1145)
>> at
>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav
>>a:615)
>> at java.lang.Thread.run(Thread.java:724)
>>
>>
>
Re: TabletServerBatchReaderIterator Java heap space error
Posted by Josh Elser <jo...@gmail.com>.
An,
The BatchScanner will query multiple servers at once, up to the
numThreads argument you specified when invoking createBatchScanner from
a Connector.
Each of these connections will write data into a shared buffer in
TabletServerBatchReaderIterator. Each server the BatchScanner is talking
to will return up to the configured table property
`table.scan.max.memory` (default 512k) amount of Key-Value pairs.
However, you'll probably not see each thread always returning the full
amount as the tabletserver will time-slice your query to ensure that
your one query thread does not consume all resources.
In short, make you have sufficient JVM heap overhead for the value of
`table.scan.max.memory` of the table you're querying (`config -t
'yourTableName'` in the Accumulo shell) multiplied by the number of
threads you're using with the BatchScanner multiplied by two. Remember
while you are processing the Key-Value pairs that you queried from
Accumulo, it will already be fetching the next batch.
- Josh
On 11/19/2013 07:10 PM, Tran, An - 0553 - MITLL wrote:
> Hi,
>
> I am currently getting TabletServerBatchReaderIterator Java heap space errors. Below are the log outputs. Do any of you know what is going on and what is the proper solution to fix this issue?
> A little detail on the use case is I have an application which uses a thread pool that does concurrent queries on Accumulo. Any insight on this issue is greatly appreciated .
>
> Thanks.
>
> 19 Nov 2013 14:13:06,845 52142 [batch scanner 7-7 looking up 4 ranges at 192.168.1.66:9997] WARN TabletServerBatchReaderIterator run
> Java heap space
> java.lang.OutOfMemoryError: Java heap space
> 19 Nov 2013 14:13:06,845 52142 [batch scanner 7-6 looking up 5 ranges at 192.168.1.68:9997] WARN TabletServerBatchReaderIterator run
> Java heap space
> java.lang.OutOfMemoryError: Java heap space
> 19 Nov 2013 14:13:06,844 52141 [batch scanner 7-3 looking up 3 ranges at 192.168.1.69:9997] WARN TabletServerBatchReaderIterator run
> Java heap space
> java.lang.OutOfMemoryError: Java heap space
> 19 Nov 2013 14:13:06,844 52141 [batch scanner 7-2 looking up 3 ranges at 192.168.1.63:9997] WARN TabletServerBatchReaderIterator run
> Java heap space
> java.lang.OutOfMemoryError: Java heap space
> 19 Nov 2013 14:13:06,843 52140 [batch scanner 6-1 looking up 5 ranges at 192.168.1.67:9997] WARN TabletServerBatchReaderIterator run
> Java heap space
> java.lang.OutOfMemoryError: Java heap space
> 19 Nov 2013 14:13:06,846 52143 [batch scanner 7-4 looking up 7 ranges at 192.168.1.67:9997] WARN TabletServerBatchReaderIterator run
> Java heap space
> java.lang.OutOfMemoryError: Java heap space
> 19 Nov 2013 14:13:06,846 52143 [batch scanner 6-3 looking up 5 ranges at 192.168.1.64:9997] WARN TabletServerBatchReaderIterator run
> Java heap space
> ava.lang.OutOfMemoryError: Java heap space
> at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.doLookup(TabletServerBatchReaderIterator.java:564)
> at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator$QueryTask.run(TabletServerBatchReaderIterator.java:338)
> at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
>
>