You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Sai Dilip Reddy Kiralam <dk...@aadhya-analytics.com> on 2016/03/23 12:07:32 UTC
java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf
exception in kafka-storm spout?
Hi,
When I Increase the kafkaconfig.fetchSizeBytes value to 150*1024*1024,I get
desired output from kafka-spout but after sometime kafka spout is throwing
the exception and my spout is failing.What should I do to handle this.Do I
need to add any lines in storm.yaml to handle this exception.
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3332)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:622)
at java.lang.StringBuilder.append(StringBuilder.java:202)
at java.util.AbstractCollection.toString(AbstractCollection.java:464)
at clojure.core$str.invoke(core.clj:513)
at clojure.core$str$fn__3896.invoke(core.clj:517)
at clojure.core$str.doInvoke(core.clj:519)
at clojure.lang.RestFn.invoke(RestFn.java:516)
at backtype.storm.daemon.task$mk_tasks_fn$fn__6307.invoke(task.clj:152)
at backtype.storm.daemon.executor$fn__6579$fn__6594$send_spout_msg__6612.invoke(executor.clj:482)
at backtype.storm.daemon.executor$fn__6579$fn$reify__6621.emit(executor.clj:526)
at backtype.storm.spout.SpoutOutputCollector.emit(SpoutOutputCollector.java:49)
at backtype.storm.spout.SpoutOutputCollector.emit(SpoutOutputCollector.java:63)
at storm.kafka.PartitionManager.next(PartitionManager.java:141)
at storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:141)
at backtype.storm.daemon.executor$fn__6579$fn__6594$fn__6623.invoke(executor.clj:565)
at backtype.storm.util$async_loop$fn__459.invoke(util.clj:463)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:745)
Need suggestions
*Best regards,*
*K.Sai Dilip Reddy.*
Re: java.lang.OutOfMemoryError: Java heap space at
java.util.Arrays.copyOf exception in kafka-storm spout?
Posted by Rajasekhar <ra...@gmail.com>.
Hi Dilip,
you need to have proper set of configs to set this. you can not have too
much data from kafka at one shot where u cant keep in jvm. So you should
figure out the magic expression that should work as per your
cluster/hardware available.
if you increase worker jvm then you cannot have too many workers for that
supervisor. which means no parallel processing.
So, please keep low and process it faster with multi worker instances.
relatively, low throughout and low latency.
On Thu, Mar 24, 2016 at 10:40 AM, Tech Id <te...@gmail.com> wrote:
> You need to increase worker's Heap Size by adding the following in each
> worker's storm.yaml file:
>
> worker.childopts: "-Xmx16g"
>
> Then restart each worker's supervisor.
>
>
>
> On Wed, Mar 23, 2016 at 4:07 AM, Sai Dilip Reddy Kiralam <
> dkiralam@aadhya-analytics.com> wrote:
>
>> Hi,
>>
>>
>>
>> When I Increase the kafkaconfig.fetchSizeBytes value to 150*1024*1024,I
>> get desired output from kafka-spout but after sometime kafka spout is
>> throwing the exception and my spout is failing.What should I do to handle
>> this.Do I need to add any lines in storm.yaml to handle this exception.
>>
>> java.lang.OutOfMemoryError: Java heap space
>> at java.util.Arrays.copyOf(Arrays.java:3332)
>> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
>> at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
>> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:622)
>> at java.lang.StringBuilder.append(StringBuilder.java:202)
>> at java.util.AbstractCollection.toString(AbstractCollection.java:464)
>> at clojure.core$str.invoke(core.clj:513)
>> at clojure.core$str$fn__3896.invoke(core.clj:517)
>> at clojure.core$str.doInvoke(core.clj:519)
>> at clojure.lang.RestFn.invoke(RestFn.java:516)
>> at backtype.storm.daemon.task$mk_tasks_fn$fn__6307.invoke(task.clj:152)
>> at backtype.storm.daemon.executor$fn__6579$fn__6594$send_spout_msg__6612.invoke(executor.clj:482)
>> at backtype.storm.daemon.executor$fn__6579$fn$reify__6621.emit(executor.clj:526)
>> at backtype.storm.spout.SpoutOutputCollector.emit(SpoutOutputCollector.java:49)
>> at backtype.storm.spout.SpoutOutputCollector.emit(SpoutOutputCollector.java:63)
>> at storm.kafka.PartitionManager.next(PartitionManager.java:141)
>> at storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:141)
>> at backtype.storm.daemon.executor$fn__6579$fn__6594$fn__6623.invoke(executor.clj:565)
>> at backtype.storm.util$async_loop$fn__459.invoke(util.clj:463)
>> at clojure.lang.AFn.run(AFn.java:24)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> Need suggestions
>>
>>
>>
>>
>> *Best regards,*
>> *K.Sai Dilip Reddy.*
>>
>
>
--
Thanks & Regards
Rajasekhar
Re: java.lang.OutOfMemoryError: Java heap space at
java.util.Arrays.copyOf exception in kafka-storm spout?
Posted by Tech Id <te...@gmail.com>.
You need to increase worker's Heap Size by adding the following in each
worker's storm.yaml file:
worker.childopts: "-Xmx16g"
Then restart each worker's supervisor.
On Wed, Mar 23, 2016 at 4:07 AM, Sai Dilip Reddy Kiralam <
dkiralam@aadhya-analytics.com> wrote:
> Hi,
>
>
>
> When I Increase the kafkaconfig.fetchSizeBytes value to 150*1024*1024,I
> get desired output from kafka-spout but after sometime kafka spout is
> throwing the exception and my spout is failing.What should I do to handle
> this.Do I need to add any lines in storm.yaml to handle this exception.
>
> java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:3332)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
> at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:622)
> at java.lang.StringBuilder.append(StringBuilder.java:202)
> at java.util.AbstractCollection.toString(AbstractCollection.java:464)
> at clojure.core$str.invoke(core.clj:513)
> at clojure.core$str$fn__3896.invoke(core.clj:517)
> at clojure.core$str.doInvoke(core.clj:519)
> at clojure.lang.RestFn.invoke(RestFn.java:516)
> at backtype.storm.daemon.task$mk_tasks_fn$fn__6307.invoke(task.clj:152)
> at backtype.storm.daemon.executor$fn__6579$fn__6594$send_spout_msg__6612.invoke(executor.clj:482)
> at backtype.storm.daemon.executor$fn__6579$fn$reify__6621.emit(executor.clj:526)
> at backtype.storm.spout.SpoutOutputCollector.emit(SpoutOutputCollector.java:49)
> at backtype.storm.spout.SpoutOutputCollector.emit(SpoutOutputCollector.java:63)
> at storm.kafka.PartitionManager.next(PartitionManager.java:141)
> at storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:141)
> at backtype.storm.daemon.executor$fn__6579$fn__6594$fn__6623.invoke(executor.clj:565)
> at backtype.storm.util$async_loop$fn__459.invoke(util.clj:463)
> at clojure.lang.AFn.run(AFn.java:24)
> at java.lang.Thread.run(Thread.java:745)
>
> Need suggestions
>
>
>
>
> *Best regards,*
> *K.Sai Dilip Reddy.*
>