Issue with Enrichment topology: java.lang.OutOfMemoryError: GC overhead limit exceeded

[Anil Donthireddy <An...@sstech.us>]

Hi,

We have been keep on getting the error "java.lang.OutOfMemoryError: GC overhead limit exceeded" at Enrichment topology in storm at several bolts defined. Please find the attached screenshot which shows the bolts in topology at which we are getting the same issue.

I tried out couple of configuration changes to provide more RAM to storm topologies from ambary UI and restarted Storm but it dint help. Can I get some configuration steps to assign more RAM to storm topologies to resolve the issue.

I see in the logs that for each record, it is trying to update the GeoIP data as below. I wonder if it is causing the issue.
o.a.m.e.a.g.GeoLiteDatabase Thread-16-geoEnrichmentBolt-executor[6 6] [INFO] [Metron] Update to GeoIP data started with /apps/metron/geo/default/GeoLite2-City.mmdb.gz

Note: I am facing the issue from the time I implemented the geo alerts rule as specified in the link<https://www.slideshare.net/Hadoop_Summit/just-the-sketch-advanced-streaming-analytics-in-apache-metron>.

Thanks,
Anil.

Re: Issue with Enrichment topology: java.lang.OutOfMemoryError: GC overhead limit exceeded

[Casey Stella <ce...@gmail.com>]

What version of Metron are you using?

On Tue, Aug 21, 2018 at 1:46 PM Otto Fowler <ot...@gmail.com> wrote:

> So, before you where doing GEO you did not have the problem?  If you took
> the GEO out it would stop?
>
>
> On August 21, 2018 at 11:04:56, Anil Donthireddy (
> anil.donthireddy@sstech.us) wrote:
>
> Hi,
>
>
>
> We have been keep on getting the error “java.lang.OutOfMemoryError: GC
> overhead limit exceeded” at Enrichment topology in storm at several bolts
> defined. Please find the attached screenshot which shows the bolts in
> topology at which we are getting the same issue.
>
>
>
> I tried out couple of configuration changes to provide more RAM to storm
> topologies from ambary UI and restarted Storm but it dint help. Can I get
> some configuration steps to assign more RAM to storm topologies to resolve
> the issue.
>
>
>
> I see in the logs that for each record, it is trying to update the GeoIP
> data as below. I wonder if it is causing the issue.
>
> o.a.m.e.a.g.GeoLiteDatabase Thread-16-geoEnrichmentBolt-executor[6 6]
> [INFO] [Metron] Update to GeoIP data started with
> /apps/metron/geo/default/GeoLite2-City.mmdb.gz
>
>
>
> Note: I am facing the issue from the time I implemented the geo alerts
> rule as specified in the link
> <https://www.slideshare.net/Hadoop_Summit/just-the-sketch-advanced-streaming-analytics-in-apache-metron>.
>
>
>
>
> Thanks,
>
> Anil.
>
>

Re: Issue with Enrichment topology: java.lang.OutOfMemoryError: GC overhead limit exceeded

[Otto Fowler <ot...@gmail.com>]

So, before you where doing GEO you did not have the problem?  If you took
the GEO out it would stop?


On August 21, 2018 at 11:04:56, Anil Donthireddy (anil.donthireddy@sstech.us)
wrote:

Hi,



We have been keep on getting the error “java.lang.OutOfMemoryError: GC
overhead limit exceeded” at Enrichment topology in storm at several bolts
defined. Please find the attached screenshot which shows the bolts in
topology at which we are getting the same issue.



I tried out couple of configuration changes to provide more RAM to storm
topologies from ambary UI and restarted Storm but it dint help. Can I get
some configuration steps to assign more RAM to storm topologies to resolve
the issue.



I see in the logs that for each record, it is trying to update the GeoIP
data as below. I wonder if it is causing the issue.

o.a.m.e.a.g.GeoLiteDatabase Thread-16-geoEnrichmentBolt-executor[6 6]
[INFO] [Metron] Update to GeoIP data started with
/apps/metron/geo/default/GeoLite2-City.mmdb.gz



Note: I am facing the issue from the time I implemented the geo alerts rule
as specified in the link
<https://www.slideshare.net/Hadoop_Summit/just-the-sketch-advanced-streaming-analytics-in-apache-metron>.




Thanks,

Anil.