You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by "Whitmore, Mattie [USA]" <Wh...@bah.com> on 2015/09/04 16:57:49 UTC
Re: GitHTTP out of memory error
Hi,
I am 8 GetHTTP processors which run nightly, however 5 hours after these run some threads out of memory:
ERROR [Timer-Driven Process Thread-23] o.a.nifi.processors.standard.GetHTTP GetHTTP[id=021562be-b123-4b3f-b5ed-2b8c47af6d4c] GetHTTP[id=021562be-b123-4b3f-b5ed-2b8c47af6d4c] failed to process session due to java.lang.OutOfMemoryError: Java heap space: java.lang.OutOfMemoryError: Java heap space
2015-09-03 07:10:41,167 ERROR [Timer-Driven Process Thread-11] o.a.nifi.processors.standard.GetHTTP GetHTTP[id=0a5158d2-f082-4007-b646-3da8fb436fd0] GetHTTP[id=0a5158d2-f082-4007-b646-3da8fb436fd0] failed to process session due to java.lang.OutOfMemoryError: Java heap space: java.lang.OutOfMemoryError: Java heap space
2015-09-03 07:10:41,168 WARN [Timer-Driven Process Thread-23] o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding GetHTTP[id=021562be-b123-4b3f-b5ed-2b8c47af6d4c] due to uncaught Exception: java.lang.OutOfMemoryError: Java heap space
2015-09-03 07:10:41,168 WARN [Timer-Driven Process Thread-11] o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding GetHTTP[id=0a5158d2-f082-4007-b646-3da8fb436fd0] due to uncaught Exception: java.lang.OutOfMemoryError: Java heap space
I am running on Amazon Linux and have tuned the server as described in the admin page.
Thanks!
Mattie Whitmore
Lead Technologist
Booz | Allen | Hamilton
________________________
Re: [External] Re: GitHTTP out of memory error
Posted by Corey Flowers <cf...@onyxpoint.com>.
Really the setting of that depends on what you are trying to do in the flow
and what else runs on your systems. I have a cluster of 4 boxes that have
12GB of Ram and I set those to 4096. That cluster runs just fine and pushes
a few hundred gigs a day at about 30% system usage. Depending on what your
custom processors are doing with the heap, I would boost that up a bit. You
should also look at ganglia or something in that family to watch for memory
leaks in the custom code. NIFI has a really good ganglia reporting task to
show you what is happening at a system level within the heap.
On Fri, Sep 4, 2015 at 11:24 AM, Whitmore, Mattie [USA] <
Whitmore_Mattie@bah.com> wrote:
> Thanks for all the speedy replies!
>
>
> 1. I will up the heap currently it is set at: java.arg.3=-Xmx512m
> 2. I have 15 GB of memory (m3.xlarge)
> 3. I do have a number of custom processors in my workflow
>
> I’ll update ya’ll about how the increase in heap affects this issue. I
> am also working on serializing these HTTP calls into one processor, do
> ya’ll have any thoughts on that solution?
>
>
> *Mattie Whitmore*
>
>
> From: Corey Flowers
> Reply-To: "users@nifi.apache.org"
> Date: Friday, September 4, 2015 at 11:05 AM
> To: "users@nifi.apache.org"
> Subject: [External] Re: GitHTTP out of memory error
>
> What is your heap space set to and how much memory do you have on the box?
>
>
> On Fri, Sep 4, 2015 at 10:57 AM, Whitmore, Mattie [USA] <
> Whitmore_Mattie@bah.com> wrote:
>
>> Hi,
>>
>> I am 8 GetHTTP processors which run nightly, however 5 hours after these
>> run some threads out of memory:
>>
>> ERROR [Timer-Driven Process Thread-23]
>> o.a.nifi.processors.standard.GetHTTP
>> GetHTTP[id=021562be-b123-4b3f-b5ed-2b8c47af6d4c]
>> GetHTTP[id=021562be-b123-4b3f-b5ed-2b8c47af6d4c] failed to process session
>> due to java.lang.OutOfMemoryError: Java heap space:
>> java.lang.OutOfMemoryError: Java heap space
>>
>> 2015-09-03 07:10:41,167 ERROR [Timer-Driven Process Thread-11]
>> o.a.nifi.processors.standard.GetHTTP
>> GetHTTP[id=0a5158d2-f082-4007-b646-3da8fb436fd0]
>> GetHTTP[id=0a5158d2-f082-4007-b646-3da8fb436fd0] failed to process session
>> due to java.lang.OutOfMemoryError: Java heap space:
>> java.lang.OutOfMemoryError: Java heap space
>>
>> 2015-09-03 07:10:41,168 WARN [Timer-Driven Process Thread-23]
>> o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding
>> GetHTTP[id=021562be-b123-4b3f-b5ed-2b8c47af6d4c] due to uncaught Exception:
>> java.lang.OutOfMemoryError: Java heap space
>>
>> 2015-09-03 07:10:41,168 WARN [Timer-Driven Process Thread-11]
>> o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding
>> GetHTTP[id=0a5158d2-f082-4007-b646-3da8fb436fd0] due to uncaught Exception:
>> java.lang.OutOfMemoryError: Java heap space
>>
>> I am running on Amazon Linux and have tuned the server as described in
>> the admin page.
>>
>> Thanks!
>>
>> *Mattie Whitmore*
>> Lead Technologist
>>
>> Booz | Allen | Hamilton
>> *________________________*
>>
>>
>>
>
>
> --
> Corey Flowers
> Vice President, Onyx Point, Inc
> (410) 541-6699
> cflowers@onyxpoint.com
>
> -- This account not approved for unencrypted proprietary information --
>
--
Corey Flowers
Vice President, Onyx Point, Inc
(410) 541-6699
cflowers@onyxpoint.com
-- This account not approved for unencrypted proprietary information --
RE: [External] Re: GitHTTP out of memory error
Posted by Mark Payne <ma...@hotmail.com>.
Mattie,
Sounds good. If you increase the heap (I'd recommend 2 or 4 GB) and you are still running into OutOfMemoryError,
then it would certainly be hinting at a memory leak of some sort. Please let us know if increasing the heap
size helps!
Thanks
-Mark
________________________________
> From: Whitmore_Mattie@bah.com
> To: users@nifi.apache.org
> Subject: Re: [External] Re: GitHTTP out of memory error
> Date: Fri, 4 Sep 2015 15:24:22 +0000
>
> Thanks for all the speedy replies!
>
>
> 1. I will up the heap currently it is set at: java.arg.3=-Xmx512m
> 2. I have 15 GB of memory (m3.xlarge)
> 3. I do have a number of custom processors in my workflow
>
> I’ll update ya’ll about how the increase in heap affects this issue. I
> am also working on serializing these HTTP calls into one processor, do
> ya’ll have any thoughts on that solution?
>
>
> Mattie Whitmore
>
>
> From: Corey Flowers
> Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>"
> Date: Friday, September 4, 2015 at 11:05 AM
> To: "users@nifi.apache.org<ma...@nifi.apache.org>"
> Subject: [External] Re: GitHTTP out of memory error
>
> What is your heap space set to and how much memory do you have on the box?
>
>
> On Fri, Sep 4, 2015 at 10:57 AM, Whitmore, Mattie [USA]
> <Wh...@bah.com>> wrote:
> Hi,
>
> I am 8 GetHTTP processors which run nightly, however 5 hours after
> these run some threads out of memory:
>
>
> ERROR [Timer-Driven Process Thread-23]
> o.a.nifi.processors.standard.GetHTTP
> GetHTTP[id=021562be-b123-4b3f-b5ed-2b8c47af6d4c]
> GetHTTP[id=021562be-b123-4b3f-b5ed-2b8c47af6d4c] failed to process
> session due to java.lang.OutOfMemoryError: Java heap space:
> java.lang.OutOfMemoryError: Java heap space
>
> 2015-09-03 07:10:41,167 ERROR [Timer-Driven Process Thread-11]
> o.a.nifi.processors.standard.GetHTTP
> GetHTTP[id=0a5158d2-f082-4007-b646-3da8fb436fd0]
> GetHTTP[id=0a5158d2-f082-4007-b646-3da8fb436fd0] failed to process
> session due to java.lang.OutOfMemoryError: Java heap space:
> java.lang.OutOfMemoryError: Java heap space
>
> 2015-09-03 07:10:41,168 WARN [Timer-Driven Process Thread-23]
> o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding
> GetHTTP[id=021562be-b123-4b3f-b5ed-2b8c47af6d4c] due to uncaught
> Exception: java.lang.OutOfMemoryError: Java heap space
>
> 2015-09-03 07:10:41,168 WARN [Timer-Driven Process Thread-11]
> o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding
> GetHTTP[id=0a5158d2-f082-4007-b646-3da8fb436fd0] due to uncaught
> Exception: java.lang.OutOfMemoryError: Java heap space
>
> I am running on Amazon Linux and have tuned the server as described in
> the admin page.
>
> Thanks!
>
> Mattie Whitmore
> Lead Technologist
>
> Booz | Allen | Hamilton
> ________________________
>
>
>
>
>
> --
> Corey Flowers
> Vice President, Onyx Point, Inc
> (410) 541-6699
> cflowers@onyxpoint.com<ma...@onyxpoint.com>
>
> -- This account not approved for unencrypted proprietary information --
Re: [External] Re: GitHTTP out of memory error
Posted by "Whitmore, Mattie [USA]" <Wh...@bah.com>.
Thanks for all the speedy replies!
1. I will up the heap currently it is set at: java.arg.3=-Xmx512m
2. I have 15 GB of memory (m3.xlarge)
3. I do have a number of custom processors in my workflow
I’ll update ya’ll about how the increase in heap affects this issue. I am also working on serializing these HTTP calls into one processor, do ya’ll have any thoughts on that solution?
Mattie Whitmore
From: Corey Flowers
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>"
Date: Friday, September 4, 2015 at 11:05 AM
To: "users@nifi.apache.org<ma...@nifi.apache.org>"
Subject: [External] Re: GitHTTP out of memory error
What is your heap space set to and how much memory do you have on the box?
On Fri, Sep 4, 2015 at 10:57 AM, Whitmore, Mattie [USA] <Wh...@bah.com>> wrote:
Hi,
I am 8 GetHTTP processors which run nightly, however 5 hours after these run some threads out of memory:
ERROR [Timer-Driven Process Thread-23] o.a.nifi.processors.standard.GetHTTP GetHTTP[id=021562be-b123-4b3f-b5ed-2b8c47af6d4c] GetHTTP[id=021562be-b123-4b3f-b5ed-2b8c47af6d4c] failed to process session due to java.lang.OutOfMemoryError: Java heap space: java.lang.OutOfMemoryError: Java heap space
2015-09-03 07:10:41,167 ERROR [Timer-Driven Process Thread-11] o.a.nifi.processors.standard.GetHTTP GetHTTP[id=0a5158d2-f082-4007-b646-3da8fb436fd0] GetHTTP[id=0a5158d2-f082-4007-b646-3da8fb436fd0] failed to process session due to java.lang.OutOfMemoryError: Java heap space: java.lang.OutOfMemoryError: Java heap space
2015-09-03 07:10:41,168 WARN [Timer-Driven Process Thread-23] o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding GetHTTP[id=021562be-b123-4b3f-b5ed-2b8c47af6d4c] due to uncaught Exception: java.lang.OutOfMemoryError: Java heap space
2015-09-03 07:10:41,168 WARN [Timer-Driven Process Thread-11] o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding GetHTTP[id=0a5158d2-f082-4007-b646-3da8fb436fd0] due to uncaught Exception: java.lang.OutOfMemoryError: Java heap space
I am running on Amazon Linux and have tuned the server as described in the admin page.
Thanks!
Mattie Whitmore
Lead Technologist
Booz | Allen | Hamilton
________________________
--
Corey Flowers
Vice President, Onyx Point, Inc
(410) 541-6699
cflowers@onyxpoint.com<ma...@onyxpoint.com>
-- This account not approved for unencrypted proprietary information --
Re: GitHTTP out of memory error
Posted by Corey Flowers <cf...@onyxpoint.com>.
What is your heap space set to and how much memory do you have on the box?
On Fri, Sep 4, 2015 at 10:57 AM, Whitmore, Mattie [USA] <
Whitmore_Mattie@bah.com> wrote:
> Hi,
>
> I am 8 GetHTTP processors which run nightly, however 5 hours after these
> run some threads out of memory:
>
> ERROR [Timer-Driven Process Thread-23]
> o.a.nifi.processors.standard.GetHTTP
> GetHTTP[id=021562be-b123-4b3f-b5ed-2b8c47af6d4c]
> GetHTTP[id=021562be-b123-4b3f-b5ed-2b8c47af6d4c] failed to process session
> due to java.lang.OutOfMemoryError: Java heap space:
> java.lang.OutOfMemoryError: Java heap space
>
> 2015-09-03 07:10:41,167 ERROR [Timer-Driven Process Thread-11]
> o.a.nifi.processors.standard.GetHTTP
> GetHTTP[id=0a5158d2-f082-4007-b646-3da8fb436fd0]
> GetHTTP[id=0a5158d2-f082-4007-b646-3da8fb436fd0] failed to process session
> due to java.lang.OutOfMemoryError: Java heap space:
> java.lang.OutOfMemoryError: Java heap space
>
> 2015-09-03 07:10:41,168 WARN [Timer-Driven Process Thread-23]
> o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding
> GetHTTP[id=021562be-b123-4b3f-b5ed-2b8c47af6d4c] due to uncaught Exception:
> java.lang.OutOfMemoryError: Java heap space
>
> 2015-09-03 07:10:41,168 WARN [Timer-Driven Process Thread-11]
> o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding
> GetHTTP[id=0a5158d2-f082-4007-b646-3da8fb436fd0] due to uncaught Exception:
> java.lang.OutOfMemoryError: Java heap space
>
> I am running on Amazon Linux and have tuned the server as described in the
> admin page.
>
> Thanks!
>
> *Mattie Whitmore*
> Lead Technologist
>
> Booz | Allen | Hamilton
> *________________________*
>
>
>
--
Corey Flowers
Vice President, Onyx Point, Inc
(410) 541-6699
cflowers@onyxpoint.com
-- This account not approved for unencrypted proprietary information --
Re: GitHTTP out of memory error
Posted by Alan Jackoway <al...@cloudera.com>.
Is it always exactly 5 hours or somewhat variable? Is there anything
unusual about your data volume at that time?
You could just try increasing heap space in bootstrap.conf, which may fix
it depending on the underlying cause.
On Fri, Sep 4, 2015 at 4:57 PM, Whitmore, Mattie [USA] <
Whitmore_Mattie@bah.com> wrote:
> Hi,
>
> I am 8 GetHTTP processors which run nightly, however 5 hours after these
> run some threads out of memory:
>
> ERROR [Timer-Driven Process Thread-23]
> o.a.nifi.processors.standard.GetHTTP
> GetHTTP[id=021562be-b123-4b3f-b5ed-2b8c47af6d4c]
> GetHTTP[id=021562be-b123-4b3f-b5ed-2b8c47af6d4c] failed to process session
> due to java.lang.OutOfMemoryError: Java heap space:
> java.lang.OutOfMemoryError: Java heap space
>
> 2015-09-03 07:10:41,167 ERROR [Timer-Driven Process Thread-11]
> o.a.nifi.processors.standard.GetHTTP
> GetHTTP[id=0a5158d2-f082-4007-b646-3da8fb436fd0]
> GetHTTP[id=0a5158d2-f082-4007-b646-3da8fb436fd0] failed to process session
> due to java.lang.OutOfMemoryError: Java heap space:
> java.lang.OutOfMemoryError: Java heap space
>
> 2015-09-03 07:10:41,168 WARN [Timer-Driven Process Thread-23]
> o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding
> GetHTTP[id=021562be-b123-4b3f-b5ed-2b8c47af6d4c] due to uncaught Exception:
> java.lang.OutOfMemoryError: Java heap space
>
> 2015-09-03 07:10:41,168 WARN [Timer-Driven Process Thread-11]
> o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding
> GetHTTP[id=0a5158d2-f082-4007-b646-3da8fb436fd0] due to uncaught Exception:
> java.lang.OutOfMemoryError: Java heap space
>
> I am running on Amazon Linux and have tuned the server as described in the
> admin page.
>
> Thanks!
>
> *Mattie Whitmore*
> Lead Technologist
>
> Booz | Allen | Hamilton
> *________________________*
>
>
>
RE: GitHTTP out of memory error
Posted by Mark Payne <ma...@hotmail.com>.
Mattie,
How large is the Java heap? This value is in the conf/bootstrap.conf file. You'll typically see something like:
java.arg.3=-Xmx512m
Do you have any custom processors running in your flow, or are you using only Apache NiFi provided processors?
Thanks
-Mark
________________________________
> From: Whitmore_Mattie@bah.com
> To: users@nifi.apache.org
> Subject: Re: GitHTTP out of memory error
> Date: Fri, 4 Sep 2015 14:57:49 +0000
>
> Hi,
>
> I am 8 GetHTTP processors which run nightly, however 5 hours after
> these run some threads out of memory:
>
>
> ERROR [Timer-Driven Process Thread-23]
> o.a.nifi.processors.standard.GetHTTP
> GetHTTP[id=021562be-b123-4b3f-b5ed-2b8c47af6d4c]
> GetHTTP[id=021562be-b123-4b3f-b5ed-2b8c47af6d4c] failed to process
> session due to java.lang.OutOfMemoryError: Java heap space:
> java.lang.OutOfMemoryError: Java heap space
>
> 2015-09-03 07:10:41,167 ERROR [Timer-Driven Process Thread-11]
> o.a.nifi.processors.standard.GetHTTP
> GetHTTP[id=0a5158d2-f082-4007-b646-3da8fb436fd0]
> GetHTTP[id=0a5158d2-f082-4007-b646-3da8fb436fd0] failed to process
> session due to java.lang.OutOfMemoryError: Java heap space:
> java.lang.OutOfMemoryError: Java heap space
>
> 2015-09-03 07:10:41,168 WARN [Timer-Driven Process Thread-23]
> o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding
> GetHTTP[id=021562be-b123-4b3f-b5ed-2b8c47af6d4c] due to uncaught
> Exception: java.lang.OutOfMemoryError: Java heap space
>
> 2015-09-03 07:10:41,168 WARN [Timer-Driven Process Thread-11]
> o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding
> GetHTTP[id=0a5158d2-f082-4007-b646-3da8fb436fd0] due to uncaught
> Exception: java.lang.OutOfMemoryError: Java heap space
>
> I am running on Amazon Linux and have tuned the server as described in
> the admin page.
>
> Thanks!
>
> Mattie Whitmore
> Lead Technologist
>
> Booz | Allen | Hamilton
> ________________________
>
>