You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by "Adams, Jeremiah" <je...@pearson.com> on 2015/07/29 00:47:58 UTC
Having Issues with MergeContent Processor
I am running into memory issues when using MergeContent. I am seeing
OutofMemoryErrors. I am using 0.1.0.
I am processing a kafka queue via GetKafka, some custom Processors and
PutKafka. I have a PutKafka Controller just prior to the MergeContent
Processor. I am passing JSON messages through the FlowFiles.
I am looking to merge the flow files until reaching a minimum size or a
time period elapses, then write them to S3. I get the memory issue with
just 15-20 messages. The MergeContent Processor continues to update the
Tasks in Tasks/Time even when not sending messages to the flow.
Here are my settings for the MergeContent Processor:
Merge Strategy: Bin-Packing Algo
Merge Format: Binary Concatenation
Attribute Strategy: Keep only Common Attributes
Correlation Attribute Name: No Value Set
Minimum Number of Entries: 1
Maximum Number of Entries: No Value Set
Minimum Group Size: 1KB
Maximum Group Size: 2 MB
Max Bin Age: 10 m
Maximum number of Bins: 100
Header File: No Value Set
Footer File: No Value Set
Demarcator File: No Value Set
Compression Level: 1
Keep Path: False
2015-07-28 14:25:17,580 WARN [NiFi Web Server-22]
org.eclipse.jetty.servlet.ServletHandler Error for
/nifi-api/controller/process-groups/caffa1c4-ef49-4f0c-b144-0b5d47264481/status
java.lang.OutOfMemoryError: PermGen space
2015-07-28 14:25:17,811 WARN [NiFi Web Server-22]
org.eclipse.jetty.server.HttpChannel
/nifi-api/controller/process-groups/caffa1c4-ef49-4f0c-b144-0b5d47264481/status?recursive=false
java.lang.OutOfMemoryError: PermGen space
2015-07-28 14:25:18,020 WARN [NiFi Web Server-22]
o.e.jetty.util.thread.QueuedThreadPool
java.lang.OutOfMemoryError: PermGen space
2015-07-28 14:25:18,020 WARN [NiFi Web Server-22]
o.e.jetty.util.thread.QueuedThreadPool Unexpected thread death:
org.eclipse.jetty.util.thread.QueuedThreadPool$3@1a4b0747 in NiFi Web
Server{STARTED,8<=9<=200,i=0,q=4}
2015-07-28 14:25:18,775 WARN [NiFi Web Server-23]
org.eclipse.jetty.servlet.ServletHandler Error for
/nifi-api/controller/process-groups/caffa1c4-ef49-4f0c-b144-0b5d47264481/status
java.lang.OutOfMemoryError: PermGen space
2015-07-28 14:25:19,005 WARN [NiFi Web Server-23]
org.eclipse.jetty.server.HttpChannel
/nifi-api/controller/process-groups/caffa1c4-ef49-4f0c-b144-0b5d47264481/status?recursive=false
java.lang.OutOfMemoryError: PermGen space
2015-07-28 14:25:19,240 WARN [NiFi Web Server-23]
o.e.jetty.util.thread.QueuedThreadPool
java.lang.OutOfMemoryError: PermGen space
2015-07-28 14:25:19,240 WARN [NiFi Web Server-23]
o.e.jetty.util.thread.QueuedThreadPool Unexpected thread death:
org.eclipse.jetty.util.thread.QueuedThreadPool$3@1a4b0747 in NiFi Web
Server{STARTED,8<=9<=200,i=0,q=4}
*Jeremiah Adams*
Senior Software Developer
Pearson
2154 East Commons Ave.
Suite 400
Centennial, CO 80122
Always Learning
Learn more at www.pearson.com
Re: Having Issues with MergeContent Processor
Posted by "Adams, Jeremiah" <je...@pearson.com>.
Awesome! Thanks Joe.
*Jeremiah Adams*
Senior Software Developer
Pearson
2154 East Commons Ave.
Suite 400
Centennial, CO 80122
Always Learning
Learn more at www.pearson.com
On Tue, Jul 28, 2015 at 4:52 PM, Joe Witt <jo...@gmail.com> wrote:
> You'll want to uncomment both of these lines actually:
>
> #java.arg.11=-XX:PermSize=128M
> #java.arg.12=-XX:MaxPermSize=128M
>
> And if you like you can make them a bit larger.
>
> Thanks
> Joe
>
> On Tue, Jul 28, 2015 at 6:52 PM, Joe Witt <jo...@gmail.com> wrote:
> > Jeremiah,
> >
> > In this case it is talking about permgen (Java 7 and where classes go).
> >
> > If you take a look at the './conf/bootstap.conf' file there is a line
> that says:
> >
> > #java.arg.11=-XX:PermSize=128M
> >
> > Uncomment that and you should be good to go. We need more than the
> > typical Java 7 default for PermGen space.
> >
> > Of note the MergeContent processor is extremely memory efficient and
> > pretty much a non-factor on the heap as it never actually loads
> > content into memory. Very different than most systems would be.
> >
> > Thanks
> > Joe
> >
> > On Tue, Jul 28, 2015 at 6:47 PM, Adams, Jeremiah
> > <je...@pearson.com> wrote:
> >> I am running into memory issues when using MergeContent. I am seeing
> >> OutofMemoryErrors. I am using 0.1.0.
> >>
> >> I am processing a kafka queue via GetKafka, some custom Processors and
> >> PutKafka. I have a PutKafka Controller just prior to the MergeContent
> >> Processor. I am passing JSON messages through the FlowFiles.
> >>
> >> I am looking to merge the flow files until reaching a minimum size or a
> >> time period elapses, then write them to S3. I get the memory issue with
> >> just 15-20 messages. The MergeContent Processor continues to update the
> >> Tasks in Tasks/Time even when not sending messages to the flow.
> >>
> >> Here are my settings for the MergeContent Processor:
> >> Merge Strategy: Bin-Packing Algo
> >> Merge Format: Binary Concatenation
> >> Attribute Strategy: Keep only Common Attributes
> >> Correlation Attribute Name: No Value Set
> >> Minimum Number of Entries: 1
> >> Maximum Number of Entries: No Value Set
> >> Minimum Group Size: 1KB
> >> Maximum Group Size: 2 MB
> >> Max Bin Age: 10 m
> >> Maximum number of Bins: 100
> >> Header File: No Value Set
> >> Footer File: No Value Set
> >> Demarcator File: No Value Set
> >> Compression Level: 1
> >> Keep Path: False
> >>
> >> 2015-07-28 14:25:17,580 WARN [NiFi Web Server-22]
> >> org.eclipse.jetty.servlet.ServletHandler Error for
> >>
> /nifi-api/controller/process-groups/caffa1c4-ef49-4f0c-b144-0b5d47264481/status
> >> java.lang.OutOfMemoryError: PermGen space
> >> 2015-07-28 14:25:17,811 WARN [NiFi Web Server-22]
> >> org.eclipse.jetty.server.HttpChannel
> >>
> /nifi-api/controller/process-groups/caffa1c4-ef49-4f0c-b144-0b5d47264481/status?recursive=false
> >> java.lang.OutOfMemoryError: PermGen space
> >> 2015-07-28 14:25:18,020 WARN [NiFi Web Server-22]
> >> o.e.jetty.util.thread.QueuedThreadPool
> >> java.lang.OutOfMemoryError: PermGen space
> >> 2015-07-28 14:25:18,020 WARN [NiFi Web Server-22]
> >> o.e.jetty.util.thread.QueuedThreadPool Unexpected thread death:
> >> org.eclipse.jetty.util.thread.QueuedThreadPool$3@1a4b0747 in NiFi Web
> >> Server{STARTED,8<=9<=200,i=0,q=4}
> >> 2015-07-28 14:25:18,775 WARN [NiFi Web Server-23]
> >> org.eclipse.jetty.servlet.ServletHandler Error for
> >>
> /nifi-api/controller/process-groups/caffa1c4-ef49-4f0c-b144-0b5d47264481/status
> >> java.lang.OutOfMemoryError: PermGen space
> >> 2015-07-28 14:25:19,005 WARN [NiFi Web Server-23]
> >> org.eclipse.jetty.server.HttpChannel
> >>
> /nifi-api/controller/process-groups/caffa1c4-ef49-4f0c-b144-0b5d47264481/status?recursive=false
> >> java.lang.OutOfMemoryError: PermGen space
> >> 2015-07-28 14:25:19,240 WARN [NiFi Web Server-23]
> >> o.e.jetty.util.thread.QueuedThreadPool
> >> java.lang.OutOfMemoryError: PermGen space
> >> 2015-07-28 14:25:19,240 WARN [NiFi Web Server-23]
> >> o.e.jetty.util.thread.QueuedThreadPool Unexpected thread death:
> >> org.eclipse.jetty.util.thread.QueuedThreadPool$3@1a4b0747 in NiFi Web
> >> Server{STARTED,8<=9<=200,i=0,q=4}
> >>
> >> *Jeremiah Adams*
> >>
> >> Senior Software Developer
> >> Pearson
> >>
> >> 2154 East Commons Ave.
> >> Suite 400
> >> Centennial, CO 80122
> >>
> >>
> >> Always Learning
> >> Learn more at www.pearson.com
>
Re: Having Issues with MergeContent Processor
Posted by Joe Witt <jo...@gmail.com>.
You'll want to uncomment both of these lines actually:
#java.arg.11=-XX:PermSize=128M
#java.arg.12=-XX:MaxPermSize=128M
And if you like you can make them a bit larger.
Thanks
Joe
On Tue, Jul 28, 2015 at 6:52 PM, Joe Witt <jo...@gmail.com> wrote:
> Jeremiah,
>
> In this case it is talking about permgen (Java 7 and where classes go).
>
> If you take a look at the './conf/bootstap.conf' file there is a line that says:
>
> #java.arg.11=-XX:PermSize=128M
>
> Uncomment that and you should be good to go. We need more than the
> typical Java 7 default for PermGen space.
>
> Of note the MergeContent processor is extremely memory efficient and
> pretty much a non-factor on the heap as it never actually loads
> content into memory. Very different than most systems would be.
>
> Thanks
> Joe
>
> On Tue, Jul 28, 2015 at 6:47 PM, Adams, Jeremiah
> <je...@pearson.com> wrote:
>> I am running into memory issues when using MergeContent. I am seeing
>> OutofMemoryErrors. I am using 0.1.0.
>>
>> I am processing a kafka queue via GetKafka, some custom Processors and
>> PutKafka. I have a PutKafka Controller just prior to the MergeContent
>> Processor. I am passing JSON messages through the FlowFiles.
>>
>> I am looking to merge the flow files until reaching a minimum size or a
>> time period elapses, then write them to S3. I get the memory issue with
>> just 15-20 messages. The MergeContent Processor continues to update the
>> Tasks in Tasks/Time even when not sending messages to the flow.
>>
>> Here are my settings for the MergeContent Processor:
>> Merge Strategy: Bin-Packing Algo
>> Merge Format: Binary Concatenation
>> Attribute Strategy: Keep only Common Attributes
>> Correlation Attribute Name: No Value Set
>> Minimum Number of Entries: 1
>> Maximum Number of Entries: No Value Set
>> Minimum Group Size: 1KB
>> Maximum Group Size: 2 MB
>> Max Bin Age: 10 m
>> Maximum number of Bins: 100
>> Header File: No Value Set
>> Footer File: No Value Set
>> Demarcator File: No Value Set
>> Compression Level: 1
>> Keep Path: False
>>
>> 2015-07-28 14:25:17,580 WARN [NiFi Web Server-22]
>> org.eclipse.jetty.servlet.ServletHandler Error for
>> /nifi-api/controller/process-groups/caffa1c4-ef49-4f0c-b144-0b5d47264481/status
>> java.lang.OutOfMemoryError: PermGen space
>> 2015-07-28 14:25:17,811 WARN [NiFi Web Server-22]
>> org.eclipse.jetty.server.HttpChannel
>> /nifi-api/controller/process-groups/caffa1c4-ef49-4f0c-b144-0b5d47264481/status?recursive=false
>> java.lang.OutOfMemoryError: PermGen space
>> 2015-07-28 14:25:18,020 WARN [NiFi Web Server-22]
>> o.e.jetty.util.thread.QueuedThreadPool
>> java.lang.OutOfMemoryError: PermGen space
>> 2015-07-28 14:25:18,020 WARN [NiFi Web Server-22]
>> o.e.jetty.util.thread.QueuedThreadPool Unexpected thread death:
>> org.eclipse.jetty.util.thread.QueuedThreadPool$3@1a4b0747 in NiFi Web
>> Server{STARTED,8<=9<=200,i=0,q=4}
>> 2015-07-28 14:25:18,775 WARN [NiFi Web Server-23]
>> org.eclipse.jetty.servlet.ServletHandler Error for
>> /nifi-api/controller/process-groups/caffa1c4-ef49-4f0c-b144-0b5d47264481/status
>> java.lang.OutOfMemoryError: PermGen space
>> 2015-07-28 14:25:19,005 WARN [NiFi Web Server-23]
>> org.eclipse.jetty.server.HttpChannel
>> /nifi-api/controller/process-groups/caffa1c4-ef49-4f0c-b144-0b5d47264481/status?recursive=false
>> java.lang.OutOfMemoryError: PermGen space
>> 2015-07-28 14:25:19,240 WARN [NiFi Web Server-23]
>> o.e.jetty.util.thread.QueuedThreadPool
>> java.lang.OutOfMemoryError: PermGen space
>> 2015-07-28 14:25:19,240 WARN [NiFi Web Server-23]
>> o.e.jetty.util.thread.QueuedThreadPool Unexpected thread death:
>> org.eclipse.jetty.util.thread.QueuedThreadPool$3@1a4b0747 in NiFi Web
>> Server{STARTED,8<=9<=200,i=0,q=4}
>>
>> *Jeremiah Adams*
>>
>> Senior Software Developer
>> Pearson
>>
>> 2154 East Commons Ave.
>> Suite 400
>> Centennial, CO 80122
>>
>>
>> Always Learning
>> Learn more at www.pearson.com
Re: Having Issues with MergeContent Processor
Posted by Joe Witt <jo...@gmail.com>.
Jeremiah,
In this case it is talking about permgen (Java 7 and where classes go).
If you take a look at the './conf/bootstap.conf' file there is a line that says:
#java.arg.11=-XX:PermSize=128M
Uncomment that and you should be good to go. We need more than the
typical Java 7 default for PermGen space.
Of note the MergeContent processor is extremely memory efficient and
pretty much a non-factor on the heap as it never actually loads
content into memory. Very different than most systems would be.
Thanks
Joe
On Tue, Jul 28, 2015 at 6:47 PM, Adams, Jeremiah
<je...@pearson.com> wrote:
> I am running into memory issues when using MergeContent. I am seeing
> OutofMemoryErrors. I am using 0.1.0.
>
> I am processing a kafka queue via GetKafka, some custom Processors and
> PutKafka. I have a PutKafka Controller just prior to the MergeContent
> Processor. I am passing JSON messages through the FlowFiles.
>
> I am looking to merge the flow files until reaching a minimum size or a
> time period elapses, then write them to S3. I get the memory issue with
> just 15-20 messages. The MergeContent Processor continues to update the
> Tasks in Tasks/Time even when not sending messages to the flow.
>
> Here are my settings for the MergeContent Processor:
> Merge Strategy: Bin-Packing Algo
> Merge Format: Binary Concatenation
> Attribute Strategy: Keep only Common Attributes
> Correlation Attribute Name: No Value Set
> Minimum Number of Entries: 1
> Maximum Number of Entries: No Value Set
> Minimum Group Size: 1KB
> Maximum Group Size: 2 MB
> Max Bin Age: 10 m
> Maximum number of Bins: 100
> Header File: No Value Set
> Footer File: No Value Set
> Demarcator File: No Value Set
> Compression Level: 1
> Keep Path: False
>
> 2015-07-28 14:25:17,580 WARN [NiFi Web Server-22]
> org.eclipse.jetty.servlet.ServletHandler Error for
> /nifi-api/controller/process-groups/caffa1c4-ef49-4f0c-b144-0b5d47264481/status
> java.lang.OutOfMemoryError: PermGen space
> 2015-07-28 14:25:17,811 WARN [NiFi Web Server-22]
> org.eclipse.jetty.server.HttpChannel
> /nifi-api/controller/process-groups/caffa1c4-ef49-4f0c-b144-0b5d47264481/status?recursive=false
> java.lang.OutOfMemoryError: PermGen space
> 2015-07-28 14:25:18,020 WARN [NiFi Web Server-22]
> o.e.jetty.util.thread.QueuedThreadPool
> java.lang.OutOfMemoryError: PermGen space
> 2015-07-28 14:25:18,020 WARN [NiFi Web Server-22]
> o.e.jetty.util.thread.QueuedThreadPool Unexpected thread death:
> org.eclipse.jetty.util.thread.QueuedThreadPool$3@1a4b0747 in NiFi Web
> Server{STARTED,8<=9<=200,i=0,q=4}
> 2015-07-28 14:25:18,775 WARN [NiFi Web Server-23]
> org.eclipse.jetty.servlet.ServletHandler Error for
> /nifi-api/controller/process-groups/caffa1c4-ef49-4f0c-b144-0b5d47264481/status
> java.lang.OutOfMemoryError: PermGen space
> 2015-07-28 14:25:19,005 WARN [NiFi Web Server-23]
> org.eclipse.jetty.server.HttpChannel
> /nifi-api/controller/process-groups/caffa1c4-ef49-4f0c-b144-0b5d47264481/status?recursive=false
> java.lang.OutOfMemoryError: PermGen space
> 2015-07-28 14:25:19,240 WARN [NiFi Web Server-23]
> o.e.jetty.util.thread.QueuedThreadPool
> java.lang.OutOfMemoryError: PermGen space
> 2015-07-28 14:25:19,240 WARN [NiFi Web Server-23]
> o.e.jetty.util.thread.QueuedThreadPool Unexpected thread death:
> org.eclipse.jetty.util.thread.QueuedThreadPool$3@1a4b0747 in NiFi Web
> Server{STARTED,8<=9<=200,i=0,q=4}
>
> *Jeremiah Adams*
>
> Senior Software Developer
> Pearson
>
> 2154 East Commons Ave.
> Suite 400
> Centennial, CO 80122
>
>
> Always Learning
> Learn more at www.pearson.com