You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Chris Mangold <cs...@gmail.com> on 2015/05/20 19:56:46 UTC
PutJMS performance reading from NIFI Queue
I am trying to push flowfile messages to ActiveMQ using a PutJMS
processor. I am generating flow files using either a GetFile or
GenerateFlowFile processor and feeding them to the PutJMS processor via a
NIFI queue.
Issue is that the PutJMS processor never keeps up with the queue and does
not push messages out any quicker then at a rate of 30 seconds. I have the
PutJMS setup as timer driven with Run Schedule of 0 secs.
Any suggestions.
Thanks,
Chris
--
Chris Mangold
RE: PutJMS performance reading from NIFI Queue
Posted by Mark Payne <ma...@hotmail.com>.
Chris,
If you're going straight from GetFile or GenerateFlowFile, neither of those is going to penalize any FlowFiles, so that's not the issue.
I have two follow-up questions:
1. How many tasks is the Processor reporting in a 5-minute window (the bottom stat that is shown on the Processor itself)?
2. Does there constantly seem to be a thread running or typically there are no threads running (in the top-right corner of the Processor it shows how many threads are currently running. If there is no number in the very top-right corner of the Processor, then no threads are running)? If you keep right-clicking on the graph and then clicking "Refresh" , I'd like to know if there is a number that stays there, or if it generally has no threads.
Thanks
-Mark
________________________________
> Date: Wed, 20 May 2015 16:44:29 -0400
> Subject: Re: PutJMS performance reading from NIFI Queue
> From: csm9105@gmail.com
> To: users@nifi.incubator.apache.org
>
> Just going straight from the GetFile or GenerateFlowFile processor (via
> a NIFI queue) to PutJMS.
>
> I guess I have to read up on FlowFile penalization, I am not familiar
> with the concept.
>
> Chris
>
>
>
> On Wed, May 20, 2015 at 3:52 PM, Mark Payne
> <ma...@hotmail.com>> wrote:
> Ohh, I totally misunderstood the problem. I thought you were saying
> that you were pushing messages at 30 per second.
>
> When you say you have a GetFile or a GenerateFlowFile feeding the
> PutJMS, are you feeding directly from those processors, or is there
> something in between?
>
> I'm wondering if the FlowFiles are possibly penalized?
>
> Thanks
> -Mark
>
> ________________________________
>> Date: Wed, 20 May 2015 15:44:37 -0400
>> Subject: Re: PutJMS performance reading from NIFI Queue
>> From: csm9105@gmail.com<ma...@gmail.com>
>> To: users@nifi.incubator.apache.org<ma...@nifi.incubator.apache.org>
>>
>> Mark:
>>
>> I have been varying the "Message Batch Size" between 10 and 1000
>> with no appreciable improvement in performance. I notice a larger
>> batch of messages going into ActiveMQ but the Processor only is getting
>> triggered every ~ 30 seconds.
>>
>> Chris
>>
>> On Wed, May 20, 2015 at 2:50 PM, Mark Payne
>>
> <ma...@hotmail.com>>>
> wrote:
>> Chris,
>>
>> What value do you have the "Batch Size" property set to? That can make
>> a really big difference in throughput.
>>
>> ________________________________
>>> Date: Wed, 20 May 2015 13:56:46 -0400
>>> Subject: PutJMS performance reading from NIFI Queue
>>> From:
> csm9105@gmail.com<ma...@gmail.com>>
>>> To:
> users@nifi.incubator.apache.org<ma...@nifi.incubator.apache.org>>
>>>
>>> I am trying to push flowfile messages to ActiveMQ using a PutJMS
>>> processor. I am generating flow files using either a GetFile or
>>> GenerateFlowFile processor and feeding them to the PutJMS processor via
>>> a NIFI queue.
>>>
>>> Issue is that the PutJMS processor never keeps up with the queue and
>>> does not push messages out any quicker then at a rate of 30 seconds. I
>>> have the PutJMS setup as timer driven with Run Schedule of 0 secs.
>>>
>>> Any suggestions.
>>>
>>> Thanks,
>>>
>>> Chris
>>>
>>>
>>>
>>> --
>>> Chris Mangold
>>>
>>
>>
>>
>>
>> --
>> Chris Mangold
>> 301-471-5758 (c)
>> 301-898-7979 (h)
>
>
>
>
> --
> Chris Mangold
> 301-471-5758 (c)
> 301-898-7979 (h)
Re: PutJMS performance reading from NIFI Queue
Posted by Joe Witt <jo...@gmail.com>.
Mark,
Happy to hear you are seeing performance more like you would
anticipate. I do wonder why the processors are having to yield and
penalize data so much. That suggests errors are occurring. Do you
see bulletins about that?
Thanks
Joe
On Wed, May 20, 2015 at 10:19 PM, Chris Mangold <cs...@gmail.com> wrote:
> Thanks Mark and Jennifer. I was able to improve my throughput by adjusting
> the Penalty and Yield durations.
>
> Chris
>
> On Wed, May 20, 2015 at 5:07 PM, Jennifer Barnabee
> <je...@gmail.com> wrote:
>>
>> Chris,
>> FlowFile penalization is discussed in the User Guide section on
>> configuring processors [1]. Here is the text, which discusses setting the
>> Penalty Duration on the Settings tab of a given processor:
>>
>> During the normal course of processing a piece of data (a FlowFile), an
>> event may occur that indicates that the data cannot be processed at this
>> time but the data may be processable at a later time. When this occurs, the
>> Processor may choose to Penalize the FlowFile. This will prevent the
>> FlowFile from being Processed for some period of time. For example, if the
>> Processor is to push the data to a remote service, but the remote service
>> already has a file with the same name, the Processor may penalize the
>> FlowFile. The ‘Penalty duration’ allows the DFM to specify how long the
>> FlowFile should be penalized. The default value is 30 seconds.
>>
>> [1] https://nifi.incubator.apache.org/docs/user-guide.html#settings-tab
>>
>> -Jenn
>>
>> On Wed, May 20, 2015 at 4:44 PM, Chris Mangold <cs...@gmail.com> wrote:
>>>
>>> Just going straight from the GetFile or GenerateFlowFile processor (via a
>>> NIFI queue) to PutJMS.
>>>
>>> I guess I have to read up on FlowFile penalization, I am not familiar
>>> with the concept.
>>>
>>> Chris
>>>
>>>
>>>
>>> On Wed, May 20, 2015 at 3:52 PM, Mark Payne <ma...@hotmail.com> wrote:
>>>>
>>>> Ohh, I totally misunderstood the problem. I thought you were saying that
>>>> you were pushing messages at 30 per second.
>>>>
>>>> When you say you have a GetFile or a GenerateFlowFile feeding the
>>>> PutJMS, are you feeding directly from those processors, or is there
>>>> something in between?
>>>>
>>>> I'm wondering if the FlowFiles are possibly penalized?
>>>>
>>>> Thanks
>>>> -Mark
>>>>
>>>> ________________________________
>>>> > Date: Wed, 20 May 2015 15:44:37 -0400
>>>> > Subject: Re: PutJMS performance reading from NIFI Queue
>>>> > From: csm9105@gmail.com
>>>> > To: users@nifi.incubator.apache.org
>>>> >
>>>> > Mark:
>>>> >
>>>> > I have been varying the "Message Batch Size" between 10 and 1000
>>>> > with no appreciable improvement in performance. I notice a larger
>>>> > batch of messages going into ActiveMQ but the Processor only is
>>>> > getting
>>>> > triggered every ~ 30 seconds.
>>>> >
>>>> > Chris
>>>> >
>>>> > On Wed, May 20, 2015 at 2:50 PM, Mark Payne
>>>> > <ma...@hotmail.com>> wrote:
>>>> > Chris,
>>>> >
>>>> > What value do you have the "Batch Size" property set to? That can make
>>>> > a really big difference in throughput.
>>>> >
>>>> > ________________________________
>>>> >> Date: Wed, 20 May 2015 13:56:46 -0400
>>>> >> Subject: PutJMS performance reading from NIFI Queue
>>>> >> From: csm9105@gmail.com<ma...@gmail.com>
>>>> >> To:
>>>> >> users@nifi.incubator.apache.org<ma...@nifi.incubator.apache.org>
>>>> >>
>>>> >> I am trying to push flowfile messages to ActiveMQ using a PutJMS
>>>> >> processor. I am generating flow files using either a GetFile or
>>>> >> GenerateFlowFile processor and feeding them to the PutJMS processor
>>>> >> via
>>>> >> a NIFI queue.
>>>> >>
>>>> >> Issue is that the PutJMS processor never keeps up with the queue and
>>>> >> does not push messages out any quicker then at a rate of 30 seconds.
>>>> >> I
>>>> >> have the PutJMS setup as timer driven with Run Schedule of 0 secs.
>>>> >>
>>>> >> Any suggestions.
>>>> >>
>>>> >> Thanks,
>>>> >>
>>>> >> Chris
>>>> >>
>>>> >>
>>>> >>
>>>> >> --
>>>> >> Chris Mangold
>>>> >>
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > Chris Mangold
>>>> > 301-471-5758 (c)
>>>> > 301-898-7979 (h)
>>>>
>>>
>>>
>>>
>>>
>>> --
>>> Chris Mangold
>>> 301-471-5758 (c)
>>> 301-898-7979 (h)
>>
>>
>
>
>
> --
> Chris Mangold
> 301-471-5758 (c)
> 301-898-7979 (h)
Re: PutJMS performance reading from NIFI Queue
Posted by Chris Mangold <cs...@gmail.com>.
Thanks Mark and Jennifer. I was able to improve my throughput by adjusting
the Penalty and Yield durations.
Chris
On Wed, May 20, 2015 at 5:07 PM, Jennifer Barnabee <
jennifer.barnabee@gmail.com> wrote:
> Chris,
> FlowFile penalization is discussed in the User Guide section on
> configuring processors [1]. Here is the text, which discusses setting the
> Penalty Duration on the Settings tab of a given processor:
>
> During the normal course of processing a piece of data (a FlowFile), an
> event may occur that indicates that the data cannot be processed at this
> time but the data may be processable at a later time. When this occurs, the
> Processor may choose to Penalize the FlowFile. This will prevent the
> FlowFile from being Processed for some period of time. For example, if the
> Processor is to push the data to a remote service, but the remote service
> already has a file with the same name, the Processor may penalize the
> FlowFile. The ‘Penalty duration’ allows the DFM to specify how long the
> FlowFile should be penalized. The default value is 30 seconds.
>
> [1] https://nifi.incubator.apache.org/docs/user-guide.html#settings-tab
>
> -Jenn
>
> On Wed, May 20, 2015 at 4:44 PM, Chris Mangold <cs...@gmail.com> wrote:
>
>> Just going straight from the GetFile or GenerateFlowFile processor (via a
>> NIFI queue) to PutJMS.
>>
>> I guess I have to read up on FlowFile penalization, I am not familiar
>> with the concept.
>>
>> Chris
>>
>>
>>
>> On Wed, May 20, 2015 at 3:52 PM, Mark Payne <ma...@hotmail.com> wrote:
>>
>>> Ohh, I totally misunderstood the problem. I thought you were saying that
>>> you were pushing messages at 30 per second.
>>>
>>> When you say you have a GetFile or a GenerateFlowFile feeding the
>>> PutJMS, are you feeding directly from those processors, or is there
>>> something in between?
>>>
>>> I'm wondering if the FlowFiles are possibly penalized?
>>>
>>> Thanks
>>> -Mark
>>>
>>> ________________________________
>>> > Date: Wed, 20 May 2015 15:44:37 -0400
>>> > Subject: Re: PutJMS performance reading from NIFI Queue
>>> > From: csm9105@gmail.com
>>> > To: users@nifi.incubator.apache.org
>>> >
>>> > Mark:
>>> >
>>> > I have been varying the "Message Batch Size" between 10 and 1000
>>> > with no appreciable improvement in performance. I notice a larger
>>> > batch of messages going into ActiveMQ but the Processor only is getting
>>> > triggered every ~ 30 seconds.
>>> >
>>> > Chris
>>> >
>>> > On Wed, May 20, 2015 at 2:50 PM, Mark Payne
>>> > <ma...@hotmail.com>> wrote:
>>> > Chris,
>>> >
>>> > What value do you have the "Batch Size" property set to? That can make
>>> > a really big difference in throughput.
>>> >
>>> > ________________________________
>>> >> Date: Wed, 20 May 2015 13:56:46 -0400
>>> >> Subject: PutJMS performance reading from NIFI Queue
>>> >> From: csm9105@gmail.com<ma...@gmail.com>
>>> >> To: users@nifi.incubator.apache.org<mailto:
>>> users@nifi.incubator.apache.org>
>>> >>
>>> >> I am trying to push flowfile messages to ActiveMQ using a PutJMS
>>> >> processor. I am generating flow files using either a GetFile or
>>> >> GenerateFlowFile processor and feeding them to the PutJMS processor
>>> via
>>> >> a NIFI queue.
>>> >>
>>> >> Issue is that the PutJMS processor never keeps up with the queue and
>>> >> does not push messages out any quicker then at a rate of 30 seconds. I
>>> >> have the PutJMS setup as timer driven with Run Schedule of 0 secs.
>>> >>
>>> >> Any suggestions.
>>> >>
>>> >> Thanks,
>>> >>
>>> >> Chris
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Chris Mangold
>>> >>
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > Chris Mangold
>>> > 301-471-5758 (c)
>>> > 301-898-7979 (h)
>>>
>>>
>>
>>
>>
>> --
>> Chris Mangold
>> 301-471-5758 (c)
>> 301-898-7979 (h)
>>
>
>
--
Chris Mangold
301-471-5758 (c)
301-898-7979 (h)
Re: PutJMS performance reading from NIFI Queue
Posted by Jennifer Barnabee <je...@gmail.com>.
Chris,
FlowFile penalization is discussed in the User Guide section on configuring
processors [1]. Here is the text, which discusses setting the Penalty
Duration on the Settings tab of a given processor:
During the normal course of processing a piece of data (a FlowFile), an
event may occur that indicates that the data cannot be processed at this
time but the data may be processable at a later time. When this occurs, the
Processor may choose to Penalize the FlowFile. This will prevent the
FlowFile from being Processed for some period of time. For example, if the
Processor is to push the data to a remote service, but the remote service
already has a file with the same name, the Processor may penalize the
FlowFile. The ‘Penalty duration’ allows the DFM to specify how long the
FlowFile should be penalized. The default value is 30 seconds.
[1] https://nifi.incubator.apache.org/docs/user-guide.html#settings-tab
-Jenn
On Wed, May 20, 2015 at 4:44 PM, Chris Mangold <cs...@gmail.com> wrote:
> Just going straight from the GetFile or GenerateFlowFile processor (via a
> NIFI queue) to PutJMS.
>
> I guess I have to read up on FlowFile penalization, I am not familiar with
> the concept.
>
> Chris
>
>
>
> On Wed, May 20, 2015 at 3:52 PM, Mark Payne <ma...@hotmail.com> wrote:
>
>> Ohh, I totally misunderstood the problem. I thought you were saying that
>> you were pushing messages at 30 per second.
>>
>> When you say you have a GetFile or a GenerateFlowFile feeding the PutJMS,
>> are you feeding directly from those processors, or is there something in
>> between?
>>
>> I'm wondering if the FlowFiles are possibly penalized?
>>
>> Thanks
>> -Mark
>>
>> ________________________________
>> > Date: Wed, 20 May 2015 15:44:37 -0400
>> > Subject: Re: PutJMS performance reading from NIFI Queue
>> > From: csm9105@gmail.com
>> > To: users@nifi.incubator.apache.org
>> >
>> > Mark:
>> >
>> > I have been varying the "Message Batch Size" between 10 and 1000
>> > with no appreciable improvement in performance. I notice a larger
>> > batch of messages going into ActiveMQ but the Processor only is getting
>> > triggered every ~ 30 seconds.
>> >
>> > Chris
>> >
>> > On Wed, May 20, 2015 at 2:50 PM, Mark Payne
>> > <ma...@hotmail.com>> wrote:
>> > Chris,
>> >
>> > What value do you have the "Batch Size" property set to? That can make
>> > a really big difference in throughput.
>> >
>> > ________________________________
>> >> Date: Wed, 20 May 2015 13:56:46 -0400
>> >> Subject: PutJMS performance reading from NIFI Queue
>> >> From: csm9105@gmail.com<ma...@gmail.com>
>> >> To: users@nifi.incubator.apache.org<mailto:
>> users@nifi.incubator.apache.org>
>> >>
>> >> I am trying to push flowfile messages to ActiveMQ using a PutJMS
>> >> processor. I am generating flow files using either a GetFile or
>> >> GenerateFlowFile processor and feeding them to the PutJMS processor via
>> >> a NIFI queue.
>> >>
>> >> Issue is that the PutJMS processor never keeps up with the queue and
>> >> does not push messages out any quicker then at a rate of 30 seconds. I
>> >> have the PutJMS setup as timer driven with Run Schedule of 0 secs.
>> >>
>> >> Any suggestions.
>> >>
>> >> Thanks,
>> >>
>> >> Chris
>> >>
>> >>
>> >>
>> >> --
>> >> Chris Mangold
>> >>
>> >
>> >
>> >
>> >
>> > --
>> > Chris Mangold
>> > 301-471-5758 (c)
>> > 301-898-7979 (h)
>>
>>
>
>
>
> --
> Chris Mangold
> 301-471-5758 (c)
> 301-898-7979 (h)
>
Re: PutJMS performance reading from NIFI Queue
Posted by Chris Mangold <cs...@gmail.com>.
Just going straight from the GetFile or GenerateFlowFile processor (via a
NIFI queue) to PutJMS.
I guess I have to read up on FlowFile penalization, I am not familiar with
the concept.
Chris
On Wed, May 20, 2015 at 3:52 PM, Mark Payne <ma...@hotmail.com> wrote:
> Ohh, I totally misunderstood the problem. I thought you were saying that
> you were pushing messages at 30 per second.
>
> When you say you have a GetFile or a GenerateFlowFile feeding the PutJMS,
> are you feeding directly from those processors, or is there something in
> between?
>
> I'm wondering if the FlowFiles are possibly penalized?
>
> Thanks
> -Mark
>
> ________________________________
> > Date: Wed, 20 May 2015 15:44:37 -0400
> > Subject: Re: PutJMS performance reading from NIFI Queue
> > From: csm9105@gmail.com
> > To: users@nifi.incubator.apache.org
> >
> > Mark:
> >
> > I have been varying the "Message Batch Size" between 10 and 1000
> > with no appreciable improvement in performance. I notice a larger
> > batch of messages going into ActiveMQ but the Processor only is getting
> > triggered every ~ 30 seconds.
> >
> > Chris
> >
> > On Wed, May 20, 2015 at 2:50 PM, Mark Payne
> > <ma...@hotmail.com>> wrote:
> > Chris,
> >
> > What value do you have the "Batch Size" property set to? That can make
> > a really big difference in throughput.
> >
> > ________________________________
> >> Date: Wed, 20 May 2015 13:56:46 -0400
> >> Subject: PutJMS performance reading from NIFI Queue
> >> From: csm9105@gmail.com<ma...@gmail.com>
> >> To: users@nifi.incubator.apache.org<mailto:
> users@nifi.incubator.apache.org>
> >>
> >> I am trying to push flowfile messages to ActiveMQ using a PutJMS
> >> processor. I am generating flow files using either a GetFile or
> >> GenerateFlowFile processor and feeding them to the PutJMS processor via
> >> a NIFI queue.
> >>
> >> Issue is that the PutJMS processor never keeps up with the queue and
> >> does not push messages out any quicker then at a rate of 30 seconds. I
> >> have the PutJMS setup as timer driven with Run Schedule of 0 secs.
> >>
> >> Any suggestions.
> >>
> >> Thanks,
> >>
> >> Chris
> >>
> >>
> >>
> >> --
> >> Chris Mangold
> >>
> >
> >
> >
> >
> > --
> > Chris Mangold
> > 301-471-5758 (c)
> > 301-898-7979 (h)
>
>
--
Chris Mangold
301-471-5758 (c)
301-898-7979 (h)
RE: PutJMS performance reading from NIFI Queue
Posted by Mark Payne <ma...@hotmail.com>.
Ohh, I totally misunderstood the problem. I thought you were saying that you were pushing messages at 30 per second.
When you say you have a GetFile or a GenerateFlowFile feeding the PutJMS, are you feeding directly from those processors, or is there something in between?
I'm wondering if the FlowFiles are possibly penalized?
Thanks
-Mark
________________________________
> Date: Wed, 20 May 2015 15:44:37 -0400
> Subject: Re: PutJMS performance reading from NIFI Queue
> From: csm9105@gmail.com
> To: users@nifi.incubator.apache.org
>
> Mark:
>
> I have been varying the "Message Batch Size" between 10 and 1000
> with no appreciable improvement in performance. I notice a larger
> batch of messages going into ActiveMQ but the Processor only is getting
> triggered every ~ 30 seconds.
>
> Chris
>
> On Wed, May 20, 2015 at 2:50 PM, Mark Payne
> <ma...@hotmail.com>> wrote:
> Chris,
>
> What value do you have the "Batch Size" property set to? That can make
> a really big difference in throughput.
>
> ________________________________
>> Date: Wed, 20 May 2015 13:56:46 -0400
>> Subject: PutJMS performance reading from NIFI Queue
>> From: csm9105@gmail.com<ma...@gmail.com>
>> To: users@nifi.incubator.apache.org<ma...@nifi.incubator.apache.org>
>>
>> I am trying to push flowfile messages to ActiveMQ using a PutJMS
>> processor. I am generating flow files using either a GetFile or
>> GenerateFlowFile processor and feeding them to the PutJMS processor via
>> a NIFI queue.
>>
>> Issue is that the PutJMS processor never keeps up with the queue and
>> does not push messages out any quicker then at a rate of 30 seconds. I
>> have the PutJMS setup as timer driven with Run Schedule of 0 secs.
>>
>> Any suggestions.
>>
>> Thanks,
>>
>> Chris
>>
>>
>>
>> --
>> Chris Mangold
>>
>
>
>
>
> --
> Chris Mangold
> 301-471-5758 (c)
> 301-898-7979 (h)
Re: PutJMS performance reading from NIFI Queue
Posted by Chris Mangold <cs...@gmail.com>.
Mark:
I have been varying the "Message Batch Size" between 10 and 1000 with no
appreciable improvement in performance. I notice a larger batch of
messages going into ActiveMQ but the Processor only is getting triggered
every ~ 30 seconds.
Chris
On Wed, May 20, 2015 at 2:50 PM, Mark Payne <ma...@hotmail.com> wrote:
> Chris,
>
> What value do you have the "Batch Size" property set to? That can make a
> really big difference in throughput.
>
> ________________________________
> > Date: Wed, 20 May 2015 13:56:46 -0400
> > Subject: PutJMS performance reading from NIFI Queue
> > From: csm9105@gmail.com
> > To: users@nifi.incubator.apache.org
> >
> > I am trying to push flowfile messages to ActiveMQ using a PutJMS
> > processor. I am generating flow files using either a GetFile or
> > GenerateFlowFile processor and feeding them to the PutJMS processor via
> > a NIFI queue.
> >
> > Issue is that the PutJMS processor never keeps up with the queue and
> > does not push messages out any quicker then at a rate of 30 seconds. I
> > have the PutJMS setup as timer driven with Run Schedule of 0 secs.
> >
> > Any suggestions.
> >
> > Thanks,
> >
> > Chris
> >
> >
> >
> > --
> > Chris Mangold
> >
>
>
--
Chris Mangold
301-471-5758 (c)
301-898-7979 (h)
RE: PutJMS performance reading from NIFI Queue
Posted by Mark Payne <ma...@hotmail.com>.
Chris,
What value do you have the "Batch Size" property set to? That can make a really big difference in throughput.
________________________________
> Date: Wed, 20 May 2015 13:56:46 -0400
> Subject: PutJMS performance reading from NIFI Queue
> From: csm9105@gmail.com
> To: users@nifi.incubator.apache.org
>
> I am trying to push flowfile messages to ActiveMQ using a PutJMS
> processor. I am generating flow files using either a GetFile or
> GenerateFlowFile processor and feeding them to the PutJMS processor via
> a NIFI queue.
>
> Issue is that the PutJMS processor never keeps up with the queue and
> does not push messages out any quicker then at a rate of 30 seconds. I
> have the PutJMS setup as timer driven with Run Schedule of 0 secs.
>
> Any suggestions.
>
> Thanks,
>
> Chris
>
>
>
> --
> Chris Mangold
>