RPG question

[roozbeh karimi <ka...@gmail.com>]

Hello All,

Are there any best practices and guides for setting up RPG? With my set up,
I am seeing a large volume of back pressure is being built up and hardly
any data is being transmitted from the source NiFi node to the nodes in
RPG. If I replace RPG with PostHttp on the source node and a listener
processor on the destination node, no back pressure is built and the data
is transmitted smoothly between the source and destination nodes.

I'd appreciate any feedback on to go about tuning my RPG set up.

Thanks
RK

Re: RPG question

[roozbeh karimi <ka...@gmail.com>]

We will try your recommendations.

thanks
RK


On Wed, Oct 3, 2018 at 10:47 AM Bryan Bende <bb...@gmail.com> wrote:

> Have you tried with smaller batch settings?
>
> I think 2000 flow files and 1000MB would be a bit high. Might try
> something like 100 flow files and 10MB.
>
> Also, if you concurrent tasks set to 7 on the RPG side, then you will
> likely want to increase the concurrent tasks on Input Port on cluster
> 2 where the data is being received.
> On Tue, Oct 2, 2018 at 5:47 PM roozbeh karimi <ka...@gmail.com>
> wrote:
> >
> > The only items that we set are:
> >
> > Remote Process Groups:
> >   URLs ...
> >   Transport Protocol: HTTP
> >    Communication Timeout: 60 sec
> >    Yield Duration: 2 sec
> >
> > Input Ports
> > Concurrent Tasks: 7
> > Batch Settings
> >    Count: 2000
> >    Size: 1000MB
> >    Duration: not set
> >
> > On Tue, Oct 2, 2018 at 3:47 PM Bryan Bende <bb...@gmail.com> wrote:
> >
> > > It looks like there may have been images attached that didn't make it
> > > through.
> > >
> > > I'm not sure why S2S would be slow, it is made to handle very high
> > > throughput.
> > >
> > > What I meant by #4 was... did you perform any specific configuration
> of the
> > > RPG on cluster 1 or the input port on cluster 2, such as changing batch
> > > settings or modifying concurrent tasks, or did you use all default
> values?
> > >
> > >
> > > On Tue, Oct 2, 2018 at 2:01 PM roozbeh karimi <ka...@gmail.com>
> > > wrote:
> > >
> > > > Hello Bryan
> > > > 1) The communication is site to site between two separate
> > > > instances/clusters of NiFi.
> > > > 2) We have tried S2S in both raw and http modes, and neither made a
> > > > difference.
> > > > 3) The data flows but very slow, and after a while the back pressure
> > > > reaches the threshold and we start losing data.
> > > > 4) We have set up the RPG input port.
> > > > 5) If we replace RPG with a posthttp processor on cluster 1 and a
> > > listener
> > > > processor on cluster 2 (where rpg nodes were), we don't see any back
> > > > pressure buildup on cluster 1 and the data flows between the two
> > > instances
> > > > very fast.
> > > >
> > > >
> > > > [image: image.png]
> > > > [image: image.png]
> > > >
> > > > On Tue, Oct 2, 2018 at 11:55 AM Bryan Bende <bb...@gmail.com>
> wrote:
> > > >
> > > >> Hello,
> > > >>
> > > >> Can you describe the setup a little bit more?
> > > >>
> > > >> Is this site-to-site between two separate NiFi instances, or is it
> > > >> within the same cluster for purposes of redistribution?
> > > >>
> > > >> If it's separate instances then what type of instances, clustered or
> > > >> standalone?
> > > >>
> > > >> Have you made any changes to the configuration of the RPG, the port
> > > >> settings within the RPG, or the Input Port on the remote side?
> > > >>
> > > >> Is some data being transferred through the RPG and it is just slow,
> or
> > > >> is no data making it through?
> > > >>
> > > >> -Bryan
> > > >>
> > > >> On Mon, Oct 1, 2018 at 11:53 PM roozbeh karimi <
> karimi97531@gmail.com>
> > > >> wrote:
> > > >> >
> > > >> > Hello All,
> > > >> >
> > > >> > Are there any best practices and guides for setting up RPG? With
> my
> > > set
> > > >> up,
> > > >> > I am seeing a large volume of back pressure is being built up and
> > > hardly
> > > >> > any data is being transmitted from the source NiFi node to the
> nodes
> > > in
> > > >> > RPG. If I replace RPG with PostHttp on the source node and a
> listener
> > > >> > processor on the destination node, no back pressure is built and
> the
> > > >> data
> > > >> > is transmitted smoothly between the source and destination nodes.
> > > >> >
> > > >> > I'd appreciate any feedback on to go about tuning my RPG set up.
> > > >> >
> > > >> > Thanks
> > > >> > RK
> > > >>
> > > >
> > >
>

Re: RPG question

[Bryan Bende <bb...@gmail.com>]

Have you tried with smaller batch settings?

I think 2000 flow files and 1000MB would be a bit high. Might try
something like 100 flow files and 10MB.

Also, if you concurrent tasks set to 7 on the RPG side, then you will
likely want to increase the concurrent tasks on Input Port on cluster
2 where the data is being received.
On Tue, Oct 2, 2018 at 5:47 PM roozbeh karimi <ka...@gmail.com> wrote:
>
> The only items that we set are:
>
> Remote Process Groups:
>   URLs ...
>   Transport Protocol: HTTP
>    Communication Timeout: 60 sec
>    Yield Duration: 2 sec
>
> Input Ports
> Concurrent Tasks: 7
> Batch Settings
>    Count: 2000
>    Size: 1000MB
>    Duration: not set
>
> On Tue, Oct 2, 2018 at 3:47 PM Bryan Bende <bb...@gmail.com> wrote:
>
> > It looks like there may have been images attached that didn't make it
> > through.
> >
> > I'm not sure why S2S would be slow, it is made to handle very high
> > throughput.
> >
> > What I meant by #4 was... did you perform any specific configuration of the
> > RPG on cluster 1 or the input port on cluster 2, such as changing batch
> > settings or modifying concurrent tasks, or did you use all default values?
> >
> >
> > On Tue, Oct 2, 2018 at 2:01 PM roozbeh karimi <ka...@gmail.com>
> > wrote:
> >
> > > Hello Bryan
> > > 1) The communication is site to site between two separate
> > > instances/clusters of NiFi.
> > > 2) We have tried S2S in both raw and http modes, and neither made a
> > > difference.
> > > 3) The data flows but very slow, and after a while the back pressure
> > > reaches the threshold and we start losing data.
> > > 4) We have set up the RPG input port.
> > > 5) If we replace RPG with a posthttp processor on cluster 1 and a
> > listener
> > > processor on cluster 2 (where rpg nodes were), we don't see any back
> > > pressure buildup on cluster 1 and the data flows between the two
> > instances
> > > very fast.
> > >
> > >
> > > [image: image.png]
> > > [image: image.png]
> > >
> > > On Tue, Oct 2, 2018 at 11:55 AM Bryan Bende <bb...@gmail.com> wrote:
> > >
> > >> Hello,
> > >>
> > >> Can you describe the setup a little bit more?
> > >>
> > >> Is this site-to-site between two separate NiFi instances, or is it
> > >> within the same cluster for purposes of redistribution?
> > >>
> > >> If it's separate instances then what type of instances, clustered or
> > >> standalone?
> > >>
> > >> Have you made any changes to the configuration of the RPG, the port
> > >> settings within the RPG, or the Input Port on the remote side?
> > >>
> > >> Is some data being transferred through the RPG and it is just slow, or
> > >> is no data making it through?
> > >>
> > >> -Bryan
> > >>
> > >> On Mon, Oct 1, 2018 at 11:53 PM roozbeh karimi <ka...@gmail.com>
> > >> wrote:
> > >> >
> > >> > Hello All,
> > >> >
> > >> > Are there any best practices and guides for setting up RPG? With my
> > set
> > >> up,
> > >> > I am seeing a large volume of back pressure is being built up and
> > hardly
> > >> > any data is being transmitted from the source NiFi node to the nodes
> > in
> > >> > RPG. If I replace RPG with PostHttp on the source node and a listener
> > >> > processor on the destination node, no back pressure is built and the
> > >> data
> > >> > is transmitted smoothly between the source and destination nodes.
> > >> >
> > >> > I'd appreciate any feedback on to go about tuning my RPG set up.
> > >> >
> > >> > Thanks
> > >> > RK
> > >>
> > >
> >

Re: RPG question

[roozbeh karimi <ka...@gmail.com>]

The only items that we set are:

Remote Process Groups:
  URLs ...
  Transport Protocol: HTTP
   Communication Timeout: 60 sec
   Yield Duration: 2 sec

Input Ports
Concurrent Tasks: 7
Batch Settings
   Count: 2000
   Size: 1000MB
   Duration: not set

On Tue, Oct 2, 2018 at 3:47 PM Bryan Bende <bb...@gmail.com> wrote:

> It looks like there may have been images attached that didn't make it
> through.
>
> I'm not sure why S2S would be slow, it is made to handle very high
> throughput.
>
> What I meant by #4 was... did you perform any specific configuration of the
> RPG on cluster 1 or the input port on cluster 2, such as changing batch
> settings or modifying concurrent tasks, or did you use all default values?
>
>
> On Tue, Oct 2, 2018 at 2:01 PM roozbeh karimi <ka...@gmail.com>
> wrote:
>
> > Hello Bryan
> > 1) The communication is site to site between two separate
> > instances/clusters of NiFi.
> > 2) We have tried S2S in both raw and http modes, and neither made a
> > difference.
> > 3) The data flows but very slow, and after a while the back pressure
> > reaches the threshold and we start losing data.
> > 4) We have set up the RPG input port.
> > 5) If we replace RPG with a posthttp processor on cluster 1 and a
> listener
> > processor on cluster 2 (where rpg nodes were), we don't see any back
> > pressure buildup on cluster 1 and the data flows between the two
> instances
> > very fast.
> >
> >
> > [image: image.png]
> > [image: image.png]
> >
> > On Tue, Oct 2, 2018 at 11:55 AM Bryan Bende <bb...@gmail.com> wrote:
> >
> >> Hello,
> >>
> >> Can you describe the setup a little bit more?
> >>
> >> Is this site-to-site between two separate NiFi instances, or is it
> >> within the same cluster for purposes of redistribution?
> >>
> >> If it's separate instances then what type of instances, clustered or
> >> standalone?
> >>
> >> Have you made any changes to the configuration of the RPG, the port
> >> settings within the RPG, or the Input Port on the remote side?
> >>
> >> Is some data being transferred through the RPG and it is just slow, or
> >> is no data making it through?
> >>
> >> -Bryan
> >>
> >> On Mon, Oct 1, 2018 at 11:53 PM roozbeh karimi <ka...@gmail.com>
> >> wrote:
> >> >
> >> > Hello All,
> >> >
> >> > Are there any best practices and guides for setting up RPG? With my
> set
> >> up,
> >> > I am seeing a large volume of back pressure is being built up and
> hardly
> >> > any data is being transmitted from the source NiFi node to the nodes
> in
> >> > RPG. If I replace RPG with PostHttp on the source node and a listener
> >> > processor on the destination node, no back pressure is built and the
> >> data
> >> > is transmitted smoothly between the source and destination nodes.
> >> >
> >> > I'd appreciate any feedback on to go about tuning my RPG set up.
> >> >
> >> > Thanks
> >> > RK
> >>
> >
>

Re: RPG question

[Bryan Bende <bb...@gmail.com>]

It looks like there may have been images attached that didn't make it
through.

I'm not sure why S2S would be slow, it is made to handle very high
throughput.

What I meant by #4 was... did you perform any specific configuration of the
RPG on cluster 1 or the input port on cluster 2, such as changing batch
settings or modifying concurrent tasks, or did you use all default values?


On Tue, Oct 2, 2018 at 2:01 PM roozbeh karimi <ka...@gmail.com> wrote:

> Hello Bryan
> 1) The communication is site to site between two separate
> instances/clusters of NiFi.
> 2) We have tried S2S in both raw and http modes, and neither made a
> difference.
> 3) The data flows but very slow, and after a while the back pressure
> reaches the threshold and we start losing data.
> 4) We have set up the RPG input port.
> 5) If we replace RPG with a posthttp processor on cluster 1 and a listener
> processor on cluster 2 (where rpg nodes were), we don't see any back
> pressure buildup on cluster 1 and the data flows between the two instances
> very fast.
>
>
> [image: image.png]
> [image: image.png]
>
> On Tue, Oct 2, 2018 at 11:55 AM Bryan Bende <bb...@gmail.com> wrote:
>
>> Hello,
>>
>> Can you describe the setup a little bit more?
>>
>> Is this site-to-site between two separate NiFi instances, or is it
>> within the same cluster for purposes of redistribution?
>>
>> If it's separate instances then what type of instances, clustered or
>> standalone?
>>
>> Have you made any changes to the configuration of the RPG, the port
>> settings within the RPG, or the Input Port on the remote side?
>>
>> Is some data being transferred through the RPG and it is just slow, or
>> is no data making it through?
>>
>> -Bryan
>>
>> On Mon, Oct 1, 2018 at 11:53 PM roozbeh karimi <ka...@gmail.com>
>> wrote:
>> >
>> > Hello All,
>> >
>> > Are there any best practices and guides for setting up RPG? With my set
>> up,
>> > I am seeing a large volume of back pressure is being built up and hardly
>> > any data is being transmitted from the source NiFi node to the nodes in
>> > RPG. If I replace RPG with PostHttp on the source node and a listener
>> > processor on the destination node, no back pressure is built and the
>> data
>> > is transmitted smoothly between the source and destination nodes.
>> >
>> > I'd appreciate any feedback on to go about tuning my RPG set up.
>> >
>> > Thanks
>> > RK
>>
>

Re: RPG question

[roozbeh karimi <ka...@gmail.com>]

Hello Bryan
1) The communication is site to site between two separate
instances/clusters of NiFi.
2) We have tried S2S in both raw and http modes, and neither made a
difference.
3) The data flows but very slow, and after a while the back pressure
reaches the threshold and we start losing data.
4) We have set up the RPG input port.
5) If we replace RPG with a posthttp processor on cluster 1 and a listener
processor on cluster 2 (where rpg nodes were), we don't see any back
pressure buildup on cluster 1 and the data flows between the two instances
very fast.


[image: image.png]
[image: image.png]

On Tue, Oct 2, 2018 at 11:55 AM Bryan Bende <bb...@gmail.com> wrote:

> Hello,
>
> Can you describe the setup a little bit more?
>
> Is this site-to-site between two separate NiFi instances, or is it
> within the same cluster for purposes of redistribution?
>
> If it's separate instances then what type of instances, clustered or
> standalone?
>
> Have you made any changes to the configuration of the RPG, the port
> settings within the RPG, or the Input Port on the remote side?
>
> Is some data being transferred through the RPG and it is just slow, or
> is no data making it through?
>
> -Bryan
>
> On Mon, Oct 1, 2018 at 11:53 PM roozbeh karimi <ka...@gmail.com>
> wrote:
> >
> > Hello All,
> >
> > Are there any best practices and guides for setting up RPG? With my set
> up,
> > I am seeing a large volume of back pressure is being built up and hardly
> > any data is being transmitted from the source NiFi node to the nodes in
> > RPG. If I replace RPG with PostHttp on the source node and a listener
> > processor on the destination node, no back pressure is built and the data
> > is transmitted smoothly between the source and destination nodes.
> >
> > I'd appreciate any feedback on to go about tuning my RPG set up.
> >
> > Thanks
> > RK
>

Re: RPG question

[Bryan Bende <bb...@gmail.com>]

Hello,

Can you describe the setup a little bit more?

Is this site-to-site between two separate NiFi instances, or is it
within the same cluster for purposes of redistribution?

If it's separate instances then what type of instances, clustered or standalone?

Have you made any changes to the configuration of the RPG, the port
settings within the RPG, or the Input Port on the remote side?

Is some data being transferred through the RPG and it is just slow, or
is no data making it through?

-Bryan

On Mon, Oct 1, 2018 at 11:53 PM roozbeh karimi <ka...@gmail.com> wrote:
>
> Hello All,
>
> Are there any best practices and guides for setting up RPG? With my set up,
> I am seeing a large volume of back pressure is being built up and hardly
> any data is being transmitted from the source NiFi node to the nodes in
> RPG. If I replace RPG with PostHttp on the source node and a listener
> processor on the destination node, no back pressure is built and the data
> is transmitted smoothly between the source and destination nodes.
>
> I'd appreciate any feedback on to go about tuning my RPG set up.
>
> Thanks
> RK