You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@nifi.apache.org by Clay Teahouse <cl...@gmail.com> on 2018/09/25 03:07:06 UTC

NiFi remote connections

Hi All,

Are the connections between the primary node and RPG persistent, and if no,
is there a way to make them persistent?

Similarly, are the http connection from PostHTTP and InvokeHTTP to the
destination persistent, meaning keep-alive is set to true?

thanks a lot

Clay

Re: NiFi remote connections

Posted by Koji Kawamura <ij...@gmail.com>.

Hi Clay,

Excuse me for the confusing response.
I looked at the source code again and do some testing to see how RPG
(HTTP S2S) manages connections.
It uses both sync and async HTTP client simultaneously and it
establishes multiple connection and those are not persistent.
I just remember that in case of RPG, it was difficult to share a
single connection with multiple HTTP requests especially with async
HTTP client while getting along with the existing S2S mechanism.
Currently HTTP S2S protocol consists of a series of requests and uses
multiple connections.

Thanks,
Koji

On Wed, Sep 26, 2018 at 1:16 PM Clay Teahouse <cl...@gmail.com> wrote:
>
> Thanks for the reply, Koji.
>
> In case of RPG, are there circumstances where the connections are not
> persistent?
>
>
> On Tue, Sep 25, 2018 at 12:14 AM Koji Kawamura <ij...@gmail.com>
> wrote:
>
> > Hi Clay,
> >
> > RPG (Site-to-Site) is a Peer-to-Peer communication protocol. There's
> > no distinction between a primary node and the remote cluster, or nodes
> > other than the primary node and the remote cluster.
> > E.g. With Cluster A (node a1, a2 and a3) and Cluster B (node b1, b2 and b3)
> > Each node must be able to communicate with every remote node. Node a1
> > will communicate with all of node b1, b2 and b3. So do node a2 and a3.
> > Those connections are persistent.
> > S2S RAW uses socket based connection. S2S HTTP uses
> > PoolingHttpClientConnectionManager internally to reuse connection.
> >
> > PostHTTP uses PoolingHttpClientConnectionManager, too.
> > InvokeHTTP uses different HTTP client library, that is okhttp. I
> > didn't check it, but I assume it supports keep-alive.
> >
> > Do you have any specific concern about keep-alive?
> > The keep-alive technology can be used to improve performance. However,
> > we should not depend on that for any load-balancing rule.
> > If you are looking for a solution to distribute FlowFiles based on
> > some rules, NIFI-5516 will be useful (under development).
> > https://issues.apache.org/jira/browse/NIFI-5516
> >
> > Hope this helps.
> >
> > Thanks,
> > Koji
> > On Tue, Sep 25, 2018 at 12:07 PM Clay Teahouse <cl...@gmail.com>
> > wrote:
> > >
> > > Hi All,
> > >
> > > Are the connections between the primary node and RPG persistent, and if
> > no,
> > > is there a way to make them persistent?
> > >
> > > Similarly, are the http connection from PostHTTP and InvokeHTTP to the
> > > destination persistent, meaning keep-alive is set to true?
> > >
> > > thanks a lot
> > >
> > > Clay
> >

Re: NiFi remote connections

Posted by Clay Teahouse <cl...@gmail.com>.

Thanks for the reply, Koji.

In case of RPG, are there circumstances where the connections are not
persistent?


On Tue, Sep 25, 2018 at 12:14 AM Koji Kawamura <ij...@gmail.com>
wrote:

> Hi Clay,
>
> RPG (Site-to-Site) is a Peer-to-Peer communication protocol. There's
> no distinction between a primary node and the remote cluster, or nodes
> other than the primary node and the remote cluster.
> E.g. With Cluster A (node a1, a2 and a3) and Cluster B (node b1, b2 and b3)
> Each node must be able to communicate with every remote node. Node a1
> will communicate with all of node b1, b2 and b3. So do node a2 and a3.
> Those connections are persistent.
> S2S RAW uses socket based connection. S2S HTTP uses
> PoolingHttpClientConnectionManager internally to reuse connection.
>
> PostHTTP uses PoolingHttpClientConnectionManager, too.
> InvokeHTTP uses different HTTP client library, that is okhttp. I
> didn't check it, but I assume it supports keep-alive.
>
> Do you have any specific concern about keep-alive?
> The keep-alive technology can be used to improve performance. However,
> we should not depend on that for any load-balancing rule.
> If you are looking for a solution to distribute FlowFiles based on
> some rules, NIFI-5516 will be useful (under development).
> https://issues.apache.org/jira/browse/NIFI-5516
>
> Hope this helps.
>
> Thanks,
> Koji
> On Tue, Sep 25, 2018 at 12:07 PM Clay Teahouse <cl...@gmail.com>
> wrote:
> >
> > Hi All,
> >
> > Are the connections between the primary node and RPG persistent, and if
> no,
> > is there a way to make them persistent?
> >
> > Similarly, are the http connection from PostHTTP and InvokeHTTP to the
> > destination persistent, meaning keep-alive is set to true?
> >
> > thanks a lot
> >
> > Clay
>

Re: NiFi remote connections

Posted by Koji Kawamura <ij...@gmail.com>.

Hi Clay,

RPG (Site-to-Site) is a Peer-to-Peer communication protocol. There's
no distinction between a primary node and the remote cluster, or nodes
other than the primary node and the remote cluster.
E.g. With Cluster A (node a1, a2 and a3) and Cluster B (node b1, b2 and b3)
Each node must be able to communicate with every remote node. Node a1
will communicate with all of node b1, b2 and b3. So do node a2 and a3.
Those connections are persistent.
S2S RAW uses socket based connection. S2S HTTP uses
PoolingHttpClientConnectionManager internally to reuse connection.

PostHTTP uses PoolingHttpClientConnectionManager, too.
InvokeHTTP uses different HTTP client library, that is okhttp. I
didn't check it, but I assume it supports keep-alive.

Do you have any specific concern about keep-alive?
The keep-alive technology can be used to improve performance. However,
we should not depend on that for any load-balancing rule.
If you are looking for a solution to distribute FlowFiles based on
some rules, NIFI-5516 will be useful (under development).
https://issues.apache.org/jira/browse/NIFI-5516

Hope this helps.

Thanks,
Koji
On Tue, Sep 25, 2018 at 12:07 PM Clay Teahouse <cl...@gmail.com> wrote:
>
> Hi All,
>
> Are the connections between the primary node and RPG persistent, and if no,
> is there a way to make them persistent?
>
> Similarly, are the http connection from PostHTTP and InvokeHTTP to the
> destination persistent, meaning keep-alive is set to true?
>
> thanks a lot
>
> Clay