You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Paresh Shah <Pa...@lifelock.com> on 2016/03/08 20:58:51 UTC
EndpointConnectionPool is not reusable.
We have a use case where we would like to perform some routing based on application defined strategy. In order to be able to do this we need to have a mechanism to pass in the selcted nodes based on the app strategy.
Ideally we would like to pass this strategy into the EndpointConnectionPool. Currently we see that the EndpointConnectionPool is not implemented as an interface and thus SocketClient explicitly contains an instance of this class. Also the logic to perform the load balancing is buried inside the EndpointConnectionPool.
Are there any plans in the near term to make these components extensible.
Thanks
Paresh
________________________________
The information contained in this transmission may contain privileged and confidential information. It is intended only for the use of the person(s) named above. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
________________________________
Re: EndpointConnectionPool is not reusable.
Posted by Paresh Shah <Pa...@lifelock.com>.
Joe,
For our use case we want to be able to do the following.
1. In certain paths we want to be able to send the same FlowFiles to all
nodes of a remote cluster ( broadcast use case )
2. In another use case send it to one of the nodes of the remote cluster (
regular RPG use case ).
What we are working on is a general purpose DistributeProcessor that works
of a application defined strategy. This strategy in-turn would determine
which nodes need to be sent the data to.
We can set up a meeting to discuss this in more detail.
Paresh
On 3/16/16, 7:34 AM, "Joe Percivall" <jo...@yahoo.com.INVALID>
wrote:
>Hello Paresh,
>
>Just to be sure I understand your use-case correctly, you want to
>transfer FlowFiles from one NiFi instance to a NiFi cluster using site to
>site and you want to have the FlowFiles go to a certain node in the
>cluster based on an application specific strategy.
>
>I have not heard any plans to make the routing extensible but that's
>primarily because I don't think any one has asked for it before.
>
>Forgive me if this is a dumb question but, why not group similar nodes
>together into smaller clusters instead of treating them all the same?
>
>Sorry no one had gotten back to you earlier,
>
>Joe
>
>- - - - - -
>Joseph Percivall
>linkedin.com/in/Percivall
>e: joepercivall@yahoo.com
>
>
>
>
>
>On Friday, March 11, 2016 10:19 AM, Paresh Shah
><Pa...@lifelock.com> wrote:
>Have not seen a response from the Nifi team on this. Can someone from the
>team, please share their views on this.
>
>Thanks
>Paresh
>
>
>From: Paresh Shah
><pa...@lifelock.com>>
>Date: Tuesday, March 8, 2016 at 11:58 AM
>To: "dev@nifi.apache.org<ma...@nifi.apache.org>"
><de...@nifi.apache.org>>
>Subject: EndpointConnectionPool is not reusable.
>
>We have a use case where we would like to perform some routing based on
>application defined strategy. In order to be able to do this we need to
>have a mechanism to pass in the selcted nodes based on the app strategy.
>
>Ideally we would like to pass this strategy into the
>EndpointConnectionPool. Currently we see that the EndpointConnectionPool
>is not implemented as an interface and thus SocketClient explicitly
>contains an instance of this class. Also the logic to perform the load
>balancing is buried inside the EndpointConnectionPool.
>
>Are there any plans in the near term to make these components extensible.
>
>Thanks
>Paresh
>
>________________________________
>The information contained in this transmission may contain privileged and
>confidential information. It is intended only for the use of the
>person(s) named above. If you are not the intended recipient, you are
>hereby notified that any review, dissemination, distribution or
>duplication of this communication is strictly prohibited. If you are not
>the intended recipient, please contact the sender by reply email and
>destroy all copies of the original message.
>________________________________
________________________________
The information contained in this transmission may contain privileged and confidential information. It is intended only for the use of the person(s) named above. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
________________________________
Re: EndpointConnectionPool is not reusable.
Posted by Joe Percivall <jo...@yahoo.com.INVALID>.
Hello Paresh,
Just to be sure I understand your use-case correctly, you want to transfer FlowFiles from one NiFi instance to a NiFi cluster using site to site and you want to have the FlowFiles go to a certain node in the cluster based on an application specific strategy.
I have not heard any plans to make the routing extensible but that's primarily because I don't think any one has asked for it before.
Forgive me if this is a dumb question but, why not group similar nodes together into smaller clusters instead of treating them all the same?
Sorry no one had gotten back to you earlier,
Joe
- - - - - -
Joseph Percivall
linkedin.com/in/Percivall
e: joepercivall@yahoo.com
On Friday, March 11, 2016 10:19 AM, Paresh Shah <Pa...@lifelock.com> wrote:
Have not seen a response from the Nifi team on this. Can someone from the team, please share their views on this.
Thanks
Paresh
From: Paresh Shah <pa...@lifelock.com>>
Date: Tuesday, March 8, 2016 at 11:58 AM
To: "dev@nifi.apache.org<ma...@nifi.apache.org>" <de...@nifi.apache.org>>
Subject: EndpointConnectionPool is not reusable.
We have a use case where we would like to perform some routing based on application defined strategy. In order to be able to do this we need to have a mechanism to pass in the selcted nodes based on the app strategy.
Ideally we would like to pass this strategy into the EndpointConnectionPool. Currently we see that the EndpointConnectionPool is not implemented as an interface and thus SocketClient explicitly contains an instance of this class. Also the logic to perform the load balancing is buried inside the EndpointConnectionPool.
Are there any plans in the near term to make these components extensible.
Thanks
Paresh
________________________________
The information contained in this transmission may contain privileged and confidential information. It is intended only for the use of the person(s) named above. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
________________________________
Re: EndpointConnectionPool is not reusable.
Posted by Paresh Shah <Pa...@lifelock.com>.
Have not seen a response from the Nifi team on this. Can someone from the team, please share their views on this.
Thanks
Paresh
From: Paresh Shah <pa...@lifelock.com>>
Date: Tuesday, March 8, 2016 at 11:58 AM
To: "dev@nifi.apache.org<ma...@nifi.apache.org>" <de...@nifi.apache.org>>
Subject: EndpointConnectionPool is not reusable.
We have a use case where we would like to perform some routing based on application defined strategy. In order to be able to do this we need to have a mechanism to pass in the selcted nodes based on the app strategy.
Ideally we would like to pass this strategy into the EndpointConnectionPool. Currently we see that the EndpointConnectionPool is not implemented as an interface and thus SocketClient explicitly contains an instance of this class. Also the logic to perform the load balancing is buried inside the EndpointConnectionPool.
Are there any plans in the near term to make these components extensible.
Thanks
Paresh
________________________________
The information contained in this transmission may contain privileged and confidential information. It is intended only for the use of the person(s) named above. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
________________________________