You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Brian Ghigiarelli <br...@gmail.com> on 2015/08/25 01:36:55 UTC
NiFi Site-to-Site
Is there any step-by-step guide to setting up NiFi Site-to-Site with Remote
Process Groups? Any details on what port range(s) needs to be available?
My setup: NiFi Producer provides data on output port "data-output" to be
picked up by NiFi Consumer.
While trying to get a simple setup running, I set
"nifi.remote.input.secure" to false and "nifi.remote.input.socket.port" to
8082 in NiFi Producer (also added TCP ingress to the firewall). The Remote
Process Group in NiFi Consumer has the "data-output" port turned on and is
simply routing to LogAttribute for verification.
Logs in NiFi Producer indicate that it's periodically establishing a socket
connection with NiFi Consumer over ports in the 54000 - 55999 range, but no
data actually comes across to NiFi Consumer. I added opened NiFi
Consumer's firewall for ingress from 54000 - 55999.
Eventually, we'll need to get it over SSL and lock down the firewall to a
minimum required range, but I'm hoping to prove it out first.
Any guidance would be greatly appreciated!
Thanks,
Brian
RE: NiFi Site-to-Site
Posted by Mark Payne <ma...@hotmail.com>.
Brian,
The node that has the Remote Process Group is the "client". The one that has the port is
the one that returns the hostname. So I believe what is happening is that when you are
pulling data, the node that is trying to pull does not recognize the hostname returned by
the sender. When you are pushing data, the host doing the pushes does recognize
the hostname returned by the receiver.
Also, if you use the "nifi.remote.input.socket.host" property, that is only used right now when
connecting to a standalone node. There is a ticket [1] to make it work in a clustered environment.
[1] https://issues.apache.org/jira/browse/NIFI-872
----------------------------------------
> Date: Thu, 27 Aug 2015 14:40:58 -0400
> Subject: Re: NiFi Site-to-Site
> From: brianghig@gmail.com
> To: dev@nifi.apache.org
>
> Hey Mark,
>
> Mixed results... Setting up an Output Port on the Producer and a Remote
> Process Group on the Receiver is still failing (with the exceptions from
> the previous email).
>
> However, setting up an Input Port on the Receiver and a Remote Process
> Group on the Producer works just fine.
>
> Any ideas what could be going on there?
>
> Thanks,
> Brian
>
> On Thu, Aug 27, 2015 at 9:43 AM, Brian Ghigiarelli <br...@gmail.com>
> wrote:
>
>> Hey Mark,
>>
>> I haven't had a chance to go back to verify, but my guess is that
>> specifying the nifi.remote.input.socket.host property will fix it. The
>> hostname on that box is not a valid hostname for the external NiFi that
>> needed to connect with it. For some reason, that property wasn't in the
>> nifi.properties file, so I didn't know that it was an available option.
>> Thanks for the pointers!
>>
>> Brian
>>
>> On Thu, Aug 27, 2015 at 8:44 AM, Mark Payne <ma...@hotmail.com> wrote:
>>
>>> Hey Brian,
>>>
>>> Just wanted to follow up on this and see if you were able to get
>>> everything resolved, or if you
>>> are still running into problems.
>>>
>>> Thanks
>>> -Mark
>>>
>>> ----------------------------------------
>>>> From: markap14@hotmail.com
>>>> To: dev@nifi.apache.org
>>>> Subject: RE: NiFi Site-to-Site
>>>> Date: Tue, 25 Aug 2015 09:10:53 -0500
>>>>
>>>> Brian,
>>>>
>>>> As part of the site-to-site communication, the client node (i.e., the
>>> NiFi node establishing the connection,
>>>> the one with the Remote Process Group) will connect and request a list
>>> of "peers"
>>>> (i.e., a list of nodes in the cluster). If you are running a very new
>>> version of 0.3.0-SNAPSHOT, you will
>>>> be able to configure what value is returned by that node by setting a
>>> value for the
>>>> "nifi.remote.input.socket.host" property in the nifi.properties file.
>>> If what you are running is a bit older,
>>>> or if that property is not set, what gets returned is the hostname that
>>> is returned by Java.
>>>>
>>>> Generally, when you see this error, it is because the hostname returned
>>> by the remote nifi instance
>>>> is not in your /etc/hosts file on the client.
>>>>
>>>> If you run the "hostname" command on the remote instance, can you
>>> verify that the value returned
>>>> is in /etc/hosts on the client instance?
>>>>
>>>> Thanks
>>>> -Mark
>>>>
>>>> ----------------------------------------
>>>>> Date: Mon, 24 Aug 2015 21:41:19 -0400
>>>>> Subject: Re: NiFi Site-to-Site
>>>>> From: brianghig@gmail.com
>>>>> To: dev@nifi.apache.org
>>>>>
>>>>> Hey Mark,
>>>>>
>>>>> Thanks for the reply and the references. I've taken a look through,
>>> and it
>>>>> seems like everything should be set up properly. I am seeing the
>>> following
>>>>> exception in the NiFi Consumer nifi-app.log. I can ping and wget to the
>>>>> NiFi Producer instance fine from the Consumer box, but Site-to-Site is
>>>>> still giving some trouble. That's why I started looking at AWS Security
>>>>> Groups and Firewall settings. Does this stack trace ring a bell with
>>>>> anything?
>>>>>
>>>>> 2015-08-25 01:34:30,709 WARN [Timer-Driven Process Thread-8]
>>>>> o.a.n.c.t.ContinuallyRunConnectableTask
>>>>> RemoteGroupPort[name=data-output,target=
>>> http://nifi-producer-host:8080/nifi]
>>>>> Administratively Pausing for 10 seconds due to processing failure:
>>>>> java.lang.RuntimeException:
>>> java.nio.channels.UnresolvedAddressException
>>>>> java.lang.RuntimeException:
>>> java.nio.channels.UnresolvedAddressException
>>>>> at
>>>>>
>>> org.apache.nifi.controller.AbstractPort.onTrigger(AbstractPort.java:234)
>>>>> ~[nifi-framework-core-api-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
>>>>> at
>>>>>
>>> org.apache.nifi.controller.tasks.ContinuallyRunConnectableTask.call(ContinuallyRunConnectableTask.java:80)
>>>>> [nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
>>>>> at
>>>>>
>>> org.apache.nifi.controller.tasks.ContinuallyRunConnectableTask.call(ContinuallyRunConnectableTask.java:40)
>>>>> [nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
>>>>> at
>>>>>
>>> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:119)
>>>>> [nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
>>>>> at
>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>> [na:1.8.0_45]
>>>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>>>>> [na:1.8.0_45]
>>>>> at
>>>>>
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>>>>> [na:1.8.0_45]
>>>>> at
>>>>>
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>>>>> [na:1.8.0_45]
>>>>> at
>>>>>
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> [na:1.8.0_45]
>>>>> at
>>>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> [na:1.8.0_45]
>>>>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
>>>>> Caused by: java.nio.channels.UnresolvedAddressException: null
>>>>> at sun.nio.ch.Net.checkAddress(Net.java:123) ~[na:1.8.0_45]
>>>>> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622)
>>>>> ~[na:1.8.0_45]
>>>>> at java.nio.channels.SocketChannel.open(SocketChannel.java:189)
>>>>> ~[na:1.8.0_45]
>>>>> at
>>>>>
>>> org.apache.nifi.remote.client.socket.EndpointConnectionPool.establishSiteToSiteConnection(EndpointConnectionPool.java:708)
>>>>> ~[nifi-site-to-site-client-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
>>>>> at
>>>>>
>>> org.apache.nifi.remote.client.socket.EndpointConnectionPool.establishSiteToSiteConnection(EndpointConnectionPool.java:682)
>>>>> ~[nifi-site-to-site-client-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
>>>>> at
>>>>>
>>> org.apache.nifi.remote.client.socket.EndpointConnectionPool.getEndpointConnection(EndpointConnectionPool.java:300)
>>>>> ~[nifi-site-to-site-client-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
>>>>> at
>>>>>
>>> org.apache.nifi.remote.client.socket.SocketClient.createTransaction(SocketClient.java:129)
>>>>> ~[nifi-site-to-site-client-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
>>>>> at
>>>>>
>>> org.apache.nifi.remote.StandardRemoteGroupPort.onTrigger(StandardRemoteGroupPort.java:167)
>>>>> ~[nifi-site-to-site-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
>>>>> at
>>>>>
>>> org.apache.nifi.controller.AbstractPort.onTrigger(AbstractPort.java:227)
>>>>> ~[nifi-framework-core-api-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
>>>>> ... 10 common frames omitted
>>>>>
>>>>> Thanks again,
>>>>> Brian
>>>>>
>>>>> On Mon, Aug 24, 2015 at 8:05 PM, Mark Payne <ma...@hotmail.com>
>>> wrote:
>>>>>
>>>>>> Hey Brian,
>>>>>>
>>>>>> The latest version of the User Guide has instructions on setting up
>>>>>> site-to-site, but that version
>>>>>> hasn't yet been posted to the website, I don't believe. If you click
>>> the
>>>>>> "help" menu in the top-right
>>>>>> corner of your NiFi instance, and go to the User Guide from there, you
>>>>>> should have the up-to-date
>>>>>> guide there.
>>>>>>
>>>>>> If there's anything missing, please let us know so that we can update
>>> the
>>>>>> guide.
>>>>>>
>>>>>> For convenience, I have pasted the text of that section below:
>>>>>>
>>>>>>
>>>>>> In order to communicate with a remote NiFi instance via Site-to-Site,
>>>>>> simply drag a Remote Process Group onto the graph and enter the URL
>>> of the
>>>>>> remote NiFi instance (for more information on the components of a
>>> Remote
>>>>>> Process Group, see Remote Process Group Transmission section of this
>>>>>> guide.) The URL is the same URL you would use to go to that
>>> instance’s User
>>>>>> Interface. At that point, you can drag a connection to or from the
>>> Remote
>>>>>> Process Group in the same way you would drag a connection to or from a
>>>>>> Processor or a local Process Group. When you drag the connection, you
>>> will
>>>>>> have a chance to choose which Port to connect to. Note that it may
>>> take up
>>>>>> to one minute for the Remote Process Group to determine which ports
>>> are
>>>>>> available.
>>>>>>
>>>>>> If the connection is dragged starting from the Remote Process Group,
>>> the
>>>>>> ports shown will be the Output Ports of the remote group, as this
>>> indicates
>>>>>> that you will be pulling data from the remote instance. If the
>>> connection
>>>>>> instead ends on the Remote Process Group, the ports shown will be the
>>> Input
>>>>>> Ports of the remote group, as this implies that you will be pushing
>>> data to
>>>>>> the remote instance.
>>>>>>
>>>>>> Note: if the remote instance is configured to use secure data
>>>>>> transmission, you will see only ports that you are authorized to
>>>>>> communicate with. For information on configuring NiFi to run
>>> securely, see
>>>>>> the Admin Guide.
>>>>>>
>>>>>> In order to allow another NiFi instance to push data to your local
>>>>>> instance, you can simply drag an Input Port onto the Root Process
>>> Group of
>>>>>> your graph. After entering a name for the port, it will be added to
>>> your
>>>>>> flow. You can now right-click on the Input Port and choose Configure
>>> in
>>>>>> order to adjust the name and the number of concurrent tasks that are
>>> used
>>>>>> for the port. If Site-to-Site is configured to run securely, you will
>>> also
>>>>>> be given the ability to adjust who has access to the port. If secure,
>>> only
>>>>>> those who have been granted access to communicate with the port will
>>> be
>>>>>> able to see that the port exists.
>>>>>>
>>>>>> After being given access to a particular port, in order to see that
>>> port,
>>>>>> the operator of a remote NiFi instance may need to right-click on
>>> their
>>>>>> Remote Process Group and choose to "Refresh" the flow.
>>>>>>
>>>>>> Similar to an Input Port, a DataFlow Manager may choose to add an
>>> Output
>>>>>> Port to the Root Process Group. The Output Port allows an authorized
>>> NiFi
>>>>>> instance to remotely connect to your instance and pull data from the
>>> Output
>>>>>> Port. Configuring the Output Port will again allow the DFM to control
>>> how
>>>>>> many concurrent tasks are allowed, as well as which NiFi instances are
>>>>>> authorized to pull data from the instance being configured.
>>>>>>
>>>>>> In addition to other instances of NiFi, some other applications may
>>> use a
>>>>>> Site-to-Site client in order to push data to or receive data from a
>>> NiFi
>>>>>> instance. For example, NiFi provides an Apache Storm spout and an
>>> Apache
>>>>>> Spark Receiver that are able to pull data from NiFi’s Root Group
>>> Output
>>>>>> Ports.
>>>>>>
>>>>>> If your instance of NiFi is running securely, the first time that a
>>> client
>>>>>> establishes a connection to your instance, the client will be
>>> forbidden and
>>>>>> a request for an account for that client will automatically be
>>> generated.
>>>>>> The client will need to be granted the NiFi role in order to
>>> communicate
>>>>>> via Site-to-Site. For more information on managing user accounts, see
>>>>>> theControlling Levels of Access section of the Admin Guide.
>>>>>>
>>>>>> For information on how to enable and configure Site-to-Site on a NiFi
>>>>>> instance, see the Site-to-Site Properties section of the Admin Guide.
>>>>>>
>>>>>> Thanks
>>>>>> -Mark
>>>>>>
>>>>>>
>>>>>> ----------------------------------------
>>>>>>> Date: Mon, 24 Aug 2015 19:36:55 -0400
>>>>>>> Subject: NiFi Site-to-Site
>>>>>>> From: brianghig@gmail.com
>>>>>>> To: dev@nifi.apache.org
>>>>>>>
>>>>>>> Is there any step-by-step guide to setting up NiFi Site-to-Site with
>>>>>> Remote
>>>>>>> Process Groups? Any details on what port range(s) needs to be
>>> available?
>>>>>>>
>>>>>>> My setup: NiFi Producer provides data on output port "data-output"
>>> to be
>>>>>>> picked up by NiFi Consumer.
>>>>>>>
>>>>>>> While trying to get a simple setup running, I set
>>>>>>> "nifi.remote.input.secure" to false and
>>> "nifi.remote.input.socket.port"
>>>>>> to
>>>>>>> 8082 in NiFi Producer (also added TCP ingress to the firewall). The
>>>>>> Remote
>>>>>>> Process Group in NiFi Consumer has the "data-output" port turned on
>>> and
>>>>>> is
>>>>>>> simply routing to LogAttribute for verification.
>>>>>>>
>>>>>>> Logs in NiFi Producer indicate that it's periodically establishing a
>>>>>> socket
>>>>>>> connection with NiFi Consumer over ports in the 54000 - 55999 range,
>>> but
>>>>>> no
>>>>>>> data actually comes across to NiFi Consumer. I added opened NiFi
>>>>>>> Consumer's firewall for ingress from 54000 - 55999.
>>>>>>>
>>>>>>> Eventually, we'll need to get it over SSL and lock down the firewall
>>> to a
>>>>>>> minimum required range, but I'm hoping to prove it out first.
>>>>>>>
>>>>>>> Any guidance would be greatly appreciated!
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Brian
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Brian Ghigiarelli
>>>>> 570-878-9139
>>>>
>>>
>>>
>>
>>
>>
>> --
>> Brian Ghigiarelli
>> 570-878-9139
>>
>
>
>
> --
> Brian Ghigiarelli
> 570-878-9139
Re: NiFi Site-to-Site
Posted by Brian Ghigiarelli <br...@gmail.com>.
Hey Mark,
Mixed results... Setting up an Output Port on the Producer and a Remote
Process Group on the Receiver is still failing (with the exceptions from
the previous email).
However, setting up an Input Port on the Receiver and a Remote Process
Group on the Producer works just fine.
Any ideas what could be going on there?
Thanks,
Brian
On Thu, Aug 27, 2015 at 9:43 AM, Brian Ghigiarelli <br...@gmail.com>
wrote:
> Hey Mark,
>
> I haven't had a chance to go back to verify, but my guess is that
> specifying the nifi.remote.input.socket.host property will fix it. The
> hostname on that box is not a valid hostname for the external NiFi that
> needed to connect with it. For some reason, that property wasn't in the
> nifi.properties file, so I didn't know that it was an available option.
> Thanks for the pointers!
>
> Brian
>
> On Thu, Aug 27, 2015 at 8:44 AM, Mark Payne <ma...@hotmail.com> wrote:
>
>> Hey Brian,
>>
>> Just wanted to follow up on this and see if you were able to get
>> everything resolved, or if you
>> are still running into problems.
>>
>> Thanks
>> -Mark
>>
>> ----------------------------------------
>> > From: markap14@hotmail.com
>> > To: dev@nifi.apache.org
>> > Subject: RE: NiFi Site-to-Site
>> > Date: Tue, 25 Aug 2015 09:10:53 -0500
>> >
>> > Brian,
>> >
>> > As part of the site-to-site communication, the client node (i.e., the
>> NiFi node establishing the connection,
>> > the one with the Remote Process Group) will connect and request a list
>> of "peers"
>> > (i.e., a list of nodes in the cluster). If you are running a very new
>> version of 0.3.0-SNAPSHOT, you will
>> > be able to configure what value is returned by that node by setting a
>> value for the
>> > "nifi.remote.input.socket.host" property in the nifi.properties file.
>> If what you are running is a bit older,
>> > or if that property is not set, what gets returned is the hostname that
>> is returned by Java.
>> >
>> > Generally, when you see this error, it is because the hostname returned
>> by the remote nifi instance
>> > is not in your /etc/hosts file on the client.
>> >
>> > If you run the "hostname" command on the remote instance, can you
>> verify that the value returned
>> > is in /etc/hosts on the client instance?
>> >
>> > Thanks
>> > -Mark
>> >
>> > ----------------------------------------
>> >> Date: Mon, 24 Aug 2015 21:41:19 -0400
>> >> Subject: Re: NiFi Site-to-Site
>> >> From: brianghig@gmail.com
>> >> To: dev@nifi.apache.org
>> >>
>> >> Hey Mark,
>> >>
>> >> Thanks for the reply and the references. I've taken a look through,
>> and it
>> >> seems like everything should be set up properly. I am seeing the
>> following
>> >> exception in the NiFi Consumer nifi-app.log. I can ping and wget to the
>> >> NiFi Producer instance fine from the Consumer box, but Site-to-Site is
>> >> still giving some trouble. That's why I started looking at AWS Security
>> >> Groups and Firewall settings. Does this stack trace ring a bell with
>> >> anything?
>> >>
>> >> 2015-08-25 01:34:30,709 WARN [Timer-Driven Process Thread-8]
>> >> o.a.n.c.t.ContinuallyRunConnectableTask
>> >> RemoteGroupPort[name=data-output,target=
>> http://nifi-producer-host:8080/nifi]
>> >> Administratively Pausing for 10 seconds due to processing failure:
>> >> java.lang.RuntimeException:
>> java.nio.channels.UnresolvedAddressException
>> >> java.lang.RuntimeException:
>> java.nio.channels.UnresolvedAddressException
>> >> at
>> >>
>> org.apache.nifi.controller.AbstractPort.onTrigger(AbstractPort.java:234)
>> >> ~[nifi-framework-core-api-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
>> >> at
>> >>
>> org.apache.nifi.controller.tasks.ContinuallyRunConnectableTask.call(ContinuallyRunConnectableTask.java:80)
>> >> [nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
>> >> at
>> >>
>> org.apache.nifi.controller.tasks.ContinuallyRunConnectableTask.call(ContinuallyRunConnectableTask.java:40)
>> >> [nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
>> >> at
>> >>
>> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:119)
>> >> [nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
>> >> at
>> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> >> [na:1.8.0_45]
>> >> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>> >> [na:1.8.0_45]
>> >> at
>> >>
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>> >> [na:1.8.0_45]
>> >> at
>> >>
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>> >> [na:1.8.0_45]
>> >> at
>> >>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> >> [na:1.8.0_45]
>> >> at
>> >>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> >> [na:1.8.0_45]
>> >> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
>> >> Caused by: java.nio.channels.UnresolvedAddressException: null
>> >> at sun.nio.ch.Net.checkAddress(Net.java:123) ~[na:1.8.0_45]
>> >> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622)
>> >> ~[na:1.8.0_45]
>> >> at java.nio.channels.SocketChannel.open(SocketChannel.java:189)
>> >> ~[na:1.8.0_45]
>> >> at
>> >>
>> org.apache.nifi.remote.client.socket.EndpointConnectionPool.establishSiteToSiteConnection(EndpointConnectionPool.java:708)
>> >> ~[nifi-site-to-site-client-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
>> >> at
>> >>
>> org.apache.nifi.remote.client.socket.EndpointConnectionPool.establishSiteToSiteConnection(EndpointConnectionPool.java:682)
>> >> ~[nifi-site-to-site-client-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
>> >> at
>> >>
>> org.apache.nifi.remote.client.socket.EndpointConnectionPool.getEndpointConnection(EndpointConnectionPool.java:300)
>> >> ~[nifi-site-to-site-client-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
>> >> at
>> >>
>> org.apache.nifi.remote.client.socket.SocketClient.createTransaction(SocketClient.java:129)
>> >> ~[nifi-site-to-site-client-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
>> >> at
>> >>
>> org.apache.nifi.remote.StandardRemoteGroupPort.onTrigger(StandardRemoteGroupPort.java:167)
>> >> ~[nifi-site-to-site-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
>> >> at
>> >>
>> org.apache.nifi.controller.AbstractPort.onTrigger(AbstractPort.java:227)
>> >> ~[nifi-framework-core-api-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
>> >> ... 10 common frames omitted
>> >>
>> >> Thanks again,
>> >> Brian
>> >>
>> >> On Mon, Aug 24, 2015 at 8:05 PM, Mark Payne <ma...@hotmail.com>
>> wrote:
>> >>
>> >>> Hey Brian,
>> >>>
>> >>> The latest version of the User Guide has instructions on setting up
>> >>> site-to-site, but that version
>> >>> hasn't yet been posted to the website, I don't believe. If you click
>> the
>> >>> "help" menu in the top-right
>> >>> corner of your NiFi instance, and go to the User Guide from there, you
>> >>> should have the up-to-date
>> >>> guide there.
>> >>>
>> >>> If there's anything missing, please let us know so that we can update
>> the
>> >>> guide.
>> >>>
>> >>> For convenience, I have pasted the text of that section below:
>> >>>
>> >>>
>> >>> In order to communicate with a remote NiFi instance via Site-to-Site,
>> >>> simply drag a Remote Process Group onto the graph and enter the URL
>> of the
>> >>> remote NiFi instance (for more information on the components of a
>> Remote
>> >>> Process Group, see Remote Process Group Transmission section of this
>> >>> guide.) The URL is the same URL you would use to go to that
>> instance’s User
>> >>> Interface. At that point, you can drag a connection to or from the
>> Remote
>> >>> Process Group in the same way you would drag a connection to or from a
>> >>> Processor or a local Process Group. When you drag the connection, you
>> will
>> >>> have a chance to choose which Port to connect to. Note that it may
>> take up
>> >>> to one minute for the Remote Process Group to determine which ports
>> are
>> >>> available.
>> >>>
>> >>> If the connection is dragged starting from the Remote Process Group,
>> the
>> >>> ports shown will be the Output Ports of the remote group, as this
>> indicates
>> >>> that you will be pulling data from the remote instance. If the
>> connection
>> >>> instead ends on the Remote Process Group, the ports shown will be the
>> Input
>> >>> Ports of the remote group, as this implies that you will be pushing
>> data to
>> >>> the remote instance.
>> >>>
>> >>> Note: if the remote instance is configured to use secure data
>> >>> transmission, you will see only ports that you are authorized to
>> >>> communicate with. For information on configuring NiFi to run
>> securely, see
>> >>> the Admin Guide.
>> >>>
>> >>> In order to allow another NiFi instance to push data to your local
>> >>> instance, you can simply drag an Input Port onto the Root Process
>> Group of
>> >>> your graph. After entering a name for the port, it will be added to
>> your
>> >>> flow. You can now right-click on the Input Port and choose Configure
>> in
>> >>> order to adjust the name and the number of concurrent tasks that are
>> used
>> >>> for the port. If Site-to-Site is configured to run securely, you will
>> also
>> >>> be given the ability to adjust who has access to the port. If secure,
>> only
>> >>> those who have been granted access to communicate with the port will
>> be
>> >>> able to see that the port exists.
>> >>>
>> >>> After being given access to a particular port, in order to see that
>> port,
>> >>> the operator of a remote NiFi instance may need to right-click on
>> their
>> >>> Remote Process Group and choose to "Refresh" the flow.
>> >>>
>> >>> Similar to an Input Port, a DataFlow Manager may choose to add an
>> Output
>> >>> Port to the Root Process Group. The Output Port allows an authorized
>> NiFi
>> >>> instance to remotely connect to your instance and pull data from the
>> Output
>> >>> Port. Configuring the Output Port will again allow the DFM to control
>> how
>> >>> many concurrent tasks are allowed, as well as which NiFi instances are
>> >>> authorized to pull data from the instance being configured.
>> >>>
>> >>> In addition to other instances of NiFi, some other applications may
>> use a
>> >>> Site-to-Site client in order to push data to or receive data from a
>> NiFi
>> >>> instance. For example, NiFi provides an Apache Storm spout and an
>> Apache
>> >>> Spark Receiver that are able to pull data from NiFi’s Root Group
>> Output
>> >>> Ports.
>> >>>
>> >>> If your instance of NiFi is running securely, the first time that a
>> client
>> >>> establishes a connection to your instance, the client will be
>> forbidden and
>> >>> a request for an account for that client will automatically be
>> generated.
>> >>> The client will need to be granted the NiFi role in order to
>> communicate
>> >>> via Site-to-Site. For more information on managing user accounts, see
>> >>> theControlling Levels of Access section of the Admin Guide.
>> >>>
>> >>> For information on how to enable and configure Site-to-Site on a NiFi
>> >>> instance, see the Site-to-Site Properties section of the Admin Guide.
>> >>>
>> >>> Thanks
>> >>> -Mark
>> >>>
>> >>>
>> >>> ----------------------------------------
>> >>>> Date: Mon, 24 Aug 2015 19:36:55 -0400
>> >>>> Subject: NiFi Site-to-Site
>> >>>> From: brianghig@gmail.com
>> >>>> To: dev@nifi.apache.org
>> >>>>
>> >>>> Is there any step-by-step guide to setting up NiFi Site-to-Site with
>> >>> Remote
>> >>>> Process Groups? Any details on what port range(s) needs to be
>> available?
>> >>>>
>> >>>> My setup: NiFi Producer provides data on output port "data-output"
>> to be
>> >>>> picked up by NiFi Consumer.
>> >>>>
>> >>>> While trying to get a simple setup running, I set
>> >>>> "nifi.remote.input.secure" to false and
>> "nifi.remote.input.socket.port"
>> >>> to
>> >>>> 8082 in NiFi Producer (also added TCP ingress to the firewall). The
>> >>> Remote
>> >>>> Process Group in NiFi Consumer has the "data-output" port turned on
>> and
>> >>> is
>> >>>> simply routing to LogAttribute for verification.
>> >>>>
>> >>>> Logs in NiFi Producer indicate that it's periodically establishing a
>> >>> socket
>> >>>> connection with NiFi Consumer over ports in the 54000 - 55999 range,
>> but
>> >>> no
>> >>>> data actually comes across to NiFi Consumer. I added opened NiFi
>> >>>> Consumer's firewall for ingress from 54000 - 55999.
>> >>>>
>> >>>> Eventually, we'll need to get it over SSL and lock down the firewall
>> to a
>> >>>> minimum required range, but I'm hoping to prove it out first.
>> >>>>
>> >>>> Any guidance would be greatly appreciated!
>> >>>>
>> >>>> Thanks,
>> >>>> Brian
>> >>>
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> Brian Ghigiarelli
>> >> 570-878-9139
>> >
>>
>>
>
>
>
> --
> Brian Ghigiarelli
> 570-878-9139
>
--
Brian Ghigiarelli
570-878-9139
Re: NiFi Site-to-Site
Posted by Brian Ghigiarelli <br...@gmail.com>.
Hey Mark,
I haven't had a chance to go back to verify, but my guess is that
specifying the nifi.remote.input.socket.host property will fix it. The
hostname on that box is not a valid hostname for the external NiFi that
needed to connect with it. For some reason, that property wasn't in the
nifi.properties file, so I didn't know that it was an available option.
Thanks for the pointers!
Brian
On Thu, Aug 27, 2015 at 8:44 AM, Mark Payne <ma...@hotmail.com> wrote:
> Hey Brian,
>
> Just wanted to follow up on this and see if you were able to get
> everything resolved, or if you
> are still running into problems.
>
> Thanks
> -Mark
>
> ----------------------------------------
> > From: markap14@hotmail.com
> > To: dev@nifi.apache.org
> > Subject: RE: NiFi Site-to-Site
> > Date: Tue, 25 Aug 2015 09:10:53 -0500
> >
> > Brian,
> >
> > As part of the site-to-site communication, the client node (i.e., the
> NiFi node establishing the connection,
> > the one with the Remote Process Group) will connect and request a list
> of "peers"
> > (i.e., a list of nodes in the cluster). If you are running a very new
> version of 0.3.0-SNAPSHOT, you will
> > be able to configure what value is returned by that node by setting a
> value for the
> > "nifi.remote.input.socket.host" property in the nifi.properties file. If
> what you are running is a bit older,
> > or if that property is not set, what gets returned is the hostname that
> is returned by Java.
> >
> > Generally, when you see this error, it is because the hostname returned
> by the remote nifi instance
> > is not in your /etc/hosts file on the client.
> >
> > If you run the "hostname" command on the remote instance, can you verify
> that the value returned
> > is in /etc/hosts on the client instance?
> >
> > Thanks
> > -Mark
> >
> > ----------------------------------------
> >> Date: Mon, 24 Aug 2015 21:41:19 -0400
> >> Subject: Re: NiFi Site-to-Site
> >> From: brianghig@gmail.com
> >> To: dev@nifi.apache.org
> >>
> >> Hey Mark,
> >>
> >> Thanks for the reply and the references. I've taken a look through, and
> it
> >> seems like everything should be set up properly. I am seeing the
> following
> >> exception in the NiFi Consumer nifi-app.log. I can ping and wget to the
> >> NiFi Producer instance fine from the Consumer box, but Site-to-Site is
> >> still giving some trouble. That's why I started looking at AWS Security
> >> Groups and Firewall settings. Does this stack trace ring a bell with
> >> anything?
> >>
> >> 2015-08-25 01:34:30,709 WARN [Timer-Driven Process Thread-8]
> >> o.a.n.c.t.ContinuallyRunConnectableTask
> >> RemoteGroupPort[name=data-output,target=
> http://nifi-producer-host:8080/nifi]
> >> Administratively Pausing for 10 seconds due to processing failure:
> >> java.lang.RuntimeException: java.nio.channels.UnresolvedAddressException
> >> java.lang.RuntimeException: java.nio.channels.UnresolvedAddressException
> >> at
> >> org.apache.nifi.controller.AbstractPort.onTrigger(AbstractPort.java:234)
> >> ~[nifi-framework-core-api-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> >> at
> >>
> org.apache.nifi.controller.tasks.ContinuallyRunConnectableTask.call(ContinuallyRunConnectableTask.java:80)
> >> [nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> >> at
> >>
> org.apache.nifi.controller.tasks.ContinuallyRunConnectableTask.call(ContinuallyRunConnectableTask.java:40)
> >> [nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> >> at
> >>
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:119)
> >> [nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> >> at
> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> >> [na:1.8.0_45]
> >> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> >> [na:1.8.0_45]
> >> at
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> >> [na:1.8.0_45]
> >> at
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> >> [na:1.8.0_45]
> >> at
> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >> [na:1.8.0_45]
> >> at
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >> [na:1.8.0_45]
> >> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> >> Caused by: java.nio.channels.UnresolvedAddressException: null
> >> at sun.nio.ch.Net.checkAddress(Net.java:123) ~[na:1.8.0_45]
> >> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622)
> >> ~[na:1.8.0_45]
> >> at java.nio.channels.SocketChannel.open(SocketChannel.java:189)
> >> ~[na:1.8.0_45]
> >> at
> >>
> org.apache.nifi.remote.client.socket.EndpointConnectionPool.establishSiteToSiteConnection(EndpointConnectionPool.java:708)
> >> ~[nifi-site-to-site-client-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> >> at
> >>
> org.apache.nifi.remote.client.socket.EndpointConnectionPool.establishSiteToSiteConnection(EndpointConnectionPool.java:682)
> >> ~[nifi-site-to-site-client-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> >> at
> >>
> org.apache.nifi.remote.client.socket.EndpointConnectionPool.getEndpointConnection(EndpointConnectionPool.java:300)
> >> ~[nifi-site-to-site-client-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> >> at
> >>
> org.apache.nifi.remote.client.socket.SocketClient.createTransaction(SocketClient.java:129)
> >> ~[nifi-site-to-site-client-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> >> at
> >>
> org.apache.nifi.remote.StandardRemoteGroupPort.onTrigger(StandardRemoteGroupPort.java:167)
> >> ~[nifi-site-to-site-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> >> at
> >> org.apache.nifi.controller.AbstractPort.onTrigger(AbstractPort.java:227)
> >> ~[nifi-framework-core-api-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> >> ... 10 common frames omitted
> >>
> >> Thanks again,
> >> Brian
> >>
> >> On Mon, Aug 24, 2015 at 8:05 PM, Mark Payne <ma...@hotmail.com>
> wrote:
> >>
> >>> Hey Brian,
> >>>
> >>> The latest version of the User Guide has instructions on setting up
> >>> site-to-site, but that version
> >>> hasn't yet been posted to the website, I don't believe. If you click
> the
> >>> "help" menu in the top-right
> >>> corner of your NiFi instance, and go to the User Guide from there, you
> >>> should have the up-to-date
> >>> guide there.
> >>>
> >>> If there's anything missing, please let us know so that we can update
> the
> >>> guide.
> >>>
> >>> For convenience, I have pasted the text of that section below:
> >>>
> >>>
> >>> In order to communicate with a remote NiFi instance via Site-to-Site,
> >>> simply drag a Remote Process Group onto the graph and enter the URL of
> the
> >>> remote NiFi instance (for more information on the components of a
> Remote
> >>> Process Group, see Remote Process Group Transmission section of this
> >>> guide.) The URL is the same URL you would use to go to that instance’s
> User
> >>> Interface. At that point, you can drag a connection to or from the
> Remote
> >>> Process Group in the same way you would drag a connection to or from a
> >>> Processor or a local Process Group. When you drag the connection, you
> will
> >>> have a chance to choose which Port to connect to. Note that it may
> take up
> >>> to one minute for the Remote Process Group to determine which ports are
> >>> available.
> >>>
> >>> If the connection is dragged starting from the Remote Process Group,
> the
> >>> ports shown will be the Output Ports of the remote group, as this
> indicates
> >>> that you will be pulling data from the remote instance. If the
> connection
> >>> instead ends on the Remote Process Group, the ports shown will be the
> Input
> >>> Ports of the remote group, as this implies that you will be pushing
> data to
> >>> the remote instance.
> >>>
> >>> Note: if the remote instance is configured to use secure data
> >>> transmission, you will see only ports that you are authorized to
> >>> communicate with. For information on configuring NiFi to run securely,
> see
> >>> the Admin Guide.
> >>>
> >>> In order to allow another NiFi instance to push data to your local
> >>> instance, you can simply drag an Input Port onto the Root Process
> Group of
> >>> your graph. After entering a name for the port, it will be added to
> your
> >>> flow. You can now right-click on the Input Port and choose Configure in
> >>> order to adjust the name and the number of concurrent tasks that are
> used
> >>> for the port. If Site-to-Site is configured to run securely, you will
> also
> >>> be given the ability to adjust who has access to the port. If secure,
> only
> >>> those who have been granted access to communicate with the port will be
> >>> able to see that the port exists.
> >>>
> >>> After being given access to a particular port, in order to see that
> port,
> >>> the operator of a remote NiFi instance may need to right-click on their
> >>> Remote Process Group and choose to "Refresh" the flow.
> >>>
> >>> Similar to an Input Port, a DataFlow Manager may choose to add an
> Output
> >>> Port to the Root Process Group. The Output Port allows an authorized
> NiFi
> >>> instance to remotely connect to your instance and pull data from the
> Output
> >>> Port. Configuring the Output Port will again allow the DFM to control
> how
> >>> many concurrent tasks are allowed, as well as which NiFi instances are
> >>> authorized to pull data from the instance being configured.
> >>>
> >>> In addition to other instances of NiFi, some other applications may
> use a
> >>> Site-to-Site client in order to push data to or receive data from a
> NiFi
> >>> instance. For example, NiFi provides an Apache Storm spout and an
> Apache
> >>> Spark Receiver that are able to pull data from NiFi’s Root Group Output
> >>> Ports.
> >>>
> >>> If your instance of NiFi is running securely, the first time that a
> client
> >>> establishes a connection to your instance, the client will be
> forbidden and
> >>> a request for an account for that client will automatically be
> generated.
> >>> The client will need to be granted the NiFi role in order to
> communicate
> >>> via Site-to-Site. For more information on managing user accounts, see
> >>> theControlling Levels of Access section of the Admin Guide.
> >>>
> >>> For information on how to enable and configure Site-to-Site on a NiFi
> >>> instance, see the Site-to-Site Properties section of the Admin Guide.
> >>>
> >>> Thanks
> >>> -Mark
> >>>
> >>>
> >>> ----------------------------------------
> >>>> Date: Mon, 24 Aug 2015 19:36:55 -0400
> >>>> Subject: NiFi Site-to-Site
> >>>> From: brianghig@gmail.com
> >>>> To: dev@nifi.apache.org
> >>>>
> >>>> Is there any step-by-step guide to setting up NiFi Site-to-Site with
> >>> Remote
> >>>> Process Groups? Any details on what port range(s) needs to be
> available?
> >>>>
> >>>> My setup: NiFi Producer provides data on output port "data-output" to
> be
> >>>> picked up by NiFi Consumer.
> >>>>
> >>>> While trying to get a simple setup running, I set
> >>>> "nifi.remote.input.secure" to false and
> "nifi.remote.input.socket.port"
> >>> to
> >>>> 8082 in NiFi Producer (also added TCP ingress to the firewall). The
> >>> Remote
> >>>> Process Group in NiFi Consumer has the "data-output" port turned on
> and
> >>> is
> >>>> simply routing to LogAttribute for verification.
> >>>>
> >>>> Logs in NiFi Producer indicate that it's periodically establishing a
> >>> socket
> >>>> connection with NiFi Consumer over ports in the 54000 - 55999 range,
> but
> >>> no
> >>>> data actually comes across to NiFi Consumer. I added opened NiFi
> >>>> Consumer's firewall for ingress from 54000 - 55999.
> >>>>
> >>>> Eventually, we'll need to get it over SSL and lock down the firewall
> to a
> >>>> minimum required range, but I'm hoping to prove it out first.
> >>>>
> >>>> Any guidance would be greatly appreciated!
> >>>>
> >>>> Thanks,
> >>>> Brian
> >>>
> >>>
> >>
> >>
> >>
> >> --
> >> Brian Ghigiarelli
> >> 570-878-9139
> >
>
>
--
Brian Ghigiarelli
570-878-9139
RE: NiFi Site-to-Site
Posted by Mark Payne <ma...@hotmail.com>.
Hey Brian,
Just wanted to follow up on this and see if you were able to get everything resolved, or if you
are still running into problems.
Thanks
-Mark
----------------------------------------
> From: markap14@hotmail.com
> To: dev@nifi.apache.org
> Subject: RE: NiFi Site-to-Site
> Date: Tue, 25 Aug 2015 09:10:53 -0500
>
> Brian,
>
> As part of the site-to-site communication, the client node (i.e., the NiFi node establishing the connection,
> the one with the Remote Process Group) will connect and request a list of "peers"
> (i.e., a list of nodes in the cluster). If you are running a very new version of 0.3.0-SNAPSHOT, you will
> be able to configure what value is returned by that node by setting a value for the
> "nifi.remote.input.socket.host" property in the nifi.properties file. If what you are running is a bit older,
> or if that property is not set, what gets returned is the hostname that is returned by Java.
>
> Generally, when you see this error, it is because the hostname returned by the remote nifi instance
> is not in your /etc/hosts file on the client.
>
> If you run the "hostname" command on the remote instance, can you verify that the value returned
> is in /etc/hosts on the client instance?
>
> Thanks
> -Mark
>
> ----------------------------------------
>> Date: Mon, 24 Aug 2015 21:41:19 -0400
>> Subject: Re: NiFi Site-to-Site
>> From: brianghig@gmail.com
>> To: dev@nifi.apache.org
>>
>> Hey Mark,
>>
>> Thanks for the reply and the references. I've taken a look through, and it
>> seems like everything should be set up properly. I am seeing the following
>> exception in the NiFi Consumer nifi-app.log. I can ping and wget to the
>> NiFi Producer instance fine from the Consumer box, but Site-to-Site is
>> still giving some trouble. That's why I started looking at AWS Security
>> Groups and Firewall settings. Does this stack trace ring a bell with
>> anything?
>>
>> 2015-08-25 01:34:30,709 WARN [Timer-Driven Process Thread-8]
>> o.a.n.c.t.ContinuallyRunConnectableTask
>> RemoteGroupPort[name=data-output,target=http://nifi-producer-host:8080/nifi]
>> Administratively Pausing for 10 seconds due to processing failure:
>> java.lang.RuntimeException: java.nio.channels.UnresolvedAddressException
>> java.lang.RuntimeException: java.nio.channels.UnresolvedAddressException
>> at
>> org.apache.nifi.controller.AbstractPort.onTrigger(AbstractPort.java:234)
>> ~[nifi-framework-core-api-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
>> at
>> org.apache.nifi.controller.tasks.ContinuallyRunConnectableTask.call(ContinuallyRunConnectableTask.java:80)
>> [nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
>> at
>> org.apache.nifi.controller.tasks.ContinuallyRunConnectableTask.call(ContinuallyRunConnectableTask.java:40)
>> [nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
>> at
>> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:119)
>> [nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> [na:1.8.0_45]
>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>> [na:1.8.0_45]
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>> [na:1.8.0_45]
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>> [na:1.8.0_45]
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> [na:1.8.0_45]
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> [na:1.8.0_45]
>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
>> Caused by: java.nio.channels.UnresolvedAddressException: null
>> at sun.nio.ch.Net.checkAddress(Net.java:123) ~[na:1.8.0_45]
>> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622)
>> ~[na:1.8.0_45]
>> at java.nio.channels.SocketChannel.open(SocketChannel.java:189)
>> ~[na:1.8.0_45]
>> at
>> org.apache.nifi.remote.client.socket.EndpointConnectionPool.establishSiteToSiteConnection(EndpointConnectionPool.java:708)
>> ~[nifi-site-to-site-client-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
>> at
>> org.apache.nifi.remote.client.socket.EndpointConnectionPool.establishSiteToSiteConnection(EndpointConnectionPool.java:682)
>> ~[nifi-site-to-site-client-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
>> at
>> org.apache.nifi.remote.client.socket.EndpointConnectionPool.getEndpointConnection(EndpointConnectionPool.java:300)
>> ~[nifi-site-to-site-client-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
>> at
>> org.apache.nifi.remote.client.socket.SocketClient.createTransaction(SocketClient.java:129)
>> ~[nifi-site-to-site-client-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
>> at
>> org.apache.nifi.remote.StandardRemoteGroupPort.onTrigger(StandardRemoteGroupPort.java:167)
>> ~[nifi-site-to-site-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
>> at
>> org.apache.nifi.controller.AbstractPort.onTrigger(AbstractPort.java:227)
>> ~[nifi-framework-core-api-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
>> ... 10 common frames omitted
>>
>> Thanks again,
>> Brian
>>
>> On Mon, Aug 24, 2015 at 8:05 PM, Mark Payne <ma...@hotmail.com> wrote:
>>
>>> Hey Brian,
>>>
>>> The latest version of the User Guide has instructions on setting up
>>> site-to-site, but that version
>>> hasn't yet been posted to the website, I don't believe. If you click the
>>> "help" menu in the top-right
>>> corner of your NiFi instance, and go to the User Guide from there, you
>>> should have the up-to-date
>>> guide there.
>>>
>>> If there's anything missing, please let us know so that we can update the
>>> guide.
>>>
>>> For convenience, I have pasted the text of that section below:
>>>
>>>
>>> In order to communicate with a remote NiFi instance via Site-to-Site,
>>> simply drag a Remote Process Group onto the graph and enter the URL of the
>>> remote NiFi instance (for more information on the components of a Remote
>>> Process Group, see Remote Process Group Transmission section of this
>>> guide.) The URL is the same URL you would use to go to that instance’s User
>>> Interface. At that point, you can drag a connection to or from the Remote
>>> Process Group in the same way you would drag a connection to or from a
>>> Processor or a local Process Group. When you drag the connection, you will
>>> have a chance to choose which Port to connect to. Note that it may take up
>>> to one minute for the Remote Process Group to determine which ports are
>>> available.
>>>
>>> If the connection is dragged starting from the Remote Process Group, the
>>> ports shown will be the Output Ports of the remote group, as this indicates
>>> that you will be pulling data from the remote instance. If the connection
>>> instead ends on the Remote Process Group, the ports shown will be the Input
>>> Ports of the remote group, as this implies that you will be pushing data to
>>> the remote instance.
>>>
>>> Note: if the remote instance is configured to use secure data
>>> transmission, you will see only ports that you are authorized to
>>> communicate with. For information on configuring NiFi to run securely, see
>>> the Admin Guide.
>>>
>>> In order to allow another NiFi instance to push data to your local
>>> instance, you can simply drag an Input Port onto the Root Process Group of
>>> your graph. After entering a name for the port, it will be added to your
>>> flow. You can now right-click on the Input Port and choose Configure in
>>> order to adjust the name and the number of concurrent tasks that are used
>>> for the port. If Site-to-Site is configured to run securely, you will also
>>> be given the ability to adjust who has access to the port. If secure, only
>>> those who have been granted access to communicate with the port will be
>>> able to see that the port exists.
>>>
>>> After being given access to a particular port, in order to see that port,
>>> the operator of a remote NiFi instance may need to right-click on their
>>> Remote Process Group and choose to "Refresh" the flow.
>>>
>>> Similar to an Input Port, a DataFlow Manager may choose to add an Output
>>> Port to the Root Process Group. The Output Port allows an authorized NiFi
>>> instance to remotely connect to your instance and pull data from the Output
>>> Port. Configuring the Output Port will again allow the DFM to control how
>>> many concurrent tasks are allowed, as well as which NiFi instances are
>>> authorized to pull data from the instance being configured.
>>>
>>> In addition to other instances of NiFi, some other applications may use a
>>> Site-to-Site client in order to push data to or receive data from a NiFi
>>> instance. For example, NiFi provides an Apache Storm spout and an Apache
>>> Spark Receiver that are able to pull data from NiFi’s Root Group Output
>>> Ports.
>>>
>>> If your instance of NiFi is running securely, the first time that a client
>>> establishes a connection to your instance, the client will be forbidden and
>>> a request for an account for that client will automatically be generated.
>>> The client will need to be granted the NiFi role in order to communicate
>>> via Site-to-Site. For more information on managing user accounts, see
>>> theControlling Levels of Access section of the Admin Guide.
>>>
>>> For information on how to enable and configure Site-to-Site on a NiFi
>>> instance, see the Site-to-Site Properties section of the Admin Guide.
>>>
>>> Thanks
>>> -Mark
>>>
>>>
>>> ----------------------------------------
>>>> Date: Mon, 24 Aug 2015 19:36:55 -0400
>>>> Subject: NiFi Site-to-Site
>>>> From: brianghig@gmail.com
>>>> To: dev@nifi.apache.org
>>>>
>>>> Is there any step-by-step guide to setting up NiFi Site-to-Site with
>>> Remote
>>>> Process Groups? Any details on what port range(s) needs to be available?
>>>>
>>>> My setup: NiFi Producer provides data on output port "data-output" to be
>>>> picked up by NiFi Consumer.
>>>>
>>>> While trying to get a simple setup running, I set
>>>> "nifi.remote.input.secure" to false and "nifi.remote.input.socket.port"
>>> to
>>>> 8082 in NiFi Producer (also added TCP ingress to the firewall). The
>>> Remote
>>>> Process Group in NiFi Consumer has the "data-output" port turned on and
>>> is
>>>> simply routing to LogAttribute for verification.
>>>>
>>>> Logs in NiFi Producer indicate that it's periodically establishing a
>>> socket
>>>> connection with NiFi Consumer over ports in the 54000 - 55999 range, but
>>> no
>>>> data actually comes across to NiFi Consumer. I added opened NiFi
>>>> Consumer's firewall for ingress from 54000 - 55999.
>>>>
>>>> Eventually, we'll need to get it over SSL and lock down the firewall to a
>>>> minimum required range, but I'm hoping to prove it out first.
>>>>
>>>> Any guidance would be greatly appreciated!
>>>>
>>>> Thanks,
>>>> Brian
>>>
>>>
>>
>>
>>
>> --
>> Brian Ghigiarelli
>> 570-878-9139
>
RE: NiFi Site-to-Site
Posted by Mark Payne <ma...@hotmail.com>.
Brian,
As part of the site-to-site communication, the client node (i.e., the NiFi node establishing the connection,
the one with the Remote Process Group) will connect and request a list of "peers"
(i.e., a list of nodes in the cluster). If you are running a very new version of 0.3.0-SNAPSHOT, you will
be able to configure what value is returned by that node by setting a value for the
"nifi.remote.input.socket.host" property in the nifi.properties file. If what you are running is a bit older,
or if that property is not set, what gets returned is the hostname that is returned by Java.
Generally, when you see this error, it is because the hostname returned by the remote nifi instance
is not in your /etc/hosts file on the client.
If you run the "hostname" command on the remote instance, can you verify that the value returned
is in /etc/hosts on the client instance?
Thanks
-Mark
----------------------------------------
> Date: Mon, 24 Aug 2015 21:41:19 -0400
> Subject: Re: NiFi Site-to-Site
> From: brianghig@gmail.com
> To: dev@nifi.apache.org
>
> Hey Mark,
>
> Thanks for the reply and the references. I've taken a look through, and it
> seems like everything should be set up properly. I am seeing the following
> exception in the NiFi Consumer nifi-app.log. I can ping and wget to the
> NiFi Producer instance fine from the Consumer box, but Site-to-Site is
> still giving some trouble. That's why I started looking at AWS Security
> Groups and Firewall settings. Does this stack trace ring a bell with
> anything?
>
> 2015-08-25 01:34:30,709 WARN [Timer-Driven Process Thread-8]
> o.a.n.c.t.ContinuallyRunConnectableTask
> RemoteGroupPort[name=data-output,target=http://nifi-producer-host:8080/nifi]
> Administratively Pausing for 10 seconds due to processing failure:
> java.lang.RuntimeException: java.nio.channels.UnresolvedAddressException
> java.lang.RuntimeException: java.nio.channels.UnresolvedAddressException
> at
> org.apache.nifi.controller.AbstractPort.onTrigger(AbstractPort.java:234)
> ~[nifi-framework-core-api-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> at
> org.apache.nifi.controller.tasks.ContinuallyRunConnectableTask.call(ContinuallyRunConnectableTask.java:80)
> [nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> at
> org.apache.nifi.controller.tasks.ContinuallyRunConnectableTask.call(ContinuallyRunConnectableTask.java:40)
> [nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:119)
> [nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_45]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> [na:1.8.0_45]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> [na:1.8.0_45]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> [na:1.8.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_45]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> Caused by: java.nio.channels.UnresolvedAddressException: null
> at sun.nio.ch.Net.checkAddress(Net.java:123) ~[na:1.8.0_45]
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622)
> ~[na:1.8.0_45]
> at java.nio.channels.SocketChannel.open(SocketChannel.java:189)
> ~[na:1.8.0_45]
> at
> org.apache.nifi.remote.client.socket.EndpointConnectionPool.establishSiteToSiteConnection(EndpointConnectionPool.java:708)
> ~[nifi-site-to-site-client-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> at
> org.apache.nifi.remote.client.socket.EndpointConnectionPool.establishSiteToSiteConnection(EndpointConnectionPool.java:682)
> ~[nifi-site-to-site-client-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> at
> org.apache.nifi.remote.client.socket.EndpointConnectionPool.getEndpointConnection(EndpointConnectionPool.java:300)
> ~[nifi-site-to-site-client-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> at
> org.apache.nifi.remote.client.socket.SocketClient.createTransaction(SocketClient.java:129)
> ~[nifi-site-to-site-client-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> at
> org.apache.nifi.remote.StandardRemoteGroupPort.onTrigger(StandardRemoteGroupPort.java:167)
> ~[nifi-site-to-site-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> at
> org.apache.nifi.controller.AbstractPort.onTrigger(AbstractPort.java:227)
> ~[nifi-framework-core-api-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> ... 10 common frames omitted
>
> Thanks again,
> Brian
>
> On Mon, Aug 24, 2015 at 8:05 PM, Mark Payne <ma...@hotmail.com> wrote:
>
>> Hey Brian,
>>
>> The latest version of the User Guide has instructions on setting up
>> site-to-site, but that version
>> hasn't yet been posted to the website, I don't believe. If you click the
>> "help" menu in the top-right
>> corner of your NiFi instance, and go to the User Guide from there, you
>> should have the up-to-date
>> guide there.
>>
>> If there's anything missing, please let us know so that we can update the
>> guide.
>>
>> For convenience, I have pasted the text of that section below:
>>
>>
>> In order to communicate with a remote NiFi instance via Site-to-Site,
>> simply drag a Remote Process Group onto the graph and enter the URL of the
>> remote NiFi instance (for more information on the components of a Remote
>> Process Group, see Remote Process Group Transmission section of this
>> guide.) The URL is the same URL you would use to go to that instance’s User
>> Interface. At that point, you can drag a connection to or from the Remote
>> Process Group in the same way you would drag a connection to or from a
>> Processor or a local Process Group. When you drag the connection, you will
>> have a chance to choose which Port to connect to. Note that it may take up
>> to one minute for the Remote Process Group to determine which ports are
>> available.
>>
>> If the connection is dragged starting from the Remote Process Group, the
>> ports shown will be the Output Ports of the remote group, as this indicates
>> that you will be pulling data from the remote instance. If the connection
>> instead ends on the Remote Process Group, the ports shown will be the Input
>> Ports of the remote group, as this implies that you will be pushing data to
>> the remote instance.
>>
>> Note: if the remote instance is configured to use secure data
>> transmission, you will see only ports that you are authorized to
>> communicate with. For information on configuring NiFi to run securely, see
>> the Admin Guide.
>>
>> In order to allow another NiFi instance to push data to your local
>> instance, you can simply drag an Input Port onto the Root Process Group of
>> your graph. After entering a name for the port, it will be added to your
>> flow. You can now right-click on the Input Port and choose Configure in
>> order to adjust the name and the number of concurrent tasks that are used
>> for the port. If Site-to-Site is configured to run securely, you will also
>> be given the ability to adjust who has access to the port. If secure, only
>> those who have been granted access to communicate with the port will be
>> able to see that the port exists.
>>
>> After being given access to a particular port, in order to see that port,
>> the operator of a remote NiFi instance may need to right-click on their
>> Remote Process Group and choose to "Refresh" the flow.
>>
>> Similar to an Input Port, a DataFlow Manager may choose to add an Output
>> Port to the Root Process Group. The Output Port allows an authorized NiFi
>> instance to remotely connect to your instance and pull data from the Output
>> Port. Configuring the Output Port will again allow the DFM to control how
>> many concurrent tasks are allowed, as well as which NiFi instances are
>> authorized to pull data from the instance being configured.
>>
>> In addition to other instances of NiFi, some other applications may use a
>> Site-to-Site client in order to push data to or receive data from a NiFi
>> instance. For example, NiFi provides an Apache Storm spout and an Apache
>> Spark Receiver that are able to pull data from NiFi’s Root Group Output
>> Ports.
>>
>> If your instance of NiFi is running securely, the first time that a client
>> establishes a connection to your instance, the client will be forbidden and
>> a request for an account for that client will automatically be generated.
>> The client will need to be granted the NiFi role in order to communicate
>> via Site-to-Site. For more information on managing user accounts, see
>> theControlling Levels of Access section of the Admin Guide.
>>
>> For information on how to enable and configure Site-to-Site on a NiFi
>> instance, see the Site-to-Site Properties section of the Admin Guide.
>>
>> Thanks
>> -Mark
>>
>>
>> ----------------------------------------
>>> Date: Mon, 24 Aug 2015 19:36:55 -0400
>>> Subject: NiFi Site-to-Site
>>> From: brianghig@gmail.com
>>> To: dev@nifi.apache.org
>>>
>>> Is there any step-by-step guide to setting up NiFi Site-to-Site with
>> Remote
>>> Process Groups? Any details on what port range(s) needs to be available?
>>>
>>> My setup: NiFi Producer provides data on output port "data-output" to be
>>> picked up by NiFi Consumer.
>>>
>>> While trying to get a simple setup running, I set
>>> "nifi.remote.input.secure" to false and "nifi.remote.input.socket.port"
>> to
>>> 8082 in NiFi Producer (also added TCP ingress to the firewall). The
>> Remote
>>> Process Group in NiFi Consumer has the "data-output" port turned on and
>> is
>>> simply routing to LogAttribute for verification.
>>>
>>> Logs in NiFi Producer indicate that it's periodically establishing a
>> socket
>>> connection with NiFi Consumer over ports in the 54000 - 55999 range, but
>> no
>>> data actually comes across to NiFi Consumer. I added opened NiFi
>>> Consumer's firewall for ingress from 54000 - 55999.
>>>
>>> Eventually, we'll need to get it over SSL and lock down the firewall to a
>>> minimum required range, but I'm hoping to prove it out first.
>>>
>>> Any guidance would be greatly appreciated!
>>>
>>> Thanks,
>>> Brian
>>
>>
>
>
>
> --
> Brian Ghigiarelli
> 570-878-9139
Re: NiFi Site-to-Site
Posted by Brian Ghigiarelli <br...@gmail.com>.
Hey Mark,
Thanks for the reply and the references. I've taken a look through, and it
seems like everything should be set up properly. I am seeing the following
exception in the NiFi Consumer nifi-app.log. I can ping and wget to the
NiFi Producer instance fine from the Consumer box, but Site-to-Site is
still giving some trouble. That's why I started looking at AWS Security
Groups and Firewall settings. Does this stack trace ring a bell with
anything?
2015-08-25 01:34:30,709 WARN [Timer-Driven Process Thread-8]
o.a.n.c.t.ContinuallyRunConnectableTask
RemoteGroupPort[name=data-output,target=http://nifi-producer-host:8080/nifi]
Administratively Pausing for 10 seconds due to processing failure:
java.lang.RuntimeException: java.nio.channels.UnresolvedAddressException
java.lang.RuntimeException: java.nio.channels.UnresolvedAddressException
at
org.apache.nifi.controller.AbstractPort.onTrigger(AbstractPort.java:234)
~[nifi-framework-core-api-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.apache.nifi.controller.tasks.ContinuallyRunConnectableTask.call(ContinuallyRunConnectableTask.java:80)
[nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.apache.nifi.controller.tasks.ContinuallyRunConnectableTask.call(ContinuallyRunConnectableTask.java:40)
[nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:119)
[nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_45]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[na:1.8.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[na:1.8.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[na:1.8.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
Caused by: java.nio.channels.UnresolvedAddressException: null
at sun.nio.ch.Net.checkAddress(Net.java:123) ~[na:1.8.0_45]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622)
~[na:1.8.0_45]
at java.nio.channels.SocketChannel.open(SocketChannel.java:189)
~[na:1.8.0_45]
at
org.apache.nifi.remote.client.socket.EndpointConnectionPool.establishSiteToSiteConnection(EndpointConnectionPool.java:708)
~[nifi-site-to-site-client-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.apache.nifi.remote.client.socket.EndpointConnectionPool.establishSiteToSiteConnection(EndpointConnectionPool.java:682)
~[nifi-site-to-site-client-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.apache.nifi.remote.client.socket.EndpointConnectionPool.getEndpointConnection(EndpointConnectionPool.java:300)
~[nifi-site-to-site-client-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.apache.nifi.remote.client.socket.SocketClient.createTransaction(SocketClient.java:129)
~[nifi-site-to-site-client-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.apache.nifi.remote.StandardRemoteGroupPort.onTrigger(StandardRemoteGroupPort.java:167)
~[nifi-site-to-site-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.apache.nifi.controller.AbstractPort.onTrigger(AbstractPort.java:227)
~[nifi-framework-core-api-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
... 10 common frames omitted
Thanks again,
Brian
On Mon, Aug 24, 2015 at 8:05 PM, Mark Payne <ma...@hotmail.com> wrote:
> Hey Brian,
>
> The latest version of the User Guide has instructions on setting up
> site-to-site, but that version
> hasn't yet been posted to the website, I don't believe. If you click the
> "help" menu in the top-right
> corner of your NiFi instance, and go to the User Guide from there, you
> should have the up-to-date
> guide there.
>
> If there's anything missing, please let us know so that we can update the
> guide.
>
> For convenience, I have pasted the text of that section below:
>
>
> In order to communicate with a remote NiFi instance via Site-to-Site,
> simply drag a Remote Process Group onto the graph and enter the URL of the
> remote NiFi instance (for more information on the components of a Remote
> Process Group, see Remote Process Group Transmission section of this
> guide.) The URL is the same URL you would use to go to that instance’s User
> Interface. At that point, you can drag a connection to or from the Remote
> Process Group in the same way you would drag a connection to or from a
> Processor or a local Process Group. When you drag the connection, you will
> have a chance to choose which Port to connect to. Note that it may take up
> to one minute for the Remote Process Group to determine which ports are
> available.
>
> If the connection is dragged starting from the Remote Process Group, the
> ports shown will be the Output Ports of the remote group, as this indicates
> that you will be pulling data from the remote instance. If the connection
> instead ends on the Remote Process Group, the ports shown will be the Input
> Ports of the remote group, as this implies that you will be pushing data to
> the remote instance.
>
> Note: if the remote instance is configured to use secure data
> transmission, you will see only ports that you are authorized to
> communicate with. For information on configuring NiFi to run securely, see
> the Admin Guide.
>
> In order to allow another NiFi instance to push data to your local
> instance, you can simply drag an Input Port onto the Root Process Group of
> your graph. After entering a name for the port, it will be added to your
> flow. You can now right-click on the Input Port and choose Configure in
> order to adjust the name and the number of concurrent tasks that are used
> for the port. If Site-to-Site is configured to run securely, you will also
> be given the ability to adjust who has access to the port. If secure, only
> those who have been granted access to communicate with the port will be
> able to see that the port exists.
>
> After being given access to a particular port, in order to see that port,
> the operator of a remote NiFi instance may need to right-click on their
> Remote Process Group and choose to "Refresh" the flow.
>
> Similar to an Input Port, a DataFlow Manager may choose to add an Output
> Port to the Root Process Group. The Output Port allows an authorized NiFi
> instance to remotely connect to your instance and pull data from the Output
> Port. Configuring the Output Port will again allow the DFM to control how
> many concurrent tasks are allowed, as well as which NiFi instances are
> authorized to pull data from the instance being configured.
>
> In addition to other instances of NiFi, some other applications may use a
> Site-to-Site client in order to push data to or receive data from a NiFi
> instance. For example, NiFi provides an Apache Storm spout and an Apache
> Spark Receiver that are able to pull data from NiFi’s Root Group Output
> Ports.
>
> If your instance of NiFi is running securely, the first time that a client
> establishes a connection to your instance, the client will be forbidden and
> a request for an account for that client will automatically be generated.
> The client will need to be granted the NiFi role in order to communicate
> via Site-to-Site. For more information on managing user accounts, see
> theControlling Levels of Access section of the Admin Guide.
>
> For information on how to enable and configure Site-to-Site on a NiFi
> instance, see the Site-to-Site Properties section of the Admin Guide.
>
> Thanks
> -Mark
>
>
> ----------------------------------------
> > Date: Mon, 24 Aug 2015 19:36:55 -0400
> > Subject: NiFi Site-to-Site
> > From: brianghig@gmail.com
> > To: dev@nifi.apache.org
> >
> > Is there any step-by-step guide to setting up NiFi Site-to-Site with
> Remote
> > Process Groups? Any details on what port range(s) needs to be available?
> >
> > My setup: NiFi Producer provides data on output port "data-output" to be
> > picked up by NiFi Consumer.
> >
> > While trying to get a simple setup running, I set
> > "nifi.remote.input.secure" to false and "nifi.remote.input.socket.port"
> to
> > 8082 in NiFi Producer (also added TCP ingress to the firewall). The
> Remote
> > Process Group in NiFi Consumer has the "data-output" port turned on and
> is
> > simply routing to LogAttribute for verification.
> >
> > Logs in NiFi Producer indicate that it's periodically establishing a
> socket
> > connection with NiFi Consumer over ports in the 54000 - 55999 range, but
> no
> > data actually comes across to NiFi Consumer. I added opened NiFi
> > Consumer's firewall for ingress from 54000 - 55999.
> >
> > Eventually, we'll need to get it over SSL and lock down the firewall to a
> > minimum required range, but I'm hoping to prove it out first.
> >
> > Any guidance would be greatly appreciated!
> >
> > Thanks,
> > Brian
>
>
--
Brian Ghigiarelli
570-878-9139
RE: NiFi Site-to-Site
Posted by Mark Payne <ma...@hotmail.com>.
Hey Brian,
The latest version of the User Guide has instructions on setting up site-to-site, but that version
hasn't yet been posted to the website, I don't believe. If you click the "help" menu in the top-right
corner of your NiFi instance, and go to the User Guide from there, you should have the up-to-date
guide there.
If there's anything missing, please let us know so that we can update the guide.
For convenience, I have pasted the text of that section below:
In order to communicate with a remote NiFi instance via Site-to-Site, simply drag a Remote Process Group onto the graph and enter the URL of the remote NiFi instance (for more information on the components of a Remote Process Group, see Remote Process Group Transmission section of this guide.) The URL is the same URL you would use to go to that instance’s User Interface. At that point, you can drag a connection to or from the Remote Process Group in the same way you would drag a connection to or from a Processor or a local Process Group. When you drag the connection, you will have a chance to choose which Port to connect to. Note that it may take up to one minute for the Remote Process Group to determine which ports are available.
If the connection is dragged starting from the Remote Process Group, the ports shown will be the Output Ports of the remote group, as this indicates that you will be pulling data from the remote instance. If the connection instead ends on the Remote Process Group, the ports shown will be the Input Ports of the remote group, as this implies that you will be pushing data to the remote instance.
Note: if the remote instance is configured to use secure data transmission, you will see only ports that you are authorized to communicate with. For information on configuring NiFi to run securely, see the Admin Guide.
In order to allow another NiFi instance to push data to your local instance, you can simply drag an Input Port onto the Root Process Group of your graph. After entering a name for the port, it will be added to your flow. You can now right-click on the Input Port and choose Configure in order to adjust the name and the number of concurrent tasks that are used for the port. If Site-to-Site is configured to run securely, you will also be given the ability to adjust who has access to the port. If secure, only those who have been granted access to communicate with the port will be able to see that the port exists.
After being given access to a particular port, in order to see that port, the operator of a remote NiFi instance may need to right-click on their Remote Process Group and choose to "Refresh" the flow.
Similar to an Input Port, a DataFlow Manager may choose to add an Output Port to the Root Process Group. The Output Port allows an authorized NiFi instance to remotely connect to your instance and pull data from the Output Port. Configuring the Output Port will again allow the DFM to control how many concurrent tasks are allowed, as well as which NiFi instances are authorized to pull data from the instance being configured.
In addition to other instances of NiFi, some other applications may use a Site-to-Site client in order to push data to or receive data from a NiFi instance. For example, NiFi provides an Apache Storm spout and an Apache Spark Receiver that are able to pull data from NiFi’s Root Group Output Ports.
If your instance of NiFi is running securely, the first time that a client establishes a connection to your instance, the client will be forbidden and a request for an account for that client will automatically be generated. The client will need to be granted the NiFi role in order to communicate via Site-to-Site. For more information on managing user accounts, see theControlling Levels of Access section of the Admin Guide.
For information on how to enable and configure Site-to-Site on a NiFi instance, see the Site-to-Site Properties section of the Admin Guide.
Thanks
-Mark
----------------------------------------
> Date: Mon, 24 Aug 2015 19:36:55 -0400
> Subject: NiFi Site-to-Site
> From: brianghig@gmail.com
> To: dev@nifi.apache.org
>
> Is there any step-by-step guide to setting up NiFi Site-to-Site with Remote
> Process Groups? Any details on what port range(s) needs to be available?
>
> My setup: NiFi Producer provides data on output port "data-output" to be
> picked up by NiFi Consumer.
>
> While trying to get a simple setup running, I set
> "nifi.remote.input.secure" to false and "nifi.remote.input.socket.port" to
> 8082 in NiFi Producer (also added TCP ingress to the firewall). The Remote
> Process Group in NiFi Consumer has the "data-output" port turned on and is
> simply routing to LogAttribute for verification.
>
> Logs in NiFi Producer indicate that it's periodically establishing a socket
> connection with NiFi Consumer over ports in the 54000 - 55999 range, but no
> data actually comes across to NiFi Consumer. I added opened NiFi
> Consumer's firewall for ingress from 54000 - 55999.
>
> Eventually, we'll need to get it over SSL and lock down the firewall to a
> minimum required range, but I'm hoping to prove it out first.
>
> Any guidance would be greatly appreciated!
>
> Thanks,
> Brian