You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Rick Braddy <rb...@softnas.com> on 2015/10/03 23:59:13 UTC
Remote process group networking
I have a question about network paths required for proper operation of remote process groups.
By default, the initial connection from source node to remote process group target node is on port 8080. Then, there's a second port (e.g., I set it to 8081 and a setting for whether it's SSL secured or not).
The question is, are the TCP connection one way, from source node where graph is running to the remote process group's node only, or are bidirectional TCP connections required?
The reason I ask is encountering problems trying to connect from data center that has open outbound firewall, but allows no incoming connections. On the target node, there is no indication in nifi-app.log of the source node even attempting connect (not sure if debug logging is required).
If there's some other information on remote process group network topology setup and/or troubleshooting, would be great to read up on it.
Thanks
Rick
Re: Remote process group networking
Posted by Joe Witt <jo...@gmail.com>.
"Rick" - sorry for the extra Y.
On Tue, Nov 3, 2015 at 9:48 AM, Joe Witt <jo...@gmail.com> wrote:
> ...wonder if we should turn this into a FAQ/explanation.
>
> Thanks for writing this up and following through with resolution Ricky.
>
> On Fri, Oct 16, 2015 at 2:36 PM, Rick Braddy <rb...@softnas.com> wrote:
>> Just to close this topic off...
>>
>> First, I found an error in my remote target node flow that was preventing proper connection from the source node and hampering troubleshooting - had a connector inside a process group, but no connector at top level of graph, which is required for Remote Process Group access.
>>
>> On firewall configuration, indeed only TCP traffic on the UI port 8080 plus the site-to-site port (e.g., 8081) need to be open on the target node for unidirectional site-to-site operation (not required to be open on the source node's firewall). No other ports are required across firewall boundaries.
>>
>> nifi.remote.input.socket.host must be set to the external (Internet) NAT firewall address is the other key configuration item, because when site-to-site connection is established, the source node must connect to the firewall (not directly to the remote target node's local IP, which is the default if this value is not configured).
>>
>> localhost must also be enabled for local operation, as the "service nifi status" (and probably other stuff) makes calls via localhost (in case you're using iptables, as I was for testing).
>>
>> Best,
>> Rick
>>
>> -----Original Message-----
>> From: Rick Braddy [mailto:rbraddy@softnas.com]
>> Sent: Monday, October 05, 2015 4:45 PM
>> To: dev@nifi.apache.org
>> Subject: RE: Remote process group networking
>>
>> Still no definitive answers...
>>
>> My testing shows that both the NiFi UI port (e.g., 8080) and the data port (e.g., 8081) must be open in both directions through a firewall. Even with those iptables rules, it seems something is missing. I will figure it out eventually, and let everyone know what's required to use Nifi across firewall boundaries.
>>
>> Rick
>>
>> -----Original Message-----
>> From: Rick Braddy
>> Sent: Monday, October 05, 2015 10:18 AM
>> To: dev@nifi.apache.org
>> Subject: RE: Remote process group networking
>>
>> Let me ask this in a simpler way... for Nifi Remote Process Group communications across firewall boundaries, which ports must be open through firewalls between a source node running the local graph processes and the Remote Process Group node?
>>
>> Rick
>>
>> -----Original Message-----
>> From: Rick Braddy [mailto:rbraddy@softnas.com]
>> Sent: Saturday, October 03, 2015 4:59 PM
>> To: dev@nifi.apache.org
>> Subject: Remote process group networking
>>
>> I have a question about network paths required for proper operation of remote process groups.
>>
>> By default, the initial connection from source node to remote process group target node is on port 8080. Then, there's a second port (e.g., I set it to 8081 and a setting for whether it's SSL secured or not).
>>
>> The question is, are the TCP connection one way, from source node where graph is running to the remote process group's node only, or are bidirectional TCP connections required?
>>
>> The reason I ask is encountering problems trying to connect from data center that has open outbound firewall, but allows no incoming connections. On the target node, there is no indication in nifi-app.log of the source node even attempting connect (not sure if debug logging is required).
>>
>> If there's some other information on remote process group network topology setup and/or troubleshooting, would be great to read up on it.
>>
>> Thanks
>> Rick
Re: Remote process group networking
Posted by Joe Witt <jo...@gmail.com>.
...wonder if we should turn this into a FAQ/explanation.
Thanks for writing this up and following through with resolution Ricky.
On Fri, Oct 16, 2015 at 2:36 PM, Rick Braddy <rb...@softnas.com> wrote:
> Just to close this topic off...
>
> First, I found an error in my remote target node flow that was preventing proper connection from the source node and hampering troubleshooting - had a connector inside a process group, but no connector at top level of graph, which is required for Remote Process Group access.
>
> On firewall configuration, indeed only TCP traffic on the UI port 8080 plus the site-to-site port (e.g., 8081) need to be open on the target node for unidirectional site-to-site operation (not required to be open on the source node's firewall). No other ports are required across firewall boundaries.
>
> nifi.remote.input.socket.host must be set to the external (Internet) NAT firewall address is the other key configuration item, because when site-to-site connection is established, the source node must connect to the firewall (not directly to the remote target node's local IP, which is the default if this value is not configured).
>
> localhost must also be enabled for local operation, as the "service nifi status" (and probably other stuff) makes calls via localhost (in case you're using iptables, as I was for testing).
>
> Best,
> Rick
>
> -----Original Message-----
> From: Rick Braddy [mailto:rbraddy@softnas.com]
> Sent: Monday, October 05, 2015 4:45 PM
> To: dev@nifi.apache.org
> Subject: RE: Remote process group networking
>
> Still no definitive answers...
>
> My testing shows that both the NiFi UI port (e.g., 8080) and the data port (e.g., 8081) must be open in both directions through a firewall. Even with those iptables rules, it seems something is missing. I will figure it out eventually, and let everyone know what's required to use Nifi across firewall boundaries.
>
> Rick
>
> -----Original Message-----
> From: Rick Braddy
> Sent: Monday, October 05, 2015 10:18 AM
> To: dev@nifi.apache.org
> Subject: RE: Remote process group networking
>
> Let me ask this in a simpler way... for Nifi Remote Process Group communications across firewall boundaries, which ports must be open through firewalls between a source node running the local graph processes and the Remote Process Group node?
>
> Rick
>
> -----Original Message-----
> From: Rick Braddy [mailto:rbraddy@softnas.com]
> Sent: Saturday, October 03, 2015 4:59 PM
> To: dev@nifi.apache.org
> Subject: Remote process group networking
>
> I have a question about network paths required for proper operation of remote process groups.
>
> By default, the initial connection from source node to remote process group target node is on port 8080. Then, there's a second port (e.g., I set it to 8081 and a setting for whether it's SSL secured or not).
>
> The question is, are the TCP connection one way, from source node where graph is running to the remote process group's node only, or are bidirectional TCP connections required?
>
> The reason I ask is encountering problems trying to connect from data center that has open outbound firewall, but allows no incoming connections. On the target node, there is no indication in nifi-app.log of the source node even attempting connect (not sure if debug logging is required).
>
> If there's some other information on remote process group network topology setup and/or troubleshooting, would be great to read up on it.
>
> Thanks
> Rick
RE: Remote process group networking
Posted by Rick Braddy <rb...@softnas.com>.
Just to close this topic off...
First, I found an error in my remote target node flow that was preventing proper connection from the source node and hampering troubleshooting - had a connector inside a process group, but no connector at top level of graph, which is required for Remote Process Group access.
On firewall configuration, indeed only TCP traffic on the UI port 8080 plus the site-to-site port (e.g., 8081) need to be open on the target node for unidirectional site-to-site operation (not required to be open on the source node's firewall). No other ports are required across firewall boundaries.
nifi.remote.input.socket.host must be set to the external (Internet) NAT firewall address is the other key configuration item, because when site-to-site connection is established, the source node must connect to the firewall (not directly to the remote target node's local IP, which is the default if this value is not configured).
localhost must also be enabled for local operation, as the "service nifi status" (and probably other stuff) makes calls via localhost (in case you're using iptables, as I was for testing).
Best,
Rick
-----Original Message-----
From: Rick Braddy [mailto:rbraddy@softnas.com]
Sent: Monday, October 05, 2015 4:45 PM
To: dev@nifi.apache.org
Subject: RE: Remote process group networking
Still no definitive answers...
My testing shows that both the NiFi UI port (e.g., 8080) and the data port (e.g., 8081) must be open in both directions through a firewall. Even with those iptables rules, it seems something is missing. I will figure it out eventually, and let everyone know what's required to use Nifi across firewall boundaries.
Rick
-----Original Message-----
From: Rick Braddy
Sent: Monday, October 05, 2015 10:18 AM
To: dev@nifi.apache.org
Subject: RE: Remote process group networking
Let me ask this in a simpler way... for Nifi Remote Process Group communications across firewall boundaries, which ports must be open through firewalls between a source node running the local graph processes and the Remote Process Group node?
Rick
-----Original Message-----
From: Rick Braddy [mailto:rbraddy@softnas.com]
Sent: Saturday, October 03, 2015 4:59 PM
To: dev@nifi.apache.org
Subject: Remote process group networking
I have a question about network paths required for proper operation of remote process groups.
By default, the initial connection from source node to remote process group target node is on port 8080. Then, there's a second port (e.g., I set it to 8081 and a setting for whether it's SSL secured or not).
The question is, are the TCP connection one way, from source node where graph is running to the remote process group's node only, or are bidirectional TCP connections required?
The reason I ask is encountering problems trying to connect from data center that has open outbound firewall, but allows no incoming connections. On the target node, there is no indication in nifi-app.log of the source node even attempting connect (not sure if debug logging is required).
If there's some other information on remote process group network topology setup and/or troubleshooting, would be great to read up on it.
Thanks
Rick
RE: Remote process group networking
Posted by Rick Braddy <rb...@softnas.com>.
Still no definitive answers...
My testing shows that both the NiFi UI port (e.g., 8080) and the data port (e.g., 8081) must be open in both directions through a firewall. Even with those iptables rules, it seems something is missing. I will figure it out eventually, and let everyone know what's required to use Nifi across firewall boundaries.
Rick
-----Original Message-----
From: Rick Braddy
Sent: Monday, October 05, 2015 10:18 AM
To: dev@nifi.apache.org
Subject: RE: Remote process group networking
Let me ask this in a simpler way... for Nifi Remote Process Group communications across firewall boundaries, which ports must be open through firewalls between a source node running the local graph processes and the Remote Process Group node?
Rick
-----Original Message-----
From: Rick Braddy [mailto:rbraddy@softnas.com]
Sent: Saturday, October 03, 2015 4:59 PM
To: dev@nifi.apache.org
Subject: Remote process group networking
I have a question about network paths required for proper operation of remote process groups.
By default, the initial connection from source node to remote process group target node is on port 8080. Then, there's a second port (e.g., I set it to 8081 and a setting for whether it's SSL secured or not).
The question is, are the TCP connection one way, from source node where graph is running to the remote process group's node only, or are bidirectional TCP connections required?
The reason I ask is encountering problems trying to connect from data center that has open outbound firewall, but allows no incoming connections. On the target node, there is no indication in nifi-app.log of the source node even attempting connect (not sure if debug logging is required).
If there's some other information on remote process group network topology setup and/or troubleshooting, would be great to read up on it.
Thanks
Rick
RE: Remote process group networking
Posted by Rick Braddy <rb...@softnas.com>.
Let me ask this in a simpler way... for Nifi Remote Process Group communications across firewall boundaries, which ports must be open through firewalls between a source node running the local graph processes and the Remote Process Group node?
Rick
-----Original Message-----
From: Rick Braddy [mailto:rbraddy@softnas.com]
Sent: Saturday, October 03, 2015 4:59 PM
To: dev@nifi.apache.org
Subject: Remote process group networking
I have a question about network paths required for proper operation of remote process groups.
By default, the initial connection from source node to remote process group target node is on port 8080. Then, there's a second port (e.g., I set it to 8081 and a setting for whether it's SSL secured or not).
The question is, are the TCP connection one way, from source node where graph is running to the remote process group's node only, or are bidirectional TCP connections required?
The reason I ask is encountering problems trying to connect from data center that has open outbound firewall, but allows no incoming connections. On the target node, there is no indication in nifi-app.log of the source node even attempting connect (not sure if debug logging is required).
If there's some other information on remote process group network topology setup and/or troubleshooting, would be great to read up on it.
Thanks
Rick