You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by "Paul Gibeault (pagibeault)" <pa...@micron.com> on 2016/08/30 22:28:45 UTC
I need help configuring Site-to-Site in Secure Mode.
Hello,
We have been attempting to set up Site-to-Site for NiFi in secure mode and have not been successful.
When I create a Remote Process Group, and enter the URL* https://servername:8443/nifi I receive an error icon. The hover status is "Unauthorized"
* - servername is the actual hostname running NiFi
Things I have tried without success:
- Closely followed the instructions in the NiFi System Administrator's Guide<https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html>
- Enabled SSL Security and Kerberos User Authentication (no self-signed certs)
- Imported public keys of remote NiFi servers into the local keystores for each instance
- Created policies on each instance to allow for full permissions to the accounts in use.
- Tried various combinations of Linux & Windows instances of NiFi.
- Connected a Site-to-Site process group to itself
- Used both 1.0.0-BETA and 1.1.0-SNAPSHOT NiFi versions
There are no warnings or errors in the log files when I attempt to connect a NiFi instance running on Linux to another instance on Linux. However, I did see something when attempting to connect a NiFi instance running on Windows to an instance running on Linux
From log of NiFi on Windows:
2016-08-30 16:11:21,173 ERROR [Remote Process Group dd2e3ac9-0156-1000-4543-5ba3d10c6130: https://servername:8443/nifi Thread-1] o.a.n.remote.StandardRemoteProcessGroup org.apache.nifi.remote.StandardRemoteProcessGroup$InitializationTask@19bde1f3 Failed to request account: got unexpected response code of 404:Not Found
From log of NiFi on Linux:
nifi-user.log:2016-08-30 16:39:53,973 INFO [NiFi Web Server-465] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (CN=pagibeault) GET https://servername:8443/nifi-api/site-to-site (source ip: 137.201.48.150)
nifi-user.log:2016-08-30 16:39:53,973 INFO [NiFi Web Server-465] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for CN=pagibeault
nifi-user.log:2016-08-30 16:39:53,974 INFO [NiFi Web Server-465] o.a.n.w.a.c.AccessDeniedExceptionMapper CN=pagibeault does not have permission to access the requested resource. Returning Forbidden response.
Any guidance would be grand.
Thanks,
[http://collab.micron.com/corp/brand/SiteAssets/Micron.png]<http://www.micron.com/>
Paul Gibeault
Sr. Software Engineer, Big Data
Enterprise Analytics & Data
Micron Technology, Inc.
Office (208) 363-3238
pagibeault@micron.com<ma...@micron.com>
[http://collab.micron.com/corp/brand/SiteAssets/LinkedIn.png] linkedin.com/in/paulgibeault<https://www.linkedin.com/in/linkedin.com/in/paulgibeault>
RE: I need help configuring Site-to-Site in Secure Mode.
Posted by "Paul Gibeault (pagibeault)" <pa...@micron.com>.
Peter has found and corrected our issue.
When setting up our truststore we failed to include the cert for the root CA. We only included the certs for the individual hosts running NiFi. This left us in a half-functional state where we could create a secure connection for site-to-site but all traffic was identified as “anonymous”
One thing that complicated our investigation is an incorrect error message being reported:
Unable to refresh Remote Group's peers due to Unable to communicate with remote NiFi cluster in order to determine which nodes exist in the remote cluster
We had not configured to cluster our NiFi instances at the time.
Once we corrected our truststore configuration this error also disappeared.
Thanks for all your help,
-Paul
From: Paul Gibeault (pagibeault)
Sent: Friday, September 02, 2016 2:46 PM
To: users@nifi.apache.org
Subject: RE: I need help configuring Site-to-Site in Secure Mode.
Bryan,
I have found that the code is looking for an attribute called “javax.servlet.request.X509Certificate” in the certificate and there does not appear to be one.
https://goo.gl/photos/SwyiAfgezfTSzDeg7
file: X509CertificateExtractor.java Line 39
This article suggests the issue is with the certificate being used.
http://stackoverflow.com/questions/24351472/getattributejavax-servlet-request-x509certificate-not-set-spring-cxf-jetty
My security ignorance will show here, but could this be due to the format we are using JKS vs. PKCS12?
Thanks,
Paul Gibeault
From: Bryan Bende [mailto:bbende@gmail.com]
Sent: Friday, September 02, 2016 11:54 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: I need help configuring Site-to-Site in Secure Mode.
Ok, I am not completely familiar with all the ins and outs of the site-to-site client code, but I know one place that it creates a connection is here:
https://github.com/apache/nifi/blob/a3586e04d9978e105cc5645e893dc6d77b79b86e/nifi-commons/nifi-site-to-site-client/src/main/java/org/apache/nifi/remote/client/socket/EndpointConnectionPool.java#L439
The SSLContext that will be used in there comes from the FlowController which takes the SSL related properties from nifi.properties:
https://github.com/apache/nifi/blob/1745c1274b274a994a92312054d8951ce1c499d0/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/FlowController.java#L442
The part I am not sure about is whether or not there is a call somewhere before the EndpointConnectionPool, but you could at least start by seeing if it is hitting there. I can look a little more later.
-Bryan
On Fri, Sep 2, 2016 at 12:25 PM, Peter Wicks (pwicks) <pw...@micron.com>> wrote:
Bryan,
In the log on the server side we see this message:
INFO [NiFi Web Server-324] o.a.n.w.a.c.AccessDeniedExceptionMapper anonymous does not have permission to access the requested resource. Returning Unauthorized response.
I forgot to mention, we tried adding a user named anonymous and granting it access. When we did this Site-to-Site started working. Obviously that is not a course of action we want to take, but it was a good exercise.
When I debugged the X509AuthenticationFilter it found no certs, here is the screenshot showing some of the stack/variables at the time.
https://goo.gl/photos/938E5A8vsb7nQ6Kh8
What would be the equivalent source code for debugging the client to find out why no cert is being sent?
Thanks,
Peter
From: Bryan Bende [mailto:bbende@gmail.com<ma...@gmail.com>]
Sent: Friday, September 02, 2016 10:09 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: I need help configuring Site-to-Site in Secure Mode.
Paul/Peter,
Having Kerberos enabled should not have any impact, you can have Kerberos or LDAP enabled, but if a certificate is provided that should always take precedence.
What do you see in the nifi-user.log on the second instance (the one where the remote process group is pointing at)?
If it found the incoming cert it should log something like:
"Attempting request for <cert of first NiFi> %s %s (source ip: %s)"
"Authentication success for <cert of first NiFi>"
If it didn't find one I think it should log:
"Rejecting access to web api: %s", ae.getMessage()
It might be help to try debugging into the X509AuthenticationFilter:
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-security/src/main/java/org/apache/nifi/web/security/x509/X509AuthenticationFilter.java
The base class it extends is:
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-security/src/main/java/org/apache/nifi/web/security/NiFiAuthenticationFilter.java
If it goes into the X509 filter and returns null from attemptAuthentication, we can at least narrow down to the fact that that the client certificate is not in the request for some reason.
It sounds like you might have already done this, but there is a line in bootstrap.conf that you can uncomment to hook up a remote debugger:
java.arg.debug=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000
-Bryan
On Fri, Sep 2, 2016 at 11:27 AM, Peter Wicks (pwicks) <pw...@micron.com>> wrote:
Matt,
That was the case on our first go round, when we only had SSL certs. We went back yesterday and got new certificates that support both Server Auth and Client Auth and rebuilt our KeyStore.
When I use keytool to look at my KeyStore I can see both of these on the certificate:
#6: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
clientAuth
serverAuth
]
Thanks,
Peter
From: Matthew Clarke [mailto:matt.clarke.138@gmail.com<ma...@gmail.com>]
Sent: Friday, September 02, 2016 9:23 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: I need help configuring Site-to-Site in Secure Mode.
Do the certs you created/obtained support being used for both client and server auth. If they were created for server auth only, this could explain your issue. NiFi instances need to act as a client and a server at times.
Thanks,
Matt
On Fri, Sep 2, 2016 at 10:59 AM, Peter Wicks (pwicks) <pw...@micron.com>> wrote:
Bryan,
We’ve fixed our certs, with no change to the outcome.
We have username/password authentication enabled, via Kerberos, are there issues having Kerberos enabled (username/password) and trying to do site-to-site? When I try to initiate site-to-site with an instance of NiFI configured for Kerberos all requests come through to the server as anonymous because no challenge appears to be sent. We’ve debugged the code and even deep down in NiFiUserUtils.getNiFiUser the request is already marked as anonymous by the Spring framework. It appears to me that the client has a cert, and is waiting for a challenge(?) from the server, and the server is configured for Kerberos and it’s waiting for a ‘bearer’ token…
We’ve debugged both client and server, the client sends the request and gets back a 401 (Unauthorized). SSL verifies good.
Server doesn’t appear to get any authorization information of any kind.
Looking for further guidance/next steps.
Thanks,
Peter
From: Bryan Bende [mailto:bbende@gmail.com<ma...@gmail.com>]
Sent: Thursday, September 01, 2016 9:44 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: I need help configuring Site-to-Site in Secure Mode.
Peter,
Yes, by no means am I saying everyone should use the TLS toolkit. I was just using that because many people are not familiar with how to create certificates, and for people trying to follow a tutorial it is the easiest option.
In your case you definitely want to be using your CA. What you described about not having a cert for client authentication definitely sounds like it would be a problem. Let us know if everything works out after getting the new certs.
-Bryan
On Thu, Sep 1, 2016 at 11:34 AM, Peter Wicks (pwicks) <pw...@micron.com>> wrote:
Bryan,
Paul and I have been working on this, and I think our issue is related to certificates.
In your blog posting you used TLS-Toolkit in your example, but I think that is unrealistic for many environments. For example, this also creates the certificates for SSL right? But these will be self-signed and thus untrusted by default in web browsers. In our environment we generated SSL certificates from our CA and loaded them into the KeyStore. We then extracted public keys for the SSL certs and put them in each of the Trust Stores. This I think is where our main problem is…
I’m making a few assumptions here, so feel free to correct me, but my understanding is that when you use TLS-Toolkit it either creates multiple certs (SSL & Client Auth), or it creates a cert that you are allowed to use for both activities. In our case we ONLY have SSL certs, and the certs are marked such that they aren’t allowed to be used for Client Authentication. I believe this is the reason why our requests are showing up as ‘Anonymous’, because there are no Client Authentication certificates in the KeyStore, just SSL certs.
I’ve asked our security team for Client Authentication certs for each server, since it would be our preference to use our CA rather than having TLS-Toolkit be its own CA.
Thoughts?
Thanks,
Peter
From: Bryan Bende [mailto:bbende@gmail.com<ma...@gmail.com>]
Sent: Thursday, September 01, 2016 9:26 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: I need help configuring Site-to-Site in Secure Mode.
Paul,
Clustering is not a requirement for site-to-site... This sounds strange since "anonymous" is used to represent a user when NiFi is not secured.
Can you double-check all your configs and make sure you have the following properties set...
nifi.remote.input.secure=true
nifi.web.https.host=
nifi.web.https.port=
nifi.security.keystore=
nifi.security.keystoreType=
nifi.security.keystorePasswd=
nifi.security.keyPasswd=
nifi.security.truststore=
nifi.security.truststoreType=
nifi.security.truststorePasswd=<your password>
After your question the other day I went through the steps of setting secure site-to-site to make sure I knew what I was talking about :)
I wrote up the steps here: http://bryanbende.com/development/2016/08/30/apache-nifi-1.0.0-secure-site-to-site
Thanks,
Bryan
On Thu, Sep 1, 2016 at 10:44 AM, Paul Gibeault (pagibeault) <pa...@micron.com>> wrote:
Bryan,
Thanks for the reply. After increasing the log level for Authentication I saw the target NiFi instance used the account “anonymous” for the Site-to-Site connection. After creating a policy for “anonymous”, I was able to view the remote ports and connect to them.
Obviously this is not ideal. We would prefer to make policies for remote hosts/users rather than anonymous.
We are using the same SSL Certificate for both Key Store and Trust Store on a NiFi Instance. This is likely the cause of the “anonymous” user as it doesn’t have a DN. We are working to correct this.
However, after getting my work flow set up across NiFi instances I see this error:
Unable to refresh Remote Group's peers due to Unable to communicate with remote NiFi cluster in order to determine which nodes exist in the remote cluster
Our NiFi servers are not set up for clustering. Is clustering required to perform Site-to-Site?
Thanks,
Paul Gibeault
From: Bryan Bende [mailto:bbende@gmail.com<ma...@gmail.com>]
Sent: Tuesday, August 30, 2016 5:09 PM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: I need help configuring Site-to-Site in Secure Mode.
Paul,
It sounds like you probably have the certificates/truststores setup correctly and just need to create the appropriate policies...
Lets say you have nifi-1 with an Remote Process Group pointing at the URL of nifi-2, and nifi-2 has an Input port to receive data.
In nifi-2 there needs to be a user for the certificate of nifi-1, and then in the global policies of nifi-2 (top right menu) there needs to be a policy for "retrieve site-to-site details" with the nifi-1 user added to the policy. I think this is what is causing the error message you are seeing since nifi-1 is not authorized to query nifi-2 for site-to-site information (available ports, etc).
I believe you also need to create a policy on the Input Port on nifi-2... select the input port and use the lock icon in the left palette and choose "receive data over site-to-site" and add the user of nifi-1. This gives nifi-1 access to the specific port.
Let us know if that works. If so we should definitely look at updating some of the documentation to explain this.
Thanks,
Bryan
On Tue, Aug 30, 2016 at 6:28 PM, Paul Gibeault (pagibeault) <pa...@micron.com>> wrote:
Hello,
We have been attempting to set up Site-to-Site for NiFi in secure mode and have not been successful.
When I create a Remote Process Group, and enter the URL* https://servername:8443/nifi I receive an error icon. The hover status is “Unauthorized”
* - servername is the actual hostname running NiFi
Things I have tried without success:
- Closely followed the instructions in the NiFi System Administrator’s Guide<https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html>
- Enabled SSL Security and Kerberos User Authentication (no self-signed certs)
- Imported public keys of remote NiFi servers into the local keystores for each instance
- Created policies on each instance to allow for full permissions to the accounts in use.
- Tried various combinations of Linux & Windows instances of NiFi.
- Connected a Site-to-Site process group to itself
- Used both 1.0.0-BETA and 1.1.0-SNAPSHOT NiFi versions
There are no warnings or errors in the log files when I attempt to connect a NiFi instance running on Linux to another instance on Linux. However, I did see something when attempting to connect a NiFi instance running on Windows to an instance running on Linux
From log of NiFi on Windows:
2016-08-30 16:11:21,173 ERROR [Remote Process Group dd2e3ac9-0156-1000-4543-5ba3d10c6130: https://servername:8443/nifi Thread-1] o.a.n.remote.StandardRemoteProcessGroup org.apache.nifi.remote.StandardRemoteProcessGroup$InitializationTask@19bde1f3<ma...@19bde1f3> Failed to request account: got unexpected response code of 404:Not Found
From log of NiFi on Linux:
nifi-user.log:2016-08-30 16:39:53,973 INFO [NiFi Web Server-465] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (CN=pagibeault) GET https://servername:8443/nifi-api/site-to-site (source ip: 137.201.48.150)
nifi-user.log:2016-08-30 16:39:53,973 INFO [NiFi Web Server-465] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for CN=pagibeault
nifi-user.log:2016-08-30 16:39:53,974 INFO [NiFi Web Server-465] o.a.n.w.a.c.AccessDeniedExceptionMapper CN=pagibeault does not have permission to access the requested resource. Returning Forbidden response.
Any guidance would be grand.
Thanks,
[http://collab.micron.com/corp/brand/SiteAssets/Micron.png]<http://www.micron.com/>
Paul Gibeault
Sr. Software Engineer, Big Data
Enterprise Analytics & Data
Micron Technology, Inc.
Office (208) 363-3238<tel:%28208%29%20363-3238>
pagibeault@micron.com<ma...@micron.com>
[http://collab.micron.com/corp/brand/SiteAssets/LinkedIn.png] linkedin.com/in/paulgibeault<https://www.linkedin.com/in/linkedin.com/in/paulgibeault>
RE: I need help configuring Site-to-Site in Secure Mode.
Posted by "Paul Gibeault (pagibeault)" <pa...@micron.com>.
Bryan,
I have found that the code is looking for an attribute called “javax.servlet.request.X509Certificate” in the certificate and there does not appear to be one.
https://goo.gl/photos/SwyiAfgezfTSzDeg7
file: X509CertificateExtractor.java Line 39
This article suggests the issue is with the certificate being used.
http://stackoverflow.com/questions/24351472/getattributejavax-servlet-request-x509certificate-not-set-spring-cxf-jetty
My security ignorance will show here, but could this be due to the format we are using JKS vs. PKCS12?
Thanks,
Paul Gibeault
From: Bryan Bende [mailto:bbende@gmail.com]
Sent: Friday, September 02, 2016 11:54 AM
To: users@nifi.apache.org
Subject: Re: I need help configuring Site-to-Site in Secure Mode.
Ok, I am not completely familiar with all the ins and outs of the site-to-site client code, but I know one place that it creates a connection is here:
https://github.com/apache/nifi/blob/a3586e04d9978e105cc5645e893dc6d77b79b86e/nifi-commons/nifi-site-to-site-client/src/main/java/org/apache/nifi/remote/client/socket/EndpointConnectionPool.java#L439
The SSLContext that will be used in there comes from the FlowController which takes the SSL related properties from nifi.properties:
https://github.com/apache/nifi/blob/1745c1274b274a994a92312054d8951ce1c499d0/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/FlowController.java#L442
The part I am not sure about is whether or not there is a call somewhere before the EndpointConnectionPool, but you could at least start by seeing if it is hitting there. I can look a little more later.
-Bryan
On Fri, Sep 2, 2016 at 12:25 PM, Peter Wicks (pwicks) <pw...@micron.com>> wrote:
Bryan,
In the log on the server side we see this message:
INFO [NiFi Web Server-324] o.a.n.w.a.c.AccessDeniedExceptionMapper anonymous does not have permission to access the requested resource. Returning Unauthorized response.
I forgot to mention, we tried adding a user named anonymous and granting it access. When we did this Site-to-Site started working. Obviously that is not a course of action we want to take, but it was a good exercise.
When I debugged the X509AuthenticationFilter it found no certs, here is the screenshot showing some of the stack/variables at the time.
https://goo.gl/photos/938E5A8vsb7nQ6Kh8
What would be the equivalent source code for debugging the client to find out why no cert is being sent?
Thanks,
Peter
From: Bryan Bende [mailto:bbende@gmail.com<ma...@gmail.com>]
Sent: Friday, September 02, 2016 10:09 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: I need help configuring Site-to-Site in Secure Mode.
Paul/Peter,
Having Kerberos enabled should not have any impact, you can have Kerberos or LDAP enabled, but if a certificate is provided that should always take precedence.
What do you see in the nifi-user.log on the second instance (the one where the remote process group is pointing at)?
If it found the incoming cert it should log something like:
"Attempting request for <cert of first NiFi> %s %s (source ip: %s)"
"Authentication success for <cert of first NiFi>"
If it didn't find one I think it should log:
"Rejecting access to web api: %s", ae.getMessage()
It might be help to try debugging into the X509AuthenticationFilter:
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-security/src/main/java/org/apache/nifi/web/security/x509/X509AuthenticationFilter.java
The base class it extends is:
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-security/src/main/java/org/apache/nifi/web/security/NiFiAuthenticationFilter.java
If it goes into the X509 filter and returns null from attemptAuthentication, we can at least narrow down to the fact that that the client certificate is not in the request for some reason.
It sounds like you might have already done this, but there is a line in bootstrap.conf that you can uncomment to hook up a remote debugger:
java.arg.debug=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000
-Bryan
On Fri, Sep 2, 2016 at 11:27 AM, Peter Wicks (pwicks) <pw...@micron.com>> wrote:
Matt,
That was the case on our first go round, when we only had SSL certs. We went back yesterday and got new certificates that support both Server Auth and Client Auth and rebuilt our KeyStore.
When I use keytool to look at my KeyStore I can see both of these on the certificate:
#6: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
clientAuth
serverAuth
]
Thanks,
Peter
From: Matthew Clarke [mailto:matt.clarke.138@gmail.com<ma...@gmail.com>]
Sent: Friday, September 02, 2016 9:23 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: I need help configuring Site-to-Site in Secure Mode.
Do the certs you created/obtained support being used for both client and server auth. If they were created for server auth only, this could explain your issue. NiFi instances need to act as a client and a server at times.
Thanks,
Matt
On Fri, Sep 2, 2016 at 10:59 AM, Peter Wicks (pwicks) <pw...@micron.com>> wrote:
Bryan,
We’ve fixed our certs, with no change to the outcome.
We have username/password authentication enabled, via Kerberos, are there issues having Kerberos enabled (username/password) and trying to do site-to-site? When I try to initiate site-to-site with an instance of NiFI configured for Kerberos all requests come through to the server as anonymous because no challenge appears to be sent. We’ve debugged the code and even deep down in NiFiUserUtils.getNiFiUser the request is already marked as anonymous by the Spring framework. It appears to me that the client has a cert, and is waiting for a challenge(?) from the server, and the server is configured for Kerberos and it’s waiting for a ‘bearer’ token…
We’ve debugged both client and server, the client sends the request and gets back a 401 (Unauthorized). SSL verifies good.
Server doesn’t appear to get any authorization information of any kind.
Looking for further guidance/next steps.
Thanks,
Peter
From: Bryan Bende [mailto:bbende@gmail.com<ma...@gmail.com>]
Sent: Thursday, September 01, 2016 9:44 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: I need help configuring Site-to-Site in Secure Mode.
Peter,
Yes, by no means am I saying everyone should use the TLS toolkit. I was just using that because many people are not familiar with how to create certificates, and for people trying to follow a tutorial it is the easiest option.
In your case you definitely want to be using your CA. What you described about not having a cert for client authentication definitely sounds like it would be a problem. Let us know if everything works out after getting the new certs.
-Bryan
On Thu, Sep 1, 2016 at 11:34 AM, Peter Wicks (pwicks) <pw...@micron.com>> wrote:
Bryan,
Paul and I have been working on this, and I think our issue is related to certificates.
In your blog posting you used TLS-Toolkit in your example, but I think that is unrealistic for many environments. For example, this also creates the certificates for SSL right? But these will be self-signed and thus untrusted by default in web browsers. In our environment we generated SSL certificates from our CA and loaded them into the KeyStore. We then extracted public keys for the SSL certs and put them in each of the Trust Stores. This I think is where our main problem is…
I’m making a few assumptions here, so feel free to correct me, but my understanding is that when you use TLS-Toolkit it either creates multiple certs (SSL & Client Auth), or it creates a cert that you are allowed to use for both activities. In our case we ONLY have SSL certs, and the certs are marked such that they aren’t allowed to be used for Client Authentication. I believe this is the reason why our requests are showing up as ‘Anonymous’, because there are no Client Authentication certificates in the KeyStore, just SSL certs.
I’ve asked our security team for Client Authentication certs for each server, since it would be our preference to use our CA rather than having TLS-Toolkit be its own CA.
Thoughts?
Thanks,
Peter
From: Bryan Bende [mailto:bbende@gmail.com<ma...@gmail.com>]
Sent: Thursday, September 01, 2016 9:26 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: I need help configuring Site-to-Site in Secure Mode.
Paul,
Clustering is not a requirement for site-to-site... This sounds strange since "anonymous" is used to represent a user when NiFi is not secured.
Can you double-check all your configs and make sure you have the following properties set...
nifi.remote.input.secure=true
nifi.web.https.host=
nifi.web.https.port=
nifi.security.keystore=
nifi.security.keystoreType=
nifi.security.keystorePasswd=
nifi.security.keyPasswd=
nifi.security.truststore=
nifi.security.truststoreType=
nifi.security.truststorePasswd=<your password>
After your question the other day I went through the steps of setting secure site-to-site to make sure I knew what I was talking about :)
I wrote up the steps here: http://bryanbende.com/development/2016/08/30/apache-nifi-1.0.0-secure-site-to-site
Thanks,
Bryan
On Thu, Sep 1, 2016 at 10:44 AM, Paul Gibeault (pagibeault) <pa...@micron.com>> wrote:
Bryan,
Thanks for the reply. After increasing the log level for Authentication I saw the target NiFi instance used the account “anonymous” for the Site-to-Site connection. After creating a policy for “anonymous”, I was able to view the remote ports and connect to them.
Obviously this is not ideal. We would prefer to make policies for remote hosts/users rather than anonymous.
We are using the same SSL Certificate for both Key Store and Trust Store on a NiFi Instance. This is likely the cause of the “anonymous” user as it doesn’t have a DN. We are working to correct this.
However, after getting my work flow set up across NiFi instances I see this error:
Unable to refresh Remote Group's peers due to Unable to communicate with remote NiFi cluster in order to determine which nodes exist in the remote cluster
Our NiFi servers are not set up for clustering. Is clustering required to perform Site-to-Site?
Thanks,
Paul Gibeault
From: Bryan Bende [mailto:bbende@gmail.com<ma...@gmail.com>]
Sent: Tuesday, August 30, 2016 5:09 PM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: I need help configuring Site-to-Site in Secure Mode.
Paul,
It sounds like you probably have the certificates/truststores setup correctly and just need to create the appropriate policies...
Lets say you have nifi-1 with an Remote Process Group pointing at the URL of nifi-2, and nifi-2 has an Input port to receive data.
In nifi-2 there needs to be a user for the certificate of nifi-1, and then in the global policies of nifi-2 (top right menu) there needs to be a policy for "retrieve site-to-site details" with the nifi-1 user added to the policy. I think this is what is causing the error message you are seeing since nifi-1 is not authorized to query nifi-2 for site-to-site information (available ports, etc).
I believe you also need to create a policy on the Input Port on nifi-2... select the input port and use the lock icon in the left palette and choose "receive data over site-to-site" and add the user of nifi-1. This gives nifi-1 access to the specific port.
Let us know if that works. If so we should definitely look at updating some of the documentation to explain this.
Thanks,
Bryan
On Tue, Aug 30, 2016 at 6:28 PM, Paul Gibeault (pagibeault) <pa...@micron.com>> wrote:
Hello,
We have been attempting to set up Site-to-Site for NiFi in secure mode and have not been successful.
When I create a Remote Process Group, and enter the URL* https://servername:8443/nifi I receive an error icon. The hover status is “Unauthorized”
* - servername is the actual hostname running NiFi
Things I have tried without success:
- Closely followed the instructions in the NiFi System Administrator’s Guide<https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html>
- Enabled SSL Security and Kerberos User Authentication (no self-signed certs)
- Imported public keys of remote NiFi servers into the local keystores for each instance
- Created policies on each instance to allow for full permissions to the accounts in use.
- Tried various combinations of Linux & Windows instances of NiFi.
- Connected a Site-to-Site process group to itself
- Used both 1.0.0-BETA and 1.1.0-SNAPSHOT NiFi versions
There are no warnings or errors in the log files when I attempt to connect a NiFi instance running on Linux to another instance on Linux. However, I did see something when attempting to connect a NiFi instance running on Windows to an instance running on Linux
From log of NiFi on Windows:
2016-08-30 16:11:21,173 ERROR [Remote Process Group dd2e3ac9-0156-1000-4543-5ba3d10c6130: https://servername:8443/nifi Thread-1] o.a.n.remote.StandardRemoteProcessGroup org.apache.nifi.remote.StandardRemoteProcessGroup$InitializationTask@19bde1f3<ma...@19bde1f3> Failed to request account: got unexpected response code of 404:Not Found
From log of NiFi on Linux:
nifi-user.log:2016-08-30 16:39:53,973 INFO [NiFi Web Server-465] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (CN=pagibeault) GET https://servername:8443/nifi-api/site-to-site (source ip: 137.201.48.150)
nifi-user.log:2016-08-30 16:39:53,973 INFO [NiFi Web Server-465] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for CN=pagibeault
nifi-user.log:2016-08-30 16:39:53,974 INFO [NiFi Web Server-465] o.a.n.w.a.c.AccessDeniedExceptionMapper CN=pagibeault does not have permission to access the requested resource. Returning Forbidden response.
Any guidance would be grand.
Thanks,
[http://collab.micron.com/corp/brand/SiteAssets/Micron.png]<http://www.micron.com/>
Paul Gibeault
Sr. Software Engineer, Big Data
Enterprise Analytics & Data
Micron Technology, Inc.
Office (208) 363-3238<tel:%28208%29%20363-3238>
pagibeault@micron.com<ma...@micron.com>
[http://collab.micron.com/corp/brand/SiteAssets/LinkedIn.png] linkedin.com/in/paulgibeault<https://www.linkedin.com/in/linkedin.com/in/paulgibeault>
Re: I need help configuring Site-to-Site in Secure Mode.
Posted by Bryan Bende <bb...@gmail.com>.
Ok, I am not completely familiar with all the ins and outs of the
site-to-site client code, but I know one place that it creates a connection
is here:
https://github.com/apache/nifi/blob/a3586e04d9978e105cc5645e893dc6d77b79b86e/nifi-commons/nifi-site-to-site-client/src/main/java/org/apache/nifi/remote/client/socket/EndpointConnectionPool.java#L439
The SSLContext that will be used in there comes from the FlowController
which takes the SSL related properties from nifi.properties:
https://github.com/apache/nifi/blob/1745c1274b274a994a92312054d8951ce1c499d0/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/FlowController.java#L442
The part I am not sure about is whether or not there is a call somewhere
before the EndpointConnectionPool, but you could at least start by seeing
if it is hitting there. I can look a little more later.
-Bryan
On Fri, Sep 2, 2016 at 12:25 PM, Peter Wicks (pwicks) <pw...@micron.com>
wrote:
> Bryan,
>
>
>
> In the log on the server side we see this message:
>
>
>
> INFO [NiFi Web Server-324] o.a.n.w.a.c.AccessDeniedExceptionMapper
> anonymous does not have permission to access the requested resource.
> Returning Unauthorized response.
>
>
>
> I forgot to mention, we tried adding a user named anonymous and granting
> it access. When we did this Site-to-Site started working. Obviously that
> is not a course of action we want to take, but it was a good exercise.
>
>
>
> When I debugged the X509AuthenticationFilter it found no certs, here is
> the screenshot showing some of the stack/variables at the time.
>
> https://goo.gl/photos/938E5A8vsb7nQ6Kh8
>
>
>
> What would be the equivalent source code for debugging the client to find
> out why no cert is being sent?
>
>
>
> Thanks,
>
> Peter
>
>
>
>
>
>
>
> *From:* Bryan Bende [mailto:bbende@gmail.com]
> *Sent:* Friday, September 02, 2016 10:09 AM
>
> *To:* users@nifi.apache.org
> *Subject:* Re: I need help configuring Site-to-Site in Secure Mode.
>
>
>
> Paul/Peter,
>
>
>
> Having Kerberos enabled should not have any impact, you can have Kerberos
> or LDAP enabled, but if a certificate is provided that should always take
> precedence.
>
>
>
> What do you see in the nifi-user.log on the second instance (the one where
> the remote process group is pointing at)?
>
>
>
> If it found the incoming cert it should log something like:
>
>
>
> "Attempting request for <cert of first NiFi> %s %s (source ip: %s)"
> "Authentication success for <cert of first NiFi>"
>
>
>
> If it didn't find one I think it should log:
>
>
> "Rejecting access to web api: %s", ae.getMessage()
>
>
>
> It might be help to try debugging into the X509AuthenticationFilter:
>
>
>
> https://github.com/apache/nifi/blob/master/nifi-nar-
> bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-
> web-security/src/main/java/org/apache/nifi/web/security/
> x509/X509AuthenticationFilter.java
>
>
>
> The base class it extends is:
>
>
>
> https://github.com/apache/nifi/blob/master/nifi-nar-
> bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-
> web-security/src/main/java/org/apache/nifi/web/security/
> NiFiAuthenticationFilter.java
>
>
>
> If it goes into the X509 filter and returns null from
> attemptAuthentication, we can at least narrow down to the fact that that
> the client certificate is not in the request for some reason.
>
>
>
> It sounds like you might have already done this, but there is a line in
> bootstrap.conf that you can uncomment to hook up a remote debugger:
>
> java.arg.debug=-agentlib:jdwp=transport=dt_socket,server=y,
> suspend=n,address=8000
>
> -Bryan
>
>
>
>
>
>
>
> On Fri, Sep 2, 2016 at 11:27 AM, Peter Wicks (pwicks) <pw...@micron.com>
> wrote:
>
> Matt,
>
>
>
> That was the case on our first go round, when we only had SSL certs. We
> went back yesterday and got new certificates that support both Server Auth
> and Client Auth and rebuilt our KeyStore.
>
>
>
> When I use keytool to look at my KeyStore I can see both of these on the
> certificate:
>
>
>
> #6: ObjectId: 2.5.29.37 Criticality=false
>
> ExtendedKeyUsages [
>
> * clientAuth*
>
> * serverAuth*
>
> ]
>
>
>
> Thanks,
>
> Peter
>
>
>
> *From:* Matthew Clarke [mailto:matt.clarke.138@gmail.com]
> *Sent:* Friday, September 02, 2016 9:23 AM
>
>
> *To:* users@nifi.apache.org
> *Subject:* Re: I need help configuring Site-to-Site in Secure Mode.
>
>
>
> Do the certs you created/obtained support being used for both client and
> server auth. If they were created for server auth only, this could explain
> your issue. NiFi instances need to act as a client and a server at times.
>
>
>
> Thanks,
>
> Matt
>
>
>
> On Fri, Sep 2, 2016 at 10:59 AM, Peter Wicks (pwicks) <pw...@micron.com>
> wrote:
>
> Bryan,
>
>
>
> We’ve fixed our certs, with no change to the outcome.
>
>
>
> We have username/password authentication enabled, via Kerberos, are there
> issues having Kerberos enabled (username/password) and trying to do
> site-to-site? When I try to initiate site-to-site with an instance of NiFI
> configured for Kerberos all requests come through to the server as
> anonymous because no challenge appears to be sent. We’ve debugged the code
> and even deep down in NiFiUserUtils.getNiFiUser the request is already
> marked as anonymous by the Spring framework. It appears to me that the
> client has a cert, and is waiting for a challenge(?) from the server, and
> the server is configured for Kerberos and it’s waiting for a ‘bearer’ token…
>
>
>
> We’ve debugged both client and server, the client sends the request and
> gets back a 401 (Unauthorized). SSL verifies good.
>
> Server doesn’t appear to get any authorization information of any kind.
>
>
>
> Looking for further guidance/next steps.
>
>
>
> Thanks,
>
> Peter
>
>
>
> *From:* Bryan Bende [mailto:bbende@gmail.com]
> *Sent:* Thursday, September 01, 2016 9:44 AM
>
>
> *To:* users@nifi.apache.org
> *Subject:* Re: I need help configuring Site-to-Site in Secure Mode.
>
>
>
> Peter,
>
>
>
> Yes, by no means am I saying everyone should use the TLS toolkit. I was
> just using that because many people are not familiar with how to create
> certificates, and for people trying to follow a tutorial it is the easiest
> option.
>
>
>
> In your case you definitely want to be using your CA. What you described
> about not having a cert for client authentication definitely sounds like it
> would be a problem. Let us know if everything works out after getting the
> new certs.
>
>
>
> -Bryan
>
>
>
>
>
> On Thu, Sep 1, 2016 at 11:34 AM, Peter Wicks (pwicks) <pw...@micron.com>
> wrote:
>
> Bryan,
>
>
>
> Paul and I have been working on this, and I think our issue is related to
> certificates.
>
>
>
> In your blog posting you used TLS-Toolkit in your example, but I think
> that is unrealistic for many environments. For example, this also creates
> the certificates for SSL right? But these will be self-signed and thus
> untrusted by default in web browsers. In our environment we generated SSL
> certificates from our CA and loaded them into the KeyStore. We then
> extracted public keys for the SSL certs and put them in each of the Trust
> Stores. This I think is where our main problem is…
>
>
>
> I’m making a few assumptions here, so feel free to correct me, but my
> understanding is that when you use TLS-Toolkit it either creates multiple
> certs (SSL & Client Auth), or it creates a cert that you are allowed to use
> for both activities. In our case we ONLY have SSL certs, and the certs are
> marked such that they aren’t allowed to be used for Client Authentication.
> I believe this is the reason why our requests are showing up as
> ‘Anonymous’, because there are no Client Authentication certificates in the
> KeyStore, just SSL certs.
>
>
>
> I’ve asked our security team for Client Authentication certs for each
> server, since it would be our preference to use our CA rather than having
> TLS-Toolkit be its own CA.
>
>
>
> Thoughts?
>
>
>
> Thanks,
>
> Peter
>
>
>
> *From:* Bryan Bende [mailto:bbende@gmail.com]
> *Sent:* Thursday, September 01, 2016 9:26 AM
>
>
> *To:* users@nifi.apache.org
> *Subject:* Re: I need help configuring Site-to-Site in Secure Mode.
>
>
>
> Paul,
>
>
>
> Clustering is not a requirement for site-to-site... This sounds strange
> since "anonymous" is used to represent a user when NiFi is not secured.
>
>
>
> Can you double-check all your configs and make sure you have the following
> properties set...
>
>
>
> nifi.remote.input.secure=true
>
> nifi.web.https.host=
> nifi.web.https.port=
>
> nifi.security.keystore=
> nifi.security.keystoreType=
> nifi.security.keystorePasswd=
> nifi.security.keyPasswd=
> nifi.security.truststore=
> nifi.security.truststoreType=
> nifi.security.truststorePasswd=<your password>
>
>
>
> After your question the other day I went through the steps of setting
> secure site-to-site to make sure I knew what I was talking about :)
>
>
>
> I wrote up the steps here: http://bryanbende.com/
> development/2016/08/30/apache-nifi-1.0.0-secure-site-to-site
>
>
>
> Thanks,
>
>
>
> Bryan
>
>
>
> On Thu, Sep 1, 2016 at 10:44 AM, Paul Gibeault (pagibeault) <
> pagibeault@micron.com> wrote:
>
> Bryan,
>
>
>
> Thanks for the reply. After increasing the log level for Authentication I
> saw the target NiFi instance used the account “anonymous” for the
> Site-to-Site connection. After creating a policy for “anonymous”, I was
> able to view the remote ports and connect to them.
>
>
>
> Obviously this is not ideal. We would prefer to make policies for remote
> hosts/users rather than anonymous.
>
>
>
> We are using the same SSL Certificate for both Key Store and Trust Store
> on a NiFi Instance. This is likely the cause of the “anonymous” user as it
> doesn’t have a DN. We are working to correct this.
>
>
>
> However, after getting my work flow set up across NiFi instances I see
> this error:
>
>
>
> Unable to refresh Remote Group's peers due to Unable to communicate with
> remote NiFi cluster in order to determine which nodes exist in the remote
> cluster
>
>
>
> Our NiFi servers are not set up for clustering. Is clustering required to
> perform Site-to-Site?
>
>
>
> Thanks,
>
> Paul Gibeault
>
>
>
> *From:* Bryan Bende [mailto:bbende@gmail.com]
> *Sent:* Tuesday, August 30, 2016 5:09 PM
> *To:* users@nifi.apache.org
> *Subject:* Re: I need help configuring Site-to-Site in Secure Mode.
>
>
>
> Paul,
>
>
>
> It sounds like you probably have the certificates/truststores setup
> correctly and just need to create the appropriate policies...
>
>
>
> Lets say you have nifi-1 with an Remote Process Group pointing at the URL
> of nifi-2, and nifi-2 has an Input port to receive data.
>
>
>
> In nifi-2 there needs to be a user for the certificate of nifi-1, and then
> in the global policies of nifi-2 (top right menu) there needs to be a
> policy for "retrieve site-to-site details" with the nifi-1 user added to
> the policy. I think this is what is causing the error message you are
> seeing since nifi-1 is not authorized to query nifi-2 for site-to-site
> information (available ports, etc).
>
>
>
> I believe you also need to create a policy on the Input Port on nifi-2...
> select the input port and use the lock icon in the left palette and choose
> "receive data over site-to-site" and add the user of nifi-1. This gives
> nifi-1 access to the specific port.
>
>
>
> Let us know if that works. If so we should definitely look at updating
> some of the documentation to explain this.
>
>
>
> Thanks,
>
>
>
> Bryan
>
>
>
>
>
> On Tue, Aug 30, 2016 at 6:28 PM, Paul Gibeault (pagibeault) <
> pagibeault@micron.com> wrote:
>
> Hello,
>
>
>
> We have been attempting to set up Site-to-Site for NiFi in secure mode and
> have not been successful.
>
>
>
> When I create a Remote Process Group, and enter the URL*
> https://servername:8443/nifi I receive an error icon. The hover status
> is “Unauthorized”
>
> ** - servername is the actual hostname running NiFi*
>
>
>
> Things I have tried without success:
>
> - Closely followed the instructions in the NiFi System Administrator’s
> Guide
> <https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html>
>
> - Enabled SSL Security and Kerberos User Authentication (no self-signed
> certs)
>
> - Imported public keys of remote NiFi servers into the local keystores for
> each instance
>
> - Created policies on each instance to allow for full permissions to the
> accounts in use.
>
> - Tried various combinations of Linux & Windows instances of NiFi.
>
> - Connected a Site-to-Site process group to itself
>
> - Used both 1.0.0-BETA and 1.1.0-SNAPSHOT NiFi versions
>
>
>
> There are no warnings or errors in the log files when I attempt to connect
> a NiFi instance running on Linux to another instance on Linux. However, I
> did see something when attempting to connect a NiFi instance running on
> Windows to an instance running on Linux
>
> From log of NiFi on Windows:
>
> 2016-08-30 16:11:21,173 ERROR [Remote Process Group
> dd2e3ac9-0156-1000-4543-5ba3d10c6130: https://servername:8443/nifi
> Thread-1] o.a.n.remote.StandardRemoteProcessGroup org.apache.nifi.remote.
> StandardRemoteProcessGroup$InitializationTask@19bde1f3 Failed to request
> account: got unexpected response code of 404:Not Found
>
>
>
> From log of NiFi on Linux:
>
> nifi-user.log:2016-08-30 16:39:53,973 INFO [NiFi Web Server-465] o.a.n.w.s.NiFiAuthenticationFilter
> Attempting request for (CN=pagibeault) GET https://servername:8443/nifi-
> api/site-to-site (source ip: 137.201.48.150)
>
> nifi-user.log:2016-08-30 16:39:53,973 INFO [NiFi Web Server-465] o.a.n.w.s.NiFiAuthenticationFilter
> Authentication success for CN=pagibeault
>
> nifi-user.log:2016-08-30 16:39:53,974 INFO [NiFi Web Server-465]
> o.a.n.w.a.c.AccessDeniedExceptionMapper CN=pagibeault does not have
> permission to access the requested resource. Returning Forbidden response.
>
>
>
> Any guidance would be grand.
>
>
>
> Thanks,
>
>
>
> [image: http://collab.micron.com/corp/brand/SiteAssets/Micron.png]
> <http://www.micron.com/>
>
> *Paul Gibeault*
> Sr. Software Engineer, Big Data
> Enterprise Analytics & Data
> Micron Technology, Inc.
>
> *Office* (208) 363-3238 <%28208%29%20363-3238>
>
>
> pagibeault@micron.com
> [image: http://collab.micron.com/corp/brand/SiteAssets/LinkedIn.png]
> linkedin.com/in/paulgibeault
> <https://www.linkedin.com/in/linkedin.com/in/paulgibeault>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
RE: I need help configuring Site-to-Site in Secure Mode.
Posted by "Peter Wicks (pwicks)" <pw...@micron.com>.
Bryan,
In the log on the server side we see this message:
INFO [NiFi Web Server-324] o.a.n.w.a.c.AccessDeniedExceptionMapper anonymous does not have permission to access the requested resource. Returning Unauthorized response.
I forgot to mention, we tried adding a user named anonymous and granting it access. When we did this Site-to-Site started working. Obviously that is not a course of action we want to take, but it was a good exercise.
When I debugged the X509AuthenticationFilter it found no certs, here is the screenshot showing some of the stack/variables at the time.
https://goo.gl/photos/938E5A8vsb7nQ6Kh8
What would be the equivalent source code for debugging the client to find out why no cert is being sent?
Thanks,
Peter
From: Bryan Bende [mailto:bbende@gmail.com]
Sent: Friday, September 02, 2016 10:09 AM
To: users@nifi.apache.org
Subject: Re: I need help configuring Site-to-Site in Secure Mode.
Paul/Peter,
Having Kerberos enabled should not have any impact, you can have Kerberos or LDAP enabled, but if a certificate is provided that should always take precedence.
What do you see in the nifi-user.log on the second instance (the one where the remote process group is pointing at)?
If it found the incoming cert it should log something like:
"Attempting request for <cert of first NiFi> %s %s (source ip: %s)"
"Authentication success for <cert of first NiFi>"
If it didn't find one I think it should log:
"Rejecting access to web api: %s", ae.getMessage()
It might be help to try debugging into the X509AuthenticationFilter:
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-security/src/main/java/org/apache/nifi/web/security/x509/X509AuthenticationFilter.java
The base class it extends is:
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-security/src/main/java/org/apache/nifi/web/security/NiFiAuthenticationFilter.java
If it goes into the X509 filter and returns null from attemptAuthentication, we can at least narrow down to the fact that that the client certificate is not in the request for some reason.
It sounds like you might have already done this, but there is a line in bootstrap.conf that you can uncomment to hook up a remote debugger:
java.arg.debug=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000
-Bryan
On Fri, Sep 2, 2016 at 11:27 AM, Peter Wicks (pwicks) <pw...@micron.com>> wrote:
Matt,
That was the case on our first go round, when we only had SSL certs. We went back yesterday and got new certificates that support both Server Auth and Client Auth and rebuilt our KeyStore.
When I use keytool to look at my KeyStore I can see both of these on the certificate:
#6: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
clientAuth
serverAuth
]
Thanks,
Peter
From: Matthew Clarke [mailto:matt.clarke.138@gmail.com<ma...@gmail.com>]
Sent: Friday, September 02, 2016 9:23 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: I need help configuring Site-to-Site in Secure Mode.
Do the certs you created/obtained support being used for both client and server auth. If they were created for server auth only, this could explain your issue. NiFi instances need to act as a client and a server at times.
Thanks,
Matt
On Fri, Sep 2, 2016 at 10:59 AM, Peter Wicks (pwicks) <pw...@micron.com>> wrote:
Bryan,
We’ve fixed our certs, with no change to the outcome.
We have username/password authentication enabled, via Kerberos, are there issues having Kerberos enabled (username/password) and trying to do site-to-site? When I try to initiate site-to-site with an instance of NiFI configured for Kerberos all requests come through to the server as anonymous because no challenge appears to be sent. We’ve debugged the code and even deep down in NiFiUserUtils.getNiFiUser the request is already marked as anonymous by the Spring framework. It appears to me that the client has a cert, and is waiting for a challenge(?) from the server, and the server is configured for Kerberos and it’s waiting for a ‘bearer’ token…
We’ve debugged both client and server, the client sends the request and gets back a 401 (Unauthorized). SSL verifies good.
Server doesn’t appear to get any authorization information of any kind.
Looking for further guidance/next steps.
Thanks,
Peter
From: Bryan Bende [mailto:bbende@gmail.com<ma...@gmail.com>]
Sent: Thursday, September 01, 2016 9:44 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: I need help configuring Site-to-Site in Secure Mode.
Peter,
Yes, by no means am I saying everyone should use the TLS toolkit. I was just using that because many people are not familiar with how to create certificates, and for people trying to follow a tutorial it is the easiest option.
In your case you definitely want to be using your CA. What you described about not having a cert for client authentication definitely sounds like it would be a problem. Let us know if everything works out after getting the new certs.
-Bryan
On Thu, Sep 1, 2016 at 11:34 AM, Peter Wicks (pwicks) <pw...@micron.com>> wrote:
Bryan,
Paul and I have been working on this, and I think our issue is related to certificates.
In your blog posting you used TLS-Toolkit in your example, but I think that is unrealistic for many environments. For example, this also creates the certificates for SSL right? But these will be self-signed and thus untrusted by default in web browsers. In our environment we generated SSL certificates from our CA and loaded them into the KeyStore. We then extracted public keys for the SSL certs and put them in each of the Trust Stores. This I think is where our main problem is…
I’m making a few assumptions here, so feel free to correct me, but my understanding is that when you use TLS-Toolkit it either creates multiple certs (SSL & Client Auth), or it creates a cert that you are allowed to use for both activities. In our case we ONLY have SSL certs, and the certs are marked such that they aren’t allowed to be used for Client Authentication. I believe this is the reason why our requests are showing up as ‘Anonymous’, because there are no Client Authentication certificates in the KeyStore, just SSL certs.
I’ve asked our security team for Client Authentication certs for each server, since it would be our preference to use our CA rather than having TLS-Toolkit be its own CA.
Thoughts?
Thanks,
Peter
From: Bryan Bende [mailto:bbende@gmail.com<ma...@gmail.com>]
Sent: Thursday, September 01, 2016 9:26 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: I need help configuring Site-to-Site in Secure Mode.
Paul,
Clustering is not a requirement for site-to-site... This sounds strange since "anonymous" is used to represent a user when NiFi is not secured.
Can you double-check all your configs and make sure you have the following properties set...
nifi.remote.input.secure=true
nifi.web.https.host=
nifi.web.https.port=
nifi.security.keystore=
nifi.security.keystoreType=
nifi.security.keystorePasswd=
nifi.security.keyPasswd=
nifi.security.truststore=
nifi.security.truststoreType=
nifi.security.truststorePasswd=<your password>
After your question the other day I went through the steps of setting secure site-to-site to make sure I knew what I was talking about :)
I wrote up the steps here: http://bryanbende.com/development/2016/08/30/apache-nifi-1.0.0-secure-site-to-site
Thanks,
Bryan
On Thu, Sep 1, 2016 at 10:44 AM, Paul Gibeault (pagibeault) <pa...@micron.com>> wrote:
Bryan,
Thanks for the reply. After increasing the log level for Authentication I saw the target NiFi instance used the account “anonymous” for the Site-to-Site connection. After creating a policy for “anonymous”, I was able to view the remote ports and connect to them.
Obviously this is not ideal. We would prefer to make policies for remote hosts/users rather than anonymous.
We are using the same SSL Certificate for both Key Store and Trust Store on a NiFi Instance. This is likely the cause of the “anonymous” user as it doesn’t have a DN. We are working to correct this.
However, after getting my work flow set up across NiFi instances I see this error:
Unable to refresh Remote Group's peers due to Unable to communicate with remote NiFi cluster in order to determine which nodes exist in the remote cluster
Our NiFi servers are not set up for clustering. Is clustering required to perform Site-to-Site?
Thanks,
Paul Gibeault
From: Bryan Bende [mailto:bbende@gmail.com<ma...@gmail.com>]
Sent: Tuesday, August 30, 2016 5:09 PM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: I need help configuring Site-to-Site in Secure Mode.
Paul,
It sounds like you probably have the certificates/truststores setup correctly and just need to create the appropriate policies...
Lets say you have nifi-1 with an Remote Process Group pointing at the URL of nifi-2, and nifi-2 has an Input port to receive data.
In nifi-2 there needs to be a user for the certificate of nifi-1, and then in the global policies of nifi-2 (top right menu) there needs to be a policy for "retrieve site-to-site details" with the nifi-1 user added to the policy. I think this is what is causing the error message you are seeing since nifi-1 is not authorized to query nifi-2 for site-to-site information (available ports, etc).
I believe you also need to create a policy on the Input Port on nifi-2... select the input port and use the lock icon in the left palette and choose "receive data over site-to-site" and add the user of nifi-1. This gives nifi-1 access to the specific port.
Let us know if that works. If so we should definitely look at updating some of the documentation to explain this.
Thanks,
Bryan
On Tue, Aug 30, 2016 at 6:28 PM, Paul Gibeault (pagibeault) <pa...@micron.com>> wrote:
Hello,
We have been attempting to set up Site-to-Site for NiFi in secure mode and have not been successful.
When I create a Remote Process Group, and enter the URL* https://servername:8443/nifi I receive an error icon. The hover status is “Unauthorized”
* - servername is the actual hostname running NiFi
Things I have tried without success:
- Closely followed the instructions in the NiFi System Administrator’s Guide<https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html>
- Enabled SSL Security and Kerberos User Authentication (no self-signed certs)
- Imported public keys of remote NiFi servers into the local keystores for each instance
- Created policies on each instance to allow for full permissions to the accounts in use.
- Tried various combinations of Linux & Windows instances of NiFi.
- Connected a Site-to-Site process group to itself
- Used both 1.0.0-BETA and 1.1.0-SNAPSHOT NiFi versions
There are no warnings or errors in the log files when I attempt to connect a NiFi instance running on Linux to another instance on Linux. However, I did see something when attempting to connect a NiFi instance running on Windows to an instance running on Linux
From log of NiFi on Windows:
2016-08-30 16:11:21,173 ERROR [Remote Process Group dd2e3ac9-0156-1000-4543-5ba3d10c6130: https://servername:8443/nifi Thread-1] o.a.n.remote.StandardRemoteProcessGroup org.apache.nifi.remote.StandardRemoteProcessGroup$InitializationTask@19bde1f3<ma...@19bde1f3> Failed to request account: got unexpected response code of 404:Not Found
From log of NiFi on Linux:
nifi-user.log:2016-08-30 16:39:53,973 INFO [NiFi Web Server-465] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (CN=pagibeault) GET https://servername:8443/nifi-api/site-to-site (source ip: 137.201.48.150)
nifi-user.log:2016-08-30 16:39:53,973 INFO [NiFi Web Server-465] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for CN=pagibeault
nifi-user.log:2016-08-30 16:39:53,974 INFO [NiFi Web Server-465] o.a.n.w.a.c.AccessDeniedExceptionMapper CN=pagibeault does not have permission to access the requested resource. Returning Forbidden response.
Any guidance would be grand.
Thanks,
[http://collab.micron.com/corp/brand/SiteAssets/Micron.png]<http://www.micron.com/>
Paul Gibeault
Sr. Software Engineer, Big Data
Enterprise Analytics & Data
Micron Technology, Inc.
Office (208) 363-3238<tel:%28208%29%20363-3238>
pagibeault@micron.com<ma...@micron.com>
[http://collab.micron.com/corp/brand/SiteAssets/LinkedIn.png] linkedin.com/in/paulgibeault<https://www.linkedin.com/in/linkedin.com/in/paulgibeault>
Re: I need help configuring Site-to-Site in Secure Mode.
Posted by Bryan Bende <bb...@gmail.com>.
Paul/Peter,
Having Kerberos enabled should not have any impact, you can have Kerberos
or LDAP enabled, but if a certificate is provided that should always take
precedence.
What do you see in the nifi-user.log on the second instance (the one where
the remote process group is pointing at)?
If it found the incoming cert it should log something like:
"Attempting request for <cert of first NiFi> %s %s (source ip: %s)"
"Authentication success for <cert of first NiFi>"
If it didn't find one I think it should log:
"Rejecting access to web api: %s", ae.getMessage()
It might be help to try debugging into the X509AuthenticationFilter:
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-security/src/main/java/org/apache/nifi/web/security/x509/X509AuthenticationFilter.java
The base class it extends is:
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-security/src/main/java/org/apache/nifi/web/security/NiFiAuthenticationFilter.java
If it goes into the X509 filter and returns null from
attemptAuthentication, we can at least narrow down to the fact that that
the client certificate is not in the request for some reason.
It sounds like you might have already done this, but there is a line in
bootstrap.conf that you can uncomment to hook up a remote debugger:
java.arg.debug=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000
-Bryan
On Fri, Sep 2, 2016 at 11:27 AM, Peter Wicks (pwicks) <pw...@micron.com>
wrote:
> Matt,
>
>
>
> That was the case on our first go round, when we only had SSL certs. We
> went back yesterday and got new certificates that support both Server Auth
> and Client Auth and rebuilt our KeyStore.
>
>
>
> When I use keytool to look at my KeyStore I can see both of these on the
> certificate:
>
>
>
> #6: ObjectId: 2.5.29.37 Criticality=false
>
> ExtendedKeyUsages [
>
> * clientAuth*
>
> * serverAuth*
>
> ]
>
>
>
> Thanks,
>
> Peter
>
>
>
> *From:* Matthew Clarke [mailto:matt.clarke.138@gmail.com]
> *Sent:* Friday, September 02, 2016 9:23 AM
>
> *To:* users@nifi.apache.org
> *Subject:* Re: I need help configuring Site-to-Site in Secure Mode.
>
>
>
> Do the certs you created/obtained support being used for both client and
> server auth. If they were created for server auth only, this could explain
> your issue. NiFi instances need to act as a client and a server at times.
>
>
>
> Thanks,
>
> Matt
>
>
>
> On Fri, Sep 2, 2016 at 10:59 AM, Peter Wicks (pwicks) <pw...@micron.com>
> wrote:
>
> Bryan,
>
>
>
> We’ve fixed our certs, with no change to the outcome.
>
>
>
> We have username/password authentication enabled, via Kerberos, are there
> issues having Kerberos enabled (username/password) and trying to do
> site-to-site? When I try to initiate site-to-site with an instance of NiFI
> configured for Kerberos all requests come through to the server as
> anonymous because no challenge appears to be sent. We’ve debugged the code
> and even deep down in NiFiUserUtils.getNiFiUser the request is already
> marked as anonymous by the Spring framework. It appears to me that the
> client has a cert, and is waiting for a challenge(?) from the server, and
> the server is configured for Kerberos and it’s waiting for a ‘bearer’ token…
>
>
>
> We’ve debugged both client and server, the client sends the request and
> gets back a 401 (Unauthorized). SSL verifies good.
>
> Server doesn’t appear to get any authorization information of any kind.
>
>
>
> Looking for further guidance/next steps.
>
>
>
> Thanks,
>
> Peter
>
>
>
> *From:* Bryan Bende [mailto:bbende@gmail.com]
> *Sent:* Thursday, September 01, 2016 9:44 AM
>
>
> *To:* users@nifi.apache.org
> *Subject:* Re: I need help configuring Site-to-Site in Secure Mode.
>
>
>
> Peter,
>
>
>
> Yes, by no means am I saying everyone should use the TLS toolkit. I was
> just using that because many people are not familiar with how to create
> certificates, and for people trying to follow a tutorial it is the easiest
> option.
>
>
>
> In your case you definitely want to be using your CA. What you described
> about not having a cert for client authentication definitely sounds like it
> would be a problem. Let us know if everything works out after getting the
> new certs.
>
>
>
> -Bryan
>
>
>
>
>
> On Thu, Sep 1, 2016 at 11:34 AM, Peter Wicks (pwicks) <pw...@micron.com>
> wrote:
>
> Bryan,
>
>
>
> Paul and I have been working on this, and I think our issue is related to
> certificates.
>
>
>
> In your blog posting you used TLS-Toolkit in your example, but I think
> that is unrealistic for many environments. For example, this also creates
> the certificates for SSL right? But these will be self-signed and thus
> untrusted by default in web browsers. In our environment we generated SSL
> certificates from our CA and loaded them into the KeyStore. We then
> extracted public keys for the SSL certs and put them in each of the Trust
> Stores. This I think is where our main problem is…
>
>
>
> I’m making a few assumptions here, so feel free to correct me, but my
> understanding is that when you use TLS-Toolkit it either creates multiple
> certs (SSL & Client Auth), or it creates a cert that you are allowed to use
> for both activities. In our case we ONLY have SSL certs, and the certs are
> marked such that they aren’t allowed to be used for Client Authentication.
> I believe this is the reason why our requests are showing up as
> ‘Anonymous’, because there are no Client Authentication certificates in the
> KeyStore, just SSL certs.
>
>
>
> I’ve asked our security team for Client Authentication certs for each
> server, since it would be our preference to use our CA rather than having
> TLS-Toolkit be its own CA.
>
>
>
> Thoughts?
>
>
>
> Thanks,
>
> Peter
>
>
>
> *From:* Bryan Bende [mailto:bbende@gmail.com]
> *Sent:* Thursday, September 01, 2016 9:26 AM
>
>
> *To:* users@nifi.apache.org
> *Subject:* Re: I need help configuring Site-to-Site in Secure Mode.
>
>
>
> Paul,
>
>
>
> Clustering is not a requirement for site-to-site... This sounds strange
> since "anonymous" is used to represent a user when NiFi is not secured.
>
>
>
> Can you double-check all your configs and make sure you have the following
> properties set...
>
>
>
> nifi.remote.input.secure=true
>
> nifi.web.https.host=
> nifi.web.https.port=
>
> nifi.security.keystore=
> nifi.security.keystoreType=
> nifi.security.keystorePasswd=
> nifi.security.keyPasswd=
> nifi.security.truststore=
> nifi.security.truststoreType=
> nifi.security.truststorePasswd=<your password>
>
>
>
> After your question the other day I went through the steps of setting
> secure site-to-site to make sure I knew what I was talking about :)
>
>
>
> I wrote up the steps here: http://bryanbende.com/
> development/2016/08/30/apache-nifi-1.0.0-secure-site-to-site
>
>
>
> Thanks,
>
>
>
> Bryan
>
>
>
> On Thu, Sep 1, 2016 at 10:44 AM, Paul Gibeault (pagibeault) <
> pagibeault@micron.com> wrote:
>
> Bryan,
>
>
>
> Thanks for the reply. After increasing the log level for Authentication I
> saw the target NiFi instance used the account “anonymous” for the
> Site-to-Site connection. After creating a policy for “anonymous”, I was
> able to view the remote ports and connect to them.
>
>
>
> Obviously this is not ideal. We would prefer to make policies for remote
> hosts/users rather than anonymous.
>
>
>
> We are using the same SSL Certificate for both Key Store and Trust Store
> on a NiFi Instance. This is likely the cause of the “anonymous” user as it
> doesn’t have a DN. We are working to correct this.
>
>
>
> However, after getting my work flow set up across NiFi instances I see
> this error:
>
>
>
> Unable to refresh Remote Group's peers due to Unable to communicate with
> remote NiFi cluster in order to determine which nodes exist in the remote
> cluster
>
>
>
> Our NiFi servers are not set up for clustering. Is clustering required to
> perform Site-to-Site?
>
>
>
> Thanks,
>
> Paul Gibeault
>
>
>
> *From:* Bryan Bende [mailto:bbende@gmail.com]
> *Sent:* Tuesday, August 30, 2016 5:09 PM
> *To:* users@nifi.apache.org
> *Subject:* Re: I need help configuring Site-to-Site in Secure Mode.
>
>
>
> Paul,
>
>
>
> It sounds like you probably have the certificates/truststores setup
> correctly and just need to create the appropriate policies...
>
>
>
> Lets say you have nifi-1 with an Remote Process Group pointing at the URL
> of nifi-2, and nifi-2 has an Input port to receive data.
>
>
>
> In nifi-2 there needs to be a user for the certificate of nifi-1, and then
> in the global policies of nifi-2 (top right menu) there needs to be a
> policy for "retrieve site-to-site details" with the nifi-1 user added to
> the policy. I think this is what is causing the error message you are
> seeing since nifi-1 is not authorized to query nifi-2 for site-to-site
> information (available ports, etc).
>
>
>
> I believe you also need to create a policy on the Input Port on nifi-2...
> select the input port and use the lock icon in the left palette and choose
> "receive data over site-to-site" and add the user of nifi-1. This gives
> nifi-1 access to the specific port.
>
>
>
> Let us know if that works. If so we should definitely look at updating
> some of the documentation to explain this.
>
>
>
> Thanks,
>
>
>
> Bryan
>
>
>
>
>
> On Tue, Aug 30, 2016 at 6:28 PM, Paul Gibeault (pagibeault) <
> pagibeault@micron.com> wrote:
>
> Hello,
>
>
>
> We have been attempting to set up Site-to-Site for NiFi in secure mode and
> have not been successful.
>
>
>
> When I create a Remote Process Group, and enter the URL*
> https://servername:8443/nifi I receive an error icon. The hover status
> is “Unauthorized”
>
> ** - servername is the actual hostname running NiFi*
>
>
>
> Things I have tried without success:
>
> - Closely followed the instructions in the NiFi System Administrator’s
> Guide
> <https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html>
>
> - Enabled SSL Security and Kerberos User Authentication (no self-signed
> certs)
>
> - Imported public keys of remote NiFi servers into the local keystores for
> each instance
>
> - Created policies on each instance to allow for full permissions to the
> accounts in use.
>
> - Tried various combinations of Linux & Windows instances of NiFi.
>
> - Connected a Site-to-Site process group to itself
>
> - Used both 1.0.0-BETA and 1.1.0-SNAPSHOT NiFi versions
>
>
>
> There are no warnings or errors in the log files when I attempt to connect
> a NiFi instance running on Linux to another instance on Linux. However, I
> did see something when attempting to connect a NiFi instance running on
> Windows to an instance running on Linux
>
> From log of NiFi on Windows:
>
> 2016-08-30 16:11:21,173 ERROR [Remote Process Group
> dd2e3ac9-0156-1000-4543-5ba3d10c6130: https://servername:8443/nifi
> Thread-1] o.a.n.remote.StandardRemoteProcessGroup org.apache.nifi.remote.
> StandardRemoteProcessGroup$InitializationTask@19bde1f3 Failed to request
> account: got unexpected response code of 404:Not Found
>
>
>
> From log of NiFi on Linux:
>
> nifi-user.log:2016-08-30 16:39:53,973 INFO [NiFi Web Server-465] o.a.n.w.s.NiFiAuthenticationFilter
> Attempting request for (CN=pagibeault) GET https://servername:8443/nifi-
> api/site-to-site (source ip: 137.201.48.150)
>
> nifi-user.log:2016-08-30 16:39:53,973 INFO [NiFi Web Server-465] o.a.n.w.s.NiFiAuthenticationFilter
> Authentication success for CN=pagibeault
>
> nifi-user.log:2016-08-30 16:39:53,974 INFO [NiFi Web Server-465]
> o.a.n.w.a.c.AccessDeniedExceptionMapper CN=pagibeault does not have
> permission to access the requested resource. Returning Forbidden response.
>
>
>
> Any guidance would be grand.
>
>
>
> Thanks,
>
>
>
> [image: http://collab.micron.com/corp/brand/SiteAssets/Micron.png]
> <http://www.micron.com/>
>
> *Paul Gibeault*
> Sr. Software Engineer, Big Data
> Enterprise Analytics & Data
> Micron Technology, Inc.
>
> *Office* (208) 363-3238 <%28208%29%20363-3238>
>
>
> pagibeault@micron.com
> [image: http://collab.micron.com/corp/brand/SiteAssets/LinkedIn.png]
> linkedin.com/in/paulgibeault
> <https://www.linkedin.com/in/linkedin.com/in/paulgibeault>
>
>
>
>
>
>
>
>
>
>
>
>
>
RE: I need help configuring Site-to-Site in Secure Mode.
Posted by "Peter Wicks (pwicks)" <pw...@micron.com>.
Matt,
That was the case on our first go round, when we only had SSL certs. We went back yesterday and got new certificates that support both Server Auth and Client Auth and rebuilt our KeyStore.
When I use keytool to look at my KeyStore I can see both of these on the certificate:
#6: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
clientAuth
serverAuth
]
Thanks,
Peter
From: Matthew Clarke [mailto:matt.clarke.138@gmail.com]
Sent: Friday, September 02, 2016 9:23 AM
To: users@nifi.apache.org
Subject: Re: I need help configuring Site-to-Site in Secure Mode.
Do the certs you created/obtained support being used for both client and server auth. If they were created for server auth only, this could explain your issue. NiFi instances need to act as a client and a server at times.
Thanks,
Matt
On Fri, Sep 2, 2016 at 10:59 AM, Peter Wicks (pwicks) <pw...@micron.com>> wrote:
Bryan,
We’ve fixed our certs, with no change to the outcome.
We have username/password authentication enabled, via Kerberos, are there issues having Kerberos enabled (username/password) and trying to do site-to-site? When I try to initiate site-to-site with an instance of NiFI configured for Kerberos all requests come through to the server as anonymous because no challenge appears to be sent. We’ve debugged the code and even deep down in NiFiUserUtils.getNiFiUser the request is already marked as anonymous by the Spring framework. It appears to me that the client has a cert, and is waiting for a challenge(?) from the server, and the server is configured for Kerberos and it’s waiting for a ‘bearer’ token…
We’ve debugged both client and server, the client sends the request and gets back a 401 (Unauthorized). SSL verifies good.
Server doesn’t appear to get any authorization information of any kind.
Looking for further guidance/next steps.
Thanks,
Peter
From: Bryan Bende [mailto:bbende@gmail.com<ma...@gmail.com>]
Sent: Thursday, September 01, 2016 9:44 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: I need help configuring Site-to-Site in Secure Mode.
Peter,
Yes, by no means am I saying everyone should use the TLS toolkit. I was just using that because many people are not familiar with how to create certificates, and for people trying to follow a tutorial it is the easiest option.
In your case you definitely want to be using your CA. What you described about not having a cert for client authentication definitely sounds like it would be a problem. Let us know if everything works out after getting the new certs.
-Bryan
On Thu, Sep 1, 2016 at 11:34 AM, Peter Wicks (pwicks) <pw...@micron.com>> wrote:
Bryan,
Paul and I have been working on this, and I think our issue is related to certificates.
In your blog posting you used TLS-Toolkit in your example, but I think that is unrealistic for many environments. For example, this also creates the certificates for SSL right? But these will be self-signed and thus untrusted by default in web browsers. In our environment we generated SSL certificates from our CA and loaded them into the KeyStore. We then extracted public keys for the SSL certs and put them in each of the Trust Stores. This I think is where our main problem is…
I’m making a few assumptions here, so feel free to correct me, but my understanding is that when you use TLS-Toolkit it either creates multiple certs (SSL & Client Auth), or it creates a cert that you are allowed to use for both activities. In our case we ONLY have SSL certs, and the certs are marked such that they aren’t allowed to be used for Client Authentication. I believe this is the reason why our requests are showing up as ‘Anonymous’, because there are no Client Authentication certificates in the KeyStore, just SSL certs.
I’ve asked our security team for Client Authentication certs for each server, since it would be our preference to use our CA rather than having TLS-Toolkit be its own CA.
Thoughts?
Thanks,
Peter
From: Bryan Bende [mailto:bbende@gmail.com<ma...@gmail.com>]
Sent: Thursday, September 01, 2016 9:26 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: I need help configuring Site-to-Site in Secure Mode.
Paul,
Clustering is not a requirement for site-to-site... This sounds strange since "anonymous" is used to represent a user when NiFi is not secured.
Can you double-check all your configs and make sure you have the following properties set...
nifi.remote.input.secure=true
nifi.web.https.host=
nifi.web.https.port=
nifi.security.keystore=
nifi.security.keystoreType=
nifi.security.keystorePasswd=
nifi.security.keyPasswd=
nifi.security.truststore=
nifi.security.truststoreType=
nifi.security.truststorePasswd=<your password>
After your question the other day I went through the steps of setting secure site-to-site to make sure I knew what I was talking about :)
I wrote up the steps here: http://bryanbende.com/development/2016/08/30/apache-nifi-1.0.0-secure-site-to-site
Thanks,
Bryan
On Thu, Sep 1, 2016 at 10:44 AM, Paul Gibeault (pagibeault) <pa...@micron.com>> wrote:
Bryan,
Thanks for the reply. After increasing the log level for Authentication I saw the target NiFi instance used the account “anonymous” for the Site-to-Site connection. After creating a policy for “anonymous”, I was able to view the remote ports and connect to them.
Obviously this is not ideal. We would prefer to make policies for remote hosts/users rather than anonymous.
We are using the same SSL Certificate for both Key Store and Trust Store on a NiFi Instance. This is likely the cause of the “anonymous” user as it doesn’t have a DN. We are working to correct this.
However, after getting my work flow set up across NiFi instances I see this error:
Unable to refresh Remote Group's peers due to Unable to communicate with remote NiFi cluster in order to determine which nodes exist in the remote cluster
Our NiFi servers are not set up for clustering. Is clustering required to perform Site-to-Site?
Thanks,
Paul Gibeault
From: Bryan Bende [mailto:bbende@gmail.com<ma...@gmail.com>]
Sent: Tuesday, August 30, 2016 5:09 PM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: I need help configuring Site-to-Site in Secure Mode.
Paul,
It sounds like you probably have the certificates/truststores setup correctly and just need to create the appropriate policies...
Lets say you have nifi-1 with an Remote Process Group pointing at the URL of nifi-2, and nifi-2 has an Input port to receive data.
In nifi-2 there needs to be a user for the certificate of nifi-1, and then in the global policies of nifi-2 (top right menu) there needs to be a policy for "retrieve site-to-site details" with the nifi-1 user added to the policy. I think this is what is causing the error message you are seeing since nifi-1 is not authorized to query nifi-2 for site-to-site information (available ports, etc).
I believe you also need to create a policy on the Input Port on nifi-2... select the input port and use the lock icon in the left palette and choose "receive data over site-to-site" and add the user of nifi-1. This gives nifi-1 access to the specific port.
Let us know if that works. If so we should definitely look at updating some of the documentation to explain this.
Thanks,
Bryan
On Tue, Aug 30, 2016 at 6:28 PM, Paul Gibeault (pagibeault) <pa...@micron.com>> wrote:
Hello,
We have been attempting to set up Site-to-Site for NiFi in secure mode and have not been successful.
When I create a Remote Process Group, and enter the URL* https://servername:8443/nifi I receive an error icon. The hover status is “Unauthorized”
* - servername is the actual hostname running NiFi
Things I have tried without success:
- Closely followed the instructions in the NiFi System Administrator’s Guide<https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html>
- Enabled SSL Security and Kerberos User Authentication (no self-signed certs)
- Imported public keys of remote NiFi servers into the local keystores for each instance
- Created policies on each instance to allow for full permissions to the accounts in use.
- Tried various combinations of Linux & Windows instances of NiFi.
- Connected a Site-to-Site process group to itself
- Used both 1.0.0-BETA and 1.1.0-SNAPSHOT NiFi versions
There are no warnings or errors in the log files when I attempt to connect a NiFi instance running on Linux to another instance on Linux. However, I did see something when attempting to connect a NiFi instance running on Windows to an instance running on Linux
From log of NiFi on Windows:
2016-08-30 16:11:21,173 ERROR [Remote Process Group dd2e3ac9-0156-1000-4543-5ba3d10c6130: https://servername:8443/nifi Thread-1] o.a.n.remote.StandardRemoteProcessGroup org.apache.nifi.remote.StandardRemoteProcessGroup$InitializationTask@19bde1f3<ma...@19bde1f3> Failed to request account: got unexpected response code of 404:Not Found
From log of NiFi on Linux:
nifi-user.log:2016-08-30 16:39:53,973 INFO [NiFi Web Server-465] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (CN=pagibeault) GET https://servername:8443/nifi-api/site-to-site (source ip: 137.201.48.150)
nifi-user.log:2016-08-30 16:39:53,973 INFO [NiFi Web Server-465] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for CN=pagibeault
nifi-user.log:2016-08-30 16:39:53,974 INFO [NiFi Web Server-465] o.a.n.w.a.c.AccessDeniedExceptionMapper CN=pagibeault does not have permission to access the requested resource. Returning Forbidden response.
Any guidance would be grand.
Thanks,
[http://collab.micron.com/corp/brand/SiteAssets/Micron.png]<http://www.micron.com/>
Paul Gibeault
Sr. Software Engineer, Big Data
Enterprise Analytics & Data
Micron Technology, Inc.
Office (208) 363-3238<tel:%28208%29%20363-3238>
pagibeault@micron.com<ma...@micron.com>
[http://collab.micron.com/corp/brand/SiteAssets/LinkedIn.png] linkedin.com/in/paulgibeault<https://www.linkedin.com/in/linkedin.com/in/paulgibeault>
Re: I need help configuring Site-to-Site in Secure Mode.
Posted by Matthew Clarke <ma...@gmail.com>.
Do the certs you created/obtained support being used for both client and
server auth. If they were created for server auth only, this could explain
your issue. NiFi instances need to act as a client and a server at times.
Thanks,
Matt
On Fri, Sep 2, 2016 at 10:59 AM, Peter Wicks (pwicks) <pw...@micron.com>
wrote:
> Bryan,
>
>
>
> We’ve fixed our certs, with no change to the outcome.
>
>
>
> We have username/password authentication enabled, via Kerberos, are there
> issues having Kerberos enabled (username/password) and trying to do
> site-to-site? When I try to initiate site-to-site with an instance of NiFI
> configured for Kerberos all requests come through to the server as
> anonymous because no challenge appears to be sent. We’ve debugged the code
> and even deep down in NiFiUserUtils.getNiFiUser the request is already
> marked as anonymous by the Spring framework. It appears to me that the
> client has a cert, and is waiting for a challenge(?) from the server, and
> the server is configured for Kerberos and it’s waiting for a ‘bearer’ token…
>
>
>
> We’ve debugged both client and server, the client sends the request and
> gets back a 401 (Unauthorized). SSL verifies good.
>
> Server doesn’t appear to get any authorization information of any kind.
>
>
>
> Looking for further guidance/next steps.
>
>
>
> Thanks,
>
> Peter
>
>
>
> *From:* Bryan Bende [mailto:bbende@gmail.com]
> *Sent:* Thursday, September 01, 2016 9:44 AM
>
> *To:* users@nifi.apache.org
> *Subject:* Re: I need help configuring Site-to-Site in Secure Mode.
>
>
>
> Peter,
>
>
>
> Yes, by no means am I saying everyone should use the TLS toolkit. I was
> just using that because many people are not familiar with how to create
> certificates, and for people trying to follow a tutorial it is the easiest
> option.
>
>
>
> In your case you definitely want to be using your CA. What you described
> about not having a cert for client authentication definitely sounds like it
> would be a problem. Let us know if everything works out after getting the
> new certs.
>
>
>
> -Bryan
>
>
>
>
>
> On Thu, Sep 1, 2016 at 11:34 AM, Peter Wicks (pwicks) <pw...@micron.com>
> wrote:
>
> Bryan,
>
>
>
> Paul and I have been working on this, and I think our issue is related to
> certificates.
>
>
>
> In your blog posting you used TLS-Toolkit in your example, but I think
> that is unrealistic for many environments. For example, this also creates
> the certificates for SSL right? But these will be self-signed and thus
> untrusted by default in web browsers. In our environment we generated SSL
> certificates from our CA and loaded them into the KeyStore. We then
> extracted public keys for the SSL certs and put them in each of the Trust
> Stores. This I think is where our main problem is…
>
>
>
> I’m making a few assumptions here, so feel free to correct me, but my
> understanding is that when you use TLS-Toolkit it either creates multiple
> certs (SSL & Client Auth), or it creates a cert that you are allowed to use
> for both activities. In our case we ONLY have SSL certs, and the certs are
> marked such that they aren’t allowed to be used for Client Authentication.
> I believe this is the reason why our requests are showing up as
> ‘Anonymous’, because there are no Client Authentication certificates in the
> KeyStore, just SSL certs.
>
>
>
> I’ve asked our security team for Client Authentication certs for each
> server, since it would be our preference to use our CA rather than having
> TLS-Toolkit be its own CA.
>
>
>
> Thoughts?
>
>
>
> Thanks,
>
> Peter
>
>
>
> *From:* Bryan Bende [mailto:bbende@gmail.com]
> *Sent:* Thursday, September 01, 2016 9:26 AM
>
>
> *To:* users@nifi.apache.org
> *Subject:* Re: I need help configuring Site-to-Site in Secure Mode.
>
>
>
> Paul,
>
>
>
> Clustering is not a requirement for site-to-site... This sounds strange
> since "anonymous" is used to represent a user when NiFi is not secured.
>
>
>
> Can you double-check all your configs and make sure you have the following
> properties set...
>
>
>
> nifi.remote.input.secure=true
>
> nifi.web.https.host=
> nifi.web.https.port=
>
> nifi.security.keystore=
> nifi.security.keystoreType=
> nifi.security.keystorePasswd=
> nifi.security.keyPasswd=
> nifi.security.truststore=
> nifi.security.truststoreType=
> nifi.security.truststorePasswd=<your password>
>
>
>
> After your question the other day I went through the steps of setting
> secure site-to-site to make sure I knew what I was talking about :)
>
>
>
> I wrote up the steps here: http://bryanbende.com/
> development/2016/08/30/apache-nifi-1.0.0-secure-site-to-site
>
>
>
> Thanks,
>
>
>
> Bryan
>
>
>
> On Thu, Sep 1, 2016 at 10:44 AM, Paul Gibeault (pagibeault) <
> pagibeault@micron.com> wrote:
>
> Bryan,
>
>
>
> Thanks for the reply. After increasing the log level for Authentication I
> saw the target NiFi instance used the account “anonymous” for the
> Site-to-Site connection. After creating a policy for “anonymous”, I was
> able to view the remote ports and connect to them.
>
>
>
> Obviously this is not ideal. We would prefer to make policies for remote
> hosts/users rather than anonymous.
>
>
>
> We are using the same SSL Certificate for both Key Store and Trust Store
> on a NiFi Instance. This is likely the cause of the “anonymous” user as it
> doesn’t have a DN. We are working to correct this.
>
>
>
> However, after getting my work flow set up across NiFi instances I see
> this error:
>
>
>
> Unable to refresh Remote Group's peers due to Unable to communicate with
> remote NiFi cluster in order to determine which nodes exist in the remote
> cluster
>
>
>
> Our NiFi servers are not set up for clustering. Is clustering required to
> perform Site-to-Site?
>
>
>
> Thanks,
>
> Paul Gibeault
>
>
>
> *From:* Bryan Bende [mailto:bbende@gmail.com]
> *Sent:* Tuesday, August 30, 2016 5:09 PM
> *To:* users@nifi.apache.org
> *Subject:* Re: I need help configuring Site-to-Site in Secure Mode.
>
>
>
> Paul,
>
>
>
> It sounds like you probably have the certificates/truststores setup
> correctly and just need to create the appropriate policies...
>
>
>
> Lets say you have nifi-1 with an Remote Process Group pointing at the URL
> of nifi-2, and nifi-2 has an Input port to receive data.
>
>
>
> In nifi-2 there needs to be a user for the certificate of nifi-1, and then
> in the global policies of nifi-2 (top right menu) there needs to be a
> policy for "retrieve site-to-site details" with the nifi-1 user added to
> the policy. I think this is what is causing the error message you are
> seeing since nifi-1 is not authorized to query nifi-2 for site-to-site
> information (available ports, etc).
>
>
>
> I believe you also need to create a policy on the Input Port on nifi-2...
> select the input port and use the lock icon in the left palette and choose
> "receive data over site-to-site" and add the user of nifi-1. This gives
> nifi-1 access to the specific port.
>
>
>
> Let us know if that works. If so we should definitely look at updating
> some of the documentation to explain this.
>
>
>
> Thanks,
>
>
>
> Bryan
>
>
>
>
>
> On Tue, Aug 30, 2016 at 6:28 PM, Paul Gibeault (pagibeault) <
> pagibeault@micron.com> wrote:
>
> Hello,
>
>
>
> We have been attempting to set up Site-to-Site for NiFi in secure mode and
> have not been successful.
>
>
>
> When I create a Remote Process Group, and enter the URL*
> https://servername:8443/nifi I receive an error icon. The hover status
> is “Unauthorized”
>
> ** - servername is the actual hostname running NiFi*
>
>
>
> Things I have tried without success:
>
> - Closely followed the instructions in the NiFi System Administrator’s
> Guide
> <https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html>
>
> - Enabled SSL Security and Kerberos User Authentication (no self-signed
> certs)
>
> - Imported public keys of remote NiFi servers into the local keystores for
> each instance
>
> - Created policies on each instance to allow for full permissions to the
> accounts in use.
>
> - Tried various combinations of Linux & Windows instances of NiFi.
>
> - Connected a Site-to-Site process group to itself
>
> - Used both 1.0.0-BETA and 1.1.0-SNAPSHOT NiFi versions
>
>
>
> There are no warnings or errors in the log files when I attempt to connect
> a NiFi instance running on Linux to another instance on Linux. However, I
> did see something when attempting to connect a NiFi instance running on
> Windows to an instance running on Linux
>
> From log of NiFi on Windows:
>
> 2016-08-30 16:11:21,173 ERROR [Remote Process Group
> dd2e3ac9-0156-1000-4543-5ba3d10c6130: https://servername:8443/nifi
> Thread-1] o.a.n.remote.StandardRemoteProcessGroup org.apache.nifi.remote.
> StandardRemoteProcessGroup$InitializationTask@19bde1f3 Failed to request
> account: got unexpected response code of 404:Not Found
>
>
>
> From log of NiFi on Linux:
>
> nifi-user.log:2016-08-30 16:39:53,973 INFO [NiFi Web Server-465] o.a.n.w.s.NiFiAuthenticationFilter
> Attempting request for (CN=pagibeault) GET https://servername:8443/nifi-
> api/site-to-site (source ip: 137.201.48.150)
>
> nifi-user.log:2016-08-30 16:39:53,973 INFO [NiFi Web Server-465] o.a.n.w.s.NiFiAuthenticationFilter
> Authentication success for CN=pagibeault
>
> nifi-user.log:2016-08-30 16:39:53,974 INFO [NiFi Web Server-465]
> o.a.n.w.a.c.AccessDeniedExceptionMapper CN=pagibeault does not have
> permission to access the requested resource. Returning Forbidden response.
>
>
>
> Any guidance would be grand.
>
>
>
> Thanks,
>
>
>
> [image: http://collab.micron.com/corp/brand/SiteAssets/Micron.png]
> <http://www.micron.com/>
>
> *Paul Gibeault*
> Sr. Software Engineer, Big Data
> Enterprise Analytics & Data
> Micron Technology, Inc.
>
> *Office* (208) 363-3238 <%28208%29%20363-3238>
>
>
> pagibeault@micron.com
> [image: http://collab.micron.com/corp/brand/SiteAssets/LinkedIn.png]
> linkedin.com/in/paulgibeault
> <https://www.linkedin.com/in/linkedin.com/in/paulgibeault>
>
>
>
>
>
>
>
>
>
>
>
RE: I need help configuring Site-to-Site in Secure Mode.
Posted by "Peter Wicks (pwicks)" <pw...@micron.com>.
Bryan,
We’ve fixed our certs, with no change to the outcome.
We have username/password authentication enabled, via Kerberos, are there issues having Kerberos enabled (username/password) and trying to do site-to-site? When I try to initiate site-to-site with an instance of NiFI configured for Kerberos all requests come through to the server as anonymous because no challenge appears to be sent. We’ve debugged the code and even deep down in NiFiUserUtils.getNiFiUser the request is already marked as anonymous by the Spring framework. It appears to me that the client has a cert, and is waiting for a challenge(?) from the server, and the server is configured for Kerberos and it’s waiting for a ‘bearer’ token…
We’ve debugged both client and server, the client sends the request and gets back a 401 (Unauthorized). SSL verifies good.
Server doesn’t appear to get any authorization information of any kind.
Looking for further guidance/next steps.
Thanks,
Peter
From: Bryan Bende [mailto:bbende@gmail.com]
Sent: Thursday, September 01, 2016 9:44 AM
To: users@nifi.apache.org
Subject: Re: I need help configuring Site-to-Site in Secure Mode.
Peter,
Yes, by no means am I saying everyone should use the TLS toolkit. I was just using that because many people are not familiar with how to create certificates, and for people trying to follow a tutorial it is the easiest option.
In your case you definitely want to be using your CA. What you described about not having a cert for client authentication definitely sounds like it would be a problem. Let us know if everything works out after getting the new certs.
-Bryan
On Thu, Sep 1, 2016 at 11:34 AM, Peter Wicks (pwicks) <pw...@micron.com>> wrote:
Bryan,
Paul and I have been working on this, and I think our issue is related to certificates.
In your blog posting you used TLS-Toolkit in your example, but I think that is unrealistic for many environments. For example, this also creates the certificates for SSL right? But these will be self-signed and thus untrusted by default in web browsers. In our environment we generated SSL certificates from our CA and loaded them into the KeyStore. We then extracted public keys for the SSL certs and put them in each of the Trust Stores. This I think is where our main problem is…
I’m making a few assumptions here, so feel free to correct me, but my understanding is that when you use TLS-Toolkit it either creates multiple certs (SSL & Client Auth), or it creates a cert that you are allowed to use for both activities. In our case we ONLY have SSL certs, and the certs are marked such that they aren’t allowed to be used for Client Authentication. I believe this is the reason why our requests are showing up as ‘Anonymous’, because there are no Client Authentication certificates in the KeyStore, just SSL certs.
I’ve asked our security team for Client Authentication certs for each server, since it would be our preference to use our CA rather than having TLS-Toolkit be its own CA.
Thoughts?
Thanks,
Peter
From: Bryan Bende [mailto:bbende@gmail.com<ma...@gmail.com>]
Sent: Thursday, September 01, 2016 9:26 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: I need help configuring Site-to-Site in Secure Mode.
Paul,
Clustering is not a requirement for site-to-site... This sounds strange since "anonymous" is used to represent a user when NiFi is not secured.
Can you double-check all your configs and make sure you have the following properties set...
nifi.remote.input.secure=true
nifi.web.https.host=
nifi.web.https.port=
nifi.security.keystore=
nifi.security.keystoreType=
nifi.security.keystorePasswd=
nifi.security.keyPasswd=
nifi.security.truststore=
nifi.security.truststoreType=
nifi.security.truststorePasswd=<your password>
After your question the other day I went through the steps of setting secure site-to-site to make sure I knew what I was talking about :)
I wrote up the steps here: http://bryanbende.com/development/2016/08/30/apache-nifi-1.0.0-secure-site-to-site
Thanks,
Bryan
On Thu, Sep 1, 2016 at 10:44 AM, Paul Gibeault (pagibeault) <pa...@micron.com>> wrote:
Bryan,
Thanks for the reply. After increasing the log level for Authentication I saw the target NiFi instance used the account “anonymous” for the Site-to-Site connection. After creating a policy for “anonymous”, I was able to view the remote ports and connect to them.
Obviously this is not ideal. We would prefer to make policies for remote hosts/users rather than anonymous.
We are using the same SSL Certificate for both Key Store and Trust Store on a NiFi Instance. This is likely the cause of the “anonymous” user as it doesn’t have a DN. We are working to correct this.
However, after getting my work flow set up across NiFi instances I see this error:
Unable to refresh Remote Group's peers due to Unable to communicate with remote NiFi cluster in order to determine which nodes exist in the remote cluster
Our NiFi servers are not set up for clustering. Is clustering required to perform Site-to-Site?
Thanks,
Paul Gibeault
From: Bryan Bende [mailto:bbende@gmail.com<ma...@gmail.com>]
Sent: Tuesday, August 30, 2016 5:09 PM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: I need help configuring Site-to-Site in Secure Mode.
Paul,
It sounds like you probably have the certificates/truststores setup correctly and just need to create the appropriate policies...
Lets say you have nifi-1 with an Remote Process Group pointing at the URL of nifi-2, and nifi-2 has an Input port to receive data.
In nifi-2 there needs to be a user for the certificate of nifi-1, and then in the global policies of nifi-2 (top right menu) there needs to be a policy for "retrieve site-to-site details" with the nifi-1 user added to the policy. I think this is what is causing the error message you are seeing since nifi-1 is not authorized to query nifi-2 for site-to-site information (available ports, etc).
I believe you also need to create a policy on the Input Port on nifi-2... select the input port and use the lock icon in the left palette and choose "receive data over site-to-site" and add the user of nifi-1. This gives nifi-1 access to the specific port.
Let us know if that works. If so we should definitely look at updating some of the documentation to explain this.
Thanks,
Bryan
On Tue, Aug 30, 2016 at 6:28 PM, Paul Gibeault (pagibeault) <pa...@micron.com>> wrote:
Hello,
We have been attempting to set up Site-to-Site for NiFi in secure mode and have not been successful.
When I create a Remote Process Group, and enter the URL* https://servername:8443/nifi I receive an error icon. The hover status is “Unauthorized”
* - servername is the actual hostname running NiFi
Things I have tried without success:
- Closely followed the instructions in the NiFi System Administrator’s Guide<https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html>
- Enabled SSL Security and Kerberos User Authentication (no self-signed certs)
- Imported public keys of remote NiFi servers into the local keystores for each instance
- Created policies on each instance to allow for full permissions to the accounts in use.
- Tried various combinations of Linux & Windows instances of NiFi.
- Connected a Site-to-Site process group to itself
- Used both 1.0.0-BETA and 1.1.0-SNAPSHOT NiFi versions
There are no warnings or errors in the log files when I attempt to connect a NiFi instance running on Linux to another instance on Linux. However, I did see something when attempting to connect a NiFi instance running on Windows to an instance running on Linux
From log of NiFi on Windows:
2016-08-30 16:11:21,173 ERROR [Remote Process Group dd2e3ac9-0156-1000-4543-5ba3d10c6130: https://servername:8443/nifi Thread-1] o.a.n.remote.StandardRemoteProcessGroup org.apache.nifi.remote.StandardRemoteProcessGroup$InitializationTask@19bde1f3<ma...@19bde1f3> Failed to request account: got unexpected response code of 404:Not Found
From log of NiFi on Linux:
nifi-user.log:2016-08-30 16:39:53,973 INFO [NiFi Web Server-465] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (CN=pagibeault) GET https://servername:8443/nifi-api/site-to-site (source ip: 137.201.48.150)
nifi-user.log:2016-08-30 16:39:53,973 INFO [NiFi Web Server-465] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for CN=pagibeault
nifi-user.log:2016-08-30 16:39:53,974 INFO [NiFi Web Server-465] o.a.n.w.a.c.AccessDeniedExceptionMapper CN=pagibeault does not have permission to access the requested resource. Returning Forbidden response.
Any guidance would be grand.
Thanks,
[http://collab.micron.com/corp/brand/SiteAssets/Micron.png]<http://www.micron.com/>
Paul Gibeault
Sr. Software Engineer, Big Data
Enterprise Analytics & Data
Micron Technology, Inc.
Office (208) 363-3238<tel:%28208%29%20363-3238>
pagibeault@micron.com<ma...@micron.com>
[http://collab.micron.com/corp/brand/SiteAssets/LinkedIn.png] linkedin.com/in/paulgibeault<https://www.linkedin.com/in/linkedin.com/in/paulgibeault>
Re: I need help configuring Site-to-Site in Secure Mode.
Posted by Bryan Bende <bb...@gmail.com>.
Peter,
Yes, by no means am I saying everyone should use the TLS toolkit. I was
just using that because many people are not familiar with how to create
certificates, and for people trying to follow a tutorial it is the easiest
option.
In your case you definitely want to be using your CA. What you described
about not having a cert for client authentication definitely sounds like it
would be a problem. Let us know if everything works out after getting the
new certs.
-Bryan
On Thu, Sep 1, 2016 at 11:34 AM, Peter Wicks (pwicks) <pw...@micron.com>
wrote:
> Bryan,
>
>
>
> Paul and I have been working on this, and I think our issue is related to
> certificates.
>
>
>
> In your blog posting you used TLS-Toolkit in your example, but I think
> that is unrealistic for many environments. For example, this also creates
> the certificates for SSL right? But these will be self-signed and thus
> untrusted by default in web browsers. In our environment we generated SSL
> certificates from our CA and loaded them into the KeyStore. We then
> extracted public keys for the SSL certs and put them in each of the Trust
> Stores. This I think is where our main problem is…
>
>
>
> I’m making a few assumptions here, so feel free to correct me, but my
> understanding is that when you use TLS-Toolkit it either creates multiple
> certs (SSL & Client Auth), or it creates a cert that you are allowed to use
> for both activities. In our case we ONLY have SSL certs, and the certs are
> marked such that they aren’t allowed to be used for Client Authentication.
> I believe this is the reason why our requests are showing up as
> ‘Anonymous’, because there are no Client Authentication certificates in the
> KeyStore, just SSL certs.
>
>
>
> I’ve asked our security team for Client Authentication certs for each
> server, since it would be our preference to use our CA rather than having
> TLS-Toolkit be its own CA.
>
>
>
> Thoughts?
>
>
>
> Thanks,
>
> Peter
>
>
>
> *From:* Bryan Bende [mailto:bbende@gmail.com]
> *Sent:* Thursday, September 01, 2016 9:26 AM
>
> *To:* users@nifi.apache.org
> *Subject:* Re: I need help configuring Site-to-Site in Secure Mode.
>
>
>
> Paul,
>
>
>
> Clustering is not a requirement for site-to-site... This sounds strange
> since "anonymous" is used to represent a user when NiFi is not secured.
>
>
>
> Can you double-check all your configs and make sure you have the following
> properties set...
>
>
>
> nifi.remote.input.secure=true
>
> nifi.web.https.host=
> nifi.web.https.port=
>
> nifi.security.keystore=
> nifi.security.keystoreType=
> nifi.security.keystorePasswd=
> nifi.security.keyPasswd=
> nifi.security.truststore=
> nifi.security.truststoreType=
> nifi.security.truststorePasswd=<your password>
>
>
>
> After your question the other day I went through the steps of setting
> secure site-to-site to make sure I knew what I was talking about :)
>
>
>
> I wrote up the steps here: http://bryanbende.com/
> development/2016/08/30/apache-nifi-1.0.0-secure-site-to-site
>
>
>
> Thanks,
>
>
>
> Bryan
>
>
>
> On Thu, Sep 1, 2016 at 10:44 AM, Paul Gibeault (pagibeault) <
> pagibeault@micron.com> wrote:
>
> Bryan,
>
>
>
> Thanks for the reply. After increasing the log level for Authentication I
> saw the target NiFi instance used the account “anonymous” for the
> Site-to-Site connection. After creating a policy for “anonymous”, I was
> able to view the remote ports and connect to them.
>
>
>
> Obviously this is not ideal. We would prefer to make policies for remote
> hosts/users rather than anonymous.
>
>
>
> We are using the same SSL Certificate for both Key Store and Trust Store
> on a NiFi Instance. This is likely the cause of the “anonymous” user as it
> doesn’t have a DN. We are working to correct this.
>
>
>
> However, after getting my work flow set up across NiFi instances I see
> this error:
>
>
>
> Unable to refresh Remote Group's peers due to Unable to communicate with
> remote NiFi cluster in order to determine which nodes exist in the remote
> cluster
>
>
>
> Our NiFi servers are not set up for clustering. Is clustering required to
> perform Site-to-Site?
>
>
>
> Thanks,
>
> Paul Gibeault
>
>
>
> *From:* Bryan Bende [mailto:bbende@gmail.com]
> *Sent:* Tuesday, August 30, 2016 5:09 PM
> *To:* users@nifi.apache.org
> *Subject:* Re: I need help configuring Site-to-Site in Secure Mode.
>
>
>
> Paul,
>
>
>
> It sounds like you probably have the certificates/truststores setup
> correctly and just need to create the appropriate policies...
>
>
>
> Lets say you have nifi-1 with an Remote Process Group pointing at the URL
> of nifi-2, and nifi-2 has an Input port to receive data.
>
>
>
> In nifi-2 there needs to be a user for the certificate of nifi-1, and then
> in the global policies of nifi-2 (top right menu) there needs to be a
> policy for "retrieve site-to-site details" with the nifi-1 user added to
> the policy. I think this is what is causing the error message you are
> seeing since nifi-1 is not authorized to query nifi-2 for site-to-site
> information (available ports, etc).
>
>
>
> I believe you also need to create a policy on the Input Port on nifi-2...
> select the input port and use the lock icon in the left palette and choose
> "receive data over site-to-site" and add the user of nifi-1. This gives
> nifi-1 access to the specific port.
>
>
>
> Let us know if that works. If so we should definitely look at updating
> some of the documentation to explain this.
>
>
>
> Thanks,
>
>
>
> Bryan
>
>
>
>
>
> On Tue, Aug 30, 2016 at 6:28 PM, Paul Gibeault (pagibeault) <
> pagibeault@micron.com> wrote:
>
> Hello,
>
>
>
> We have been attempting to set up Site-to-Site for NiFi in secure mode and
> have not been successful.
>
>
>
> When I create a Remote Process Group, and enter the URL*
> https://servername:8443/nifi I receive an error icon. The hover status
> is “Unauthorized”
>
> ** - servername is the actual hostname running NiFi*
>
>
>
> Things I have tried without success:
>
> - Closely followed the instructions in the NiFi System Administrator’s
> Guide
> <https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html>
>
> - Enabled SSL Security and Kerberos User Authentication (no self-signed
> certs)
>
> - Imported public keys of remote NiFi servers into the local keystores for
> each instance
>
> - Created policies on each instance to allow for full permissions to the
> accounts in use.
>
> - Tried various combinations of Linux & Windows instances of NiFi.
>
> - Connected a Site-to-Site process group to itself
>
> - Used both 1.0.0-BETA and 1.1.0-SNAPSHOT NiFi versions
>
>
>
> There are no warnings or errors in the log files when I attempt to connect
> a NiFi instance running on Linux to another instance on Linux. However, I
> did see something when attempting to connect a NiFi instance running on
> Windows to an instance running on Linux
>
> From log of NiFi on Windows:
>
> 2016-08-30 16:11:21,173 ERROR [Remote Process Group
> dd2e3ac9-0156-1000-4543-5ba3d10c6130: https://servername:8443/nifi
> Thread-1] o.a.n.remote.StandardRemoteProcessGroup org.apache.nifi.remote.
> StandardRemoteProcessGroup$InitializationTask@19bde1f3 Failed to request
> account: got unexpected response code of 404:Not Found
>
>
>
> From log of NiFi on Linux:
>
> nifi-user.log:2016-08-30 16:39:53,973 INFO [NiFi Web Server-465] o.a.n.w.s.NiFiAuthenticationFilter
> Attempting request for (CN=pagibeault) GET https://servername:8443/nifi-
> api/site-to-site (source ip: 137.201.48.150)
>
> nifi-user.log:2016-08-30 16:39:53,973 INFO [NiFi Web Server-465] o.a.n.w.s.NiFiAuthenticationFilter
> Authentication success for CN=pagibeault
>
> nifi-user.log:2016-08-30 16:39:53,974 INFO [NiFi Web Server-465]
> o.a.n.w.a.c.AccessDeniedExceptionMapper CN=pagibeault does not have
> permission to access the requested resource. Returning Forbidden response.
>
>
>
> Any guidance would be grand.
>
>
>
> Thanks,
>
>
>
> [image: http://collab.micron.com/corp/brand/SiteAssets/Micron.png]
> <http://www.micron.com/>
>
> *Paul Gibeault*
> Sr. Software Engineer, Big Data
> Enterprise Analytics & Data
> Micron Technology, Inc.
>
> *Office* (208) 363-3238 <%28208%29%20363-3238>
>
>
> pagibeault@micron.com
> [image: http://collab.micron.com/corp/brand/SiteAssets/LinkedIn.png]
> linkedin.com/in/paulgibeault
> <https://www.linkedin.com/in/linkedin.com/in/paulgibeault>
>
>
>
>
>
>
>
>
>
RE: I need help configuring Site-to-Site in Secure Mode.
Posted by "Peter Wicks (pwicks)" <pw...@micron.com>.
Bryan,
Paul and I have been working on this, and I think our issue is related to certificates.
In your blog posting you used TLS-Toolkit in your example, but I think that is unrealistic for many environments. For example, this also creates the certificates for SSL right? But these will be self-signed and thus untrusted by default in web browsers. In our environment we generated SSL certificates from our CA and loaded them into the KeyStore. We then extracted public keys for the SSL certs and put them in each of the Trust Stores. This I think is where our main problem is…
I’m making a few assumptions here, so feel free to correct me, but my understanding is that when you use TLS-Toolkit it either creates multiple certs (SSL & Client Auth), or it creates a cert that you are allowed to use for both activities. In our case we ONLY have SSL certs, and the certs are marked such that they aren’t allowed to be used for Client Authentication. I believe this is the reason why our requests are showing up as ‘Anonymous’, because there are no Client Authentication certificates in the KeyStore, just SSL certs.
I’ve asked our security team for Client Authentication certs for each server, since it would be our preference to use our CA rather than having TLS-Toolkit be its own CA.
Thoughts?
Thanks,
Peter
From: Bryan Bende [mailto:bbende@gmail.com]
Sent: Thursday, September 01, 2016 9:26 AM
To: users@nifi.apache.org
Subject: Re: I need help configuring Site-to-Site in Secure Mode.
Paul,
Clustering is not a requirement for site-to-site... This sounds strange since "anonymous" is used to represent a user when NiFi is not secured.
Can you double-check all your configs and make sure you have the following properties set...
nifi.remote.input.secure=true
nifi.web.https.host=
nifi.web.https.port=
nifi.security.keystore=
nifi.security.keystoreType=
nifi.security.keystorePasswd=
nifi.security.keyPasswd=
nifi.security.truststore=
nifi.security.truststoreType=
nifi.security.truststorePasswd=<your password>
After your question the other day I went through the steps of setting secure site-to-site to make sure I knew what I was talking about :)
I wrote up the steps here: http://bryanbende.com/development/2016/08/30/apache-nifi-1.0.0-secure-site-to-site
Thanks,
Bryan
On Thu, Sep 1, 2016 at 10:44 AM, Paul Gibeault (pagibeault) <pa...@micron.com>> wrote:
Bryan,
Thanks for the reply. After increasing the log level for Authentication I saw the target NiFi instance used the account “anonymous” for the Site-to-Site connection. After creating a policy for “anonymous”, I was able to view the remote ports and connect to them.
Obviously this is not ideal. We would prefer to make policies for remote hosts/users rather than anonymous.
We are using the same SSL Certificate for both Key Store and Trust Store on a NiFi Instance. This is likely the cause of the “anonymous” user as it doesn’t have a DN. We are working to correct this.
However, after getting my work flow set up across NiFi instances I see this error:
Unable to refresh Remote Group's peers due to Unable to communicate with remote NiFi cluster in order to determine which nodes exist in the remote cluster
Our NiFi servers are not set up for clustering. Is clustering required to perform Site-to-Site?
Thanks,
Paul Gibeault
From: Bryan Bende [mailto:bbende@gmail.com<ma...@gmail.com>]
Sent: Tuesday, August 30, 2016 5:09 PM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: I need help configuring Site-to-Site in Secure Mode.
Paul,
It sounds like you probably have the certificates/truststores setup correctly and just need to create the appropriate policies...
Lets say you have nifi-1 with an Remote Process Group pointing at the URL of nifi-2, and nifi-2 has an Input port to receive data.
In nifi-2 there needs to be a user for the certificate of nifi-1, and then in the global policies of nifi-2 (top right menu) there needs to be a policy for "retrieve site-to-site details" with the nifi-1 user added to the policy. I think this is what is causing the error message you are seeing since nifi-1 is not authorized to query nifi-2 for site-to-site information (available ports, etc).
I believe you also need to create a policy on the Input Port on nifi-2... select the input port and use the lock icon in the left palette and choose "receive data over site-to-site" and add the user of nifi-1. This gives nifi-1 access to the specific port.
Let us know if that works. If so we should definitely look at updating some of the documentation to explain this.
Thanks,
Bryan
On Tue, Aug 30, 2016 at 6:28 PM, Paul Gibeault (pagibeault) <pa...@micron.com>> wrote:
Hello,
We have been attempting to set up Site-to-Site for NiFi in secure mode and have not been successful.
When I create a Remote Process Group, and enter the URL* https://servername:8443/nifi I receive an error icon. The hover status is “Unauthorized”
* - servername is the actual hostname running NiFi
Things I have tried without success:
- Closely followed the instructions in the NiFi System Administrator’s Guide<https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html>
- Enabled SSL Security and Kerberos User Authentication (no self-signed certs)
- Imported public keys of remote NiFi servers into the local keystores for each instance
- Created policies on each instance to allow for full permissions to the accounts in use.
- Tried various combinations of Linux & Windows instances of NiFi.
- Connected a Site-to-Site process group to itself
- Used both 1.0.0-BETA and 1.1.0-SNAPSHOT NiFi versions
There are no warnings or errors in the log files when I attempt to connect a NiFi instance running on Linux to another instance on Linux. However, I did see something when attempting to connect a NiFi instance running on Windows to an instance running on Linux
From log of NiFi on Windows:
2016-08-30 16:11:21,173 ERROR [Remote Process Group dd2e3ac9-0156-1000-4543-5ba3d10c6130: https://servername:8443/nifi Thread-1] o.a.n.remote.StandardRemoteProcessGroup org.apache.nifi.remote.StandardRemoteProcessGroup$InitializationTask@19bde1f3<ma...@19bde1f3> Failed to request account: got unexpected response code of 404:Not Found
From log of NiFi on Linux:
nifi-user.log:2016-08-30 16:39:53,973 INFO [NiFi Web Server-465] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (CN=pagibeault) GET https://servername:8443/nifi-api/site-to-site (source ip: 137.201.48.150)
nifi-user.log:2016-08-30 16:39:53,973 INFO [NiFi Web Server-465] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for CN=pagibeault
nifi-user.log:2016-08-30 16:39:53,974 INFO [NiFi Web Server-465] o.a.n.w.a.c.AccessDeniedExceptionMapper CN=pagibeault does not have permission to access the requested resource. Returning Forbidden response.
Any guidance would be grand.
Thanks,
[http://collab.micron.com/corp/brand/SiteAssets/Micron.png]<http://www.micron.com/>
Paul Gibeault
Sr. Software Engineer, Big Data
Enterprise Analytics & Data
Micron Technology, Inc.
Office (208) 363-3238<tel:%28208%29%20363-3238>
pagibeault@micron.com<ma...@micron.com>
[http://collab.micron.com/corp/brand/SiteAssets/LinkedIn.png] linkedin.com/in/paulgibeault<https://www.linkedin.com/in/linkedin.com/in/paulgibeault>
Re: I need help configuring Site-to-Site in Secure Mode.
Posted by Bryan Bende <bb...@gmail.com>.
Paul,
Clustering is not a requirement for site-to-site... This sounds strange
since "anonymous" is used to represent a user when NiFi is not secured.
Can you double-check all your configs and make sure you have the following
properties set...
nifi.remote.input.secure=true
nifi.web.https.host=
nifi.web.https.port=
nifi.security.keystore=
nifi.security.keystoreType=
nifi.security.keystorePasswd=
nifi.security.keyPasswd=
nifi.security.truststore=
nifi.security.truststoreType=
nifi.security.truststorePasswd=<your password>
After your question the other day I went through the steps of setting
secure site-to-site to make sure I knew what I was talking about :)
I wrote up the steps here:
http://bryanbende.com/development/2016/08/30/apache-nifi-1.0.0-secure-site-to-site
Thanks,
Bryan
On Thu, Sep 1, 2016 at 10:44 AM, Paul Gibeault (pagibeault) <
pagibeault@micron.com> wrote:
> Bryan,
>
>
>
> Thanks for the reply. After increasing the log level for Authentication I
> saw the target NiFi instance used the account “anonymous” for the
> Site-to-Site connection. After creating a policy for “anonymous”, I was
> able to view the remote ports and connect to them.
>
>
>
> Obviously this is not ideal. We would prefer to make policies for remote
> hosts/users rather than anonymous.
>
>
>
> We are using the same SSL Certificate for both Key Store and Trust Store
> on a NiFi Instance. This is likely the cause of the “anonymous” user as it
> doesn’t have a DN. We are working to correct this.
>
>
>
> However, after getting my work flow set up across NiFi instances I see
> this error:
>
>
>
> Unable to refresh Remote Group's peers due to Unable to communicate with
> remote NiFi cluster in order to determine which nodes exist in the remote
> cluster
>
>
>
> Our NiFi servers are not set up for clustering. Is clustering required to
> perform Site-to-Site?
>
>
>
> Thanks,
>
> Paul Gibeault
>
>
>
> *From:* Bryan Bende [mailto:bbende@gmail.com]
> *Sent:* Tuesday, August 30, 2016 5:09 PM
> *To:* users@nifi.apache.org
> *Subject:* Re: I need help configuring Site-to-Site in Secure Mode.
>
>
>
> Paul,
>
>
>
> It sounds like you probably have the certificates/truststores setup
> correctly and just need to create the appropriate policies...
>
>
>
> Lets say you have nifi-1 with an Remote Process Group pointing at the URL
> of nifi-2, and nifi-2 has an Input port to receive data.
>
>
>
> In nifi-2 there needs to be a user for the certificate of nifi-1, and then
> in the global policies of nifi-2 (top right menu) there needs to be a
> policy for "retrieve site-to-site details" with the nifi-1 user added to
> the policy. I think this is what is causing the error message you are
> seeing since nifi-1 is not authorized to query nifi-2 for site-to-site
> information (available ports, etc).
>
>
>
> I believe you also need to create a policy on the Input Port on nifi-2...
> select the input port and use the lock icon in the left palette and choose
> "receive data over site-to-site" and add the user of nifi-1. This gives
> nifi-1 access to the specific port.
>
>
>
> Let us know if that works. If so we should definitely look at updating
> some of the documentation to explain this.
>
>
>
> Thanks,
>
>
>
> Bryan
>
>
>
>
>
> On Tue, Aug 30, 2016 at 6:28 PM, Paul Gibeault (pagibeault) <
> pagibeault@micron.com> wrote:
>
> Hello,
>
>
>
> We have been attempting to set up Site-to-Site for NiFi in secure mode and
> have not been successful.
>
>
>
> When I create a Remote Process Group, and enter the URL*
> https://servername:8443/nifi I receive an error icon. The hover status
> is “Unauthorized”
>
> ** - servername is the actual hostname running NiFi*
>
>
>
> Things I have tried without success:
>
> - Closely followed the instructions in the NiFi System Administrator’s
> Guide
> <https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html>
>
> - Enabled SSL Security and Kerberos User Authentication (no self-signed
> certs)
>
> - Imported public keys of remote NiFi servers into the local keystores for
> each instance
>
> - Created policies on each instance to allow for full permissions to the
> accounts in use.
>
> - Tried various combinations of Linux & Windows instances of NiFi.
>
> - Connected a Site-to-Site process group to itself
>
> - Used both 1.0.0-BETA and 1.1.0-SNAPSHOT NiFi versions
>
>
>
> There are no warnings or errors in the log files when I attempt to connect
> a NiFi instance running on Linux to another instance on Linux. However, I
> did see something when attempting to connect a NiFi instance running on
> Windows to an instance running on Linux
>
> From log of NiFi on Windows:
>
> 2016-08-30 16:11:21,173 ERROR [Remote Process Group
> dd2e3ac9-0156-1000-4543-5ba3d10c6130: https://servername:8443/nifi
> Thread-1] o.a.n.remote.StandardRemoteProcessGroup org.apache.nifi.remote.
> StandardRemoteProcessGroup$InitializationTask@19bde1f3 Failed to request
> account: got unexpected response code of 404:Not Found
>
>
>
> From log of NiFi on Linux:
>
> nifi-user.log:2016-08-30 16:39:53,973 INFO [NiFi Web Server-465] o.a.n.w.s.NiFiAuthenticationFilter
> Attempting request for (CN=pagibeault) GET https://servername:8443/nifi-
> api/site-to-site (source ip: 137.201.48.150)
>
> nifi-user.log:2016-08-30 16:39:53,973 INFO [NiFi Web Server-465] o.a.n.w.s.NiFiAuthenticationFilter
> Authentication success for CN=pagibeault
>
> nifi-user.log:2016-08-30 16:39:53,974 INFO [NiFi Web Server-465]
> o.a.n.w.a.c.AccessDeniedExceptionMapper CN=pagibeault does not have
> permission to access the requested resource. Returning Forbidden response.
>
>
>
> Any guidance would be grand.
>
>
>
> Thanks,
>
>
>
> [image: http://collab.micron.com/corp/brand/SiteAssets/Micron.png]
> <http://www.micron.com/>
>
> *Paul Gibeault*
> Sr. Software Engineer, Big Data
> Enterprise Analytics & Data
> Micron Technology, Inc.
>
> *Office* (208) 363-3238 <%28208%29%20363-3238>
>
>
> pagibeault@micron.com
> [image: http://collab.micron.com/corp/brand/SiteAssets/LinkedIn.png]
> linkedin.com/in/paulgibeault
> <https://www.linkedin.com/in/linkedin.com/in/paulgibeault>
>
>
>
>
>
>
>
RE: I need help configuring Site-to-Site in Secure Mode.
Posted by "Paul Gibeault (pagibeault)" <pa...@micron.com>.
Bryan,
Thanks for the reply. After increasing the log level for Authentication I saw the target NiFi instance used the account “anonymous” for the Site-to-Site connection. After creating a policy for “anonymous”, I was able to view the remote ports and connect to them.
Obviously this is not ideal. We would prefer to make policies for remote hosts/users rather than anonymous.
We are using the same SSL Certificate for both Key Store and Trust Store on a NiFi Instance. This is likely the cause of the “anonymous” user as it doesn’t have a DN. We are working to correct this.
However, after getting my work flow set up across NiFi instances I see this error:
Unable to refresh Remote Group's peers due to Unable to communicate with remote NiFi cluster in order to determine which nodes exist in the remote cluster
Our NiFi servers are not set up for clustering. Is clustering required to perform Site-to-Site?
Thanks,
Paul Gibeault
From: Bryan Bende [mailto:bbende@gmail.com]
Sent: Tuesday, August 30, 2016 5:09 PM
To: users@nifi.apache.org
Subject: Re: I need help configuring Site-to-Site in Secure Mode.
Paul,
It sounds like you probably have the certificates/truststores setup correctly and just need to create the appropriate policies...
Lets say you have nifi-1 with an Remote Process Group pointing at the URL of nifi-2, and nifi-2 has an Input port to receive data.
In nifi-2 there needs to be a user for the certificate of nifi-1, and then in the global policies of nifi-2 (top right menu) there needs to be a policy for "retrieve site-to-site details" with the nifi-1 user added to the policy. I think this is what is causing the error message you are seeing since nifi-1 is not authorized to query nifi-2 for site-to-site information (available ports, etc).
I believe you also need to create a policy on the Input Port on nifi-2... select the input port and use the lock icon in the left palette and choose "receive data over site-to-site" and add the user of nifi-1. This gives nifi-1 access to the specific port.
Let us know if that works. If so we should definitely look at updating some of the documentation to explain this.
Thanks,
Bryan
On Tue, Aug 30, 2016 at 6:28 PM, Paul Gibeault (pagibeault) <pa...@micron.com>> wrote:
Hello,
We have been attempting to set up Site-to-Site for NiFi in secure mode and have not been successful.
When I create a Remote Process Group, and enter the URL* https://servername:8443/nifi I receive an error icon. The hover status is “Unauthorized”
* - servername is the actual hostname running NiFi
Things I have tried without success:
- Closely followed the instructions in the NiFi System Administrator’s Guide<https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html>
- Enabled SSL Security and Kerberos User Authentication (no self-signed certs)
- Imported public keys of remote NiFi servers into the local keystores for each instance
- Created policies on each instance to allow for full permissions to the accounts in use.
- Tried various combinations of Linux & Windows instances of NiFi.
- Connected a Site-to-Site process group to itself
- Used both 1.0.0-BETA and 1.1.0-SNAPSHOT NiFi versions
There are no warnings or errors in the log files when I attempt to connect a NiFi instance running on Linux to another instance on Linux. However, I did see something when attempting to connect a NiFi instance running on Windows to an instance running on Linux
From log of NiFi on Windows:
2016-08-30 16:11:21,173 ERROR [Remote Process Group dd2e3ac9-0156-1000-4543-5ba3d10c6130: https://servername:8443/nifi Thread-1] o.a.n.remote.StandardRemoteProcessGroup org.apache.nifi.remote.StandardRemoteProcessGroup$InitializationTask@19bde1f3<ma...@19bde1f3> Failed to request account: got unexpected response code of 404:Not Found
From log of NiFi on Linux:
nifi-user.log:2016-08-30 16:39:53,973 INFO [NiFi Web Server-465] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (CN=pagibeault) GET https://servername:8443/nifi-api/site-to-site (source ip: 137.201.48.150)
nifi-user.log:2016-08-30 16:39:53,973 INFO [NiFi Web Server-465] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for CN=pagibeault
nifi-user.log:2016-08-30 16:39:53,974 INFO [NiFi Web Server-465] o.a.n.w.a.c.AccessDeniedExceptionMapper CN=pagibeault does not have permission to access the requested resource. Returning Forbidden response.
Any guidance would be grand.
Thanks,
[http://collab.micron.com/corp/brand/SiteAssets/Micron.png]<http://www.micron.com/>
Paul Gibeault
Sr. Software Engineer, Big Data
Enterprise Analytics & Data
Micron Technology, Inc.
Office (208) 363-3238<tel:%28208%29%20363-3238>
pagibeault@micron.com<ma...@micron.com>
[http://collab.micron.com/corp/brand/SiteAssets/LinkedIn.png] linkedin.com/in/paulgibeault<https://www.linkedin.com/in/linkedin.com/in/paulgibeault>
Re: I need help configuring Site-to-Site in Secure Mode.
Posted by Bryan Bende <bb...@gmail.com>.
Paul,
It sounds like you probably have the certificates/truststores setup
correctly and just need to create the appropriate policies...
Lets say you have nifi-1 with an Remote Process Group pointing at the URL
of nifi-2, and nifi-2 has an Input port to receive data.
In nifi-2 there needs to be a user for the certificate of nifi-1, and then
in the global policies of nifi-2 (top right menu) there needs to be a
policy for "retrieve site-to-site details" with the nifi-1 user added to
the policy. I think this is what is causing the error message you are
seeing since nifi-1 is not authorized to query nifi-2 for site-to-site
information (available ports, etc).
I believe you also need to create a policy on the Input Port on nifi-2...
select the input port and use the lock icon in the left palette and choose
"receive data over site-to-site" and add the user of nifi-1. This gives
nifi-1 access to the specific port.
Let us know if that works. If so we should definitely look at updating some
of the documentation to explain this.
Thanks,
Bryan
On Tue, Aug 30, 2016 at 6:28 PM, Paul Gibeault (pagibeault) <
pagibeault@micron.com> wrote:
> Hello,
>
>
>
> We have been attempting to set up Site-to-Site for NiFi in secure mode and
> have not been successful.
>
>
>
> When I create a Remote Process Group, and enter the URL*
> https://servername:8443/nifi I receive an error icon. The hover status
> is “Unauthorized”
>
> ** - servername is the actual hostname running NiFi*
>
>
>
> Things I have tried without success:
>
> - Closely followed the instructions in the NiFi System Administrator’s
> Guide
> <https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html>
>
> - Enabled SSL Security and Kerberos User Authentication (no self-signed
> certs)
>
> - Imported public keys of remote NiFi servers into the local keystores for
> each instance
>
> - Created policies on each instance to allow for full permissions to the
> accounts in use.
>
> - Tried various combinations of Linux & Windows instances of NiFi.
>
> - Connected a Site-to-Site process group to itself
>
> - Used both 1.0.0-BETA and 1.1.0-SNAPSHOT NiFi versions
>
>
>
> There are no warnings or errors in the log files when I attempt to connect
> a NiFi instance running on Linux to another instance on Linux. However, I
> did see something when attempting to connect a NiFi instance running on
> Windows to an instance running on Linux
>
> From log of NiFi on Windows:
>
> 2016-08-30 16:11:21,173 ERROR [Remote Process Group
> dd2e3ac9-0156-1000-4543-5ba3d10c6130: https://servername:8443/nifi
> Thread-1] o.a.n.remote.StandardRemoteProcessGroup org.apache.nifi.remote.
> StandardRemoteProcessGroup$InitializationTask@19bde1f3 Failed to request
> account: got unexpected response code of 404:Not Found
>
>
>
> From log of NiFi on Linux:
>
> nifi-user.log:2016-08-30 16:39:53,973 INFO [NiFi Web Server-465] o.a.n.w.s.NiFiAuthenticationFilter
> Attempting request for (CN=pagibeault) GET https://servername:8443/nifi-
> api/site-to-site (source ip: 137.201.48.150)
>
> nifi-user.log:2016-08-30 16:39:53,973 INFO [NiFi Web Server-465] o.a.n.w.s.NiFiAuthenticationFilter
> Authentication success for CN=pagibeault
>
> nifi-user.log:2016-08-30 16:39:53,974 INFO [NiFi Web Server-465]
> o.a.n.w.a.c.AccessDeniedExceptionMapper CN=pagibeault does not have
> permission to access the requested resource. Returning Forbidden response.
>
>
>
> Any guidance would be grand.
>
>
>
> Thanks,
>
>
>
> [image: http://collab.micron.com/corp/brand/SiteAssets/Micron.png]
> <http://www.micron.com/>
>
> *Paul Gibeault*
> Sr. Software Engineer, Big Data
> Enterprise Analytics & Data
> Micron Technology, Inc.
>
> *Office* (208) 363-3238
>
>
> pagibeault@micron.com
> [image: http://collab.micron.com/corp/brand/SiteAssets/LinkedIn.png]
> linkedin.com/in/paulgibeault
> <https://www.linkedin.com/in/linkedin.com/in/paulgibeault>
>
>
>
>
>