You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Ricky Saltzer <ri...@cloudera.com> on 2016/11/02 21:25:01 UTC
Secure Cluster Mode Issues
Hey all -
I'm using NiFi 1.0 and I'm having an issue using secure mode with a local
key store while in clustered mode. If I set the node in clustered mode, and
also provide a valid keystore, I receive a KeyStoreException [1]. If I set
the configuration to not use clustered mode, NiFi will start up fine with
the provided key store. Am I supposed to be storing this key store in
Zookeeper somewhere?
[1]
Caused by: java.security.KeyStoreException: not found
at java.security.KeyStore.getInstance(KeyStore.java:839)
~[na:1.8.0_11]
at
org.apache.nifi.io.socket.SSLContextFactory.<init>(SSLContextFactory.java:61)
~[nifi-socket-utils-1.0.0.jar:1.0.0]
at
org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFactoryBean.getObject(ServerSocketConfigurationFactoryBean.java:45)
~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
at
org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFactoryBean.getObject(ServerSocketConfigurationFactoryBean.java:30)
~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
at
org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
... 69 common frames omitted
Caused by: java.security.NoSuchAlgorithmException: KeyStore not available
at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
~[na:1.8.0_11]
at java.security.Security.getImpl(Security.java:695) ~[na:1.8.0_11]
at java.security.KeyStore.getInstance(KeyStore.java:836)
~[na:1.8.0_11]
... 73 common frames omitted
Re: Secure Cluster Mode Issues
Posted by Ricky Saltzer <ri...@cloudera.com>.
Andy -
I'm using version 1.1.0 (official binary). I'm using kerberos
authentication, and able to log in using my internal Kerberos principal. To
be clear, the only difference between blanking out the *keyPasswd *and
*keystorePasswd* was that I was allowed to access the UI without manually
importing the certificate, but instead agreeing to proceed even though I
know the certificate was untrusted.
[image: Inline image 1]
[image: Inline image 2]
Ricky
On Wed, Nov 30, 2016 at 7:43 PM, Andy LoPresto <al...@apache.org> wrote:
> Ricky,
>
> Removing the redundant key password property shouldn’t have an impact
> (although you may be running a legacy version before NIFI-2222 [1] and
> NIFI-2466 [2] were fixed). Can you look at the top right of your NiFi UI
> and see what user is accessing the system? It should look like the
> screenshot I have attached. This, and the contents of logs/nifi-user.log,
> will indicate the authenticated user. That should help you figure out how
> the authentication is occurring (client certificate, LDAP, or Kerberos). If
> you still cannot determine it, you can update conf/logback.xml and change
> the logging level for the following loggers from INFO to DEBUG:
>
> <logger name="org.apache.nifi.web.security" level="*INFO*"
> additivity="false">
> <appender-ref ref="USER_FILE"/>
> </logger>
> <logger name="org.apache.nifi.web.api.config" level="*INFO*"
> additivity="false">
> <appender-ref ref="USER_FILE"/>
> </logger>
> <logger name="org.apache.nifi.authorization" level="*INFO*"
> additivity="false">
> <appender-ref ref="USER_FILE"/>
> </logger>
>
>
> I only ask for this information because your results do not make sense and
> I fear that they will not be reproducible for the rest of your team when
> you try to deploy the system and let them access NiFi and I would hope we
> can provide the best experience from the beginning.
>
> [1] https://issues.apache.org/jira/browse/NIFI-2222
> [2] https://issues.apache.org/jira/browse/NIFI-2466
>
>
> Andy LoPresto
> alopresto@apache.org
> *alopresto.apache@gmail.com <al...@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> On Nov 30, 2016, at 11:12 AM, Ricky Saltzer <ri...@cloudera.com> wrote:
>
> Hey Andy -
>
> I think I may have figured out the problem. Although the keystorePasswd and
> keyPasswd are the same, after completely removing the value for
> nifi.security.keyPasswd,and restarting NiFi...I'm able to access the web UI
> without manually importing the certificate.
>
> On Tue, Nov 29, 2016 at 2:19 PM, Andy LoPresto <al...@apache.org>
> wrote:
>
> Ricky,
>
> When using HTTPS in non-cluster mode, NiFi still requires user
> authentication — this can be either client certificate (perhaps you already
> had one loaded?), LDAP, or Kerberos. If you are able to access the NiFi UI
> over HTTPS without presenting some authentication, something is seriously
> broken. The warning in the browser is because the CA certificate that
> signed the server certificate (the one being presented *to* the browser
> by the application) is not trusted in the browser’s pre-installed trust
> chain. If, for example, that CA cert had been imported to the browser ahead
> of time, or if it was signed by a publicly known entity like DigiCert,
> Verisign, Comodo, etc., you would not receive a warning.
>
> For small teams, client certificates can be manageable, but if you want to
> allow multiple users to connect with minimal identity management, I
> recommend setting up an LDAP server (OpenLDAP, Microsoft ActiveDirectory,
> Apache Directory Studio, etc.) and administering users there. Then the
> users will just enter a username and password into a login field on their
> first connection to NiFi and be authenticated.
>
>
> Andy LoPresto
> alopresto@apache.org
> *alopresto.apache@gmail.com <al...@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> On Nov 29, 2016, at 11:07 AM, Ricky Saltzer <ri...@cloudera.com> wrote:
>
> Hey Andy -
>
> Thanks for the reply, I used the openssl command you provided and indeed
> the return code was *OK*. Before proceeding with the recommendation of
>
> importing the key into my OSX keychain, I would like to understand why this
> required. When using HTTPS mode in non-clustered mode, it does not require
> clients to have a special key or cert imported to their machine. Instead,
> the client is given a warning in the browser, and it's up to them to
> proceed. This UI will serve as an endpoint to several users, and I would
> really like to avoid the cumbersomeness of having members of multiple teams
> follow instructions for importing keys just so they can access a web UI.
>
> On Tue, Nov 29, 2016 at 1:55 PM, Andy LoPresto <al...@apache.org>
> wrote:
>
> Hi Ricky,
>
> The ERR_CONNECTION_CLOSED is likely because you are not sending a client
> certificate on the HTTP request. By default, a secured cluster requires
> client certificate authentication unless LDAP or Kerberos are configured as
> identity providers [1]. The TLS Toolkit provides a quick way to generate a
> valid client certificate which you can load into your browser in order to
> access the site.
>
> First, verify the cluster is running and accepting incoming connections
> (we’re going to cheat here just to be quick about it; disclaimer that this
> is not the RIGHT way to do this):
>
> In the directory where you ran the toolkit, you noted there was a
> “nifi-cert.pem” and “nifi-cert.key” file. The pem file is the PEM-encoded
> public certificate of the NiFi CA cert that was generated by the toolkit,
> and the key file is the PEM-encoded private key. Because this is the same
> certificate that signed the NiFi server key loaded in the keystore, it is
> also loaded into the truststore. That means the server will accept an
> incoming connection with any certificate signed by the CA cert.
> Coincidentally, the CA cert is self-signed, so…
>
> $ openssl s_client -connect <host:port> -debug -state -cert nifi-cert.pem
> -key nifi-key.key -CAfile nifi-cert.pem
>
> That command will attempt to negotiate a TLS connection to your server by
> presenting the CA cert and key as the client. Again, not semantically
> correct, but technically will work. You’ll get a long output, but it
> should end in a section like this:
>
> ---
> New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384
> Server public key is 2048 bit
> Secure Renegotiation IS supported
> Compression: NONE
> Expansion: NONE
> No ALPN negotiated
> SSL-Session:
> Protocol : TLSv1.2
> Cipher : ECDHE-RSA-AES256-SHA384
> Session-ID: 583DCD...9E828C
> Session-ID-ctx:
> Master-Key: 5477C0...A51E85
> Key-Arg : None
> PSK identity: None
> PSK identity hint: None
> SRP username: None
> Start Time: 1480445265
> Timeout : 300 (sec)
> Verify return code: 0 (ok)
> ---
>
> The important part is the last line — you want the *Verify return code* to
>
> be 0 for success. Once you have verified this, run the TLS toolkit again to
> generate a valid client certificate:
>
> $ ./bin/tls-toolkit.sh standalone -C 'CN=Ricky Saltzer, OU=Apache NiFi'
> -B thisIsABadPassword
>
> This will generate a PKCS12 keystore (*.p12) containing your public
> certificate AND the private key, as well as an additional file (.password)
> containing the password you provided for -B.
>
> hw12203:...lkit/nifi-toolkit-assembly/target/nifi-toolkit-
> 1.1.0-bin/nifi-toolkit-1.1.0
> (master) alopresto
> 🔓 146s @ 10:50:14 $ ./bin/tls-toolkit.sh standalone -C 'CN=Ricky Saltzer,
> OU=Apache NiFi' -B password
> 2016/11/29 10:50:20 INFO [main] org.apache.nifi.toolkit.tls.standalone.
> TlsToolkitStandaloneCommandLine: No nifiPropertiesFile specified, using
> embedded one.
> 2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.
> TlsToolkitStandalone:
> Running standalone certificate generation with output directory
> ../nifi-toolkit-1.1.0
> 2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.
> TlsToolkitStandalone:
> Using existing CA certificate ../nifi-toolkit-1.1.0/nifi-cert.pem and key
> ../nifi-toolkit-1.1.0/nifi-key.key
> 2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.
> TlsToolkitStandalone:
> No hostnames specified, not generating any host certificates or
> configuration.
> 2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.
> TlsToolkitStandalone:
> Generating new client certificate ../nifi-toolkit-1.1.0/CN=
> Ricky_Saltzer_OU=Apache_NiFi.p12
> 2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.
> TlsToolkitStandalone:
> Successfully generated client certificate ../nifi-toolkit-1.1.0/CN=
> Ricky_Saltzer_OU=Apache_NiFi.p12
> 2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.
> TlsToolkitStandalone:
> tls-toolkit standalone completed successfully
> hw12203:...lkit/nifi-toolkit-assembly/target/nifi-toolkit-
> 1.1.0-bin/nifi-toolkit-1.1.0
> (master) alopresto
> 🔓 7s @ 10:50:22 $ ll
> total 136
> drwxr-xr-x 20 alopresto staff 680B Nov 29 10:50 ./
> drwxr-xr-x 4 alopresto staff 136B Nov 28 17:16 ../
> -rw------- 1 alopresto staff 3.4K Nov 29 10:50
> CN=Ricky_Saltzer_OU=Apache_NiFi.p12
> -rw------- 1 alopresto staff 8B Nov 29 10:50
> CN=Ricky_Saltzer_OU=Apache_NiFi.password
> ...
> -rw------- 1 alopresto staff 1.2K Nov 28 16:31 nifi-cert.pem
> -rw------- 1 alopresto staff 1.6K Nov 28 16:31 nifi-key.key
> hw12203:...lkit/nifi-toolkit-assembly/target/nifi-toolkit-
> 1.1.0-bin/nifi-toolkit-1.1.0
> (master) alopresto
> 🔓 4s @ 10:50:27 $
>
> You can double-click on the *.p12 file to open it in your default handler
> — on OS X for example, this is Keychain Access. You can also manually load
> it into Firefox to keep it isolated from your system certificates.
>
> Now when you visit the UI through the browser, you should receive a prompt
> for which certificate to present, and select this entry from the presented
> list.
>
> If you get a NiFi error message in the browser that you do not have
> sufficient access, remember to update conf/authorizers.xml with an Initial
> Admin Identity, which should match EXACTLY the DN of this certificate —
> “CN=Ricky Saltzer, OU=Apache NiFi”. You can copy it from the failed
> authentication message in logs/nifi-user.log to be sure.
>
> Andy LoPresto
> alopresto@apache.org
> *alopresto.apache@gmail.com <al...@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> On Nov 29, 2016, at 10:27 AM, Ricky Saltzer <ri...@cloudera.com> wrote:
>
> Hi Andy -
>
> Thanks again for the help, sorry I've been a little preoccupied with some
> other tasks. I took your advice and used the tls-toolkit and things are
> looking a lot better. I generated a keystore/truststore for both of my
> nodes. It also generated a pem and key file. I was able to get NiFi to
> start in clustered mode (horray)! The only issue I'm seeing now is that I
> can't actually connect to the web UI using HTTPS, as it immediately throws
> a "ERR_CONNECTION_CLOSED" message in my browser.
>
> Judging by the logs, NiFi is seemingly running. A leader election took
> place and the node is heart beating to itself. Am I missing a step where
> I'm supposed to be doing something with the pem/key files?
>
> Thanks again,
> Ricky
>
> On Tue, Nov 8, 2016 at 8:22 PM, Andy LoPresto <al...@apache.org>
> wrote:
>
> Hi Ricky,
>
> Sorry for the delay in my response. I see a couple things which could be
> causing an issue:
>
> 1. You do not specify a hostname for the NiFi node (nifi.web.https.host=)
> which you should do. I have not set up a one-node cluster before, but my
> suspicion is that this field needs to be populated, and the value needs to
> match the CN for the issued server certificate. This value does not need to
> be unique (i.e. it could be nifi.cloudera.com or localhost and you could
> run multiple instances on the same machine, identified by the same
> certificate and different ports), but I think it has to be populated.
> 1.a. While you have nifi.security.needClientAuth=false set in both
> cluster and standalone mode, I am not sure if this allows for the
> truststore to be missing completely. I would have to explore this further.
> There is a current Jira for clarifying cluster, UI/API, and S2S TLS
> configurations [1]. You can try using the default JRE truststore, located
> at “$JAVA_HOME/jre/lib/security/cacerts” with password “changeit”.
> 2. The certificate you have put into the keystore appears to be a
> certificate identifying you (Ricky the person) rather than a server entity.
> You should create a certificate identifying the server itself, so the CN is
> the FQDN as mentioned above. Then import into (or generate directly inside)
> the keystore. You can also use the provided NiFi TLS Toolkit to automate
> this entire process [2].
> Certificate chain
> 0 s:/C=CA/ST=CA/L=Palo Alto/O=Cloudera/OU=EDH Team, Cloudera/CN=Ricky
> Saltzer
> i:/C=CA/ST=CA/L=Palo Alto/O=Cloudera/OU=EDH Team, Cloudera/CN=Ricky
> Saltzercompare
> to an example connection to google.com:443:
>
> Certificate chain
> 0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/*CN=*.google.com
> <http://google.com>*
> i:/C=US/O=Google Inc/CN=Google Internet Authority G2
> 1 s:/C=US/O=Google Inc/CN=Google Internet Authority G2
> i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
> 2 s:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
> i:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority
>
> 3. It appears the certificate you are using is expired. If you look at the
> output of the OpenSSL command you ran, you can see that the result code was
> *10*, not *0* as would be in the case of a successful connection (I
> understand that it looks successful because it negotiates a key and
> encrypts the channel).
> New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384
> Server public key is 2048 bit
> Secure Renegotiation IS supported
> Compression: NONE
> Expansion: NONE
> SSL-Session:
> Protocol : TLSv1.2
> Cipher : ECDHE-RSA-AES256-SHA384
> Session-ID: 581F5A5...41153B
> Session-ID-ctx:
> Master-Key: CEEED2...944C30
> Key-Arg : None
> PSK identity: None
> PSK identity hint: None
> SRP username: None
> Start Time: 1478449748
> Timeout : 300 (sec)
> Verify return code: *10* *(certificate has expired)*
>
> ---
> HTTP/1.1 400 No URI
> Content-Length: 0
> Connection: close
> Server: Jetty(9.3.9.v20160517)
> I rebuilt your certificate from the Base64-encoded version in the output,
> and it appears it expired on October 25, 2016. You can verify this by
> copying the section between "-----BEGIN CERTIFICATE-----“ and "-----END
> CERTIFICATE-----“ (including these markers) into a text file and naming it
> “ricky.pem” and then running the following command:
>
> hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
> 🔓 11s @ 17:09:30 $ more ricky.pem
> -----BEGIN CERTIFICATE-----
> MIIDizCCAnOgAwIBAgIEHPLwBzANBgkqhkiG9w0BAQsFADB2MQswCQYDVQQGEwJD
> ...
> 45GrxLz7n/ZXZk8q9aXcrSGaj+HHCyrO0/9NYtMNP2V5wpYcBRiHO9f3sHKgnzU=
> -----END CERTIFICATE-----
> hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
> 🔓 5s @ 17:09:35 $ openssl x509 -in ricky.pem -text -noout
> Certificate:
> Data:
> Version: 3 (0x2)
> Serial Number: 485683207 (0x1cf2f007)
> Signature Algorithm: sha256WithRSAEncryption
> Issuer: C=CA, ST=CA, L=Palo Alto, O=Cloudera, OU=EDH Team,
> Cloudera, CN=Ricky Saltzer
> Validity
> Not Before: Jul 27 17:15:46 2016 GMT
> *Not After : Oct 25 17:15:46 2016 GMT*
>
> Subject: C=CA, ST=CA, L=Palo Alto, O=Cloudera, OU=EDH Team,
> Cloudera, CN=Ricky Saltzer
> Subject Public Key Info:
> Public Key Algorithm: rsaEncryption
> Public-Key: (2048 bit)
> Modulus:
> 00:f0:4c:fe:7e:af:69:89:f0:4b:a0:e0:df:e8:8c:
> 46:6e:d7:17:4d:17:eb:ee:89:25:ab:5b:35:6a:b2:
> 1e:07:84:95:5f:39:1e:c3:d9:5c:d4:54:09:0c:1a:
> 2a:60:df:9f:ce:7d:87:00:d7:31:23:37:62:8c:02:
> c9:98:a3:d4:f5:3e:71:4e:1d:fb:bb:5b:86:35:af:
> ce:6d:9d:ee:fb:65:d1:01:22:e3:f8:df:c0:81:d7:
> fe:fd:f3:c8:48:c2:bd:23:e2:44:1e:8d:48:5f:81:
> ed:10:62:77:71:3e:15:82:5a:c2:49:70:bb:ef:95:
> d9:ff:f2:37:6c:06:63:fc:3c:98:a3:c9:49:3a:1d:
> b8:37:e3:0b:50:0f:23:5f:26:32:19:c8:59:4e:5a:
> 09:5a:87:7f:b6:40:b4:37:18:20:74:fb:d2:f8:e9:
> 18:e8:d7:23:fa:83:be:2a:c3:33:8d:17:a8:b3:7c:
> 52:03:03:7c:24:80:c5:9c:a5:0d:21:3d:57:7c:d4:
> 0f:78:00:fd:be:69:dd:d8:fe:07:a0:09:77:81:b7:
> bf:fd:0f:bd:8c:9a:35:42:4f:89:8e:31:fd:00:22:
> 16:bb:76:0b:51:b0:c4:81:64:69:f7:c4:5d:37:1e:
> 79:b7:08:d2:5a:6a:a7:43:98:d3:81:a9:08:00:57:
> 2b:c5
> Exponent: 65537 (0x10001)
> X509v3 extensions:
> X509v3 Subject Key Identifier:
> 1C:DD:E1:16:5F:41:CC:7C:69:27:
> E3:B1:6B:78:EF:FA:16:DA:1F:6F
> Signature Algorithm: sha256WithRSAEncryption
> da:5a:31:4b:3b:a6:bf:84:8d:6a:56:2a:82:88:2c:e0:95:49:
> 30:f2:3a:31:ed:b5:bc:5b:92:e6:ad:b3:5b:9f:7a:71:bf:62:
> 0f:38:27:4c:71:02:a1:2f:7f:fa:c0:46:da:78:64:36:35:34:
> 1d:91:b3:f8:e2:23:22:1a:e8:e1:14:2b:15:2c:aa:36:71:39:
> 05:d7:c0:2a:e0:cd:4d:f9:72:07:03:b9:f2:c8:6d:59:fb:c9:
> 25:49:ae:3f:a2:c2:fe:d0:54:1e:45:13:8d:95:05:86:8b:a4:
> c4:7c:df:e1:7a:35:cc:48:e3:cd:20:87:c5:50:a7:43:d2:66:
> 94:81:89:b1:a2:66:20:44:ee:cb:ea:08:c9:7c:aa:61:c7:24:
> 5f:71:ca:6e:c0:a8:15:1f:4a:d3:79:e3:19:fd:7f:86:96:1a:
> 4d:db:9d:38:6f:b3:16:fd:39:68:cf:1c:57:86:26:93:6d:0e:
> 76:6b:aa:b8:9a:e7:c4:05:c6:c5:ae:5a:e1:a1:2a:ad:aa:e3:
> 5b:9a:f1:70:a5:f0:54:76:b3:0e:00:e3:91:ab:c4:bc:fb:9f:
> f6:57:66:4f:2a:f5:a5:dc:ad:21:9a:8f:e1:c7:0b:2a:ce:d3:
> ff:4d:62:d3:0d:3f:65:79:c2:96:1c:05:18:87:3b:d7:f7:b0:
> 72:a0:9f:35
> hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
> 🔓 11s @ 17:09:47 $
>
> I would recommend using the TLS Toolkit to generate a valid CA, server
> certificate, client certificate, and keystore & truststore (seriously, one
> command does all of this) and then re-trying. From this stable baseline, I
> think I’ll be able to better help iron out the cluster/standalone issues.
>
> [1] https://issues.apache.org/jira/browse/NIFI-1990
> [2] https://nifi.apache.org/docs/nifi-docs/html/
> administration-guide.html#tls-generation-toolkit
>
>
> Andy LoPresto
> alopresto@apache.org
> *alopresto.apache@gmail.com <al...@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> On Nov 8, 2016, at 1:31 AM, Andre <an...@fucs.org> wrote:
>
> Rick,
>
> Can you confirm the certificate has a chain of trust with the default JDK
> trusted certs? (i.e. trusted by the JVM)
>
> Cheers
>
>
> On Mon, Nov 7, 2016 at 3:38 AM, Ricky Saltzer <ri...@cloudera.com> wrote:
>
> Hey Andy -
>
> Thanks again for the help.
>
> The error message seems indicative that it doesn't seem to properly read
> the keystore file. One thing to note, if I point the nifi properties to a
> bogus keystore location, then it actually throws a FileNotFound exception.
> This is really odd behavior, because as I mentioned I'm able to start it in
> standalone mode using the correct keystore location, just as I try to do in
> clustered mode.
>
> I've attached both the clustered [1] nifi.properties, which doesn't work,
> and the standalone [2] which does work. . I restored it to a more basic
> configuration without the encrypted configuration, but with SSL still
> enabled. I also added a diff [3] of both the standalone and clustered
> properties file. Note that I I only have NiFi configured to use the
> keystore and not a truststore. I've redacted a few of the values in the
> property files, but be assured that the keystore is most definitely valid
> and is readable / locatable, as starting in standalone works just fine.
>
> I ran the SSL command [4] you gave me, minus the three PEM file arguments
> as I don't have any of those on hand. I hope that is fine. The output still
> looks good.
>
> [1] https://gist.github.com/rickysaltzer/712aa6586592fe6628db2d57cec7a562
> [2] https://gist.github.com/rickysaltzer/fe11c8233e4434eacedd7fd0a083d950
> [3] https://gist.github.com/rickysaltzer/d715c7451eb554a54f14ec6e64da8558
> [4] https://gist.github.com/rickysaltzer/5d7cdeff8868bfc1f47010189735411a
>
>
>
>
> On Fri, Nov 4, 2016 at 7:48 PM, Andy LoPresto <al...@apache.org>
> wrote:
>
> Hi Ricky,
>
> Sorry, should have noted that the debug output goes to
>
> nifi-bootstrap.log,
>
> so thanks Mark for jumping in to help there.
>
> If you look at the top of that log, you’ll note that there is no keystore
> file provided and the truststore loaded is the default JRE cacerts
> truststore. Can you please provide your nifi.properties file in a Gist,
>
> **taking
>
> care to redact any sensitive values** like keystore/truststore passwords,
> although I think from looking at your log output, you are taking
>
> advantage
>
> of the encrypted configuration feature, so even viewing the encrypted
> values should be ok. Could you also please provide the directory listing
> where the keystore and truststore are located including the permissions
>
> and
>
> ownership information?
>
> There may be a bug in the logic between cluster and standalone mode, but
>
> I
>
> haven’t encountered this behavior before. If you can start NiFi in
> standalone mode, could you please provide the output of the following
> command run from the terminal? It will simulate an HTTPS connection to
>
> the
>
> server and verify the key and certificate presented by NiFi.
>
> * host — the NiFi hostname
> * port — the port NiFi is running on
> * path_to_your_cert.pem — the public key certificate identifying the
> client/user (i.e. what you load into your browser to authenticate)
> * path_to_your_key.key — the private key identifying the client/user
> * path_to_your_CA_cert.pem — the public key certificate identifying the
>
> CA
>
> which signed your NiFi server certificate (if self-signed, provide that
> certificate)
>
> $ openssl s_client -connect <host:port> -debug -state -cert
> <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile
> <path_to_your_CA_cert.pem>
>
> Andy LoPresto
> alopresto@apache.org
> *alopresto.apache@gmail.com <al...@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> On Nov 4, 2016, at 11:21 AM, Ricky Saltzer <ri...@cloudera.com> wrote:
>
> Hey guys -
>
> I went ahead and uploaded the boostrap log. I took a look at it and it
> seems to be the same error [1]
>
> [1]:
> https://gist.githubusercontent.com/rickysaltzer/
> b156594f92066873f80928eddf84e7bb/raw/4d0e018038b332f4fdf14644910dfa
> 9e70c57e49/gistfile1.txt
>
> On Fri, Nov 4, 2016 at 2:14 PM, Mark Payne <ma...@hotmail.com> wrote:
>
> Hey Ricky,
>
> When you enable debug logging for SSL, it writes to StdErr (or StdOut?)
>
> so
>
> it will end up in your logs/nifi-bootstrap.log instead of nifi-app.log.
> Can you give that a look?
>
> Thanks
> -Mark
>
> On Nov 4, 2016, at 2:07 PM, Ricky Saltzer <ri...@cloudera.com> wrote:
>
> Hey Andy -
>
> Thanks for the response. I'm currently just trying to get one node in
> clustered mode before adding a second. The keystore is stored locally and
> I've confirmed it's readable, as it was able to start once I took it out
>
> of
>
> clustered mode. I added that line to the bootstrap.conf, but I don't
> believe any additional logging was produced in regards to troubleshooting
> this problem. Just in case, I've attached the entire log [1].
>
> [1]:
> https://gist.githubusercontent.com/rickysaltzer/
>
> ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0
> fedabc88bb/gistfile1.txt <https://gist.githubusercontent.com/
>
> rickysaltzer/
>
> ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0
> fedabc88bb/gistfile1.txt>
>
>
> On Wed, Nov 2, 2016 at 7:08 PM, Andy LoPresto <alopresto@apache.org
>
> <mailto:alopresto@apache.org <al...@apache.org> <al...@apache.org>
> <al...@apache.org>
> <al...@apache.org>>>
> wrote:
>
>
> Hi Ricky,
>
> Sorry to hear you are having this issue. Is the keystore available on
>
> all
>
> nodes of the cluster? It appears from the log message that the keystore
>
> is
>
> not found during startup. To further debug, you can add the following
>
> line
>
> in bootstrap.conf to provide additional logging:
>
> java.arg.15=-Djavax.net.debug=ssl,handshake
>
> Andy LoPresto
> alopresto@apache.org <mailto:alopresto@apache.org <al...@apache.org> <
> alopresto@apache.org> <
> alopresto@apache.org> <
> alopresto@apache.org>>
> *alopresto.apache@gmail.com <mailto:alopresto.apache@gmail.com
> <al...@gmail.com>
> <al...@gmail.com>
> <al...@gmail.com>
> <al...@gmail.com>> <
>
> alopresto.apache@gmail.com <mailto:alopresto.apache@gmail.com
> <al...@gmail.com>
> <al...@gmail.com>
>
> <al...@gmail.com>
>
> <al...@gmail.com>>>*
>
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> On Nov 2, 2016, at 2:25 PM, Ricky Saltzer <ri...@cloudera.com> wrote:
>
> Hey all -
>
> I'm using NiFi 1.0 and I'm having an issue using secure mode with a
>
> local
>
> key store while in clustered mode. If I set the node in clustered mode,
>
> and
>
> also provide a valid keystore, I receive a KeyStoreException [1]. If I
>
> set
>
> the configuration to not use clustered mode, NiFi will start up fine
>
> with
>
> the provided key store. Am I supposed to be storing this key store in
> Zookeeper somewhere?
>
>
> [1]
>
>
> Caused by: java.security.KeyStoreException: not found
>
>
> at java.security.KeyStore.getInstance(KeyStore.java:839)
> ~[na:1.8.0_11]
>
> at
> org.apache.nifi.io.socket.SSLContextFactory.<init>(
> SSLContextFactory.java:61)
> ~[nifi-socket-utils-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
> ryBean.getObject(ServerSocketConfigurationFactoryBean.java:45)
> ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
> ryBean.getObject(ServerSocketConfigurationFactoryBean.java:30)
> ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
>
> at
> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.
> doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
> ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
>
> ... 69 common frames omitted
>
> Caused by: java.security.NoSuchAlgorithmException: KeyStore not
>
> available
>
>
> at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
> ~[na:1.8.0_11]
>
> at java.security.Security.getImpl(Security.java:695)
>
> ~[na:1.8.0_11]
>
>
> at java.security.KeyStore.getInstance(KeyStore.java:836)
> ~[na:1.8.0_11]
>
> ... 73 common frames omitted
>
>
>
>
>
> --
> Ricky Saltzer
> http://www.cloudera.com <http://www.cloudera.com/>
>
>
>
>
>
> --
> Ricky Saltzer
> http://www.cloudera.com
>
>
>
>
>
> --
> Ricky Saltzer
> http://www.cloudera.com
>
>
>
>
>
> --
> Ricky Saltzer
> http://www.cloudera.com
>
>
>
>
>
> --
> Ricky Saltzer
> http://www.cloudera.com
>
>
>
>
>
> --
> Ricky Saltzer
> http://www.cloudera.com
>
>
>
--
Ricky Saltzer
http://www.cloudera.com
Re: Secure Cluster Mode Issues
Posted by Andy LoPresto <al...@apache.org>.
Ricky,
Removing the redundant key password property shouldn’t have an impact (although you may be running a legacy version before NIFI-2222 [1] and NIFI-2466 [2] were fixed). Can you look at the top right of your NiFi UI and see what user is accessing the system? It should look like the screenshot I have attached. This, and the contents of logs/nifi-user.log, will indicate the authenticated user. That should help you figure out how the authentication is occurring (client certificate, LDAP, or Kerberos). If you still cannot determine it, you can update conf/logback.xml and change the logging level for the following loggers from INFO to DEBUG:
<logger name="org.apache.nifi.web.security" level="INFO" additivity="false">
<appender-ref ref="USER_FILE"/>
</logger>
<logger name="org.apache.nifi.web.api.config" level="INFO" additivity="false">
<appender-ref ref="USER_FILE"/>
</logger>
<logger name="org.apache.nifi.authorization" level="INFO" additivity="false">
<appender-ref ref="USER_FILE"/>
</logger>
I only ask for this information because your results do not make sense and I fear that they will not be reproducible for the rest of your team when you try to deploy the system and let them access NiFi and I would hope we can provide the best experience from the beginning.
[1] https://issues.apache.org/jira/browse/NIFI-2222
[2] https://issues.apache.org/jira/browse/NIFI-2466
Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Nov 30, 2016, at 11:12 AM, Ricky Saltzer <ri...@cloudera.com> wrote:
>
> Hey Andy -
>
> I think I may have figured out the problem. Although the keystorePasswd and
> keyPasswd are the same, after completely removing the value for
> nifi.security.keyPasswd,and restarting NiFi...I'm able to access the web UI
> without manually importing the certificate.
>
> On Tue, Nov 29, 2016 at 2:19 PM, Andy LoPresto <alopresto@apache.org <ma...@apache.org>> wrote:
>
>> Ricky,
>>
>> When using HTTPS in non-cluster mode, NiFi still requires user
>> authentication — this can be either client certificate (perhaps you already
>> had one loaded?), LDAP, or Kerberos. If you are able to access the NiFi UI
>> over HTTPS without presenting some authentication, something is seriously
>> broken. The warning in the browser is because the CA certificate that
>> signed the server certificate (the one being presented *to* the browser
>> by the application) is not trusted in the browser’s pre-installed trust
>> chain. If, for example, that CA cert had been imported to the browser ahead
>> of time, or if it was signed by a publicly known entity like DigiCert,
>> Verisign, Comodo, etc., you would not receive a warning.
>>
>> For small teams, client certificates can be manageable, but if you want to
>> allow multiple users to connect with minimal identity management, I
>> recommend setting up an LDAP server (OpenLDAP, Microsoft ActiveDirectory,
>> Apache Directory Studio, etc.) and administering users there. Then the
>> users will just enter a username and password into a login field on their
>> first connection to NiFi and be authenticated.
>>
>>
>> Andy LoPresto
>> alopresto@apache.org
>> *alopresto.apache@gmail.com <ma...@gmail.com> <alopresto.apache@gmail.com <ma...@gmail.com>>*
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>>
>> On Nov 29, 2016, at 11:07 AM, Ricky Saltzer <ricky@cloudera.com <ma...@cloudera.com>> wrote:
>>
>> Hey Andy -
>>
>> Thanks for the reply, I used the openssl command you provided and indeed
>> the return code was *OK*. Before proceeding with the recommendation of
>>
>> importing the key into my OSX keychain, I would like to understand why this
>> required. When using HTTPS mode in non-clustered mode, it does not require
>> clients to have a special key or cert imported to their machine. Instead,
>> the client is given a warning in the browser, and it's up to them to
>> proceed. This UI will serve as an endpoint to several users, and I would
>> really like to avoid the cumbersomeness of having members of multiple teams
>> follow instructions for importing keys just so they can access a web UI.
>>
>> On Tue, Nov 29, 2016 at 1:55 PM, Andy LoPresto <alopresto@apache.org <ma...@apache.org>>
>> wrote:
>>
>> Hi Ricky,
>>
>> The ERR_CONNECTION_CLOSED is likely because you are not sending a client
>> certificate on the HTTP request. By default, a secured cluster requires
>> client certificate authentication unless LDAP or Kerberos are configured as
>> identity providers [1]. The TLS Toolkit provides a quick way to generate a
>> valid client certificate which you can load into your browser in order to
>> access the site.
>>
>> First, verify the cluster is running and accepting incoming connections
>> (we’re going to cheat here just to be quick about it; disclaimer that this
>> is not the RIGHT way to do this):
>>
>> In the directory where you ran the toolkit, you noted there was a
>> “nifi-cert.pem” and “nifi-cert.key” file. The pem file is the PEM-encoded
>> public certificate of the NiFi CA cert that was generated by the toolkit,
>> and the key file is the PEM-encoded private key. Because this is the same
>> certificate that signed the NiFi server key loaded in the keystore, it is
>> also loaded into the truststore. That means the server will accept an
>> incoming connection with any certificate signed by the CA cert.
>> Coincidentally, the CA cert is self-signed, so…
>>
>> $ openssl s_client -connect <host:port> -debug -state -cert nifi-cert.pem
>> -key nifi-key.key -CAfile nifi-cert.pem
>>
>> That command will attempt to negotiate a TLS connection to your server by
>> presenting the CA cert and key as the client. Again, not semantically
>> correct, but technically will work. You’ll get a long output, but it
>> should end in a section like this:
>>
>> ---
>> New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384
>> Server public key is 2048 bit
>> Secure Renegotiation IS supported
>> Compression: NONE
>> Expansion: NONE
>> No ALPN negotiated
>> SSL-Session:
>> Protocol : TLSv1.2
>> Cipher : ECDHE-RSA-AES256-SHA384
>> Session-ID: 583DCD...9E828C
>> Session-ID-ctx:
>> Master-Key: 5477C0...A51E85
>> Key-Arg : None
>> PSK identity: None
>> PSK identity hint: None
>> SRP username: None
>> Start Time: 1480445265
>> Timeout : 300 (sec)
>> Verify return code: 0 (ok)
>> ---
>>
>> The important part is the last line — you want the *Verify return code* to
>>
>> be 0 for success. Once you have verified this, run the TLS toolkit again to
>> generate a valid client certificate:
>>
>> $ ./bin/tls-toolkit.sh standalone -C 'CN=Ricky Saltzer, OU=Apache NiFi'
>> -B thisIsABadPassword
>>
>> This will generate a PKCS12 keystore (*.p12) containing your public
>> certificate AND the private key, as well as an additional file (.password)
>> containing the password you provided for -B.
>>
>> hw12203:...lkit/nifi-toolkit-assembly/target/nifi-toolkit-
>> 1.1.0-bin/nifi-toolkit-1.1.0
>> (master) alopresto
>> 🔓 146s @ 10:50:14 $ ./bin/tls-toolkit.sh standalone -C 'CN=Ricky Saltzer,
>> OU=Apache NiFi' -B password
>> 2016/11/29 10:50:20 INFO [main] org.apache.nifi.toolkit.tls.standalone.
>> TlsToolkitStandaloneCommandLine: No nifiPropertiesFile specified, using
>> embedded one.
>> 2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.
>> TlsToolkitStandalone:
>> Running standalone certificate generation with output directory
>> ../nifi-toolkit-1.1.0
>> 2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.
>> TlsToolkitStandalone:
>> Using existing CA certificate ../nifi-toolkit-1.1.0/nifi-cert.pem and key
>> ../nifi-toolkit-1.1.0/nifi-key.key
>> 2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.
>> TlsToolkitStandalone:
>> No hostnames specified, not generating any host certificates or
>> configuration.
>> 2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.
>> TlsToolkitStandalone:
>> Generating new client certificate ../nifi-toolkit-1.1.0/CN=
>> Ricky_Saltzer_OU=Apache_NiFi.p12
>> 2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.
>> TlsToolkitStandalone:
>> Successfully generated client certificate ../nifi-toolkit-1.1.0/CN=
>> Ricky_Saltzer_OU=Apache_NiFi.p12
>> 2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.
>> TlsToolkitStandalone:
>> tls-toolkit standalone completed successfully
>> hw12203:...lkit/nifi-toolkit-assembly/target/nifi-toolkit-
>> 1.1.0-bin/nifi-toolkit-1.1.0
>> (master) alopresto
>> 🔓 7s @ 10:50:22 $ ll
>> total 136
>> drwxr-xr-x 20 alopresto staff 680B Nov 29 10:50 ./
>> drwxr-xr-x 4 alopresto staff 136B Nov 28 17:16 ../
>> -rw------- 1 alopresto staff 3.4K Nov 29 10:50
>> CN=Ricky_Saltzer_OU=Apache_NiFi.p12
>> -rw------- 1 alopresto staff 8B Nov 29 10:50
>> CN=Ricky_Saltzer_OU=Apache_NiFi.password
>> ...
>> -rw------- 1 alopresto staff 1.2K Nov 28 16:31 nifi-cert.pem
>> -rw------- 1 alopresto staff 1.6K Nov 28 16:31 nifi-key.key
>> hw12203:...lkit/nifi-toolkit-assembly/target/nifi-toolkit-
>> 1.1.0-bin/nifi-toolkit-1.1.0
>> (master) alopresto
>> 🔓 4s @ 10:50:27 $
>>
>> You can double-click on the *.p12 file to open it in your default handler
>> — on OS X for example, this is Keychain Access. You can also manually load
>> it into Firefox to keep it isolated from your system certificates.
>>
>> Now when you visit the UI through the browser, you should receive a prompt
>> for which certificate to present, and select this entry from the presented
>> list.
>>
>> If you get a NiFi error message in the browser that you do not have
>> sufficient access, remember to update conf/authorizers.xml with an Initial
>> Admin Identity, which should match EXACTLY the DN of this certificate —
>> “CN=Ricky Saltzer, OU=Apache NiFi”. You can copy it from the failed
>> authentication message in logs/nifi-user.log to be sure.
>>
>> Andy LoPresto
>> alopresto@apache.org <ma...@apache.org>
>> *alopresto.apache@gmail.com <ma...@gmail.com> <alopresto.apache@gmail.com <ma...@gmail.com>>*
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>>
>> On Nov 29, 2016, at 10:27 AM, Ricky Saltzer <ricky@cloudera.com <ma...@cloudera.com>> wrote:
>>
>> Hi Andy -
>>
>> Thanks again for the help, sorry I've been a little preoccupied with some
>> other tasks. I took your advice and used the tls-toolkit and things are
>> looking a lot better. I generated a keystore/truststore for both of my
>> nodes. It also generated a pem and key file. I was able to get NiFi to
>> start in clustered mode (horray)! The only issue I'm seeing now is that I
>> can't actually connect to the web UI using HTTPS, as it immediately throws
>> a "ERR_CONNECTION_CLOSED" message in my browser.
>>
>> Judging by the logs, NiFi is seemingly running. A leader election took
>> place and the node is heart beating to itself. Am I missing a step where
>> I'm supposed to be doing something with the pem/key files?
>>
>> Thanks again,
>> Ricky
>>
>> On Tue, Nov 8, 2016 at 8:22 PM, Andy LoPresto <alopresto@apache.org <ma...@apache.org>>
>> wrote:
>>
>> Hi Ricky,
>>
>> Sorry for the delay in my response. I see a couple things which could be
>> causing an issue:
>>
>> 1. You do not specify a hostname for the NiFi node (nifi.web.https.host=)
>> which you should do. I have not set up a one-node cluster before, but my
>> suspicion is that this field needs to be populated, and the value needs to
>> match the CN for the issued server certificate. This value does not need to
>> be unique (i.e. it could be nifi.cloudera.com <http://nifi.cloudera.com/> or localhost and you could
>> run multiple instances on the same machine, identified by the same
>> certificate and different ports), but I think it has to be populated.
>> 1.a. While you have nifi.security.needClientAuth=false set in both
>> cluster and standalone mode, I am not sure if this allows for the
>> truststore to be missing completely. I would have to explore this further.
>> There is a current Jira for clarifying cluster, UI/API, and S2S TLS
>> configurations [1]. You can try using the default JRE truststore, located
>> at “$JAVA_HOME/jre/lib/security/cacerts” with password “changeit”.
>> 2. The certificate you have put into the keystore appears to be a
>> certificate identifying you (Ricky the person) rather than a server entity.
>> You should create a certificate identifying the server itself, so the CN is
>> the FQDN as mentioned above. Then import into (or generate directly inside)
>> the keystore. You can also use the provided NiFi TLS Toolkit to automate
>> this entire process [2].
>> Certificate chain
>> 0 s:/C=CA/ST=CA/L=Palo Alto/O=Cloudera/OU=EDH Team, Cloudera/CN=Ricky
>> Saltzer
>> i:/C=CA/ST=CA/L=Palo Alto/O=Cloudera/OU=EDH Team, Cloudera/CN=Ricky
>> Saltzercompare
>> to an example connection to google.com <http://google.com/>:443:
>>
>> Certificate chain
>> 0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/*CN=*.google.com <http://google.com/>
>> <http://google.com <http://google.com/>>*
>> i:/C=US/O=Google Inc/CN=Google Internet Authority G2
>> 1 s:/C=US/O=Google Inc/CN=Google Internet Authority G2
>> i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
>> 2 s:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
>> i:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority
>>
>> 3. It appears the certificate you are using is expired. If you look at the
>> output of the OpenSSL command you ran, you can see that the result code was
>> *10*, not *0* as would be in the case of a successful connection (I
>> understand that it looks successful because it negotiates a key and
>> encrypts the channel).
>> New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384
>> Server public key is 2048 bit
>> Secure Renegotiation IS supported
>> Compression: NONE
>> Expansion: NONE
>> SSL-Session:
>> Protocol : TLSv1.2
>> Cipher : ECDHE-RSA-AES256-SHA384
>> Session-ID: 581F5A5...41153B
>> Session-ID-ctx:
>> Master-Key: CEEED2...944C30
>> Key-Arg : None
>> PSK identity: None
>> PSK identity hint: None
>> SRP username: None
>> Start Time: 1478449748
>> Timeout : 300 (sec)
>> Verify return code: *10* *(certificate has expired)*
>>
>> ---
>> HTTP/1.1 400 No URI
>> Content-Length: 0
>> Connection: close
>> Server: Jetty(9.3.9.v20160517)
>> I rebuilt your certificate from the Base64-encoded version in the output,
>> and it appears it expired on October 25, 2016. You can verify this by
>> copying the section between "-----BEGIN CERTIFICATE-----“ and "-----END
>> CERTIFICATE-----“ (including these markers) into a text file and naming it
>> “ricky.pem” and then running the following command:
>>
>> hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
>> 🔓 11s @ 17:09:30 $ more ricky.pem
>> -----BEGIN CERTIFICATE-----
>> MIIDizCCAnOgAwIBAgIEHPLwBzANBgkqhkiG9w0BAQsFADB2MQswCQYDVQQGEwJD
>> ...
>> 45GrxLz7n/ZXZk8q9aXcrSGaj+HHCyrO0/9NYtMNP2V5wpYcBRiHO9f3sHKgnzU=
>> -----END CERTIFICATE-----
>> hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
>> 🔓 5s @ 17:09:35 $ openssl x509 -in ricky.pem -text -noout
>> Certificate:
>> Data:
>> Version: 3 (0x2)
>> Serial Number: 485683207 (0x1cf2f007)
>> Signature Algorithm: sha256WithRSAEncryption
>> Issuer: C=CA, ST=CA, L=Palo Alto, O=Cloudera, OU=EDH Team,
>> Cloudera, CN=Ricky Saltzer
>> Validity
>> Not Before: Jul 27 17:15:46 2016 GMT
>> *Not After : Oct 25 17:15:46 2016 GMT*
>>
>> Subject: C=CA, ST=CA, L=Palo Alto, O=Cloudera, OU=EDH Team,
>> Cloudera, CN=Ricky Saltzer
>> Subject Public Key Info:
>> Public Key Algorithm: rsaEncryption
>> Public-Key: (2048 bit)
>> Modulus:
>> 00:f0:4c:fe:7e:af:69:89:f0:4b:a0:e0:df:e8:8c:
>> 46:6e:d7:17:4d:17:eb:ee:89:25:ab:5b:35:6a:b2:
>> 1e:07:84:95:5f:39:1e:c3:d9:5c:d4:54:09:0c:1a:
>> 2a:60:df:9f:ce:7d:87:00:d7:31:23:37:62:8c:02:
>> c9:98:a3:d4:f5:3e:71:4e:1d:fb:bb:5b:86:35:af:
>> ce:6d:9d:ee:fb:65:d1:01:22:e3:f8:df:c0:81:d7:
>> fe:fd:f3:c8:48:c2:bd:23:e2:44:1e:8d:48:5f:81:
>> ed:10:62:77:71:3e:15:82:5a:c2:49:70:bb:ef:95:
>> d9:ff:f2:37:6c:06:63:fc:3c:98:a3:c9:49:3a:1d:
>> b8:37:e3:0b:50:0f:23:5f:26:32:19:c8:59:4e:5a:
>> 09:5a:87:7f:b6:40:b4:37:18:20:74:fb:d2:f8:e9:
>> 18:e8:d7:23:fa:83:be:2a:c3:33:8d:17:a8:b3:7c:
>> 52:03:03:7c:24:80:c5:9c:a5:0d:21:3d:57:7c:d4:
>> 0f:78:00:fd:be:69:dd:d8:fe:07:a0:09:77:81:b7:
>> bf:fd:0f:bd:8c:9a:35:42:4f:89:8e:31:fd:00:22:
>> 16:bb:76:0b:51:b0:c4:81:64:69:f7:c4:5d:37:1e:
>> 79:b7:08:d2:5a:6a:a7:43:98:d3:81:a9:08:00:57:
>> 2b:c5
>> Exponent: 65537 (0x10001)
>> X509v3 extensions:
>> X509v3 Subject Key Identifier:
>> 1C:DD:E1:16:5F:41:CC:7C:69:27:
>> E3:B1:6B:78:EF:FA:16:DA:1F:6F
>> Signature Algorithm: sha256WithRSAEncryption
>> da:5a:31:4b:3b:a6:bf:84:8d:6a:56:2a:82:88:2c:e0:95:49:
>> 30:f2:3a:31:ed:b5:bc:5b:92:e6:ad:b3:5b:9f:7a:71:bf:62:
>> 0f:38:27:4c:71:02:a1:2f:7f:fa:c0:46:da:78:64:36:35:34:
>> 1d:91:b3:f8:e2:23:22:1a:e8:e1:14:2b:15:2c:aa:36:71:39:
>> 05:d7:c0:2a:e0:cd:4d:f9:72:07:03:b9:f2:c8:6d:59:fb:c9:
>> 25:49:ae:3f:a2:c2:fe:d0:54:1e:45:13:8d:95:05:86:8b:a4:
>> c4:7c:df:e1:7a:35:cc:48:e3:cd:20:87:c5:50:a7:43:d2:66:
>> 94:81:89:b1:a2:66:20:44:ee:cb:ea:08:c9:7c:aa:61:c7:24:
>> 5f:71:ca:6e:c0:a8:15:1f:4a:d3:79:e3:19:fd:7f:86:96:1a:
>> 4d:db:9d:38:6f:b3:16:fd:39:68:cf:1c:57:86:26:93:6d:0e:
>> 76:6b:aa:b8:9a:e7:c4:05:c6:c5:ae:5a:e1:a1:2a:ad:aa:e3:
>> 5b:9a:f1:70:a5:f0:54:76:b3:0e:00:e3:91:ab:c4:bc:fb:9f:
>> f6:57:66:4f:2a:f5:a5:dc:ad:21:9a:8f:e1:c7:0b:2a:ce:d3:
>> ff:4d:62:d3:0d:3f:65:79:c2:96:1c:05:18:87:3b:d7:f7:b0:
>> 72:a0:9f:35
>> hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
>> 🔓 11s @ 17:09:47 $
>>
>> I would recommend using the TLS Toolkit to generate a valid CA, server
>> certificate, client certificate, and keystore & truststore (seriously, one
>> command does all of this) and then re-trying. From this stable baseline, I
>> think I’ll be able to better help iron out the cluster/standalone issues.
>>
>> [1] https://issues.apache.org/jira/browse/NIFI-1990 <https://issues.apache.org/jira/browse/NIFI-1990>
>> [2] https://nifi.apache.org/docs/nifi-docs/html/ <https://nifi.apache.org/docs/nifi-docs/html/>
>> administration-guide.html#tls-generation-toolkit
>>
>>
>> Andy LoPresto
>> alopresto@apache.org <ma...@apache.org>
>> *alopresto.apache@gmail.com <ma...@gmail.com> <alopresto.apache@gmail.com <ma...@gmail.com>>*
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>>
>> On Nov 8, 2016, at 1:31 AM, Andre <andre-lists@fucs.org <ma...@fucs.org>> wrote:
>>
>> Rick,
>>
>> Can you confirm the certificate has a chain of trust with the default JDK
>> trusted certs? (i.e. trusted by the JVM)
>>
>> Cheers
>>
>>
>> On Mon, Nov 7, 2016 at 3:38 AM, Ricky Saltzer <ricky@cloudera.com <ma...@cloudera.com>> wrote:
>>
>> Hey Andy -
>>
>> Thanks again for the help.
>>
>> The error message seems indicative that it doesn't seem to properly read
>> the keystore file. One thing to note, if I point the nifi properties to a
>> bogus keystore location, then it actually throws a FileNotFound exception.
>> This is really odd behavior, because as I mentioned I'm able to start it in
>> standalone mode using the correct keystore location, just as I try to do in
>> clustered mode.
>>
>> I've attached both the clustered [1] nifi.properties, which doesn't work,
>> and the standalone [2] which does work. . I restored it to a more basic
>> configuration without the encrypted configuration, but with SSL still
>> enabled. I also added a diff [3] of both the standalone and clustered
>> properties file. Note that I I only have NiFi configured to use the
>> keystore and not a truststore. I've redacted a few of the values in the
>> property files, but be assured that the keystore is most definitely valid
>> and is readable / locatable, as starting in standalone works just fine.
>>
>> I ran the SSL command [4] you gave me, minus the three PEM file arguments
>> as I don't have any of those on hand. I hope that is fine. The output still
>> looks good.
>>
>> [1] https://gist.github.com/rickysaltzer/712aa6586592fe6628db2d57cec7a562 <https://gist.github.com/rickysaltzer/712aa6586592fe6628db2d57cec7a562>
>> [2] https://gist.github.com/rickysaltzer/fe11c8233e4434eacedd7fd0a083d950 <https://gist.github.com/rickysaltzer/fe11c8233e4434eacedd7fd0a083d950>
>> [3] https://gist.github.com/rickysaltzer/d715c7451eb554a54f14ec6e64da8558 <https://gist.github.com/rickysaltzer/d715c7451eb554a54f14ec6e64da8558>
>> [4] https://gist.github.com/rickysaltzer/5d7cdeff8868bfc1f47010189735411a <https://gist.github.com/rickysaltzer/5d7cdeff8868bfc1f47010189735411a>
>>
>>
>>
>>
>> On Fri, Nov 4, 2016 at 7:48 PM, Andy LoPresto <alopresto@apache.org <ma...@apache.org>>
>> wrote:
>>
>> Hi Ricky,
>>
>> Sorry, should have noted that the debug output goes to
>>
>> nifi-bootstrap.log,
>>
>> so thanks Mark for jumping in to help there.
>>
>> If you look at the top of that log, you’ll note that there is no keystore
>> file provided and the truststore loaded is the default JRE cacerts
>> truststore. Can you please provide your nifi.properties file in a Gist,
>>
>> **taking
>>
>> care to redact any sensitive values** like keystore/truststore passwords,
>> although I think from looking at your log output, you are taking
>>
>> advantage
>>
>> of the encrypted configuration feature, so even viewing the encrypted
>> values should be ok. Could you also please provide the directory listing
>> where the keystore and truststore are located including the permissions
>>
>> and
>>
>> ownership information?
>>
>> There may be a bug in the logic between cluster and standalone mode, but
>>
>> I
>>
>> haven’t encountered this behavior before. If you can start NiFi in
>> standalone mode, could you please provide the output of the following
>> command run from the terminal? It will simulate an HTTPS connection to
>>
>> the
>>
>> server and verify the key and certificate presented by NiFi.
>>
>> * host — the NiFi hostname
>> * port — the port NiFi is running on
>> * path_to_your_cert.pem — the public key certificate identifying the
>> client/user (i.e. what you load into your browser to authenticate)
>> * path_to_your_key.key — the private key identifying the client/user
>> * path_to_your_CA_cert.pem — the public key certificate identifying the
>>
>> CA
>>
>> which signed your NiFi server certificate (if self-signed, provide that
>> certificate)
>>
>> $ openssl s_client -connect <host:port> -debug -state -cert
>> <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile
>> <path_to_your_CA_cert.pem>
>>
>> Andy LoPresto
>> alopresto@apache.org <ma...@apache.org>
>> *alopresto.apache@gmail.com <ma...@gmail.com> <alopresto.apache@gmail.com <ma...@gmail.com>>*
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>>
>> On Nov 4, 2016, at 11:21 AM, Ricky Saltzer <ricky@cloudera.com <ma...@cloudera.com>> wrote:
>>
>> Hey guys -
>>
>> I went ahead and uploaded the boostrap log. I took a look at it and it
>> seems to be the same error [1]
>>
>> [1]:
>> https://gist.githubusercontent.com/rickysaltzer/ <https://gist.githubusercontent.com/rickysaltzer/>
>> b156594f92066873f80928eddf84e7bb/raw/4d0e018038b332f4fdf14644910dfa
>> 9e70c57e49/gistfile1.txt
>>
>> On Fri, Nov 4, 2016 at 2:14 PM, Mark Payne <markap14@hotmail.com <ma...@hotmail.com>> wrote:
>>
>> Hey Ricky,
>>
>> When you enable debug logging for SSL, it writes to StdErr (or StdOut?)
>>
>> so
>>
>> it will end up in your logs/nifi-bootstrap.log instead of nifi-app.log.
>> Can you give that a look?
>>
>> Thanks
>> -Mark
>>
>> On Nov 4, 2016, at 2:07 PM, Ricky Saltzer <ricky@cloudera.com <ma...@cloudera.com>> wrote:
>>
>> Hey Andy -
>>
>> Thanks for the response. I'm currently just trying to get one node in
>> clustered mode before adding a second. The keystore is stored locally and
>> I've confirmed it's readable, as it was able to start once I took it out
>>
>> of
>>
>> clustered mode. I added that line to the bootstrap.conf, but I don't
>> believe any additional logging was produced in regards to troubleshooting
>> this problem. Just in case, I've attached the entire log [1].
>>
>> [1]:
>> https://gist.githubusercontent.com/rickysaltzer/ <https://gist.githubusercontent.com/rickysaltzer/>
>>
>> ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0
>> fedabc88bb/gistfile1.txt <https://gist.githubusercontent.com/ <https://gist.githubusercontent.com/>
>>
>> rickysaltzer/
>>
>> ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0
>> fedabc88bb/gistfile1.txt>
>>
>>
>> On Wed, Nov 2, 2016 at 7:08 PM, Andy LoPresto <alopresto@apache.org <ma...@apache.org>
>>
>> <mailto:alopresto@apache.org <ma...@apache.org> <alopresto@apache.org <ma...@apache.org>> <alopresto@apache.org <ma...@apache.org>>
>> <alopresto@apache.org <ma...@apache.org>>>>
>> wrote:
>>
>>
>> Hi Ricky,
>>
>> Sorry to hear you are having this issue. Is the keystore available on
>>
>> all
>>
>> nodes of the cluster? It appears from the log message that the keystore
>>
>> is
>>
>> not found during startup. To further debug, you can add the following
>>
>> line
>>
>> in bootstrap.conf to provide additional logging:
>>
>> java.arg.15=-Djavax.net.debug=ssl,handshake
>>
>> Andy LoPresto
>> alopresto@apache.org <ma...@apache.org> <mailto:alopresto@apache.org <ma...@apache.org> <alopresto@apache.org <ma...@apache.org>> <
>> alopresto@apache.org <ma...@apache.org>> <
>> alopresto@apache.org <ma...@apache.org>>>
>> *alopresto.apache@gmail.com <ma...@gmail.com> <mailto:alopresto.apache@gmail.com <ma...@gmail.com>
>> <alopresto.apache@gmail.com <ma...@gmail.com>>
>> <alopresto.apache@gmail.com <ma...@gmail.com>>
>> <alopresto.apache@gmail.com <ma...@gmail.com>>> <
>>
>> alopresto.apache@gmail.com <ma...@gmail.com> <mailto:alopresto.apache@gmail.com <ma...@gmail.com>
>> <alopresto.apache@gmail.com <ma...@gmail.com>>
>>
>> <alopresto.apache@gmail.com <ma...@gmail.com>>
>>
>> <al...@gmail.com>>>*
>>
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>>
>> On Nov 2, 2016, at 2:25 PM, Ricky Saltzer <ri...@cloudera.com> wrote:
>>
>> Hey all -
>>
>> I'm using NiFi 1.0 and I'm having an issue using secure mode with a
>>
>> local
>>
>> key store while in clustered mode. If I set the node in clustered mode,
>>
>> and
>>
>> also provide a valid keystore, I receive a KeyStoreException [1]. If I
>>
>> set
>>
>> the configuration to not use clustered mode, NiFi will start up fine
>>
>> with
>>
>> the provided key store. Am I supposed to be storing this key store in
>> Zookeeper somewhere?
>>
>>
>> [1]
>>
>>
>> Caused by: java.security.KeyStoreException: not found
>>
>>
>> at java.security.KeyStore.getInstance(KeyStore.java:839)
>> ~[na:1.8.0_11]
>>
>> at
>> org.apache.nifi.io.socket.SSLContextFactory.<init>(
>> SSLContextFactory.java:61)
>> ~[nifi-socket-utils-1.0.0.jar:1.0.0]
>>
>> at
>> org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
>> ryBean.getObject(ServerSocketConfigurationFactoryBean.java:45)
>> ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
>>
>> at
>> org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
>> ryBean.getObject(ServerSocketConfigurationFactoryBean.java:30)
>> ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
>>
>> at
>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.
>> doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
>> ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
>>
>> ... 69 common frames omitted
>>
>> Caused by: java.security.NoSuchAlgorithmException: KeyStore not
>>
>> available
>>
>>
>> at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
>> ~[na:1.8.0_11]
>>
>> at java.security.Security.getImpl(Security.java:695)
>>
>> ~[na:1.8.0_11]
>>
>>
>> at java.security.KeyStore.getInstance(KeyStore.java:836)
>> ~[na:1.8.0_11]
>>
>> ... 73 common frames omitted
>>
>>
>>
>>
>>
>> --
>> Ricky Saltzer
>> http://www.cloudera.com <http://www.cloudera.com/>
>>
>>
>>
>>
>>
>> --
>> Ricky Saltzer
>> http://www.cloudera.com
>>
>>
>>
>>
>>
>> --
>> Ricky Saltzer
>> http://www.cloudera.com
>>
>>
>>
>>
>>
>> --
>> Ricky Saltzer
>> http://www.cloudera.com
>>
>>
>>
>>
>>
>> --
>> Ricky Saltzer
>> http://www.cloudera.com
>>
>>
>>
>
>
> --
> Ricky Saltzer
> http://www.cloudera.com <http://www.cloudera.com/>
Re: Secure Cluster Mode Issues
Posted by Ricky Saltzer <ri...@cloudera.com>.
Hey Andy -
I think I may have figured out the problem. Although the keystorePasswd and
keyPasswd are the same, after completely removing the value for
nifi.security.keyPasswd,and restarting NiFi...I'm able to access the web UI
without manually importing the certificate.
On Tue, Nov 29, 2016 at 2:19 PM, Andy LoPresto <al...@apache.org> wrote:
> Ricky,
>
> When using HTTPS in non-cluster mode, NiFi still requires user
> authentication — this can be either client certificate (perhaps you already
> had one loaded?), LDAP, or Kerberos. If you are able to access the NiFi UI
> over HTTPS without presenting some authentication, something is seriously
> broken. The warning in the browser is because the CA certificate that
> signed the server certificate (the one being presented *to* the browser
> by the application) is not trusted in the browser’s pre-installed trust
> chain. If, for example, that CA cert had been imported to the browser ahead
> of time, or if it was signed by a publicly known entity like DigiCert,
> Verisign, Comodo, etc., you would not receive a warning.
>
> For small teams, client certificates can be manageable, but if you want to
> allow multiple users to connect with minimal identity management, I
> recommend setting up an LDAP server (OpenLDAP, Microsoft ActiveDirectory,
> Apache Directory Studio, etc.) and administering users there. Then the
> users will just enter a username and password into a login field on their
> first connection to NiFi and be authenticated.
>
>
> Andy LoPresto
> alopresto@apache.org
> *alopresto.apache@gmail.com <al...@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> On Nov 29, 2016, at 11:07 AM, Ricky Saltzer <ri...@cloudera.com> wrote:
>
> Hey Andy -
>
> Thanks for the reply, I used the openssl command you provided and indeed
> the return code was *OK*. Before proceeding with the recommendation of
>
> importing the key into my OSX keychain, I would like to understand why this
> required. When using HTTPS mode in non-clustered mode, it does not require
> clients to have a special key or cert imported to their machine. Instead,
> the client is given a warning in the browser, and it's up to them to
> proceed. This UI will serve as an endpoint to several users, and I would
> really like to avoid the cumbersomeness of having members of multiple teams
> follow instructions for importing keys just so they can access a web UI.
>
> On Tue, Nov 29, 2016 at 1:55 PM, Andy LoPresto <al...@apache.org>
> wrote:
>
> Hi Ricky,
>
> The ERR_CONNECTION_CLOSED is likely because you are not sending a client
> certificate on the HTTP request. By default, a secured cluster requires
> client certificate authentication unless LDAP or Kerberos are configured as
> identity providers [1]. The TLS Toolkit provides a quick way to generate a
> valid client certificate which you can load into your browser in order to
> access the site.
>
> First, verify the cluster is running and accepting incoming connections
> (we’re going to cheat here just to be quick about it; disclaimer that this
> is not the RIGHT way to do this):
>
> In the directory where you ran the toolkit, you noted there was a
> “nifi-cert.pem” and “nifi-cert.key” file. The pem file is the PEM-encoded
> public certificate of the NiFi CA cert that was generated by the toolkit,
> and the key file is the PEM-encoded private key. Because this is the same
> certificate that signed the NiFi server key loaded in the keystore, it is
> also loaded into the truststore. That means the server will accept an
> incoming connection with any certificate signed by the CA cert.
> Coincidentally, the CA cert is self-signed, so…
>
> $ openssl s_client -connect <host:port> -debug -state -cert nifi-cert.pem
> -key nifi-key.key -CAfile nifi-cert.pem
>
> That command will attempt to negotiate a TLS connection to your server by
> presenting the CA cert and key as the client. Again, not semantically
> correct, but technically will work. You’ll get a long output, but it
> should end in a section like this:
>
> ---
> New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384
> Server public key is 2048 bit
> Secure Renegotiation IS supported
> Compression: NONE
> Expansion: NONE
> No ALPN negotiated
> SSL-Session:
> Protocol : TLSv1.2
> Cipher : ECDHE-RSA-AES256-SHA384
> Session-ID: 583DCD...9E828C
> Session-ID-ctx:
> Master-Key: 5477C0...A51E85
> Key-Arg : None
> PSK identity: None
> PSK identity hint: None
> SRP username: None
> Start Time: 1480445265
> Timeout : 300 (sec)
> Verify return code: 0 (ok)
> ---
>
> The important part is the last line — you want the *Verify return code* to
>
> be 0 for success. Once you have verified this, run the TLS toolkit again to
> generate a valid client certificate:
>
> $ ./bin/tls-toolkit.sh standalone -C 'CN=Ricky Saltzer, OU=Apache NiFi'
> -B thisIsABadPassword
>
> This will generate a PKCS12 keystore (*.p12) containing your public
> certificate AND the private key, as well as an additional file (.password)
> containing the password you provided for -B.
>
> hw12203:...lkit/nifi-toolkit-assembly/target/nifi-toolkit-
> 1.1.0-bin/nifi-toolkit-1.1.0
> (master) alopresto
> 🔓 146s @ 10:50:14 $ ./bin/tls-toolkit.sh standalone -C 'CN=Ricky Saltzer,
> OU=Apache NiFi' -B password
> 2016/11/29 10:50:20 INFO [main] org.apache.nifi.toolkit.tls.standalone.
> TlsToolkitStandaloneCommandLine: No nifiPropertiesFile specified, using
> embedded one.
> 2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.
> TlsToolkitStandalone:
> Running standalone certificate generation with output directory
> ../nifi-toolkit-1.1.0
> 2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.
> TlsToolkitStandalone:
> Using existing CA certificate ../nifi-toolkit-1.1.0/nifi-cert.pem and key
> ../nifi-toolkit-1.1.0/nifi-key.key
> 2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.
> TlsToolkitStandalone:
> No hostnames specified, not generating any host certificates or
> configuration.
> 2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.
> TlsToolkitStandalone:
> Generating new client certificate ../nifi-toolkit-1.1.0/CN=
> Ricky_Saltzer_OU=Apache_NiFi.p12
> 2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.
> TlsToolkitStandalone:
> Successfully generated client certificate ../nifi-toolkit-1.1.0/CN=
> Ricky_Saltzer_OU=Apache_NiFi.p12
> 2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.
> TlsToolkitStandalone:
> tls-toolkit standalone completed successfully
> hw12203:...lkit/nifi-toolkit-assembly/target/nifi-toolkit-
> 1.1.0-bin/nifi-toolkit-1.1.0
> (master) alopresto
> 🔓 7s @ 10:50:22 $ ll
> total 136
> drwxr-xr-x 20 alopresto staff 680B Nov 29 10:50 ./
> drwxr-xr-x 4 alopresto staff 136B Nov 28 17:16 ../
> -rw------- 1 alopresto staff 3.4K Nov 29 10:50
> CN=Ricky_Saltzer_OU=Apache_NiFi.p12
> -rw------- 1 alopresto staff 8B Nov 29 10:50
> CN=Ricky_Saltzer_OU=Apache_NiFi.password
> ...
> -rw------- 1 alopresto staff 1.2K Nov 28 16:31 nifi-cert.pem
> -rw------- 1 alopresto staff 1.6K Nov 28 16:31 nifi-key.key
> hw12203:...lkit/nifi-toolkit-assembly/target/nifi-toolkit-
> 1.1.0-bin/nifi-toolkit-1.1.0
> (master) alopresto
> 🔓 4s @ 10:50:27 $
>
> You can double-click on the *.p12 file to open it in your default handler
> — on OS X for example, this is Keychain Access. You can also manually load
> it into Firefox to keep it isolated from your system certificates.
>
> Now when you visit the UI through the browser, you should receive a prompt
> for which certificate to present, and select this entry from the presented
> list.
>
> If you get a NiFi error message in the browser that you do not have
> sufficient access, remember to update conf/authorizers.xml with an Initial
> Admin Identity, which should match EXACTLY the DN of this certificate —
> “CN=Ricky Saltzer, OU=Apache NiFi”. You can copy it from the failed
> authentication message in logs/nifi-user.log to be sure.
>
> Andy LoPresto
> alopresto@apache.org
> *alopresto.apache@gmail.com <al...@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> On Nov 29, 2016, at 10:27 AM, Ricky Saltzer <ri...@cloudera.com> wrote:
>
> Hi Andy -
>
> Thanks again for the help, sorry I've been a little preoccupied with some
> other tasks. I took your advice and used the tls-toolkit and things are
> looking a lot better. I generated a keystore/truststore for both of my
> nodes. It also generated a pem and key file. I was able to get NiFi to
> start in clustered mode (horray)! The only issue I'm seeing now is that I
> can't actually connect to the web UI using HTTPS, as it immediately throws
> a "ERR_CONNECTION_CLOSED" message in my browser.
>
> Judging by the logs, NiFi is seemingly running. A leader election took
> place and the node is heart beating to itself. Am I missing a step where
> I'm supposed to be doing something with the pem/key files?
>
> Thanks again,
> Ricky
>
> On Tue, Nov 8, 2016 at 8:22 PM, Andy LoPresto <al...@apache.org>
> wrote:
>
> Hi Ricky,
>
> Sorry for the delay in my response. I see a couple things which could be
> causing an issue:
>
> 1. You do not specify a hostname for the NiFi node (nifi.web.https.host=)
> which you should do. I have not set up a one-node cluster before, but my
> suspicion is that this field needs to be populated, and the value needs to
> match the CN for the issued server certificate. This value does not need to
> be unique (i.e. it could be nifi.cloudera.com or localhost and you could
> run multiple instances on the same machine, identified by the same
> certificate and different ports), but I think it has to be populated.
> 1.a. While you have nifi.security.needClientAuth=false set in both
> cluster and standalone mode, I am not sure if this allows for the
> truststore to be missing completely. I would have to explore this further.
> There is a current Jira for clarifying cluster, UI/API, and S2S TLS
> configurations [1]. You can try using the default JRE truststore, located
> at “$JAVA_HOME/jre/lib/security/cacerts” with password “changeit”.
> 2. The certificate you have put into the keystore appears to be a
> certificate identifying you (Ricky the person) rather than a server entity.
> You should create a certificate identifying the server itself, so the CN is
> the FQDN as mentioned above. Then import into (or generate directly inside)
> the keystore. You can also use the provided NiFi TLS Toolkit to automate
> this entire process [2].
> Certificate chain
> 0 s:/C=CA/ST=CA/L=Palo Alto/O=Cloudera/OU=EDH Team, Cloudera/CN=Ricky
> Saltzer
> i:/C=CA/ST=CA/L=Palo Alto/O=Cloudera/OU=EDH Team, Cloudera/CN=Ricky
> Saltzercompare
> to an example connection to google.com:443:
>
> Certificate chain
> 0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/*CN=*.google.com
> <http://google.com>*
> i:/C=US/O=Google Inc/CN=Google Internet Authority G2
> 1 s:/C=US/O=Google Inc/CN=Google Internet Authority G2
> i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
> 2 s:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
> i:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority
>
> 3. It appears the certificate you are using is expired. If you look at the
> output of the OpenSSL command you ran, you can see that the result code was
> *10*, not *0* as would be in the case of a successful connection (I
> understand that it looks successful because it negotiates a key and
> encrypts the channel).
> New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384
> Server public key is 2048 bit
> Secure Renegotiation IS supported
> Compression: NONE
> Expansion: NONE
> SSL-Session:
> Protocol : TLSv1.2
> Cipher : ECDHE-RSA-AES256-SHA384
> Session-ID: 581F5A5...41153B
> Session-ID-ctx:
> Master-Key: CEEED2...944C30
> Key-Arg : None
> PSK identity: None
> PSK identity hint: None
> SRP username: None
> Start Time: 1478449748
> Timeout : 300 (sec)
> Verify return code: *10* *(certificate has expired)*
>
> ---
> HTTP/1.1 400 No URI
> Content-Length: 0
> Connection: close
> Server: Jetty(9.3.9.v20160517)
> I rebuilt your certificate from the Base64-encoded version in the output,
> and it appears it expired on October 25, 2016. You can verify this by
> copying the section between "-----BEGIN CERTIFICATE-----“ and "-----END
> CERTIFICATE-----“ (including these markers) into a text file and naming it
> “ricky.pem” and then running the following command:
>
> hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
> 🔓 11s @ 17:09:30 $ more ricky.pem
> -----BEGIN CERTIFICATE-----
> MIIDizCCAnOgAwIBAgIEHPLwBzANBgkqhkiG9w0BAQsFADB2MQswCQYDVQQGEwJD
> ...
> 45GrxLz7n/ZXZk8q9aXcrSGaj+HHCyrO0/9NYtMNP2V5wpYcBRiHO9f3sHKgnzU=
> -----END CERTIFICATE-----
> hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
> 🔓 5s @ 17:09:35 $ openssl x509 -in ricky.pem -text -noout
> Certificate:
> Data:
> Version: 3 (0x2)
> Serial Number: 485683207 (0x1cf2f007)
> Signature Algorithm: sha256WithRSAEncryption
> Issuer: C=CA, ST=CA, L=Palo Alto, O=Cloudera, OU=EDH Team,
> Cloudera, CN=Ricky Saltzer
> Validity
> Not Before: Jul 27 17:15:46 2016 GMT
> *Not After : Oct 25 17:15:46 2016 GMT*
>
> Subject: C=CA, ST=CA, L=Palo Alto, O=Cloudera, OU=EDH Team,
> Cloudera, CN=Ricky Saltzer
> Subject Public Key Info:
> Public Key Algorithm: rsaEncryption
> Public-Key: (2048 bit)
> Modulus:
> 00:f0:4c:fe:7e:af:69:89:f0:4b:a0:e0:df:e8:8c:
> 46:6e:d7:17:4d:17:eb:ee:89:25:ab:5b:35:6a:b2:
> 1e:07:84:95:5f:39:1e:c3:d9:5c:d4:54:09:0c:1a:
> 2a:60:df:9f:ce:7d:87:00:d7:31:23:37:62:8c:02:
> c9:98:a3:d4:f5:3e:71:4e:1d:fb:bb:5b:86:35:af:
> ce:6d:9d:ee:fb:65:d1:01:22:e3:f8:df:c0:81:d7:
> fe:fd:f3:c8:48:c2:bd:23:e2:44:1e:8d:48:5f:81:
> ed:10:62:77:71:3e:15:82:5a:c2:49:70:bb:ef:95:
> d9:ff:f2:37:6c:06:63:fc:3c:98:a3:c9:49:3a:1d:
> b8:37:e3:0b:50:0f:23:5f:26:32:19:c8:59:4e:5a:
> 09:5a:87:7f:b6:40:b4:37:18:20:74:fb:d2:f8:e9:
> 18:e8:d7:23:fa:83:be:2a:c3:33:8d:17:a8:b3:7c:
> 52:03:03:7c:24:80:c5:9c:a5:0d:21:3d:57:7c:d4:
> 0f:78:00:fd:be:69:dd:d8:fe:07:a0:09:77:81:b7:
> bf:fd:0f:bd:8c:9a:35:42:4f:89:8e:31:fd:00:22:
> 16:bb:76:0b:51:b0:c4:81:64:69:f7:c4:5d:37:1e:
> 79:b7:08:d2:5a:6a:a7:43:98:d3:81:a9:08:00:57:
> 2b:c5
> Exponent: 65537 (0x10001)
> X509v3 extensions:
> X509v3 Subject Key Identifier:
> 1C:DD:E1:16:5F:41:CC:7C:69:27:
> E3:B1:6B:78:EF:FA:16:DA:1F:6F
> Signature Algorithm: sha256WithRSAEncryption
> da:5a:31:4b:3b:a6:bf:84:8d:6a:56:2a:82:88:2c:e0:95:49:
> 30:f2:3a:31:ed:b5:bc:5b:92:e6:ad:b3:5b:9f:7a:71:bf:62:
> 0f:38:27:4c:71:02:a1:2f:7f:fa:c0:46:da:78:64:36:35:34:
> 1d:91:b3:f8:e2:23:22:1a:e8:e1:14:2b:15:2c:aa:36:71:39:
> 05:d7:c0:2a:e0:cd:4d:f9:72:07:03:b9:f2:c8:6d:59:fb:c9:
> 25:49:ae:3f:a2:c2:fe:d0:54:1e:45:13:8d:95:05:86:8b:a4:
> c4:7c:df:e1:7a:35:cc:48:e3:cd:20:87:c5:50:a7:43:d2:66:
> 94:81:89:b1:a2:66:20:44:ee:cb:ea:08:c9:7c:aa:61:c7:24:
> 5f:71:ca:6e:c0:a8:15:1f:4a:d3:79:e3:19:fd:7f:86:96:1a:
> 4d:db:9d:38:6f:b3:16:fd:39:68:cf:1c:57:86:26:93:6d:0e:
> 76:6b:aa:b8:9a:e7:c4:05:c6:c5:ae:5a:e1:a1:2a:ad:aa:e3:
> 5b:9a:f1:70:a5:f0:54:76:b3:0e:00:e3:91:ab:c4:bc:fb:9f:
> f6:57:66:4f:2a:f5:a5:dc:ad:21:9a:8f:e1:c7:0b:2a:ce:d3:
> ff:4d:62:d3:0d:3f:65:79:c2:96:1c:05:18:87:3b:d7:f7:b0:
> 72:a0:9f:35
> hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
> 🔓 11s @ 17:09:47 $
>
> I would recommend using the TLS Toolkit to generate a valid CA, server
> certificate, client certificate, and keystore & truststore (seriously, one
> command does all of this) and then re-trying. From this stable baseline, I
> think I’ll be able to better help iron out the cluster/standalone issues.
>
> [1] https://issues.apache.org/jira/browse/NIFI-1990
> [2] https://nifi.apache.org/docs/nifi-docs/html/
> administration-guide.html#tls-generation-toolkit
>
>
> Andy LoPresto
> alopresto@apache.org
> *alopresto.apache@gmail.com <al...@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> On Nov 8, 2016, at 1:31 AM, Andre <an...@fucs.org> wrote:
>
> Rick,
>
> Can you confirm the certificate has a chain of trust with the default JDK
> trusted certs? (i.e. trusted by the JVM)
>
> Cheers
>
>
> On Mon, Nov 7, 2016 at 3:38 AM, Ricky Saltzer <ri...@cloudera.com> wrote:
>
> Hey Andy -
>
> Thanks again for the help.
>
> The error message seems indicative that it doesn't seem to properly read
> the keystore file. One thing to note, if I point the nifi properties to a
> bogus keystore location, then it actually throws a FileNotFound exception.
> This is really odd behavior, because as I mentioned I'm able to start it in
> standalone mode using the correct keystore location, just as I try to do in
> clustered mode.
>
> I've attached both the clustered [1] nifi.properties, which doesn't work,
> and the standalone [2] which does work. . I restored it to a more basic
> configuration without the encrypted configuration, but with SSL still
> enabled. I also added a diff [3] of both the standalone and clustered
> properties file. Note that I I only have NiFi configured to use the
> keystore and not a truststore. I've redacted a few of the values in the
> property files, but be assured that the keystore is most definitely valid
> and is readable / locatable, as starting in standalone works just fine.
>
> I ran the SSL command [4] you gave me, minus the three PEM file arguments
> as I don't have any of those on hand. I hope that is fine. The output still
> looks good.
>
> [1] https://gist.github.com/rickysaltzer/712aa6586592fe6628db2d57cec7a562
> [2] https://gist.github.com/rickysaltzer/fe11c8233e4434eacedd7fd0a083d950
> [3] https://gist.github.com/rickysaltzer/d715c7451eb554a54f14ec6e64da8558
> [4] https://gist.github.com/rickysaltzer/5d7cdeff8868bfc1f47010189735411a
>
>
>
>
> On Fri, Nov 4, 2016 at 7:48 PM, Andy LoPresto <al...@apache.org>
> wrote:
>
> Hi Ricky,
>
> Sorry, should have noted that the debug output goes to
>
> nifi-bootstrap.log,
>
> so thanks Mark for jumping in to help there.
>
> If you look at the top of that log, you’ll note that there is no keystore
> file provided and the truststore loaded is the default JRE cacerts
> truststore. Can you please provide your nifi.properties file in a Gist,
>
> **taking
>
> care to redact any sensitive values** like keystore/truststore passwords,
> although I think from looking at your log output, you are taking
>
> advantage
>
> of the encrypted configuration feature, so even viewing the encrypted
> values should be ok. Could you also please provide the directory listing
> where the keystore and truststore are located including the permissions
>
> and
>
> ownership information?
>
> There may be a bug in the logic between cluster and standalone mode, but
>
> I
>
> haven’t encountered this behavior before. If you can start NiFi in
> standalone mode, could you please provide the output of the following
> command run from the terminal? It will simulate an HTTPS connection to
>
> the
>
> server and verify the key and certificate presented by NiFi.
>
> * host — the NiFi hostname
> * port — the port NiFi is running on
> * path_to_your_cert.pem — the public key certificate identifying the
> client/user (i.e. what you load into your browser to authenticate)
> * path_to_your_key.key — the private key identifying the client/user
> * path_to_your_CA_cert.pem — the public key certificate identifying the
>
> CA
>
> which signed your NiFi server certificate (if self-signed, provide that
> certificate)
>
> $ openssl s_client -connect <host:port> -debug -state -cert
> <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile
> <path_to_your_CA_cert.pem>
>
> Andy LoPresto
> alopresto@apache.org
> *alopresto.apache@gmail.com <al...@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> On Nov 4, 2016, at 11:21 AM, Ricky Saltzer <ri...@cloudera.com> wrote:
>
> Hey guys -
>
> I went ahead and uploaded the boostrap log. I took a look at it and it
> seems to be the same error [1]
>
> [1]:
> https://gist.githubusercontent.com/rickysaltzer/
> b156594f92066873f80928eddf84e7bb/raw/4d0e018038b332f4fdf14644910dfa
> 9e70c57e49/gistfile1.txt
>
> On Fri, Nov 4, 2016 at 2:14 PM, Mark Payne <ma...@hotmail.com> wrote:
>
> Hey Ricky,
>
> When you enable debug logging for SSL, it writes to StdErr (or StdOut?)
>
> so
>
> it will end up in your logs/nifi-bootstrap.log instead of nifi-app.log.
> Can you give that a look?
>
> Thanks
> -Mark
>
> On Nov 4, 2016, at 2:07 PM, Ricky Saltzer <ri...@cloudera.com> wrote:
>
> Hey Andy -
>
> Thanks for the response. I'm currently just trying to get one node in
> clustered mode before adding a second. The keystore is stored locally and
> I've confirmed it's readable, as it was able to start once I took it out
>
> of
>
> clustered mode. I added that line to the bootstrap.conf, but I don't
> believe any additional logging was produced in regards to troubleshooting
> this problem. Just in case, I've attached the entire log [1].
>
> [1]:
> https://gist.githubusercontent.com/rickysaltzer/
>
> ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0
> fedabc88bb/gistfile1.txt <https://gist.githubusercontent.com/
>
> rickysaltzer/
>
> ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0
> fedabc88bb/gistfile1.txt>
>
>
> On Wed, Nov 2, 2016 at 7:08 PM, Andy LoPresto <alopresto@apache.org
>
> <mailto:alopresto@apache.org <al...@apache.org> <al...@apache.org>
> <al...@apache.org>>>
> wrote:
>
>
> Hi Ricky,
>
> Sorry to hear you are having this issue. Is the keystore available on
>
> all
>
> nodes of the cluster? It appears from the log message that the keystore
>
> is
>
> not found during startup. To further debug, you can add the following
>
> line
>
> in bootstrap.conf to provide additional logging:
>
> java.arg.15=-Djavax.net.debug=ssl,handshake
>
> Andy LoPresto
> alopresto@apache.org <mailto:alopresto@apache.org <al...@apache.org> <
> alopresto@apache.org> <
> alopresto@apache.org>>
> *alopresto.apache@gmail.com <mailto:alopresto.apache@gmail.com
> <al...@gmail.com>
> <al...@gmail.com>
> <al...@gmail.com>> <
>
> alopresto.apache@gmail.com <mailto:alopresto.apache@gmail.com
> <al...@gmail.com>
>
> <al...@gmail.com>
>
> <al...@gmail.com>>>*
>
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> On Nov 2, 2016, at 2:25 PM, Ricky Saltzer <ri...@cloudera.com> wrote:
>
> Hey all -
>
> I'm using NiFi 1.0 and I'm having an issue using secure mode with a
>
> local
>
> key store while in clustered mode. If I set the node in clustered mode,
>
> and
>
> also provide a valid keystore, I receive a KeyStoreException [1]. If I
>
> set
>
> the configuration to not use clustered mode, NiFi will start up fine
>
> with
>
> the provided key store. Am I supposed to be storing this key store in
> Zookeeper somewhere?
>
>
> [1]
>
>
> Caused by: java.security.KeyStoreException: not found
>
>
> at java.security.KeyStore.getInstance(KeyStore.java:839)
> ~[na:1.8.0_11]
>
> at
> org.apache.nifi.io.socket.SSLContextFactory.<init>(
> SSLContextFactory.java:61)
> ~[nifi-socket-utils-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
> ryBean.getObject(ServerSocketConfigurationFactoryBean.java:45)
> ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
> ryBean.getObject(ServerSocketConfigurationFactoryBean.java:30)
> ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
>
> at
> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.
> doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
> ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
>
> ... 69 common frames omitted
>
> Caused by: java.security.NoSuchAlgorithmException: KeyStore not
>
> available
>
>
> at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
> ~[na:1.8.0_11]
>
> at java.security.Security.getImpl(Security.java:695)
>
> ~[na:1.8.0_11]
>
>
> at java.security.KeyStore.getInstance(KeyStore.java:836)
> ~[na:1.8.0_11]
>
> ... 73 common frames omitted
>
>
>
>
>
> --
> Ricky Saltzer
> http://www.cloudera.com <http://www.cloudera.com/>
>
>
>
>
>
> --
> Ricky Saltzer
> http://www.cloudera.com
>
>
>
>
>
> --
> Ricky Saltzer
> http://www.cloudera.com
>
>
>
>
>
> --
> Ricky Saltzer
> http://www.cloudera.com
>
>
>
>
>
> --
> Ricky Saltzer
> http://www.cloudera.com
>
>
>
--
Ricky Saltzer
http://www.cloudera.com
Re: Secure Cluster Mode Issues
Posted by Andy LoPresto <al...@apache.org>.
Ricky,
When using HTTPS in non-cluster mode, NiFi still requires user authentication — this can be either client certificate (perhaps you already had one loaded?), LDAP, or Kerberos. If you are able to access the NiFi UI over HTTPS without presenting some authentication, something is seriously broken. The warning in the browser is because the CA certificate that signed the server certificate (the one being presented to the browser by the application) is not trusted in the browser’s pre-installed trust chain. If, for example, that CA cert had been imported to the browser ahead of time, or if it was signed by a publicly known entity like DigiCert, Verisign, Comodo, etc., you would not receive a warning.
For small teams, client certificates can be manageable, but if you want to allow multiple users to connect with minimal identity management, I recommend setting up an LDAP server (OpenLDAP, Microsoft ActiveDirectory, Apache Directory Studio, etc.) and administering users there. Then the users will just enter a username and password into a login field on their first connection to NiFi and be authenticated.
Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Nov 29, 2016, at 11:07 AM, Ricky Saltzer <ri...@cloudera.com> wrote:
>
> Hey Andy -
>
> Thanks for the reply, I used the openssl command you provided and indeed
> the return code was *OK*. Before proceeding with the recommendation of
> importing the key into my OSX keychain, I would like to understand why this
> required. When using HTTPS mode in non-clustered mode, it does not require
> clients to have a special key or cert imported to their machine. Instead,
> the client is given a warning in the browser, and it's up to them to
> proceed. This UI will serve as an endpoint to several users, and I would
> really like to avoid the cumbersomeness of having members of multiple teams
> follow instructions for importing keys just so they can access a web UI.
>
> On Tue, Nov 29, 2016 at 1:55 PM, Andy LoPresto <alopresto@apache.org <ma...@apache.org>> wrote:
>
>> Hi Ricky,
>>
>> The ERR_CONNECTION_CLOSED is likely because you are not sending a client
>> certificate on the HTTP request. By default, a secured cluster requires
>> client certificate authentication unless LDAP or Kerberos are configured as
>> identity providers [1]. The TLS Toolkit provides a quick way to generate a
>> valid client certificate which you can load into your browser in order to
>> access the site.
>>
>> First, verify the cluster is running and accepting incoming connections
>> (we’re going to cheat here just to be quick about it; disclaimer that this
>> is not the RIGHT way to do this):
>>
>> In the directory where you ran the toolkit, you noted there was a
>> “nifi-cert.pem” and “nifi-cert.key” file. The pem file is the PEM-encoded
>> public certificate of the NiFi CA cert that was generated by the toolkit,
>> and the key file is the PEM-encoded private key. Because this is the same
>> certificate that signed the NiFi server key loaded in the keystore, it is
>> also loaded into the truststore. That means the server will accept an
>> incoming connection with any certificate signed by the CA cert.
>> Coincidentally, the CA cert is self-signed, so…
>>
>> $ openssl s_client -connect <host:port> -debug -state -cert nifi-cert.pem
>> -key nifi-key.key -CAfile nifi-cert.pem
>>
>> That command will attempt to negotiate a TLS connection to your server by
>> presenting the CA cert and key as the client. Again, not semantically
>> correct, but technically will work. You’ll get a long output, but it
>> should end in a section like this:
>>
>> ---
>> New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384
>> Server public key is 2048 bit
>> Secure Renegotiation IS supported
>> Compression: NONE
>> Expansion: NONE
>> No ALPN negotiated
>> SSL-Session:
>> Protocol : TLSv1.2
>> Cipher : ECDHE-RSA-AES256-SHA384
>> Session-ID: 583DCD...9E828C
>> Session-ID-ctx:
>> Master-Key: 5477C0...A51E85
>> Key-Arg : None
>> PSK identity: None
>> PSK identity hint: None
>> SRP username: None
>> Start Time: 1480445265
>> Timeout : 300 (sec)
>> Verify return code: 0 (ok)
>> ---
>>
>> The important part is the last line — you want the *Verify return code* to
>> be 0 for success. Once you have verified this, run the TLS toolkit again to
>> generate a valid client certificate:
>>
>> $ ./bin/tls-toolkit.sh standalone -C 'CN=Ricky Saltzer, OU=Apache NiFi'
>> -B thisIsABadPassword
>>
>> This will generate a PKCS12 keystore (*.p12) containing your public
>> certificate AND the private key, as well as an additional file (.password)
>> containing the password you provided for -B.
>>
>> hw12203:...lkit/nifi-toolkit-assembly/target/nifi-toolkit-1.1.0-bin/nifi-toolkit-1.1.0
>> (master) alopresto
>> 🔓 146s @ 10:50:14 $ ./bin/tls-toolkit.sh standalone -C 'CN=Ricky Saltzer,
>> OU=Apache NiFi' -B password
>> 2016/11/29 10:50:20 INFO [main] org.apache.nifi.toolkit.tls.standalone.
>> TlsToolkitStandaloneCommandLine: No nifiPropertiesFile specified, using
>> embedded one.
>> 2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone:
>> Running standalone certificate generation with output directory
>> ../nifi-toolkit-1.1.0
>> 2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone:
>> Using existing CA certificate ../nifi-toolkit-1.1.0/nifi-cert.pem and key
>> ../nifi-toolkit-1.1.0/nifi-key.key
>> 2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone:
>> No hostnames specified, not generating any host certificates or
>> configuration.
>> 2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone:
>> Generating new client certificate ../nifi-toolkit-1.1.0/CN=
>> Ricky_Saltzer_OU=Apache_NiFi.p12
>> 2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone:
>> Successfully generated client certificate ../nifi-toolkit-1.1.0/CN=
>> Ricky_Saltzer_OU=Apache_NiFi.p12
>> 2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone:
>> tls-toolkit standalone completed successfully
>> hw12203:...lkit/nifi-toolkit-assembly/target/nifi-toolkit-1.1.0-bin/nifi-toolkit-1.1.0
>> (master) alopresto
>> 🔓 7s @ 10:50:22 $ ll
>> total 136
>> drwxr-xr-x 20 alopresto staff 680B Nov 29 10:50 ./
>> drwxr-xr-x 4 alopresto staff 136B Nov 28 17:16 ../
>> -rw------- 1 alopresto staff 3.4K Nov 29 10:50
>> CN=Ricky_Saltzer_OU=Apache_NiFi.p12
>> -rw------- 1 alopresto staff 8B Nov 29 10:50
>> CN=Ricky_Saltzer_OU=Apache_NiFi.password
>> ...
>> -rw------- 1 alopresto staff 1.2K Nov 28 16:31 nifi-cert.pem
>> -rw------- 1 alopresto staff 1.6K Nov 28 16:31 nifi-key.key
>> hw12203:...lkit/nifi-toolkit-assembly/target/nifi-toolkit-1.1.0-bin/nifi-toolkit-1.1.0
>> (master) alopresto
>> 🔓 4s @ 10:50:27 $
>>
>> You can double-click on the *.p12 file to open it in your default handler
>> — on OS X for example, this is Keychain Access. You can also manually load
>> it into Firefox to keep it isolated from your system certificates.
>>
>> Now when you visit the UI through the browser, you should receive a prompt
>> for which certificate to present, and select this entry from the presented
>> list.
>>
>> If you get a NiFi error message in the browser that you do not have
>> sufficient access, remember to update conf/authorizers.xml with an Initial
>> Admin Identity, which should match EXACTLY the DN of this certificate —
>> “CN=Ricky Saltzer, OU=Apache NiFi”. You can copy it from the failed
>> authentication message in logs/nifi-user.log to be sure.
>>
>> Andy LoPresto
>> alopresto@apache.org
>> *alopresto.apache@gmail.com <ma...@gmail.com> <alopresto.apache@gmail.com <ma...@gmail.com>>*
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>>
>> On Nov 29, 2016, at 10:27 AM, Ricky Saltzer <ricky@cloudera.com <ma...@cloudera.com>> wrote:
>>
>> Hi Andy -
>>
>> Thanks again for the help, sorry I've been a little preoccupied with some
>> other tasks. I took your advice and used the tls-toolkit and things are
>> looking a lot better. I generated a keystore/truststore for both of my
>> nodes. It also generated a pem and key file. I was able to get NiFi to
>> start in clustered mode (horray)! The only issue I'm seeing now is that I
>> can't actually connect to the web UI using HTTPS, as it immediately throws
>> a "ERR_CONNECTION_CLOSED" message in my browser.
>>
>> Judging by the logs, NiFi is seemingly running. A leader election took
>> place and the node is heart beating to itself. Am I missing a step where
>> I'm supposed to be doing something with the pem/key files?
>>
>> Thanks again,
>> Ricky
>>
>> On Tue, Nov 8, 2016 at 8:22 PM, Andy LoPresto <alopresto@apache.org <ma...@apache.org>>
>> wrote:
>>
>> Hi Ricky,
>>
>> Sorry for the delay in my response. I see a couple things which could be
>> causing an issue:
>>
>> 1. You do not specify a hostname for the NiFi node (nifi.web.https.host=)
>> which you should do. I have not set up a one-node cluster before, but my
>> suspicion is that this field needs to be populated, and the value needs to
>> match the CN for the issued server certificate. This value does not need to
>> be unique (i.e. it could be nifi.cloudera.com <http://nifi.cloudera.com/> or localhost and you could
>> run multiple instances on the same machine, identified by the same
>> certificate and different ports), but I think it has to be populated.
>> 1.a. While you have nifi.security.needClientAuth=false set in both
>> cluster and standalone mode, I am not sure if this allows for the
>> truststore to be missing completely. I would have to explore this further.
>> There is a current Jira for clarifying cluster, UI/API, and S2S TLS
>> configurations [1]. You can try using the default JRE truststore, located
>> at “$JAVA_HOME/jre/lib/security/cacerts” with password “changeit”.
>> 2. The certificate you have put into the keystore appears to be a
>> certificate identifying you (Ricky the person) rather than a server entity.
>> You should create a certificate identifying the server itself, so the CN is
>> the FQDN as mentioned above. Then import into (or generate directly inside)
>> the keystore. You can also use the provided NiFi TLS Toolkit to automate
>> this entire process [2].
>> Certificate chain
>> 0 s:/C=CA/ST=CA/L=Palo Alto/O=Cloudera/OU=EDH Team, Cloudera/CN=Ricky
>> Saltzer
>> i:/C=CA/ST=CA/L=Palo Alto/O=Cloudera/OU=EDH Team, Cloudera/CN=Ricky
>> Saltzercompare
>> to an example connection to google.com <http://google.com/>:443:
>>
>> Certificate chain
>> 0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/*CN=*.google.com <http://google.com/>
>> <http://google.com <http://google.com/>>*
>> i:/C=US/O=Google Inc/CN=Google Internet Authority G2
>> 1 s:/C=US/O=Google Inc/CN=Google Internet Authority G2
>> i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
>> 2 s:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
>> i:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority
>>
>> 3. It appears the certificate you are using is expired. If you look at the
>> output of the OpenSSL command you ran, you can see that the result code was
>> *10*, not *0* as would be in the case of a successful connection (I
>> understand that it looks successful because it negotiates a key and
>> encrypts the channel).
>> New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384
>> Server public key is 2048 bit
>> Secure Renegotiation IS supported
>> Compression: NONE
>> Expansion: NONE
>> SSL-Session:
>> Protocol : TLSv1.2
>> Cipher : ECDHE-RSA-AES256-SHA384
>> Session-ID: 581F5A5...41153B
>> Session-ID-ctx:
>> Master-Key: CEEED2...944C30
>> Key-Arg : None
>> PSK identity: None
>> PSK identity hint: None
>> SRP username: None
>> Start Time: 1478449748
>> Timeout : 300 (sec)
>> Verify return code: *10* *(certificate has expired)*
>>
>> ---
>> HTTP/1.1 400 No URI
>> Content-Length: 0
>> Connection: close
>> Server: Jetty(9.3.9.v20160517)
>> I rebuilt your certificate from the Base64-encoded version in the output,
>> and it appears it expired on October 25, 2016. You can verify this by
>> copying the section between "-----BEGIN CERTIFICATE-----“ and "-----END
>> CERTIFICATE-----“ (including these markers) into a text file and naming it
>> “ricky.pem” and then running the following command:
>>
>> hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
>> 🔓 11s @ 17:09:30 $ more ricky.pem
>> -----BEGIN CERTIFICATE-----
>> MIIDizCCAnOgAwIBAgIEHPLwBzANBgkqhkiG9w0BAQsFADB2MQswCQYDVQQGEwJD
>> ...
>> 45GrxLz7n/ZXZk8q9aXcrSGaj+HHCyrO0/9NYtMNP2V5wpYcBRiHO9f3sHKgnzU=
>> -----END CERTIFICATE-----
>> hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
>> 🔓 5s @ 17:09:35 $ openssl x509 -in ricky.pem -text -noout
>> Certificate:
>> Data:
>> Version: 3 (0x2)
>> Serial Number: 485683207 (0x1cf2f007)
>> Signature Algorithm: sha256WithRSAEncryption
>> Issuer: C=CA, ST=CA, L=Palo Alto, O=Cloudera, OU=EDH Team,
>> Cloudera, CN=Ricky Saltzer
>> Validity
>> Not Before: Jul 27 17:15:46 2016 GMT
>> *Not After : Oct 25 17:15:46 2016 GMT*
>>
>> Subject: C=CA, ST=CA, L=Palo Alto, O=Cloudera, OU=EDH Team,
>> Cloudera, CN=Ricky Saltzer
>> Subject Public Key Info:
>> Public Key Algorithm: rsaEncryption
>> Public-Key: (2048 bit)
>> Modulus:
>> 00:f0:4c:fe:7e:af:69:89:f0:4b:a0:e0:df:e8:8c:
>> 46:6e:d7:17:4d:17:eb:ee:89:25:ab:5b:35:6a:b2:
>> 1e:07:84:95:5f:39:1e:c3:d9:5c:d4:54:09:0c:1a:
>> 2a:60:df:9f:ce:7d:87:00:d7:31:23:37:62:8c:02:
>> c9:98:a3:d4:f5:3e:71:4e:1d:fb:bb:5b:86:35:af:
>> ce:6d:9d:ee:fb:65:d1:01:22:e3:f8:df:c0:81:d7:
>> fe:fd:f3:c8:48:c2:bd:23:e2:44:1e:8d:48:5f:81:
>> ed:10:62:77:71:3e:15:82:5a:c2:49:70:bb:ef:95:
>> d9:ff:f2:37:6c:06:63:fc:3c:98:a3:c9:49:3a:1d:
>> b8:37:e3:0b:50:0f:23:5f:26:32:19:c8:59:4e:5a:
>> 09:5a:87:7f:b6:40:b4:37:18:20:74:fb:d2:f8:e9:
>> 18:e8:d7:23:fa:83:be:2a:c3:33:8d:17:a8:b3:7c:
>> 52:03:03:7c:24:80:c5:9c:a5:0d:21:3d:57:7c:d4:
>> 0f:78:00:fd:be:69:dd:d8:fe:07:a0:09:77:81:b7:
>> bf:fd:0f:bd:8c:9a:35:42:4f:89:8e:31:fd:00:22:
>> 16:bb:76:0b:51:b0:c4:81:64:69:f7:c4:5d:37:1e:
>> 79:b7:08:d2:5a:6a:a7:43:98:d3:81:a9:08:00:57:
>> 2b:c5
>> Exponent: 65537 (0x10001)
>> X509v3 extensions:
>> X509v3 Subject Key Identifier:
>> 1C:DD:E1:16:5F:41:CC:7C:69:27:
>> E3:B1:6B:78:EF:FA:16:DA:1F:6F
>> Signature Algorithm: sha256WithRSAEncryption
>> da:5a:31:4b:3b:a6:bf:84:8d:6a:56:2a:82:88:2c:e0:95:49:
>> 30:f2:3a:31:ed:b5:bc:5b:92:e6:ad:b3:5b:9f:7a:71:bf:62:
>> 0f:38:27:4c:71:02:a1:2f:7f:fa:c0:46:da:78:64:36:35:34:
>> 1d:91:b3:f8:e2:23:22:1a:e8:e1:14:2b:15:2c:aa:36:71:39:
>> 05:d7:c0:2a:e0:cd:4d:f9:72:07:03:b9:f2:c8:6d:59:fb:c9:
>> 25:49:ae:3f:a2:c2:fe:d0:54:1e:45:13:8d:95:05:86:8b:a4:
>> c4:7c:df:e1:7a:35:cc:48:e3:cd:20:87:c5:50:a7:43:d2:66:
>> 94:81:89:b1:a2:66:20:44:ee:cb:ea:08:c9:7c:aa:61:c7:24:
>> 5f:71:ca:6e:c0:a8:15:1f:4a:d3:79:e3:19:fd:7f:86:96:1a:
>> 4d:db:9d:38:6f:b3:16:fd:39:68:cf:1c:57:86:26:93:6d:0e:
>> 76:6b:aa:b8:9a:e7:c4:05:c6:c5:ae:5a:e1:a1:2a:ad:aa:e3:
>> 5b:9a:f1:70:a5:f0:54:76:b3:0e:00:e3:91:ab:c4:bc:fb:9f:
>> f6:57:66:4f:2a:f5:a5:dc:ad:21:9a:8f:e1:c7:0b:2a:ce:d3:
>> ff:4d:62:d3:0d:3f:65:79:c2:96:1c:05:18:87:3b:d7:f7:b0:
>> 72:a0:9f:35
>> hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
>> 🔓 11s @ 17:09:47 $
>>
>> I would recommend using the TLS Toolkit to generate a valid CA, server
>> certificate, client certificate, and keystore & truststore (seriously, one
>> command does all of this) and then re-trying. From this stable baseline, I
>> think I’ll be able to better help iron out the cluster/standalone issues.
>>
>> [1] https://issues.apache.org/jira/browse/NIFI-1990 <https://issues.apache.org/jira/browse/NIFI-1990>
>> [2] https://nifi.apache.org/docs/nifi-docs/html/ <https://nifi.apache.org/docs/nifi-docs/html/>
>> administration-guide.html#tls-generation-toolkit
>>
>>
>> Andy LoPresto
>> alopresto@apache.org <ma...@apache.org>
>> *alopresto.apache@gmail.com <ma...@gmail.com> <alopresto.apache@gmail.com <ma...@gmail.com>>*
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>>
>> On Nov 8, 2016, at 1:31 AM, Andre <andre-lists@fucs.org <ma...@fucs.org>> wrote:
>>
>> Rick,
>>
>> Can you confirm the certificate has a chain of trust with the default JDK
>> trusted certs? (i.e. trusted by the JVM)
>>
>> Cheers
>>
>>
>> On Mon, Nov 7, 2016 at 3:38 AM, Ricky Saltzer <ricky@cloudera.com <ma...@cloudera.com>> wrote:
>>
>> Hey Andy -
>>
>> Thanks again for the help.
>>
>> The error message seems indicative that it doesn't seem to properly read
>> the keystore file. One thing to note, if I point the nifi properties to a
>> bogus keystore location, then it actually throws a FileNotFound exception.
>> This is really odd behavior, because as I mentioned I'm able to start it in
>> standalone mode using the correct keystore location, just as I try to do in
>> clustered mode.
>>
>> I've attached both the clustered [1] nifi.properties, which doesn't work,
>> and the standalone [2] which does work. . I restored it to a more basic
>> configuration without the encrypted configuration, but with SSL still
>> enabled. I also added a diff [3] of both the standalone and clustered
>> properties file. Note that I I only have NiFi configured to use the
>> keystore and not a truststore. I've redacted a few of the values in the
>> property files, but be assured that the keystore is most definitely valid
>> and is readable / locatable, as starting in standalone works just fine.
>>
>> I ran the SSL command [4] you gave me, minus the three PEM file arguments
>> as I don't have any of those on hand. I hope that is fine. The output still
>> looks good.
>>
>> [1] https://gist.github.com/rickysaltzer/712aa6586592fe6628db2d57cec7a562 <https://gist.github.com/rickysaltzer/712aa6586592fe6628db2d57cec7a562>
>> [2] https://gist.github.com/rickysaltzer/fe11c8233e4434eacedd7fd0a083d950 <https://gist.github.com/rickysaltzer/fe11c8233e4434eacedd7fd0a083d950>
>> [3] https://gist.github.com/rickysaltzer/d715c7451eb554a54f14ec6e64da8558 <https://gist.github.com/rickysaltzer/d715c7451eb554a54f14ec6e64da8558>
>> [4] https://gist.github.com/rickysaltzer/5d7cdeff8868bfc1f47010189735411a <https://gist.github.com/rickysaltzer/5d7cdeff8868bfc1f47010189735411a>
>>
>>
>>
>>
>> On Fri, Nov 4, 2016 at 7:48 PM, Andy LoPresto <alopresto@apache.org <ma...@apache.org>>
>> wrote:
>>
>> Hi Ricky,
>>
>> Sorry, should have noted that the debug output goes to
>>
>> nifi-bootstrap.log,
>>
>> so thanks Mark for jumping in to help there.
>>
>> If you look at the top of that log, you’ll note that there is no keystore
>> file provided and the truststore loaded is the default JRE cacerts
>> truststore. Can you please provide your nifi.properties file in a Gist,
>>
>> **taking
>>
>> care to redact any sensitive values** like keystore/truststore passwords,
>> although I think from looking at your log output, you are taking
>>
>> advantage
>>
>> of the encrypted configuration feature, so even viewing the encrypted
>> values should be ok. Could you also please provide the directory listing
>> where the keystore and truststore are located including the permissions
>>
>> and
>>
>> ownership information?
>>
>> There may be a bug in the logic between cluster and standalone mode, but
>>
>> I
>>
>> haven’t encountered this behavior before. If you can start NiFi in
>> standalone mode, could you please provide the output of the following
>> command run from the terminal? It will simulate an HTTPS connection to
>>
>> the
>>
>> server and verify the key and certificate presented by NiFi.
>>
>> * host — the NiFi hostname
>> * port — the port NiFi is running on
>> * path_to_your_cert.pem — the public key certificate identifying the
>> client/user (i.e. what you load into your browser to authenticate)
>> * path_to_your_key.key — the private key identifying the client/user
>> * path_to_your_CA_cert.pem — the public key certificate identifying the
>>
>> CA
>>
>> which signed your NiFi server certificate (if self-signed, provide that
>> certificate)
>>
>> $ openssl s_client -connect <host:port> -debug -state -cert
>> <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile
>> <path_to_your_CA_cert.pem>
>>
>> Andy LoPresto
>> alopresto@apache.org <ma...@apache.org>
>> *alopresto.apache@gmail.com <ma...@gmail.com> <alopresto.apache@gmail.com <ma...@gmail.com>>*
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>>
>> On Nov 4, 2016, at 11:21 AM, Ricky Saltzer <ricky@cloudera.com <ma...@cloudera.com>> wrote:
>>
>> Hey guys -
>>
>> I went ahead and uploaded the boostrap log. I took a look at it and it
>> seems to be the same error [1]
>>
>> [1]:
>> https://gist.githubusercontent.com/rickysaltzer/ <https://gist.githubusercontent.com/rickysaltzer/>
>> b156594f92066873f80928eddf84e7bb/raw/4d0e018038b332f4fdf14644910dfa
>> 9e70c57e49/gistfile1.txt
>>
>> On Fri, Nov 4, 2016 at 2:14 PM, Mark Payne <markap14@hotmail.com <ma...@hotmail.com>> wrote:
>>
>> Hey Ricky,
>>
>> When you enable debug logging for SSL, it writes to StdErr (or StdOut?)
>>
>> so
>>
>> it will end up in your logs/nifi-bootstrap.log instead of nifi-app.log.
>> Can you give that a look?
>>
>> Thanks
>> -Mark
>>
>> On Nov 4, 2016, at 2:07 PM, Ricky Saltzer <ricky@cloudera.com <ma...@cloudera.com>> wrote:
>>
>> Hey Andy -
>>
>> Thanks for the response. I'm currently just trying to get one node in
>> clustered mode before adding a second. The keystore is stored locally and
>> I've confirmed it's readable, as it was able to start once I took it out
>>
>> of
>>
>> clustered mode. I added that line to the bootstrap.conf, but I don't
>> believe any additional logging was produced in regards to troubleshooting
>> this problem. Just in case, I've attached the entire log [1].
>>
>> [1]:
>> https://gist.githubusercontent.com/rickysaltzer/ <https://gist.githubusercontent.com/rickysaltzer/>
>>
>> ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0
>> fedabc88bb/gistfile1.txt <https://gist.githubusercontent.com/ <https://gist.githubusercontent.com/>
>>
>> rickysaltzer/
>>
>> ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0
>> fedabc88bb/gistfile1.txt>
>>
>>
>> On Wed, Nov 2, 2016 at 7:08 PM, Andy LoPresto <alopresto@apache.org <ma...@apache.org>
>>
>> <mailto:alopresto@apache.org <ma...@apache.org> <alopresto@apache.org <ma...@apache.org>> <alopresto@apache.org <ma...@apache.org>>>>
>> wrote:
>>
>>
>> Hi Ricky,
>>
>> Sorry to hear you are having this issue. Is the keystore available on
>>
>> all
>>
>> nodes of the cluster? It appears from the log message that the keystore
>>
>> is
>>
>> not found during startup. To further debug, you can add the following
>>
>> line
>>
>> in bootstrap.conf to provide additional logging:
>>
>> java.arg.15=-Djavax.net.debug=ssl,handshake
>>
>> Andy LoPresto
>> alopresto@apache.org <ma...@apache.org> <mailto:alopresto@apache.org <ma...@apache.org> <alopresto@apache.org <ma...@apache.org>> <
>> alopresto@apache.org <ma...@apache.org>>>
>> *alopresto.apache@gmail.com <ma...@gmail.com> <mailto:alopresto.apache@gmail.com <ma...@gmail.com>
>> <alopresto.apache@gmail.com <ma...@gmail.com>>
>> <alopresto.apache@gmail.com <ma...@gmail.com>>> <
>>
>> alopresto.apache@gmail.com <ma...@gmail.com> <mailto:alopresto.apache@gmail.com <ma...@gmail.com>
>> <al...@gmail.com>
>>
>> <al...@gmail.com>>>*
>>
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>>
>> On Nov 2, 2016, at 2:25 PM, Ricky Saltzer <ri...@cloudera.com> wrote:
>>
>> Hey all -
>>
>> I'm using NiFi 1.0 and I'm having an issue using secure mode with a
>>
>> local
>>
>> key store while in clustered mode. If I set the node in clustered mode,
>>
>> and
>>
>> also provide a valid keystore, I receive a KeyStoreException [1]. If I
>>
>> set
>>
>> the configuration to not use clustered mode, NiFi will start up fine
>>
>> with
>>
>> the provided key store. Am I supposed to be storing this key store in
>> Zookeeper somewhere?
>>
>>
>> [1]
>>
>>
>> Caused by: java.security.KeyStoreException: not found
>>
>>
>> at java.security.KeyStore.getInstance(KeyStore.java:839)
>> ~[na:1.8.0_11]
>>
>> at
>> org.apache.nifi.io.socket.SSLContextFactory.<init>(
>> SSLContextFactory.java:61)
>> ~[nifi-socket-utils-1.0.0.jar:1.0.0]
>>
>> at
>> org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
>> ryBean.getObject(ServerSocketConfigurationFactoryBean.java:45)
>> ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
>>
>> at
>> org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
>> ryBean.getObject(ServerSocketConfigurationFactoryBean.java:30)
>> ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
>>
>> at
>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.
>> doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
>> ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
>>
>> ... 69 common frames omitted
>>
>> Caused by: java.security.NoSuchAlgorithmException: KeyStore not
>>
>> available
>>
>>
>> at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
>> ~[na:1.8.0_11]
>>
>> at java.security.Security.getImpl(Security.java:695)
>>
>> ~[na:1.8.0_11]
>>
>>
>> at java.security.KeyStore.getInstance(KeyStore.java:836)
>> ~[na:1.8.0_11]
>>
>> ... 73 common frames omitted
>>
>>
>>
>>
>>
>> --
>> Ricky Saltzer
>> http://www.cloudera.com <http://www.cloudera.com/>
>>
>>
>>
>>
>>
>> --
>> Ricky Saltzer
>> http://www.cloudera.com
>>
>>
>>
>>
>>
>> --
>> Ricky Saltzer
>> http://www.cloudera.com
>>
>>
>>
>>
>>
>> --
>> Ricky Saltzer
>> http://www.cloudera.com
>>
>>
>>
>
>
> --
> Ricky Saltzer
> http://www.cloudera.com <http://www.cloudera.com/>
Re: Secure Cluster Mode Issues
Posted by Ricky Saltzer <ri...@cloudera.com>.
Hey Andy -
Thanks for the reply, I used the openssl command you provided and indeed
the return code was *OK*. Before proceeding with the recommendation of
importing the key into my OSX keychain, I would like to understand why this
required. When using HTTPS mode in non-clustered mode, it does not require
clients to have a special key or cert imported to their machine. Instead,
the client is given a warning in the browser, and it's up to them to
proceed. This UI will serve as an endpoint to several users, and I would
really like to avoid the cumbersomeness of having members of multiple teams
follow instructions for importing keys just so they can access a web UI.
On Tue, Nov 29, 2016 at 1:55 PM, Andy LoPresto <al...@apache.org> wrote:
> Hi Ricky,
>
> The ERR_CONNECTION_CLOSED is likely because you are not sending a client
> certificate on the HTTP request. By default, a secured cluster requires
> client certificate authentication unless LDAP or Kerberos are configured as
> identity providers [1]. The TLS Toolkit provides a quick way to generate a
> valid client certificate which you can load into your browser in order to
> access the site.
>
> First, verify the cluster is running and accepting incoming connections
> (we’re going to cheat here just to be quick about it; disclaimer that this
> is not the RIGHT way to do this):
>
> In the directory where you ran the toolkit, you noted there was a
> “nifi-cert.pem” and “nifi-cert.key” file. The pem file is the PEM-encoded
> public certificate of the NiFi CA cert that was generated by the toolkit,
> and the key file is the PEM-encoded private key. Because this is the same
> certificate that signed the NiFi server key loaded in the keystore, it is
> also loaded into the truststore. That means the server will accept an
> incoming connection with any certificate signed by the CA cert.
> Coincidentally, the CA cert is self-signed, so…
>
> $ openssl s_client -connect <host:port> -debug -state -cert nifi-cert.pem
> -key nifi-key.key -CAfile nifi-cert.pem
>
> That command will attempt to negotiate a TLS connection to your server by
> presenting the CA cert and key as the client. Again, not semantically
> correct, but technically will work. You’ll get a long output, but it
> should end in a section like this:
>
> ---
> New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384
> Server public key is 2048 bit
> Secure Renegotiation IS supported
> Compression: NONE
> Expansion: NONE
> No ALPN negotiated
> SSL-Session:
> Protocol : TLSv1.2
> Cipher : ECDHE-RSA-AES256-SHA384
> Session-ID: 583DCD...9E828C
> Session-ID-ctx:
> Master-Key: 5477C0...A51E85
> Key-Arg : None
> PSK identity: None
> PSK identity hint: None
> SRP username: None
> Start Time: 1480445265
> Timeout : 300 (sec)
> Verify return code: 0 (ok)
> ---
>
> The important part is the last line — you want the *Verify return code* to
> be 0 for success. Once you have verified this, run the TLS toolkit again to
> generate a valid client certificate:
>
> $ ./bin/tls-toolkit.sh standalone -C 'CN=Ricky Saltzer, OU=Apache NiFi'
> -B thisIsABadPassword
>
> This will generate a PKCS12 keystore (*.p12) containing your public
> certificate AND the private key, as well as an additional file (.password)
> containing the password you provided for -B.
>
> hw12203:...lkit/nifi-toolkit-assembly/target/nifi-toolkit-1.1.0-bin/nifi-toolkit-1.1.0
> (master) alopresto
> 🔓 146s @ 10:50:14 $ ./bin/tls-toolkit.sh standalone -C 'CN=Ricky Saltzer,
> OU=Apache NiFi' -B password
> 2016/11/29 10:50:20 INFO [main] org.apache.nifi.toolkit.tls.standalone.
> TlsToolkitStandaloneCommandLine: No nifiPropertiesFile specified, using
> embedded one.
> 2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone:
> Running standalone certificate generation with output directory
> ../nifi-toolkit-1.1.0
> 2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone:
> Using existing CA certificate ../nifi-toolkit-1.1.0/nifi-cert.pem and key
> ../nifi-toolkit-1.1.0/nifi-key.key
> 2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone:
> No hostnames specified, not generating any host certificates or
> configuration.
> 2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone:
> Generating new client certificate ../nifi-toolkit-1.1.0/CN=
> Ricky_Saltzer_OU=Apache_NiFi.p12
> 2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone:
> Successfully generated client certificate ../nifi-toolkit-1.1.0/CN=
> Ricky_Saltzer_OU=Apache_NiFi.p12
> 2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone:
> tls-toolkit standalone completed successfully
> hw12203:...lkit/nifi-toolkit-assembly/target/nifi-toolkit-1.1.0-bin/nifi-toolkit-1.1.0
> (master) alopresto
> 🔓 7s @ 10:50:22 $ ll
> total 136
> drwxr-xr-x 20 alopresto staff 680B Nov 29 10:50 ./
> drwxr-xr-x 4 alopresto staff 136B Nov 28 17:16 ../
> -rw------- 1 alopresto staff 3.4K Nov 29 10:50
> CN=Ricky_Saltzer_OU=Apache_NiFi.p12
> -rw------- 1 alopresto staff 8B Nov 29 10:50
> CN=Ricky_Saltzer_OU=Apache_NiFi.password
> ...
> -rw------- 1 alopresto staff 1.2K Nov 28 16:31 nifi-cert.pem
> -rw------- 1 alopresto staff 1.6K Nov 28 16:31 nifi-key.key
> hw12203:...lkit/nifi-toolkit-assembly/target/nifi-toolkit-1.1.0-bin/nifi-toolkit-1.1.0
> (master) alopresto
> 🔓 4s @ 10:50:27 $
>
> You can double-click on the *.p12 file to open it in your default handler
> — on OS X for example, this is Keychain Access. You can also manually load
> it into Firefox to keep it isolated from your system certificates.
>
> Now when you visit the UI through the browser, you should receive a prompt
> for which certificate to present, and select this entry from the presented
> list.
>
> If you get a NiFi error message in the browser that you do not have
> sufficient access, remember to update conf/authorizers.xml with an Initial
> Admin Identity, which should match EXACTLY the DN of this certificate —
> “CN=Ricky Saltzer, OU=Apache NiFi”. You can copy it from the failed
> authentication message in logs/nifi-user.log to be sure.
>
> Andy LoPresto
> alopresto@apache.org
> *alopresto.apache@gmail.com <al...@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> On Nov 29, 2016, at 10:27 AM, Ricky Saltzer <ri...@cloudera.com> wrote:
>
> Hi Andy -
>
> Thanks again for the help, sorry I've been a little preoccupied with some
> other tasks. I took your advice and used the tls-toolkit and things are
> looking a lot better. I generated a keystore/truststore for both of my
> nodes. It also generated a pem and key file. I was able to get NiFi to
> start in clustered mode (horray)! The only issue I'm seeing now is that I
> can't actually connect to the web UI using HTTPS, as it immediately throws
> a "ERR_CONNECTION_CLOSED" message in my browser.
>
> Judging by the logs, NiFi is seemingly running. A leader election took
> place and the node is heart beating to itself. Am I missing a step where
> I'm supposed to be doing something with the pem/key files?
>
> Thanks again,
> Ricky
>
> On Tue, Nov 8, 2016 at 8:22 PM, Andy LoPresto <al...@apache.org>
> wrote:
>
> Hi Ricky,
>
> Sorry for the delay in my response. I see a couple things which could be
> causing an issue:
>
> 1. You do not specify a hostname for the NiFi node (nifi.web.https.host=)
> which you should do. I have not set up a one-node cluster before, but my
> suspicion is that this field needs to be populated, and the value needs to
> match the CN for the issued server certificate. This value does not need to
> be unique (i.e. it could be nifi.cloudera.com or localhost and you could
> run multiple instances on the same machine, identified by the same
> certificate and different ports), but I think it has to be populated.
> 1.a. While you have nifi.security.needClientAuth=false set in both
> cluster and standalone mode, I am not sure if this allows for the
> truststore to be missing completely. I would have to explore this further.
> There is a current Jira for clarifying cluster, UI/API, and S2S TLS
> configurations [1]. You can try using the default JRE truststore, located
> at “$JAVA_HOME/jre/lib/security/cacerts” with password “changeit”.
> 2. The certificate you have put into the keystore appears to be a
> certificate identifying you (Ricky the person) rather than a server entity.
> You should create a certificate identifying the server itself, so the CN is
> the FQDN as mentioned above. Then import into (or generate directly inside)
> the keystore. You can also use the provided NiFi TLS Toolkit to automate
> this entire process [2].
> Certificate chain
> 0 s:/C=CA/ST=CA/L=Palo Alto/O=Cloudera/OU=EDH Team, Cloudera/CN=Ricky
> Saltzer
> i:/C=CA/ST=CA/L=Palo Alto/O=Cloudera/OU=EDH Team, Cloudera/CN=Ricky
> Saltzercompare
> to an example connection to google.com:443:
>
> Certificate chain
> 0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/*CN=*.google.com
> <http://google.com>*
> i:/C=US/O=Google Inc/CN=Google Internet Authority G2
> 1 s:/C=US/O=Google Inc/CN=Google Internet Authority G2
> i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
> 2 s:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
> i:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority
>
> 3. It appears the certificate you are using is expired. If you look at the
> output of the OpenSSL command you ran, you can see that the result code was
> *10*, not *0* as would be in the case of a successful connection (I
> understand that it looks successful because it negotiates a key and
> encrypts the channel).
> New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384
> Server public key is 2048 bit
> Secure Renegotiation IS supported
> Compression: NONE
> Expansion: NONE
> SSL-Session:
> Protocol : TLSv1.2
> Cipher : ECDHE-RSA-AES256-SHA384
> Session-ID: 581F5A5...41153B
> Session-ID-ctx:
> Master-Key: CEEED2...944C30
> Key-Arg : None
> PSK identity: None
> PSK identity hint: None
> SRP username: None
> Start Time: 1478449748
> Timeout : 300 (sec)
> Verify return code: *10* *(certificate has expired)*
>
> ---
> HTTP/1.1 400 No URI
> Content-Length: 0
> Connection: close
> Server: Jetty(9.3.9.v20160517)
> I rebuilt your certificate from the Base64-encoded version in the output,
> and it appears it expired on October 25, 2016. You can verify this by
> copying the section between "-----BEGIN CERTIFICATE-----“ and "-----END
> CERTIFICATE-----“ (including these markers) into a text file and naming it
> “ricky.pem” and then running the following command:
>
> hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
> 🔓 11s @ 17:09:30 $ more ricky.pem
> -----BEGIN CERTIFICATE-----
> MIIDizCCAnOgAwIBAgIEHPLwBzANBgkqhkiG9w0BAQsFADB2MQswCQYDVQQGEwJD
> ...
> 45GrxLz7n/ZXZk8q9aXcrSGaj+HHCyrO0/9NYtMNP2V5wpYcBRiHO9f3sHKgnzU=
> -----END CERTIFICATE-----
> hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
> 🔓 5s @ 17:09:35 $ openssl x509 -in ricky.pem -text -noout
> Certificate:
> Data:
> Version: 3 (0x2)
> Serial Number: 485683207 (0x1cf2f007)
> Signature Algorithm: sha256WithRSAEncryption
> Issuer: C=CA, ST=CA, L=Palo Alto, O=Cloudera, OU=EDH Team,
> Cloudera, CN=Ricky Saltzer
> Validity
> Not Before: Jul 27 17:15:46 2016 GMT
> *Not After : Oct 25 17:15:46 2016 GMT*
>
> Subject: C=CA, ST=CA, L=Palo Alto, O=Cloudera, OU=EDH Team,
> Cloudera, CN=Ricky Saltzer
> Subject Public Key Info:
> Public Key Algorithm: rsaEncryption
> Public-Key: (2048 bit)
> Modulus:
> 00:f0:4c:fe:7e:af:69:89:f0:4b:a0:e0:df:e8:8c:
> 46:6e:d7:17:4d:17:eb:ee:89:25:ab:5b:35:6a:b2:
> 1e:07:84:95:5f:39:1e:c3:d9:5c:d4:54:09:0c:1a:
> 2a:60:df:9f:ce:7d:87:00:d7:31:23:37:62:8c:02:
> c9:98:a3:d4:f5:3e:71:4e:1d:fb:bb:5b:86:35:af:
> ce:6d:9d:ee:fb:65:d1:01:22:e3:f8:df:c0:81:d7:
> fe:fd:f3:c8:48:c2:bd:23:e2:44:1e:8d:48:5f:81:
> ed:10:62:77:71:3e:15:82:5a:c2:49:70:bb:ef:95:
> d9:ff:f2:37:6c:06:63:fc:3c:98:a3:c9:49:3a:1d:
> b8:37:e3:0b:50:0f:23:5f:26:32:19:c8:59:4e:5a:
> 09:5a:87:7f:b6:40:b4:37:18:20:74:fb:d2:f8:e9:
> 18:e8:d7:23:fa:83:be:2a:c3:33:8d:17:a8:b3:7c:
> 52:03:03:7c:24:80:c5:9c:a5:0d:21:3d:57:7c:d4:
> 0f:78:00:fd:be:69:dd:d8:fe:07:a0:09:77:81:b7:
> bf:fd:0f:bd:8c:9a:35:42:4f:89:8e:31:fd:00:22:
> 16:bb:76:0b:51:b0:c4:81:64:69:f7:c4:5d:37:1e:
> 79:b7:08:d2:5a:6a:a7:43:98:d3:81:a9:08:00:57:
> 2b:c5
> Exponent: 65537 (0x10001)
> X509v3 extensions:
> X509v3 Subject Key Identifier:
> 1C:DD:E1:16:5F:41:CC:7C:69:27:
> E3:B1:6B:78:EF:FA:16:DA:1F:6F
> Signature Algorithm: sha256WithRSAEncryption
> da:5a:31:4b:3b:a6:bf:84:8d:6a:56:2a:82:88:2c:e0:95:49:
> 30:f2:3a:31:ed:b5:bc:5b:92:e6:ad:b3:5b:9f:7a:71:bf:62:
> 0f:38:27:4c:71:02:a1:2f:7f:fa:c0:46:da:78:64:36:35:34:
> 1d:91:b3:f8:e2:23:22:1a:e8:e1:14:2b:15:2c:aa:36:71:39:
> 05:d7:c0:2a:e0:cd:4d:f9:72:07:03:b9:f2:c8:6d:59:fb:c9:
> 25:49:ae:3f:a2:c2:fe:d0:54:1e:45:13:8d:95:05:86:8b:a4:
> c4:7c:df:e1:7a:35:cc:48:e3:cd:20:87:c5:50:a7:43:d2:66:
> 94:81:89:b1:a2:66:20:44:ee:cb:ea:08:c9:7c:aa:61:c7:24:
> 5f:71:ca:6e:c0:a8:15:1f:4a:d3:79:e3:19:fd:7f:86:96:1a:
> 4d:db:9d:38:6f:b3:16:fd:39:68:cf:1c:57:86:26:93:6d:0e:
> 76:6b:aa:b8:9a:e7:c4:05:c6:c5:ae:5a:e1:a1:2a:ad:aa:e3:
> 5b:9a:f1:70:a5:f0:54:76:b3:0e:00:e3:91:ab:c4:bc:fb:9f:
> f6:57:66:4f:2a:f5:a5:dc:ad:21:9a:8f:e1:c7:0b:2a:ce:d3:
> ff:4d:62:d3:0d:3f:65:79:c2:96:1c:05:18:87:3b:d7:f7:b0:
> 72:a0:9f:35
> hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
> 🔓 11s @ 17:09:47 $
>
> I would recommend using the TLS Toolkit to generate a valid CA, server
> certificate, client certificate, and keystore & truststore (seriously, one
> command does all of this) and then re-trying. From this stable baseline, I
> think I’ll be able to better help iron out the cluster/standalone issues.
>
> [1] https://issues.apache.org/jira/browse/NIFI-1990
> [2] https://nifi.apache.org/docs/nifi-docs/html/
> administration-guide.html#tls-generation-toolkit
>
>
> Andy LoPresto
> alopresto@apache.org
> *alopresto.apache@gmail.com <al...@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> On Nov 8, 2016, at 1:31 AM, Andre <an...@fucs.org> wrote:
>
> Rick,
>
> Can you confirm the certificate has a chain of trust with the default JDK
> trusted certs? (i.e. trusted by the JVM)
>
> Cheers
>
>
> On Mon, Nov 7, 2016 at 3:38 AM, Ricky Saltzer <ri...@cloudera.com> wrote:
>
> Hey Andy -
>
> Thanks again for the help.
>
> The error message seems indicative that it doesn't seem to properly read
> the keystore file. One thing to note, if I point the nifi properties to a
> bogus keystore location, then it actually throws a FileNotFound exception.
> This is really odd behavior, because as I mentioned I'm able to start it in
> standalone mode using the correct keystore location, just as I try to do in
> clustered mode.
>
> I've attached both the clustered [1] nifi.properties, which doesn't work,
> and the standalone [2] which does work. . I restored it to a more basic
> configuration without the encrypted configuration, but with SSL still
> enabled. I also added a diff [3] of both the standalone and clustered
> properties file. Note that I I only have NiFi configured to use the
> keystore and not a truststore. I've redacted a few of the values in the
> property files, but be assured that the keystore is most definitely valid
> and is readable / locatable, as starting in standalone works just fine.
>
> I ran the SSL command [4] you gave me, minus the three PEM file arguments
> as I don't have any of those on hand. I hope that is fine. The output still
> looks good.
>
> [1] https://gist.github.com/rickysaltzer/712aa6586592fe6628db2d57cec7a562
> [2] https://gist.github.com/rickysaltzer/fe11c8233e4434eacedd7fd0a083d950
> [3] https://gist.github.com/rickysaltzer/d715c7451eb554a54f14ec6e64da8558
> [4] https://gist.github.com/rickysaltzer/5d7cdeff8868bfc1f47010189735411a
>
>
>
>
> On Fri, Nov 4, 2016 at 7:48 PM, Andy LoPresto <al...@apache.org>
> wrote:
>
> Hi Ricky,
>
> Sorry, should have noted that the debug output goes to
>
> nifi-bootstrap.log,
>
> so thanks Mark for jumping in to help there.
>
> If you look at the top of that log, you’ll note that there is no keystore
> file provided and the truststore loaded is the default JRE cacerts
> truststore. Can you please provide your nifi.properties file in a Gist,
>
> **taking
>
> care to redact any sensitive values** like keystore/truststore passwords,
> although I think from looking at your log output, you are taking
>
> advantage
>
> of the encrypted configuration feature, so even viewing the encrypted
> values should be ok. Could you also please provide the directory listing
> where the keystore and truststore are located including the permissions
>
> and
>
> ownership information?
>
> There may be a bug in the logic between cluster and standalone mode, but
>
> I
>
> haven’t encountered this behavior before. If you can start NiFi in
> standalone mode, could you please provide the output of the following
> command run from the terminal? It will simulate an HTTPS connection to
>
> the
>
> server and verify the key and certificate presented by NiFi.
>
> * host — the NiFi hostname
> * port — the port NiFi is running on
> * path_to_your_cert.pem — the public key certificate identifying the
> client/user (i.e. what you load into your browser to authenticate)
> * path_to_your_key.key — the private key identifying the client/user
> * path_to_your_CA_cert.pem — the public key certificate identifying the
>
> CA
>
> which signed your NiFi server certificate (if self-signed, provide that
> certificate)
>
> $ openssl s_client -connect <host:port> -debug -state -cert
> <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile
> <path_to_your_CA_cert.pem>
>
> Andy LoPresto
> alopresto@apache.org
> *alopresto.apache@gmail.com <al...@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> On Nov 4, 2016, at 11:21 AM, Ricky Saltzer <ri...@cloudera.com> wrote:
>
> Hey guys -
>
> I went ahead and uploaded the boostrap log. I took a look at it and it
> seems to be the same error [1]
>
> [1]:
> https://gist.githubusercontent.com/rickysaltzer/
> b156594f92066873f80928eddf84e7bb/raw/4d0e018038b332f4fdf14644910dfa
> 9e70c57e49/gistfile1.txt
>
> On Fri, Nov 4, 2016 at 2:14 PM, Mark Payne <ma...@hotmail.com> wrote:
>
> Hey Ricky,
>
> When you enable debug logging for SSL, it writes to StdErr (or StdOut?)
>
> so
>
> it will end up in your logs/nifi-bootstrap.log instead of nifi-app.log.
> Can you give that a look?
>
> Thanks
> -Mark
>
> On Nov 4, 2016, at 2:07 PM, Ricky Saltzer <ri...@cloudera.com> wrote:
>
> Hey Andy -
>
> Thanks for the response. I'm currently just trying to get one node in
> clustered mode before adding a second. The keystore is stored locally and
> I've confirmed it's readable, as it was able to start once I took it out
>
> of
>
> clustered mode. I added that line to the bootstrap.conf, but I don't
> believe any additional logging was produced in regards to troubleshooting
> this problem. Just in case, I've attached the entire log [1].
>
> [1]:
> https://gist.githubusercontent.com/rickysaltzer/
>
> ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0
> fedabc88bb/gistfile1.txt <https://gist.githubusercontent.com/
>
> rickysaltzer/
>
> ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0
> fedabc88bb/gistfile1.txt>
>
>
> On Wed, Nov 2, 2016 at 7:08 PM, Andy LoPresto <alopresto@apache.org
>
> <mailto:alopresto@apache.org <al...@apache.org> <al...@apache.org>>>
> wrote:
>
>
> Hi Ricky,
>
> Sorry to hear you are having this issue. Is the keystore available on
>
> all
>
> nodes of the cluster? It appears from the log message that the keystore
>
> is
>
> not found during startup. To further debug, you can add the following
>
> line
>
> in bootstrap.conf to provide additional logging:
>
> java.arg.15=-Djavax.net.debug=ssl,handshake
>
> Andy LoPresto
> alopresto@apache.org <mailto:alopresto@apache.org <al...@apache.org> <
> alopresto@apache.org>>
> *alopresto.apache@gmail.com <mailto:alopresto.apache@gmail.com
> <al...@gmail.com>
> <al...@gmail.com>> <
>
> alopresto.apache@gmail.com <mailto:alopresto.apache@gmail.com
> <al...@gmail.com>
>
> <al...@gmail.com>>>*
>
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> On Nov 2, 2016, at 2:25 PM, Ricky Saltzer <ri...@cloudera.com> wrote:
>
> Hey all -
>
> I'm using NiFi 1.0 and I'm having an issue using secure mode with a
>
> local
>
> key store while in clustered mode. If I set the node in clustered mode,
>
> and
>
> also provide a valid keystore, I receive a KeyStoreException [1]. If I
>
> set
>
> the configuration to not use clustered mode, NiFi will start up fine
>
> with
>
> the provided key store. Am I supposed to be storing this key store in
> Zookeeper somewhere?
>
>
> [1]
>
>
> Caused by: java.security.KeyStoreException: not found
>
>
> at java.security.KeyStore.getInstance(KeyStore.java:839)
> ~[na:1.8.0_11]
>
> at
> org.apache.nifi.io.socket.SSLContextFactory.<init>(
> SSLContextFactory.java:61)
> ~[nifi-socket-utils-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
> ryBean.getObject(ServerSocketConfigurationFactoryBean.java:45)
> ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
> ryBean.getObject(ServerSocketConfigurationFactoryBean.java:30)
> ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
>
> at
> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.
> doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
> ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
>
> ... 69 common frames omitted
>
> Caused by: java.security.NoSuchAlgorithmException: KeyStore not
>
> available
>
>
> at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
> ~[na:1.8.0_11]
>
> at java.security.Security.getImpl(Security.java:695)
>
> ~[na:1.8.0_11]
>
>
> at java.security.KeyStore.getInstance(KeyStore.java:836)
> ~[na:1.8.0_11]
>
> ... 73 common frames omitted
>
>
>
>
>
> --
> Ricky Saltzer
> http://www.cloudera.com <http://www.cloudera.com/>
>
>
>
>
>
> --
> Ricky Saltzer
> http://www.cloudera.com
>
>
>
>
>
> --
> Ricky Saltzer
> http://www.cloudera.com
>
>
>
>
>
> --
> Ricky Saltzer
> http://www.cloudera.com
>
>
>
--
Ricky Saltzer
http://www.cloudera.com
Re: Secure Cluster Mode Issues
Posted by Andy LoPresto <al...@apache.org>.
Hi Ricky,
The ERR_CONNECTION_CLOSED is likely because you are not sending a client certificate on the HTTP request. By default, a secured cluster requires client certificate authentication unless LDAP or Kerberos are configured as identity providers [1]. The TLS Toolkit provides a quick way to generate a valid client certificate which you can load into your browser in order to access the site.
First, verify the cluster is running and accepting incoming connections (we’re going to cheat here just to be quick about it; disclaimer that this is not the RIGHT way to do this):
In the directory where you ran the toolkit, you noted there was a “nifi-cert.pem” and “nifi-cert.key” file. The pem file is the PEM-encoded public certificate of the NiFi CA cert that was generated by the toolkit, and the key file is the PEM-encoded private key. Because this is the same certificate that signed the NiFi server key loaded in the keystore, it is also loaded into the truststore. That means the server will accept an incoming connection with any certificate signed by the CA cert. Coincidentally, the CA cert is self-signed, so…
$ openssl s_client -connect <host:port> -debug -state -cert nifi-cert.pem -key nifi-key.key -CAfile nifi-cert.pem
That command will attempt to negotiate a TLS connection to your server by presenting the CA cert and key as the client. Again, not semantically correct, but technically will work. You’ll get a long output, but it should end in a section like this:
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-SHA384
Session-ID: 583DCD...9E828C
Session-ID-ctx:
Master-Key: 5477C0...A51E85
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1480445265
Timeout : 300 (sec)
Verify return code: 0 (ok)
---
The important part is the last line — you want the Verify return code to be 0 for success. Once you have verified this, run the TLS toolkit again to generate a valid client certificate:
$ ./bin/tls-toolkit.sh standalone -C 'CN=Ricky Saltzer, OU=Apache NiFi' -B thisIsABadPassword
This will generate a PKCS12 keystore (*.p12) containing your public certificate AND the private key, as well as an additional file (.password) containing the password you provided for -B.
hw12203:...lkit/nifi-toolkit-assembly/target/nifi-toolkit-1.1.0-bin/nifi-toolkit-1.1.0 (master) alopresto
🔓 146s @ 10:50:14 $ ./bin/tls-toolkit.sh standalone -C 'CN=Ricky Saltzer, OU=Apache NiFi' -B password
2016/11/29 10:50:20 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandaloneCommandLine: No nifiPropertiesFile specified, using embedded one.
2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone: Running standalone certificate generation with output directory ../nifi-toolkit-1.1.0
2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone: Using existing CA certificate ../nifi-toolkit-1.1.0/nifi-cert.pem and key ../nifi-toolkit-1.1.0/nifi-key.key
2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone: No hostnames specified, not generating any host certificates or configuration.
2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone: Generating new client certificate ../nifi-toolkit-1.1.0/CN=Ricky_Saltzer_OU=Apache_NiFi.p12
2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone: Successfully generated client certificate ../nifi-toolkit-1.1.0/CN=Ricky_Saltzer_OU=Apache_NiFi.p12
2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone: tls-toolkit standalone completed successfully
hw12203:...lkit/nifi-toolkit-assembly/target/nifi-toolkit-1.1.0-bin/nifi-toolkit-1.1.0 (master) alopresto
🔓 7s @ 10:50:22 $ ll
total 136
drwxr-xr-x 20 alopresto staff 680B Nov 29 10:50 ./
drwxr-xr-x 4 alopresto staff 136B Nov 28 17:16 ../
-rw------- 1 alopresto staff 3.4K Nov 29 10:50 CN=Ricky_Saltzer_OU=Apache_NiFi.p12
-rw------- 1 alopresto staff 8B Nov 29 10:50 CN=Ricky_Saltzer_OU=Apache_NiFi.password
...
-rw------- 1 alopresto staff 1.2K Nov 28 16:31 nifi-cert.pem
-rw------- 1 alopresto staff 1.6K Nov 28 16:31 nifi-key.key
hw12203:...lkit/nifi-toolkit-assembly/target/nifi-toolkit-1.1.0-bin/nifi-toolkit-1.1.0 (master) alopresto
🔓 4s @ 10:50:27 $
You can double-click on the *.p12 file to open it in your default handler — on OS X for example, this is Keychain Access. You can also manually load it into Firefox to keep it isolated from your system certificates.
Now when you visit the UI through the browser, you should receive a prompt for which certificate to present, and select this entry from the presented list.
If you get a NiFi error message in the browser that you do not have sufficient access, remember to update conf/authorizers.xml with an Initial Admin Identity, which should match EXACTLY the DN of this certificate — “CN=Ricky Saltzer, OU=Apache NiFi”. You can copy it from the failed authentication message in logs/nifi-user.log to be sure.
Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Nov 29, 2016, at 10:27 AM, Ricky Saltzer <ri...@cloudera.com> wrote:
>
> Hi Andy -
>
> Thanks again for the help, sorry I've been a little preoccupied with some
> other tasks. I took your advice and used the tls-toolkit and things are
> looking a lot better. I generated a keystore/truststore for both of my
> nodes. It also generated a pem and key file. I was able to get NiFi to
> start in clustered mode (horray)! The only issue I'm seeing now is that I
> can't actually connect to the web UI using HTTPS, as it immediately throws
> a "ERR_CONNECTION_CLOSED" message in my browser.
>
> Judging by the logs, NiFi is seemingly running. A leader election took
> place and the node is heart beating to itself. Am I missing a step where
> I'm supposed to be doing something with the pem/key files?
>
> Thanks again,
> Ricky
>
> On Tue, Nov 8, 2016 at 8:22 PM, Andy LoPresto <alopresto@apache.org <ma...@apache.org>> wrote:
>
>> Hi Ricky,
>>
>> Sorry for the delay in my response. I see a couple things which could be
>> causing an issue:
>>
>> 1. You do not specify a hostname for the NiFi node (nifi.web.https.host=)
>> which you should do. I have not set up a one-node cluster before, but my
>> suspicion is that this field needs to be populated, and the value needs to
>> match the CN for the issued server certificate. This value does not need to
>> be unique (i.e. it could be nifi.cloudera.com or localhost and you could
>> run multiple instances on the same machine, identified by the same
>> certificate and different ports), but I think it has to be populated.
>> 1.a. While you have nifi.security.needClientAuth=false set in both
>> cluster and standalone mode, I am not sure if this allows for the
>> truststore to be missing completely. I would have to explore this further.
>> There is a current Jira for clarifying cluster, UI/API, and S2S TLS
>> configurations [1]. You can try using the default JRE truststore, located
>> at “$JAVA_HOME/jre/lib/security/cacerts” with password “changeit”.
>> 2. The certificate you have put into the keystore appears to be a
>> certificate identifying you (Ricky the person) rather than a server entity.
>> You should create a certificate identifying the server itself, so the CN is
>> the FQDN as mentioned above. Then import into (or generate directly inside)
>> the keystore. You can also use the provided NiFi TLS Toolkit to automate
>> this entire process [2].
>> Certificate chain
>> 0 s:/C=CA/ST=CA/L=Palo Alto/O=Cloudera/OU=EDH Team, Cloudera/CN=Ricky
>> Saltzer
>> i:/C=CA/ST=CA/L=Palo Alto/O=Cloudera/OU=EDH Team, Cloudera/CN=Ricky Saltzercompare
>> to an example connection to google.com:443:
>>
>> Certificate chain
>> 0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/*CN=*.google.com
>> <http://google.com <http://google.com/>>*
>> i:/C=US/O=Google Inc/CN=Google Internet Authority G2
>> 1 s:/C=US/O=Google Inc/CN=Google Internet Authority G2
>> i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
>> 2 s:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
>> i:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority
>>
>> 3. It appears the certificate you are using is expired. If you look at the
>> output of the OpenSSL command you ran, you can see that the result code was
>> *10*, not *0* as would be in the case of a successful connection (I
>> understand that it looks successful because it negotiates a key and
>> encrypts the channel).
>> New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384
>> Server public key is 2048 bit
>> Secure Renegotiation IS supported
>> Compression: NONE
>> Expansion: NONE
>> SSL-Session:
>> Protocol : TLSv1.2
>> Cipher : ECDHE-RSA-AES256-SHA384
>> Session-ID: 581F5A5...41153B
>> Session-ID-ctx:
>> Master-Key: CEEED2...944C30
>> Key-Arg : None
>> PSK identity: None
>> PSK identity hint: None
>> SRP username: None
>> Start Time: 1478449748
>> Timeout : 300 (sec)
>> Verify return code: *10* *(certificate has expired)*
>> ---
>> HTTP/1.1 400 No URI
>> Content-Length: 0
>> Connection: close
>> Server: Jetty(9.3.9.v20160517)
>> I rebuilt your certificate from the Base64-encoded version in the output,
>> and it appears it expired on October 25, 2016. You can verify this by
>> copying the section between "-----BEGIN CERTIFICATE-----“ and "-----END
>> CERTIFICATE-----“ (including these markers) into a text file and naming it
>> “ricky.pem” and then running the following command:
>>
>> hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
>> 🔓 11s @ 17:09:30 $ more ricky.pem
>> -----BEGIN CERTIFICATE-----
>> MIIDizCCAnOgAwIBAgIEHPLwBzANBgkqhkiG9w0BAQsFADB2MQswCQYDVQQGEwJD
>> ...
>> 45GrxLz7n/ZXZk8q9aXcrSGaj+HHCyrO0/9NYtMNP2V5wpYcBRiHO9f3sHKgnzU=
>> -----END CERTIFICATE-----
>> hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
>> 🔓 5s @ 17:09:35 $ openssl x509 -in ricky.pem -text -noout
>> Certificate:
>> Data:
>> Version: 3 (0x2)
>> Serial Number: 485683207 (0x1cf2f007)
>> Signature Algorithm: sha256WithRSAEncryption
>> Issuer: C=CA, ST=CA, L=Palo Alto, O=Cloudera, OU=EDH Team,
>> Cloudera, CN=Ricky Saltzer
>> Validity
>> Not Before: Jul 27 17:15:46 2016 GMT
>> *Not After : Oct 25 17:15:46 2016 GMT*
>> Subject: C=CA, ST=CA, L=Palo Alto, O=Cloudera, OU=EDH Team,
>> Cloudera, CN=Ricky Saltzer
>> Subject Public Key Info:
>> Public Key Algorithm: rsaEncryption
>> Public-Key: (2048 bit)
>> Modulus:
>> 00:f0:4c:fe:7e:af:69:89:f0:4b:a0:e0:df:e8:8c:
>> 46:6e:d7:17:4d:17:eb:ee:89:25:ab:5b:35:6a:b2:
>> 1e:07:84:95:5f:39:1e:c3:d9:5c:d4:54:09:0c:1a:
>> 2a:60:df:9f:ce:7d:87:00:d7:31:23:37:62:8c:02:
>> c9:98:a3:d4:f5:3e:71:4e:1d:fb:bb:5b:86:35:af:
>> ce:6d:9d:ee:fb:65:d1:01:22:e3:f8:df:c0:81:d7:
>> fe:fd:f3:c8:48:c2:bd:23:e2:44:1e:8d:48:5f:81:
>> ed:10:62:77:71:3e:15:82:5a:c2:49:70:bb:ef:95:
>> d9:ff:f2:37:6c:06:63:fc:3c:98:a3:c9:49:3a:1d:
>> b8:37:e3:0b:50:0f:23:5f:26:32:19:c8:59:4e:5a:
>> 09:5a:87:7f:b6:40:b4:37:18:20:74:fb:d2:f8:e9:
>> 18:e8:d7:23:fa:83:be:2a:c3:33:8d:17:a8:b3:7c:
>> 52:03:03:7c:24:80:c5:9c:a5:0d:21:3d:57:7c:d4:
>> 0f:78:00:fd:be:69:dd:d8:fe:07:a0:09:77:81:b7:
>> bf:fd:0f:bd:8c:9a:35:42:4f:89:8e:31:fd:00:22:
>> 16:bb:76:0b:51:b0:c4:81:64:69:f7:c4:5d:37:1e:
>> 79:b7:08:d2:5a:6a:a7:43:98:d3:81:a9:08:00:57:
>> 2b:c5
>> Exponent: 65537 (0x10001)
>> X509v3 extensions:
>> X509v3 Subject Key Identifier:
>> 1C:DD:E1:16:5F:41:CC:7C:69:27:
>> E3:B1:6B:78:EF:FA:16:DA:1F:6F
>> Signature Algorithm: sha256WithRSAEncryption
>> da:5a:31:4b:3b:a6:bf:84:8d:6a:56:2a:82:88:2c:e0:95:49:
>> 30:f2:3a:31:ed:b5:bc:5b:92:e6:ad:b3:5b:9f:7a:71:bf:62:
>> 0f:38:27:4c:71:02:a1:2f:7f:fa:c0:46:da:78:64:36:35:34:
>> 1d:91:b3:f8:e2:23:22:1a:e8:e1:14:2b:15:2c:aa:36:71:39:
>> 05:d7:c0:2a:e0:cd:4d:f9:72:07:03:b9:f2:c8:6d:59:fb:c9:
>> 25:49:ae:3f:a2:c2:fe:d0:54:1e:45:13:8d:95:05:86:8b:a4:
>> c4:7c:df:e1:7a:35:cc:48:e3:cd:20:87:c5:50:a7:43:d2:66:
>> 94:81:89:b1:a2:66:20:44:ee:cb:ea:08:c9:7c:aa:61:c7:24:
>> 5f:71:ca:6e:c0:a8:15:1f:4a:d3:79:e3:19:fd:7f:86:96:1a:
>> 4d:db:9d:38:6f:b3:16:fd:39:68:cf:1c:57:86:26:93:6d:0e:
>> 76:6b:aa:b8:9a:e7:c4:05:c6:c5:ae:5a:e1:a1:2a:ad:aa:e3:
>> 5b:9a:f1:70:a5:f0:54:76:b3:0e:00:e3:91:ab:c4:bc:fb:9f:
>> f6:57:66:4f:2a:f5:a5:dc:ad:21:9a:8f:e1:c7:0b:2a:ce:d3:
>> ff:4d:62:d3:0d:3f:65:79:c2:96:1c:05:18:87:3b:d7:f7:b0:
>> 72:a0:9f:35
>> hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
>> 🔓 11s @ 17:09:47 $
>>
>> I would recommend using the TLS Toolkit to generate a valid CA, server
>> certificate, client certificate, and keystore & truststore (seriously, one
>> command does all of this) and then re-trying. From this stable baseline, I
>> think I’ll be able to better help iron out the cluster/standalone issues.
>>
>> [1] https://issues.apache.org/jira/browse/NIFI-1990
>> [2] https://nifi.apache.org/docs/nifi-docs/html/
>> administration-guide.html#tls-generation-toolkit
>>
>>
>> Andy LoPresto
>> alopresto@apache.org
>> *alopresto.apache@gmail.com <ma...@gmail.com> <alopresto.apache@gmail.com <ma...@gmail.com>>*
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>>
>> On Nov 8, 2016, at 1:31 AM, Andre <andre-lists@fucs.org <ma...@fucs.org>> wrote:
>>
>> Rick,
>>
>> Can you confirm the certificate has a chain of trust with the default JDK
>> trusted certs? (i.e. trusted by the JVM)
>>
>> Cheers
>>
>>
>> On Mon, Nov 7, 2016 at 3:38 AM, Ricky Saltzer <ricky@cloudera.com <ma...@cloudera.com>> wrote:
>>
>> Hey Andy -
>>
>> Thanks again for the help.
>>
>> The error message seems indicative that it doesn't seem to properly read
>> the keystore file. One thing to note, if I point the nifi properties to a
>> bogus keystore location, then it actually throws a FileNotFound exception.
>> This is really odd behavior, because as I mentioned I'm able to start it in
>> standalone mode using the correct keystore location, just as I try to do in
>> clustered mode.
>>
>> I've attached both the clustered [1] nifi.properties, which doesn't work,
>> and the standalone [2] which does work. . I restored it to a more basic
>> configuration without the encrypted configuration, but with SSL still
>> enabled. I also added a diff [3] of both the standalone and clustered
>> properties file. Note that I I only have NiFi configured to use the
>> keystore and not a truststore. I've redacted a few of the values in the
>> property files, but be assured that the keystore is most definitely valid
>> and is readable / locatable, as starting in standalone works just fine.
>>
>> I ran the SSL command [4] you gave me, minus the three PEM file arguments
>> as I don't have any of those on hand. I hope that is fine. The output still
>> looks good.
>>
>> [1] https://gist.github.com/rickysaltzer/712aa6586592fe6628db2d57cec7a562 <https://gist.github.com/rickysaltzer/712aa6586592fe6628db2d57cec7a562>
>> [2] https://gist.github.com/rickysaltzer/fe11c8233e4434eacedd7fd0a083d950 <https://gist.github.com/rickysaltzer/fe11c8233e4434eacedd7fd0a083d950>
>> [3] https://gist.github.com/rickysaltzer/d715c7451eb554a54f14ec6e64da8558 <https://gist.github.com/rickysaltzer/d715c7451eb554a54f14ec6e64da8558>
>> [4] https://gist.github.com/rickysaltzer/5d7cdeff8868bfc1f47010189735411a <https://gist.github.com/rickysaltzer/5d7cdeff8868bfc1f47010189735411a>
>>
>>
>>
>>
>> On Fri, Nov 4, 2016 at 7:48 PM, Andy LoPresto <alopresto@apache.org <ma...@apache.org>>
>> wrote:
>>
>> Hi Ricky,
>>
>> Sorry, should have noted that the debug output goes to
>>
>> nifi-bootstrap.log,
>>
>> so thanks Mark for jumping in to help there.
>>
>> If you look at the top of that log, you’ll note that there is no keystore
>> file provided and the truststore loaded is the default JRE cacerts
>> truststore. Can you please provide your nifi.properties file in a Gist,
>>
>> **taking
>>
>> care to redact any sensitive values** like keystore/truststore passwords,
>> although I think from looking at your log output, you are taking
>>
>> advantage
>>
>> of the encrypted configuration feature, so even viewing the encrypted
>> values should be ok. Could you also please provide the directory listing
>> where the keystore and truststore are located including the permissions
>>
>> and
>>
>> ownership information?
>>
>> There may be a bug in the logic between cluster and standalone mode, but
>>
>> I
>>
>> haven’t encountered this behavior before. If you can start NiFi in
>> standalone mode, could you please provide the output of the following
>> command run from the terminal? It will simulate an HTTPS connection to
>>
>> the
>>
>> server and verify the key and certificate presented by NiFi.
>>
>> * host — the NiFi hostname
>> * port — the port NiFi is running on
>> * path_to_your_cert.pem — the public key certificate identifying the
>> client/user (i.e. what you load into your browser to authenticate)
>> * path_to_your_key.key — the private key identifying the client/user
>> * path_to_your_CA_cert.pem — the public key certificate identifying the
>>
>> CA
>>
>> which signed your NiFi server certificate (if self-signed, provide that
>> certificate)
>>
>> $ openssl s_client -connect <host:port> -debug -state -cert
>> <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile
>> <path_to_your_CA_cert.pem>
>>
>> Andy LoPresto
>> alopresto@apache.org <ma...@apache.org>
>> *alopresto.apache@gmail.com <ma...@gmail.com> <alopresto.apache@gmail.com <ma...@gmail.com>>*
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>>
>> On Nov 4, 2016, at 11:21 AM, Ricky Saltzer <ricky@cloudera.com <ma...@cloudera.com>> wrote:
>>
>> Hey guys -
>>
>> I went ahead and uploaded the boostrap log. I took a look at it and it
>> seems to be the same error [1]
>>
>> [1]:
>> https://gist.githubusercontent.com/rickysaltzer/ <https://gist.githubusercontent.com/rickysaltzer/>
>> b156594f92066873f80928eddf84e7bb/raw/4d0e018038b332f4fdf14644910dfa
>> 9e70c57e49/gistfile1.txt
>>
>> On Fri, Nov 4, 2016 at 2:14 PM, Mark Payne <markap14@hotmail.com <ma...@hotmail.com>> wrote:
>>
>> Hey Ricky,
>>
>> When you enable debug logging for SSL, it writes to StdErr (or StdOut?)
>>
>> so
>>
>> it will end up in your logs/nifi-bootstrap.log instead of nifi-app.log.
>> Can you give that a look?
>>
>> Thanks
>> -Mark
>>
>> On Nov 4, 2016, at 2:07 PM, Ricky Saltzer <ricky@cloudera.com <ma...@cloudera.com>> wrote:
>>
>> Hey Andy -
>>
>> Thanks for the response. I'm currently just trying to get one node in
>> clustered mode before adding a second. The keystore is stored locally and
>> I've confirmed it's readable, as it was able to start once I took it out
>>
>> of
>>
>> clustered mode. I added that line to the bootstrap.conf, but I don't
>> believe any additional logging was produced in regards to troubleshooting
>> this problem. Just in case, I've attached the entire log [1].
>>
>> [1]:
>> https://gist.githubusercontent.com/rickysaltzer/ <https://gist.githubusercontent.com/rickysaltzer/>
>>
>> ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0
>> fedabc88bb/gistfile1.txt <https://gist.githubusercontent.com/ <https://gist.githubusercontent.com/>
>>
>> rickysaltzer/
>>
>> ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0
>> fedabc88bb/gistfile1.txt>
>>
>>
>> On Wed, Nov 2, 2016 at 7:08 PM, Andy LoPresto <alopresto@apache.org <ma...@apache.org>
>>
>> <mailto:alopresto@apache.org <ma...@apache.org> <alopresto@apache.org <ma...@apache.org>>>> wrote:
>>
>>
>> Hi Ricky,
>>
>> Sorry to hear you are having this issue. Is the keystore available on
>>
>> all
>>
>> nodes of the cluster? It appears from the log message that the keystore
>>
>> is
>>
>> not found during startup. To further debug, you can add the following
>>
>> line
>>
>> in bootstrap.conf to provide additional logging:
>>
>> java.arg.15=-Djavax.net.debug=ssl,handshake
>>
>> Andy LoPresto
>> alopresto@apache.org <ma...@apache.org> <mailto:alopresto@apache.org <ma...@apache.org> <alopresto@apache.org <ma...@apache.org>>>
>> *alopresto.apache@gmail.com <ma...@gmail.com> <mailto:alopresto.apache@gmail.com <ma...@gmail.com>
>> <alopresto.apache@gmail.com <ma...@gmail.com>>> <
>>
>> alopresto.apache@gmail.com <ma...@gmail.com> <mailto:alopresto.apache@gmail.com <ma...@gmail.com>
>> <alopresto.apache@gmail.com <ma...@gmail.com>>>>*
>>
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>>
>> On Nov 2, 2016, at 2:25 PM, Ricky Saltzer <ri...@cloudera.com> wrote:
>>
>> Hey all -
>>
>> I'm using NiFi 1.0 and I'm having an issue using secure mode with a
>>
>> local
>>
>> key store while in clustered mode. If I set the node in clustered mode,
>>
>> and
>>
>> also provide a valid keystore, I receive a KeyStoreException [1]. If I
>>
>> set
>>
>> the configuration to not use clustered mode, NiFi will start up fine
>>
>> with
>>
>> the provided key store. Am I supposed to be storing this key store in
>> Zookeeper somewhere?
>>
>>
>> [1]
>>
>>
>> Caused by: java.security.KeyStoreException: not found
>>
>>
>> at java.security.KeyStore.getInstance(KeyStore.java:839)
>> ~[na:1.8.0_11]
>>
>> at
>> org.apache.nifi.io.socket.SSLContextFactory.<init>(
>> SSLContextFactory.java:61)
>> ~[nifi-socket-utils-1.0.0.jar:1.0.0]
>>
>> at
>> org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
>> ryBean.getObject(ServerSocketConfigurationFactoryBean.java:45)
>> ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
>>
>> at
>> org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
>> ryBean.getObject(ServerSocketConfigurationFactoryBean.java:30)
>> ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
>>
>> at
>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.
>> doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
>> ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
>>
>> ... 69 common frames omitted
>>
>> Caused by: java.security.NoSuchAlgorithmException: KeyStore not
>>
>> available
>>
>>
>> at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
>> ~[na:1.8.0_11]
>>
>> at java.security.Security.getImpl(Security.java:695)
>>
>> ~[na:1.8.0_11]
>>
>>
>> at java.security.KeyStore.getInstance(KeyStore.java:836)
>> ~[na:1.8.0_11]
>>
>> ... 73 common frames omitted
>>
>>
>>
>>
>>
>> --
>> Ricky Saltzer
>> http://www.cloudera.com <http://www.cloudera.com/>
>>
>>
>>
>>
>>
>> --
>> Ricky Saltzer
>> http://www.cloudera.com
>>
>>
>>
>>
>>
>> --
>> Ricky Saltzer
>> http://www.cloudera.com
>>
>>
>>
>
>
> --
> Ricky Saltzer
> http://www.cloudera.com <http://www.cloudera.com/>
Re: Secure Cluster Mode Issues
Posted by Ricky Saltzer <ri...@cloudera.com>.
Hi Andy -
Thanks again for the help, sorry I've been a little preoccupied with some
other tasks. I took your advice and used the tls-toolkit and things are
looking a lot better. I generated a keystore/truststore for both of my
nodes. It also generated a pem and key file. I was able to get NiFi to
start in clustered mode (horray)! The only issue I'm seeing now is that I
can't actually connect to the web UI using HTTPS, as it immediately throws
a "ERR_CONNECTION_CLOSED" message in my browser.
Judging by the logs, NiFi is seemingly running. A leader election took
place and the node is heart beating to itself. Am I missing a step where
I'm supposed to be doing something with the pem/key files?
Thanks again,
Ricky
On Tue, Nov 8, 2016 at 8:22 PM, Andy LoPresto <al...@apache.org> wrote:
> Hi Ricky,
>
> Sorry for the delay in my response. I see a couple things which could be
> causing an issue:
>
> 1. You do not specify a hostname for the NiFi node (nifi.web.https.host=)
> which you should do. I have not set up a one-node cluster before, but my
> suspicion is that this field needs to be populated, and the value needs to
> match the CN for the issued server certificate. This value does not need to
> be unique (i.e. it could be nifi.cloudera.com or localhost and you could
> run multiple instances on the same machine, identified by the same
> certificate and different ports), but I think it has to be populated.
> 1.a. While you have nifi.security.needClientAuth=false set in both
> cluster and standalone mode, I am not sure if this allows for the
> truststore to be missing completely. I would have to explore this further.
> There is a current Jira for clarifying cluster, UI/API, and S2S TLS
> configurations [1]. You can try using the default JRE truststore, located
> at “$JAVA_HOME/jre/lib/security/cacerts” with password “changeit”.
> 2. The certificate you have put into the keystore appears to be a
> certificate identifying you (Ricky the person) rather than a server entity.
> You should create a certificate identifying the server itself, so the CN is
> the FQDN as mentioned above. Then import into (or generate directly inside)
> the keystore. You can also use the provided NiFi TLS Toolkit to automate
> this entire process [2].
> Certificate chain
> 0 s:/C=CA/ST=CA/L=Palo Alto/O=Cloudera/OU=EDH Team, Cloudera/CN=Ricky
> Saltzer
> i:/C=CA/ST=CA/L=Palo Alto/O=Cloudera/OU=EDH Team, Cloudera/CN=Ricky Saltzercompare
> to an example connection to google.com:443:
>
> Certificate chain
> 0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/*CN=*.google.com
> <http://google.com>*
> i:/C=US/O=Google Inc/CN=Google Internet Authority G2
> 1 s:/C=US/O=Google Inc/CN=Google Internet Authority G2
> i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
> 2 s:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
> i:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority
>
> 3. It appears the certificate you are using is expired. If you look at the
> output of the OpenSSL command you ran, you can see that the result code was
> *10*, not *0* as would be in the case of a successful connection (I
> understand that it looks successful because it negotiates a key and
> encrypts the channel).
> New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384
> Server public key is 2048 bit
> Secure Renegotiation IS supported
> Compression: NONE
> Expansion: NONE
> SSL-Session:
> Protocol : TLSv1.2
> Cipher : ECDHE-RSA-AES256-SHA384
> Session-ID: 581F5A5...41153B
> Session-ID-ctx:
> Master-Key: CEEED2...944C30
> Key-Arg : None
> PSK identity: None
> PSK identity hint: None
> SRP username: None
> Start Time: 1478449748
> Timeout : 300 (sec)
> Verify return code: *10* *(certificate has expired)*
> ---
> HTTP/1.1 400 No URI
> Content-Length: 0
> Connection: close
> Server: Jetty(9.3.9.v20160517)
> I rebuilt your certificate from the Base64-encoded version in the output,
> and it appears it expired on October 25, 2016. You can verify this by
> copying the section between "-----BEGIN CERTIFICATE-----“ and "-----END
> CERTIFICATE-----“ (including these markers) into a text file and naming it
> “ricky.pem” and then running the following command:
>
> hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
> 🔓 11s @ 17:09:30 $ more ricky.pem
> -----BEGIN CERTIFICATE-----
> MIIDizCCAnOgAwIBAgIEHPLwBzANBgkqhkiG9w0BAQsFADB2MQswCQYDVQQGEwJD
> ...
> 45GrxLz7n/ZXZk8q9aXcrSGaj+HHCyrO0/9NYtMNP2V5wpYcBRiHO9f3sHKgnzU=
> -----END CERTIFICATE-----
> hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
> 🔓 5s @ 17:09:35 $ openssl x509 -in ricky.pem -text -noout
> Certificate:
> Data:
> Version: 3 (0x2)
> Serial Number: 485683207 (0x1cf2f007)
> Signature Algorithm: sha256WithRSAEncryption
> Issuer: C=CA, ST=CA, L=Palo Alto, O=Cloudera, OU=EDH Team,
> Cloudera, CN=Ricky Saltzer
> Validity
> Not Before: Jul 27 17:15:46 2016 GMT
> *Not After : Oct 25 17:15:46 2016 GMT*
> Subject: C=CA, ST=CA, L=Palo Alto, O=Cloudera, OU=EDH Team,
> Cloudera, CN=Ricky Saltzer
> Subject Public Key Info:
> Public Key Algorithm: rsaEncryption
> Public-Key: (2048 bit)
> Modulus:
> 00:f0:4c:fe:7e:af:69:89:f0:4b:a0:e0:df:e8:8c:
> 46:6e:d7:17:4d:17:eb:ee:89:25:ab:5b:35:6a:b2:
> 1e:07:84:95:5f:39:1e:c3:d9:5c:d4:54:09:0c:1a:
> 2a:60:df:9f:ce:7d:87:00:d7:31:23:37:62:8c:02:
> c9:98:a3:d4:f5:3e:71:4e:1d:fb:bb:5b:86:35:af:
> ce:6d:9d:ee:fb:65:d1:01:22:e3:f8:df:c0:81:d7:
> fe:fd:f3:c8:48:c2:bd:23:e2:44:1e:8d:48:5f:81:
> ed:10:62:77:71:3e:15:82:5a:c2:49:70:bb:ef:95:
> d9:ff:f2:37:6c:06:63:fc:3c:98:a3:c9:49:3a:1d:
> b8:37:e3:0b:50:0f:23:5f:26:32:19:c8:59:4e:5a:
> 09:5a:87:7f:b6:40:b4:37:18:20:74:fb:d2:f8:e9:
> 18:e8:d7:23:fa:83:be:2a:c3:33:8d:17:a8:b3:7c:
> 52:03:03:7c:24:80:c5:9c:a5:0d:21:3d:57:7c:d4:
> 0f:78:00:fd:be:69:dd:d8:fe:07:a0:09:77:81:b7:
> bf:fd:0f:bd:8c:9a:35:42:4f:89:8e:31:fd:00:22:
> 16:bb:76:0b:51:b0:c4:81:64:69:f7:c4:5d:37:1e:
> 79:b7:08:d2:5a:6a:a7:43:98:d3:81:a9:08:00:57:
> 2b:c5
> Exponent: 65537 (0x10001)
> X509v3 extensions:
> X509v3 Subject Key Identifier:
> 1C:DD:E1:16:5F:41:CC:7C:69:27:
> E3:B1:6B:78:EF:FA:16:DA:1F:6F
> Signature Algorithm: sha256WithRSAEncryption
> da:5a:31:4b:3b:a6:bf:84:8d:6a:56:2a:82:88:2c:e0:95:49:
> 30:f2:3a:31:ed:b5:bc:5b:92:e6:ad:b3:5b:9f:7a:71:bf:62:
> 0f:38:27:4c:71:02:a1:2f:7f:fa:c0:46:da:78:64:36:35:34:
> 1d:91:b3:f8:e2:23:22:1a:e8:e1:14:2b:15:2c:aa:36:71:39:
> 05:d7:c0:2a:e0:cd:4d:f9:72:07:03:b9:f2:c8:6d:59:fb:c9:
> 25:49:ae:3f:a2:c2:fe:d0:54:1e:45:13:8d:95:05:86:8b:a4:
> c4:7c:df:e1:7a:35:cc:48:e3:cd:20:87:c5:50:a7:43:d2:66:
> 94:81:89:b1:a2:66:20:44:ee:cb:ea:08:c9:7c:aa:61:c7:24:
> 5f:71:ca:6e:c0:a8:15:1f:4a:d3:79:e3:19:fd:7f:86:96:1a:
> 4d:db:9d:38:6f:b3:16:fd:39:68:cf:1c:57:86:26:93:6d:0e:
> 76:6b:aa:b8:9a:e7:c4:05:c6:c5:ae:5a:e1:a1:2a:ad:aa:e3:
> 5b:9a:f1:70:a5:f0:54:76:b3:0e:00:e3:91:ab:c4:bc:fb:9f:
> f6:57:66:4f:2a:f5:a5:dc:ad:21:9a:8f:e1:c7:0b:2a:ce:d3:
> ff:4d:62:d3:0d:3f:65:79:c2:96:1c:05:18:87:3b:d7:f7:b0:
> 72:a0:9f:35
> hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
> 🔓 11s @ 17:09:47 $
>
> I would recommend using the TLS Toolkit to generate a valid CA, server
> certificate, client certificate, and keystore & truststore (seriously, one
> command does all of this) and then re-trying. From this stable baseline, I
> think I’ll be able to better help iron out the cluster/standalone issues.
>
> [1] https://issues.apache.org/jira/browse/NIFI-1990
> [2] https://nifi.apache.org/docs/nifi-docs/html/
> administration-guide.html#tls-generation-toolkit
>
>
> Andy LoPresto
> alopresto@apache.org
> *alopresto.apache@gmail.com <al...@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> On Nov 8, 2016, at 1:31 AM, Andre <an...@fucs.org> wrote:
>
> Rick,
>
> Can you confirm the certificate has a chain of trust with the default JDK
> trusted certs? (i.e. trusted by the JVM)
>
> Cheers
>
>
> On Mon, Nov 7, 2016 at 3:38 AM, Ricky Saltzer <ri...@cloudera.com> wrote:
>
> Hey Andy -
>
> Thanks again for the help.
>
> The error message seems indicative that it doesn't seem to properly read
> the keystore file. One thing to note, if I point the nifi properties to a
> bogus keystore location, then it actually throws a FileNotFound exception.
> This is really odd behavior, because as I mentioned I'm able to start it in
> standalone mode using the correct keystore location, just as I try to do in
> clustered mode.
>
> I've attached both the clustered [1] nifi.properties, which doesn't work,
> and the standalone [2] which does work. . I restored it to a more basic
> configuration without the encrypted configuration, but with SSL still
> enabled. I also added a diff [3] of both the standalone and clustered
> properties file. Note that I I only have NiFi configured to use the
> keystore and not a truststore. I've redacted a few of the values in the
> property files, but be assured that the keystore is most definitely valid
> and is readable / locatable, as starting in standalone works just fine.
>
> I ran the SSL command [4] you gave me, minus the three PEM file arguments
> as I don't have any of those on hand. I hope that is fine. The output still
> looks good.
>
> [1] https://gist.github.com/rickysaltzer/712aa6586592fe6628db2d57cec7a562
> [2] https://gist.github.com/rickysaltzer/fe11c8233e4434eacedd7fd0a083d950
> [3] https://gist.github.com/rickysaltzer/d715c7451eb554a54f14ec6e64da8558
> [4] https://gist.github.com/rickysaltzer/5d7cdeff8868bfc1f47010189735411a
>
>
>
>
> On Fri, Nov 4, 2016 at 7:48 PM, Andy LoPresto <al...@apache.org>
> wrote:
>
> Hi Ricky,
>
> Sorry, should have noted that the debug output goes to
>
> nifi-bootstrap.log,
>
> so thanks Mark for jumping in to help there.
>
> If you look at the top of that log, you’ll note that there is no keystore
> file provided and the truststore loaded is the default JRE cacerts
> truststore. Can you please provide your nifi.properties file in a Gist,
>
> **taking
>
> care to redact any sensitive values** like keystore/truststore passwords,
> although I think from looking at your log output, you are taking
>
> advantage
>
> of the encrypted configuration feature, so even viewing the encrypted
> values should be ok. Could you also please provide the directory listing
> where the keystore and truststore are located including the permissions
>
> and
>
> ownership information?
>
> There may be a bug in the logic between cluster and standalone mode, but
>
> I
>
> haven’t encountered this behavior before. If you can start NiFi in
> standalone mode, could you please provide the output of the following
> command run from the terminal? It will simulate an HTTPS connection to
>
> the
>
> server and verify the key and certificate presented by NiFi.
>
> * host — the NiFi hostname
> * port — the port NiFi is running on
> * path_to_your_cert.pem — the public key certificate identifying the
> client/user (i.e. what you load into your browser to authenticate)
> * path_to_your_key.key — the private key identifying the client/user
> * path_to_your_CA_cert.pem — the public key certificate identifying the
>
> CA
>
> which signed your NiFi server certificate (if self-signed, provide that
> certificate)
>
> $ openssl s_client -connect <host:port> -debug -state -cert
> <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile
> <path_to_your_CA_cert.pem>
>
> Andy LoPresto
> alopresto@apache.org
> *alopresto.apache@gmail.com <al...@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> On Nov 4, 2016, at 11:21 AM, Ricky Saltzer <ri...@cloudera.com> wrote:
>
> Hey guys -
>
> I went ahead and uploaded the boostrap log. I took a look at it and it
> seems to be the same error [1]
>
> [1]:
> https://gist.githubusercontent.com/rickysaltzer/
> b156594f92066873f80928eddf84e7bb/raw/4d0e018038b332f4fdf14644910dfa
> 9e70c57e49/gistfile1.txt
>
> On Fri, Nov 4, 2016 at 2:14 PM, Mark Payne <ma...@hotmail.com> wrote:
>
> Hey Ricky,
>
> When you enable debug logging for SSL, it writes to StdErr (or StdOut?)
>
> so
>
> it will end up in your logs/nifi-bootstrap.log instead of nifi-app.log.
> Can you give that a look?
>
> Thanks
> -Mark
>
> On Nov 4, 2016, at 2:07 PM, Ricky Saltzer <ri...@cloudera.com> wrote:
>
> Hey Andy -
>
> Thanks for the response. I'm currently just trying to get one node in
> clustered mode before adding a second. The keystore is stored locally and
> I've confirmed it's readable, as it was able to start once I took it out
>
> of
>
> clustered mode. I added that line to the bootstrap.conf, but I don't
> believe any additional logging was produced in regards to troubleshooting
> this problem. Just in case, I've attached the entire log [1].
>
> [1]:
> https://gist.githubusercontent.com/rickysaltzer/
>
> ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0
> fedabc88bb/gistfile1.txt <https://gist.githubusercontent.com/
>
> rickysaltzer/
>
> ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0
> fedabc88bb/gistfile1.txt>
>
>
> On Wed, Nov 2, 2016 at 7:08 PM, Andy LoPresto <alopresto@apache.org
>
> <mailto:alopresto@apache.org <al...@apache.org>>> wrote:
>
>
> Hi Ricky,
>
> Sorry to hear you are having this issue. Is the keystore available on
>
> all
>
> nodes of the cluster? It appears from the log message that the keystore
>
> is
>
> not found during startup. To further debug, you can add the following
>
> line
>
> in bootstrap.conf to provide additional logging:
>
> java.arg.15=-Djavax.net.debug=ssl,handshake
>
> Andy LoPresto
> alopresto@apache.org <mailto:alopresto@apache.org <al...@apache.org>>
> *alopresto.apache@gmail.com <mailto:alopresto.apache@gmail.com
> <al...@gmail.com>> <
>
> alopresto.apache@gmail.com <mailto:alopresto.apache@gmail.com
> <al...@gmail.com>>>*
>
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> On Nov 2, 2016, at 2:25 PM, Ricky Saltzer <ri...@cloudera.com> wrote:
>
> Hey all -
>
> I'm using NiFi 1.0 and I'm having an issue using secure mode with a
>
> local
>
> key store while in clustered mode. If I set the node in clustered mode,
>
> and
>
> also provide a valid keystore, I receive a KeyStoreException [1]. If I
>
> set
>
> the configuration to not use clustered mode, NiFi will start up fine
>
> with
>
> the provided key store. Am I supposed to be storing this key store in
> Zookeeper somewhere?
>
>
> [1]
>
>
> Caused by: java.security.KeyStoreException: not found
>
>
> at java.security.KeyStore.getInstance(KeyStore.java:839)
> ~[na:1.8.0_11]
>
> at
> org.apache.nifi.io.socket.SSLContextFactory.<init>(
> SSLContextFactory.java:61)
> ~[nifi-socket-utils-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
> ryBean.getObject(ServerSocketConfigurationFactoryBean.java:45)
> ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
> ryBean.getObject(ServerSocketConfigurationFactoryBean.java:30)
> ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
>
> at
> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.
> doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
> ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
>
> ... 69 common frames omitted
>
> Caused by: java.security.NoSuchAlgorithmException: KeyStore not
>
> available
>
>
> at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
> ~[na:1.8.0_11]
>
> at java.security.Security.getImpl(Security.java:695)
>
> ~[na:1.8.0_11]
>
>
> at java.security.KeyStore.getInstance(KeyStore.java:836)
> ~[na:1.8.0_11]
>
> ... 73 common frames omitted
>
>
>
>
>
> --
> Ricky Saltzer
> http://www.cloudera.com <http://www.cloudera.com/>
>
>
>
>
>
> --
> Ricky Saltzer
> http://www.cloudera.com
>
>
>
>
>
> --
> Ricky Saltzer
> http://www.cloudera.com
>
>
>
--
Ricky Saltzer
http://www.cloudera.com
Re: Secure Cluster Mode Issues
Posted by Andy LoPresto <al...@apache.org>.
Hi Ricky,
Sorry for the delay in my response. I see a couple things which could be causing an issue:
1. You do not specify a hostname for the NiFi node (nifi.web.https.host=) which you should do. I have not set up a one-node cluster before, but my suspicion is that this field needs to be populated, and the value needs to match the CN for the issued server certificate. This value does not need to be unique (i.e. it could be nifi.cloudera.com <http://nifi.cloudera.com/> or localhost and you could run multiple instances on the same machine, identified by the same certificate and different ports), but I think it has to be populated.
1.a. While you have nifi.security.needClientAuth=false set in both cluster and standalone mode, I am not sure if this allows for the truststore to be missing completely. I would have to explore this further. There is a current Jira for clarifying cluster, UI/API, and S2S TLS configurations [1]. You can try using the default JRE truststore, located at “$JAVA_HOME/jre/lib/security/cacerts” with password “changeit”.
2. The certificate you have put into the keystore appears to be a certificate identifying you (Ricky the person) rather than a server entity. You should create a certificate identifying the server itself, so the CN is the FQDN as mentioned above. Then import into (or generate directly inside) the keystore. You can also use the provided NiFi TLS Toolkit to automate this entire process [2].
Certificate chain
0 s:/C=CA/ST=CA/L=Palo Alto/O=Cloudera/OU=EDH Team, Cloudera/CN=Ricky Saltzer
i:/C=CA/ST=CA/L=Palo Alto/O=Cloudera/OU=EDH Team, Cloudera/CN=Ricky Saltzer
compare to an example connection to google.com <http://google.com/>:443:
Certificate chain
0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/CN=*.google.com
i:/C=US/O=Google Inc/CN=Google Internet Authority G2
1 s:/C=US/O=Google Inc/CN=Google Internet Authority G2
i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
2 s:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
i:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority
3. It appears the certificate you are using is expired. If you look at the output of the OpenSSL command you ran, you can see that the result code was 10, not 0 as would be in the case of a successful connection (I understand that it looks successful because it negotiates a key and encrypts the channel).
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-SHA384
Session-ID: 581F5A5...41153B
Session-ID-ctx:
Master-Key: CEEED2...944C30
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1478449748
Timeout : 300 (sec)
Verify return code: 10 (certificate has expired)
---
HTTP/1.1 400 No URI
Content-Length: 0
Connection: close
Server: Jetty(9.3.9.v20160517)
I rebuilt your certificate from the Base64-encoded version in the output, and it appears it expired on October 25, 2016. You can verify this by copying the section between "-----BEGIN CERTIFICATE-----“ and "-----END CERTIFICATE-----“ (including these markers) into a text file and naming it “ricky.pem” and then running the following command:
hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
🔓 11s @ 17:09:30 $ more ricky.pem
-----BEGIN CERTIFICATE-----
MIIDizCCAnOgAwIBAgIEHPLwBzANBgkqhkiG9w0BAQsFADB2MQswCQYDVQQGEwJD
...
45GrxLz7n/ZXZk8q9aXcrSGaj+HHCyrO0/9NYtMNP2V5wpYcBRiHO9f3sHKgnzU=
-----END CERTIFICATE-----
hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
🔓 5s @ 17:09:35 $ openssl x509 -in ricky.pem -text -noout
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 485683207 (0x1cf2f007)
Signature Algorithm: sha256WithRSAEncryption
Issuer: C=CA, ST=CA, L=Palo Alto, O=Cloudera, OU=EDH Team, Cloudera, CN=Ricky Saltzer
Validity
Not Before: Jul 27 17:15:46 2016 GMT
Not After : Oct 25 17:15:46 2016 GMT
Subject: C=CA, ST=CA, L=Palo Alto, O=Cloudera, OU=EDH Team, Cloudera, CN=Ricky Saltzer
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:f0:4c:fe:7e:af:69:89:f0:4b:a0:e0:df:e8:8c:
46:6e:d7:17:4d:17:eb:ee:89:25:ab:5b:35:6a:b2:
1e:07:84:95:5f:39:1e:c3:d9:5c:d4:54:09:0c:1a:
2a:60:df:9f:ce:7d:87:00:d7:31:23:37:62:8c:02:
c9:98:a3:d4:f5:3e:71:4e:1d:fb:bb:5b:86:35:af:
ce:6d:9d:ee:fb:65:d1:01:22:e3:f8:df:c0:81:d7:
fe:fd:f3:c8:48:c2:bd:23:e2:44:1e:8d:48:5f:81:
ed:10:62:77:71:3e:15:82:5a:c2:49:70:bb:ef:95:
d9:ff:f2:37:6c:06:63:fc:3c:98:a3:c9:49:3a:1d:
b8:37:e3:0b:50:0f:23:5f:26:32:19:c8:59:4e:5a:
09:5a:87:7f:b6:40:b4:37:18:20:74:fb:d2:f8:e9:
18:e8:d7:23:fa:83:be:2a:c3:33:8d:17:a8:b3:7c:
52:03:03:7c:24:80:c5:9c:a5:0d:21:3d:57:7c:d4:
0f:78:00:fd:be:69:dd:d8:fe:07:a0:09:77:81:b7:
bf:fd:0f:bd:8c:9a:35:42:4f:89:8e:31:fd:00:22:
16:bb:76:0b:51:b0:c4:81:64:69:f7:c4:5d:37:1e:
79:b7:08:d2:5a:6a:a7:43:98:d3:81:a9:08:00:57:
2b:c5
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Subject Key Identifier:
1C:DD:E1:16:5F:41:CC:7C:69:27:E3:B1:6B:78:EF:FA:16:DA:1F:6F
Signature Algorithm: sha256WithRSAEncryption
da:5a:31:4b:3b:a6:bf:84:8d:6a:56:2a:82:88:2c:e0:95:49:
30:f2:3a:31:ed:b5:bc:5b:92:e6:ad:b3:5b:9f:7a:71:bf:62:
0f:38:27:4c:71:02:a1:2f:7f:fa:c0:46:da:78:64:36:35:34:
1d:91:b3:f8:e2:23:22:1a:e8:e1:14:2b:15:2c:aa:36:71:39:
05:d7:c0:2a:e0:cd:4d:f9:72:07:03:b9:f2:c8:6d:59:fb:c9:
25:49:ae:3f:a2:c2:fe:d0:54:1e:45:13:8d:95:05:86:8b:a4:
c4:7c:df:e1:7a:35:cc:48:e3:cd:20:87:c5:50:a7:43:d2:66:
94:81:89:b1:a2:66:20:44:ee:cb:ea:08:c9:7c:aa:61:c7:24:
5f:71:ca:6e:c0:a8:15:1f:4a:d3:79:e3:19:fd:7f:86:96:1a:
4d:db:9d:38:6f:b3:16:fd:39:68:cf:1c:57:86:26:93:6d:0e:
76:6b:aa:b8:9a:e7:c4:05:c6:c5:ae:5a:e1:a1:2a:ad:aa:e3:
5b:9a:f1:70:a5:f0:54:76:b3:0e:00:e3:91:ab:c4:bc:fb:9f:
f6:57:66:4f:2a:f5:a5:dc:ad:21:9a:8f:e1:c7:0b:2a:ce:d3:
ff:4d:62:d3:0d:3f:65:79:c2:96:1c:05:18:87:3b:d7:f7:b0:
72:a0:9f:35
hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
🔓 11s @ 17:09:47 $
I would recommend using the TLS Toolkit to generate a valid CA, server certificate, client certificate, and keystore & truststore (seriously, one command does all of this) and then re-trying. From this stable baseline, I think I’ll be able to better help iron out the cluster/standalone issues.
[1] https://issues.apache.org/jira/browse/NIFI-1990
[2] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#tls-generation-toolkit
Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Nov 8, 2016, at 1:31 AM, Andre <an...@fucs.org> wrote:
>
> Rick,
>
> Can you confirm the certificate has a chain of trust with the default JDK
> trusted certs? (i.e. trusted by the JVM)
>
> Cheers
>
>
> On Mon, Nov 7, 2016 at 3:38 AM, Ricky Saltzer <ri...@cloudera.com> wrote:
>
>> Hey Andy -
>>
>> Thanks again for the help.
>>
>> The error message seems indicative that it doesn't seem to properly read
>> the keystore file. One thing to note, if I point the nifi properties to a
>> bogus keystore location, then it actually throws a FileNotFound exception.
>> This is really odd behavior, because as I mentioned I'm able to start it in
>> standalone mode using the correct keystore location, just as I try to do in
>> clustered mode.
>>
>> I've attached both the clustered [1] nifi.properties, which doesn't work,
>> and the standalone [2] which does work. . I restored it to a more basic
>> configuration without the encrypted configuration, but with SSL still
>> enabled. I also added a diff [3] of both the standalone and clustered
>> properties file. Note that I I only have NiFi configured to use the
>> keystore and not a truststore. I've redacted a few of the values in the
>> property files, but be assured that the keystore is most definitely valid
>> and is readable / locatable, as starting in standalone works just fine.
>>
>> I ran the SSL command [4] you gave me, minus the three PEM file arguments
>> as I don't have any of those on hand. I hope that is fine. The output still
>> looks good.
>>
>> [1] https://gist.github.com/rickysaltzer/712aa6586592fe6628db2d57cec7a562
>> [2] https://gist.github.com/rickysaltzer/fe11c8233e4434eacedd7fd0a083d950
>> [3] https://gist.github.com/rickysaltzer/d715c7451eb554a54f14ec6e64da8558
>> [4] https://gist.github.com/rickysaltzer/5d7cdeff8868bfc1f47010189735411a
>>
>>
>>
>>
>> On Fri, Nov 4, 2016 at 7:48 PM, Andy LoPresto <al...@apache.org>
>> wrote:
>>
>>> Hi Ricky,
>>>
>>> Sorry, should have noted that the debug output goes to
>> nifi-bootstrap.log,
>>> so thanks Mark for jumping in to help there.
>>>
>>> If you look at the top of that log, you’ll note that there is no keystore
>>> file provided and the truststore loaded is the default JRE cacerts
>>> truststore. Can you please provide your nifi.properties file in a Gist,
>> **taking
>>> care to redact any sensitive values** like keystore/truststore passwords,
>>> although I think from looking at your log output, you are taking
>> advantage
>>> of the encrypted configuration feature, so even viewing the encrypted
>>> values should be ok. Could you also please provide the directory listing
>>> where the keystore and truststore are located including the permissions
>> and
>>> ownership information?
>>>
>>> There may be a bug in the logic between cluster and standalone mode, but
>> I
>>> haven’t encountered this behavior before. If you can start NiFi in
>>> standalone mode, could you please provide the output of the following
>>> command run from the terminal? It will simulate an HTTPS connection to
>> the
>>> server and verify the key and certificate presented by NiFi.
>>>
>>> * host — the NiFi hostname
>>> * port — the port NiFi is running on
>>> * path_to_your_cert.pem — the public key certificate identifying the
>>> client/user (i.e. what you load into your browser to authenticate)
>>> * path_to_your_key.key — the private key identifying the client/user
>>> * path_to_your_CA_cert.pem — the public key certificate identifying the
>> CA
>>> which signed your NiFi server certificate (if self-signed, provide that
>>> certificate)
>>>
>>> $ openssl s_client -connect <host:port> -debug -state -cert
>>> <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile
>>> <path_to_your_CA_cert.pem>
>>>
>>> Andy LoPresto
>>> alopresto@apache.org
>>> *alopresto.apache@gmail.com <al...@gmail.com>*
>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>>>
>>> On Nov 4, 2016, at 11:21 AM, Ricky Saltzer <ri...@cloudera.com> wrote:
>>>
>>> Hey guys -
>>>
>>> I went ahead and uploaded the boostrap log. I took a look at it and it
>>> seems to be the same error [1]
>>>
>>> [1]:
>>> https://gist.githubusercontent.com/rickysaltzer/
>>> b156594f92066873f80928eddf84e7bb/raw/4d0e018038b332f4fdf14644910dfa
>>> 9e70c57e49/gistfile1.txt
>>>
>>> On Fri, Nov 4, 2016 at 2:14 PM, Mark Payne <ma...@hotmail.com> wrote:
>>>
>>> Hey Ricky,
>>>
>>> When you enable debug logging for SSL, it writes to StdErr (or StdOut?)
>> so
>>> it will end up in your logs/nifi-bootstrap.log instead of nifi-app.log.
>>> Can you give that a look?
>>>
>>> Thanks
>>> -Mark
>>>
>>> On Nov 4, 2016, at 2:07 PM, Ricky Saltzer <ri...@cloudera.com> wrote:
>>>
>>> Hey Andy -
>>>
>>> Thanks for the response. I'm currently just trying to get one node in
>>> clustered mode before adding a second. The keystore is stored locally and
>>> I've confirmed it's readable, as it was able to start once I took it out
>>>
>>> of
>>>
>>> clustered mode. I added that line to the bootstrap.conf, but I don't
>>> believe any additional logging was produced in regards to troubleshooting
>>> this problem. Just in case, I've attached the entire log [1].
>>>
>>> [1]:
>>> https://gist.githubusercontent.com/rickysaltzer/
>>>
>>> ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0
>>> fedabc88bb/gistfile1.txt <https://gist.githubusercontent.com/
>> rickysaltzer/
>>> ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0
>>> fedabc88bb/gistfile1.txt>
>>>
>>>
>>> On Wed, Nov 2, 2016 at 7:08 PM, Andy LoPresto <alopresto@apache.org
>>>
>>> <ma...@apache.org>> wrote:
>>>
>>>
>>> Hi Ricky,
>>>
>>> Sorry to hear you are having this issue. Is the keystore available on
>>>
>>> all
>>>
>>> nodes of the cluster? It appears from the log message that the keystore
>>>
>>> is
>>>
>>> not found during startup. To further debug, you can add the following
>>>
>>> line
>>>
>>> in bootstrap.conf to provide additional logging:
>>>
>>> java.arg.15=-Djavax.net.debug=ssl,handshake
>>>
>>> Andy LoPresto
>>> alopresto@apache.org <ma...@apache.org>
>>> *alopresto.apache@gmail.com <ma...@gmail.com> <
>>>
>>> alopresto.apache@gmail.com <ma...@gmail.com>>*
>>>
>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>>>
>>> On Nov 2, 2016, at 2:25 PM, Ricky Saltzer <ri...@cloudera.com> wrote:
>>>
>>> Hey all -
>>>
>>> I'm using NiFi 1.0 and I'm having an issue using secure mode with a
>>>
>>> local
>>>
>>> key store while in clustered mode. If I set the node in clustered mode,
>>>
>>> and
>>>
>>> also provide a valid keystore, I receive a KeyStoreException [1]. If I
>>>
>>> set
>>>
>>> the configuration to not use clustered mode, NiFi will start up fine
>>>
>>> with
>>>
>>> the provided key store. Am I supposed to be storing this key store in
>>> Zookeeper somewhere?
>>>
>>>
>>> [1]
>>>
>>>
>>> Caused by: java.security.KeyStoreException: not found
>>>
>>>
>>> at java.security.KeyStore.getInstance(KeyStore.java:839)
>>> ~[na:1.8.0_11]
>>>
>>> at
>>> org.apache.nifi.io.socket.SSLContextFactory.<init>(
>>> SSLContextFactory.java:61)
>>> ~[nifi-socket-utils-1.0.0.jar:1.0.0]
>>>
>>> at
>>> org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
>>> ryBean.getObject(ServerSocketConfigurationFactoryBean.java:45)
>>> ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
>>>
>>> at
>>> org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
>>> ryBean.getObject(ServerSocketConfigurationFactoryBean.java:30)
>>> ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
>>>
>>> at
>>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.
>>> doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
>>> ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
>>>
>>> ... 69 common frames omitted
>>>
>>> Caused by: java.security.NoSuchAlgorithmException: KeyStore not
>>>
>>> available
>>>
>>>
>>> at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
>>> ~[na:1.8.0_11]
>>>
>>> at java.security.Security.getImpl(Security.java:695)
>>>
>>> ~[na:1.8.0_11]
>>>
>>>
>>> at java.security.KeyStore.getInstance(KeyStore.java:836)
>>> ~[na:1.8.0_11]
>>>
>>> ... 73 common frames omitted
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Ricky Saltzer
>>> http://www.cloudera.com <http://www.cloudera.com/>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Ricky Saltzer
>>> http://www.cloudera.com
>>>
>>>
>>>
>>
>>
>> --
>> Ricky Saltzer
>> http://www.cloudera.com
>>
Re: Secure Cluster Mode Issues
Posted by Andre <an...@fucs.org>.
Rick,
Can you confirm the certificate has a chain of trust with the default JDK
trusted certs? (i.e. trusted by the JVM)
Cheers
On Mon, Nov 7, 2016 at 3:38 AM, Ricky Saltzer <ri...@cloudera.com> wrote:
> Hey Andy -
>
> Thanks again for the help.
>
> The error message seems indicative that it doesn't seem to properly read
> the keystore file. One thing to note, if I point the nifi properties to a
> bogus keystore location, then it actually throws a FileNotFound exception.
> This is really odd behavior, because as I mentioned I'm able to start it in
> standalone mode using the correct keystore location, just as I try to do in
> clustered mode.
>
> I've attached both the clustered [1] nifi.properties, which doesn't work,
> and the standalone [2] which does work. . I restored it to a more basic
> configuration without the encrypted configuration, but with SSL still
> enabled. I also added a diff [3] of both the standalone and clustered
> properties file. Note that I I only have NiFi configured to use the
> keystore and not a truststore. I've redacted a few of the values in the
> property files, but be assured that the keystore is most definitely valid
> and is readable / locatable, as starting in standalone works just fine.
>
> I ran the SSL command [4] you gave me, minus the three PEM file arguments
> as I don't have any of those on hand. I hope that is fine. The output still
> looks good.
>
> [1] https://gist.github.com/rickysaltzer/712aa6586592fe6628db2d57cec7a562
> [2] https://gist.github.com/rickysaltzer/fe11c8233e4434eacedd7fd0a083d950
> [3] https://gist.github.com/rickysaltzer/d715c7451eb554a54f14ec6e64da8558
> [4] https://gist.github.com/rickysaltzer/5d7cdeff8868bfc1f47010189735411a
>
>
>
>
> On Fri, Nov 4, 2016 at 7:48 PM, Andy LoPresto <al...@apache.org>
> wrote:
>
> > Hi Ricky,
> >
> > Sorry, should have noted that the debug output goes to
> nifi-bootstrap.log,
> > so thanks Mark for jumping in to help there.
> >
> > If you look at the top of that log, you’ll note that there is no keystore
> > file provided and the truststore loaded is the default JRE cacerts
> > truststore. Can you please provide your nifi.properties file in a Gist,
> **taking
> > care to redact any sensitive values** like keystore/truststore passwords,
> > although I think from looking at your log output, you are taking
> advantage
> > of the encrypted configuration feature, so even viewing the encrypted
> > values should be ok. Could you also please provide the directory listing
> > where the keystore and truststore are located including the permissions
> and
> > ownership information?
> >
> > There may be a bug in the logic between cluster and standalone mode, but
> I
> > haven’t encountered this behavior before. If you can start NiFi in
> > standalone mode, could you please provide the output of the following
> > command run from the terminal? It will simulate an HTTPS connection to
> the
> > server and verify the key and certificate presented by NiFi.
> >
> > * host — the NiFi hostname
> > * port — the port NiFi is running on
> > * path_to_your_cert.pem — the public key certificate identifying the
> > client/user (i.e. what you load into your browser to authenticate)
> > * path_to_your_key.key — the private key identifying the client/user
> > * path_to_your_CA_cert.pem — the public key certificate identifying the
> CA
> > which signed your NiFi server certificate (if self-signed, provide that
> > certificate)
> >
> > $ openssl s_client -connect <host:port> -debug -state -cert
> > <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile
> > <path_to_your_CA_cert.pem>
> >
> > Andy LoPresto
> > alopresto@apache.org
> > *alopresto.apache@gmail.com <al...@gmail.com>*
> > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> >
> > On Nov 4, 2016, at 11:21 AM, Ricky Saltzer <ri...@cloudera.com> wrote:
> >
> > Hey guys -
> >
> > I went ahead and uploaded the boostrap log. I took a look at it and it
> > seems to be the same error [1]
> >
> > [1]:
> > https://gist.githubusercontent.com/rickysaltzer/
> > b156594f92066873f80928eddf84e7bb/raw/4d0e018038b332f4fdf14644910dfa
> > 9e70c57e49/gistfile1.txt
> >
> > On Fri, Nov 4, 2016 at 2:14 PM, Mark Payne <ma...@hotmail.com> wrote:
> >
> > Hey Ricky,
> >
> > When you enable debug logging for SSL, it writes to StdErr (or StdOut?)
> so
> > it will end up in your logs/nifi-bootstrap.log instead of nifi-app.log.
> > Can you give that a look?
> >
> > Thanks
> > -Mark
> >
> > On Nov 4, 2016, at 2:07 PM, Ricky Saltzer <ri...@cloudera.com> wrote:
> >
> > Hey Andy -
> >
> > Thanks for the response. I'm currently just trying to get one node in
> > clustered mode before adding a second. The keystore is stored locally and
> > I've confirmed it's readable, as it was able to start once I took it out
> >
> > of
> >
> > clustered mode. I added that line to the bootstrap.conf, but I don't
> > believe any additional logging was produced in regards to troubleshooting
> > this problem. Just in case, I've attached the entire log [1].
> >
> > [1]:
> > https://gist.githubusercontent.com/rickysaltzer/
> >
> > ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0
> > fedabc88bb/gistfile1.txt <https://gist.githubusercontent.com/
> rickysaltzer/
> > ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0
> > fedabc88bb/gistfile1.txt>
> >
> >
> > On Wed, Nov 2, 2016 at 7:08 PM, Andy LoPresto <alopresto@apache.org
> >
> > <ma...@apache.org>> wrote:
> >
> >
> > Hi Ricky,
> >
> > Sorry to hear you are having this issue. Is the keystore available on
> >
> > all
> >
> > nodes of the cluster? It appears from the log message that the keystore
> >
> > is
> >
> > not found during startup. To further debug, you can add the following
> >
> > line
> >
> > in bootstrap.conf to provide additional logging:
> >
> > java.arg.15=-Djavax.net.debug=ssl,handshake
> >
> > Andy LoPresto
> > alopresto@apache.org <ma...@apache.org>
> > *alopresto.apache@gmail.com <ma...@gmail.com> <
> >
> > alopresto.apache@gmail.com <ma...@gmail.com>>*
> >
> > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> >
> > On Nov 2, 2016, at 2:25 PM, Ricky Saltzer <ri...@cloudera.com> wrote:
> >
> > Hey all -
> >
> > I'm using NiFi 1.0 and I'm having an issue using secure mode with a
> >
> > local
> >
> > key store while in clustered mode. If I set the node in clustered mode,
> >
> > and
> >
> > also provide a valid keystore, I receive a KeyStoreException [1]. If I
> >
> > set
> >
> > the configuration to not use clustered mode, NiFi will start up fine
> >
> > with
> >
> > the provided key store. Am I supposed to be storing this key store in
> > Zookeeper somewhere?
> >
> >
> > [1]
> >
> >
> > Caused by: java.security.KeyStoreException: not found
> >
> >
> > at java.security.KeyStore.getInstance(KeyStore.java:839)
> > ~[na:1.8.0_11]
> >
> > at
> > org.apache.nifi.io.socket.SSLContextFactory.<init>(
> > SSLContextFactory.java:61)
> > ~[nifi-socket-utils-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
> > ryBean.getObject(ServerSocketConfigurationFactoryBean.java:45)
> > ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
> > ryBean.getObject(ServerSocketConfigurationFactoryBean.java:30)
> > ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
> >
> > at
> > org.springframework.beans.factory.support.FactoryBeanRegistrySupport.
> > doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
> > ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
> >
> > ... 69 common frames omitted
> >
> > Caused by: java.security.NoSuchAlgorithmException: KeyStore not
> >
> > available
> >
> >
> > at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
> > ~[na:1.8.0_11]
> >
> > at java.security.Security.getImpl(Security.java:695)
> >
> > ~[na:1.8.0_11]
> >
> >
> > at java.security.KeyStore.getInstance(KeyStore.java:836)
> > ~[na:1.8.0_11]
> >
> > ... 73 common frames omitted
> >
> >
> >
> >
> >
> > --
> > Ricky Saltzer
> > http://www.cloudera.com <http://www.cloudera.com/>
> >
> >
> >
> >
> >
> > --
> > Ricky Saltzer
> > http://www.cloudera.com
> >
> >
> >
>
>
> --
> Ricky Saltzer
> http://www.cloudera.com
>
Re: Secure Cluster Mode Issues
Posted by Ricky Saltzer <ri...@cloudera.com>.
Hey Andy -
Thanks again for the help.
The error message seems indicative that it doesn't seem to properly read
the keystore file. One thing to note, if I point the nifi properties to a
bogus keystore location, then it actually throws a FileNotFound exception.
This is really odd behavior, because as I mentioned I'm able to start it in
standalone mode using the correct keystore location, just as I try to do in
clustered mode.
I've attached both the clustered [1] nifi.properties, which doesn't work,
and the standalone [2] which does work. . I restored it to a more basic
configuration without the encrypted configuration, but with SSL still
enabled. I also added a diff [3] of both the standalone and clustered
properties file. Note that I I only have NiFi configured to use the
keystore and not a truststore. I've redacted a few of the values in the
property files, but be assured that the keystore is most definitely valid
and is readable / locatable, as starting in standalone works just fine.
I ran the SSL command [4] you gave me, minus the three PEM file arguments
as I don't have any of those on hand. I hope that is fine. The output still
looks good.
[1] https://gist.github.com/rickysaltzer/712aa6586592fe6628db2d57cec7a562
[2] https://gist.github.com/rickysaltzer/fe11c8233e4434eacedd7fd0a083d950
[3] https://gist.github.com/rickysaltzer/d715c7451eb554a54f14ec6e64da8558
[4] https://gist.github.com/rickysaltzer/5d7cdeff8868bfc1f47010189735411a
On Fri, Nov 4, 2016 at 7:48 PM, Andy LoPresto <al...@apache.org> wrote:
> Hi Ricky,
>
> Sorry, should have noted that the debug output goes to nifi-bootstrap.log,
> so thanks Mark for jumping in to help there.
>
> If you look at the top of that log, you’ll note that there is no keystore
> file provided and the truststore loaded is the default JRE cacerts
> truststore. Can you please provide your nifi.properties file in a Gist, **taking
> care to redact any sensitive values** like keystore/truststore passwords,
> although I think from looking at your log output, you are taking advantage
> of the encrypted configuration feature, so even viewing the encrypted
> values should be ok. Could you also please provide the directory listing
> where the keystore and truststore are located including the permissions and
> ownership information?
>
> There may be a bug in the logic between cluster and standalone mode, but I
> haven’t encountered this behavior before. If you can start NiFi in
> standalone mode, could you please provide the output of the following
> command run from the terminal? It will simulate an HTTPS connection to the
> server and verify the key and certificate presented by NiFi.
>
> * host — the NiFi hostname
> * port — the port NiFi is running on
> * path_to_your_cert.pem — the public key certificate identifying the
> client/user (i.e. what you load into your browser to authenticate)
> * path_to_your_key.key — the private key identifying the client/user
> * path_to_your_CA_cert.pem — the public key certificate identifying the CA
> which signed your NiFi server certificate (if self-signed, provide that
> certificate)
>
> $ openssl s_client -connect <host:port> -debug -state -cert
> <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile
> <path_to_your_CA_cert.pem>
>
> Andy LoPresto
> alopresto@apache.org
> *alopresto.apache@gmail.com <al...@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> On Nov 4, 2016, at 11:21 AM, Ricky Saltzer <ri...@cloudera.com> wrote:
>
> Hey guys -
>
> I went ahead and uploaded the boostrap log. I took a look at it and it
> seems to be the same error [1]
>
> [1]:
> https://gist.githubusercontent.com/rickysaltzer/
> b156594f92066873f80928eddf84e7bb/raw/4d0e018038b332f4fdf14644910dfa
> 9e70c57e49/gistfile1.txt
>
> On Fri, Nov 4, 2016 at 2:14 PM, Mark Payne <ma...@hotmail.com> wrote:
>
> Hey Ricky,
>
> When you enable debug logging for SSL, it writes to StdErr (or StdOut?) so
> it will end up in your logs/nifi-bootstrap.log instead of nifi-app.log.
> Can you give that a look?
>
> Thanks
> -Mark
>
> On Nov 4, 2016, at 2:07 PM, Ricky Saltzer <ri...@cloudera.com> wrote:
>
> Hey Andy -
>
> Thanks for the response. I'm currently just trying to get one node in
> clustered mode before adding a second. The keystore is stored locally and
> I've confirmed it's readable, as it was able to start once I took it out
>
> of
>
> clustered mode. I added that line to the bootstrap.conf, but I don't
> believe any additional logging was produced in regards to troubleshooting
> this problem. Just in case, I've attached the entire log [1].
>
> [1]:
> https://gist.githubusercontent.com/rickysaltzer/
>
> ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0
> fedabc88bb/gistfile1.txt <https://gist.githubusercontent.com/rickysaltzer/
> ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0
> fedabc88bb/gistfile1.txt>
>
>
> On Wed, Nov 2, 2016 at 7:08 PM, Andy LoPresto <alopresto@apache.org
>
> <ma...@apache.org>> wrote:
>
>
> Hi Ricky,
>
> Sorry to hear you are having this issue. Is the keystore available on
>
> all
>
> nodes of the cluster? It appears from the log message that the keystore
>
> is
>
> not found during startup. To further debug, you can add the following
>
> line
>
> in bootstrap.conf to provide additional logging:
>
> java.arg.15=-Djavax.net.debug=ssl,handshake
>
> Andy LoPresto
> alopresto@apache.org <ma...@apache.org>
> *alopresto.apache@gmail.com <ma...@gmail.com> <
>
> alopresto.apache@gmail.com <ma...@gmail.com>>*
>
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> On Nov 2, 2016, at 2:25 PM, Ricky Saltzer <ri...@cloudera.com> wrote:
>
> Hey all -
>
> I'm using NiFi 1.0 and I'm having an issue using secure mode with a
>
> local
>
> key store while in clustered mode. If I set the node in clustered mode,
>
> and
>
> also provide a valid keystore, I receive a KeyStoreException [1]. If I
>
> set
>
> the configuration to not use clustered mode, NiFi will start up fine
>
> with
>
> the provided key store. Am I supposed to be storing this key store in
> Zookeeper somewhere?
>
>
> [1]
>
>
> Caused by: java.security.KeyStoreException: not found
>
>
> at java.security.KeyStore.getInstance(KeyStore.java:839)
> ~[na:1.8.0_11]
>
> at
> org.apache.nifi.io.socket.SSLContextFactory.<init>(
> SSLContextFactory.java:61)
> ~[nifi-socket-utils-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
> ryBean.getObject(ServerSocketConfigurationFactoryBean.java:45)
> ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
> ryBean.getObject(ServerSocketConfigurationFactoryBean.java:30)
> ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
>
> at
> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.
> doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
> ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
>
> ... 69 common frames omitted
>
> Caused by: java.security.NoSuchAlgorithmException: KeyStore not
>
> available
>
>
> at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
> ~[na:1.8.0_11]
>
> at java.security.Security.getImpl(Security.java:695)
>
> ~[na:1.8.0_11]
>
>
> at java.security.KeyStore.getInstance(KeyStore.java:836)
> ~[na:1.8.0_11]
>
> ... 73 common frames omitted
>
>
>
>
>
> --
> Ricky Saltzer
> http://www.cloudera.com <http://www.cloudera.com/>
>
>
>
>
>
> --
> Ricky Saltzer
> http://www.cloudera.com
>
>
>
--
Ricky Saltzer
http://www.cloudera.com
Re: Secure Cluster Mode Issues
Posted by Andy LoPresto <al...@apache.org>.
Hi Ricky,
Sorry, should have noted that the debug output goes to nifi-bootstrap.log, so thanks Mark for jumping in to help there.
If you look at the top of that log, you’ll note that there is no keystore file provided and the truststore loaded is the default JRE cacerts truststore. Can you please provide your nifi.properties file in a Gist, *taking care to redact any sensitive values* like keystore/truststore passwords, although I think from looking at your log output, you are taking advantage of the encrypted configuration feature, so even viewing the encrypted values should be ok. Could you also please provide the directory listing where the keystore and truststore are located including the permissions and ownership information?
There may be a bug in the logic between cluster and standalone mode, but I haven’t encountered this behavior before. If you can start NiFi in standalone mode, could you please provide the output of the following command run from the terminal? It will simulate an HTTPS connection to the server and verify the key and certificate presented by NiFi.
* host — the NiFi hostname
* port — the port NiFi is running on
* path_to_your_cert.pem — the public key certificate identifying the client/user (i.e. what you load into your browser to authenticate)
* path_to_your_key.key — the private key identifying the client/user
* path_to_your_CA_cert.pem — the public key certificate identifying the CA which signed your NiFi server certificate (if self-signed, provide that certificate)
$ openssl s_client -connect <host:port> -debug -state -cert <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile <path_to_your_CA_cert.pem>
Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Nov 4, 2016, at 11:21 AM, Ricky Saltzer <ri...@cloudera.com> wrote:
>
> Hey guys -
>
> I went ahead and uploaded the boostrap log. I took a look at it and it
> seems to be the same error [1]
>
> [1]:
> https://gist.githubusercontent.com/rickysaltzer/b156594f92066873f80928eddf84e7bb/raw/4d0e018038b332f4fdf14644910dfa9e70c57e49/gistfile1.txt
>
> On Fri, Nov 4, 2016 at 2:14 PM, Mark Payne <ma...@hotmail.com> wrote:
>
>> Hey Ricky,
>>
>> When you enable debug logging for SSL, it writes to StdErr (or StdOut?) so
>> it will end up in your logs/nifi-bootstrap.log instead of nifi-app.log.
>> Can you give that a look?
>>
>> Thanks
>> -Mark
>>
>>> On Nov 4, 2016, at 2:07 PM, Ricky Saltzer <ri...@cloudera.com> wrote:
>>>
>>> Hey Andy -
>>>
>>> Thanks for the response. I'm currently just trying to get one node in
>>> clustered mode before adding a second. The keystore is stored locally and
>>> I've confirmed it's readable, as it was able to start once I took it out
>> of
>>> clustered mode. I added that line to the bootstrap.conf, but I don't
>>> believe any additional logging was produced in regards to troubleshooting
>>> this problem. Just in case, I've attached the entire log [1].
>>>
>>> [1]:
>>> https://gist.githubusercontent.com/rickysaltzer/
>> ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0
>> fedabc88bb/gistfile1.txt <https://gist.githubusercontent.com/rickysaltzer/
>> ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0
>> fedabc88bb/gistfile1.txt>
>>>
>>> On Wed, Nov 2, 2016 at 7:08 PM, Andy LoPresto <alopresto@apache.org
>> <ma...@apache.org>> wrote:
>>>
>>>> Hi Ricky,
>>>>
>>>> Sorry to hear you are having this issue. Is the keystore available on
>> all
>>>> nodes of the cluster? It appears from the log message that the keystore
>> is
>>>> not found during startup. To further debug, you can add the following
>> line
>>>> in bootstrap.conf to provide additional logging:
>>>>
>>>> java.arg.15=-Djavax.net.debug=ssl,handshake
>>>>
>>>> Andy LoPresto
>>>> alopresto@apache.org <ma...@apache.org>
>>>> *alopresto.apache@gmail.com <ma...@gmail.com> <
>> alopresto.apache@gmail.com <ma...@gmail.com>>*
>>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>>>>
>>>> On Nov 2, 2016, at 2:25 PM, Ricky Saltzer <ri...@cloudera.com> wrote:
>>>>
>>>> Hey all -
>>>>
>>>> I'm using NiFi 1.0 and I'm having an issue using secure mode with a
>> local
>>>> key store while in clustered mode. If I set the node in clustered mode,
>> and
>>>> also provide a valid keystore, I receive a KeyStoreException [1]. If I
>> set
>>>> the configuration to not use clustered mode, NiFi will start up fine
>> with
>>>> the provided key store. Am I supposed to be storing this key store in
>>>> Zookeeper somewhere?
>>>>
>>>>
>>>> [1]
>>>>
>>>>
>>>> Caused by: java.security.KeyStoreException: not found
>>>>
>>>>
>>>> at java.security.KeyStore.getInstance(KeyStore.java:839)
>>>> ~[na:1.8.0_11]
>>>>
>>>> at
>>>> org.apache.nifi.io.socket.SSLContextFactory.<init>(
>>>> SSLContextFactory.java:61)
>>>> ~[nifi-socket-utils-1.0.0.jar:1.0.0]
>>>>
>>>> at
>>>> org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
>>>> ryBean.getObject(ServerSocketConfigurationFactoryBean.java:45)
>>>> ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
>>>>
>>>> at
>>>> org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
>>>> ryBean.getObject(ServerSocketConfigurationFactoryBean.java:30)
>>>> ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
>>>>
>>>> at
>>>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.
>>>> doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
>>>> ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
>>>>
>>>> ... 69 common frames omitted
>>>>
>>>> Caused by: java.security.NoSuchAlgorithmException: KeyStore not
>> available
>>>>
>>>> at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
>>>> ~[na:1.8.0_11]
>>>>
>>>> at java.security.Security.getImpl(Security.java:695)
>> ~[na:1.8.0_11]
>>>>
>>>> at java.security.KeyStore.getInstance(KeyStore.java:836)
>>>> ~[na:1.8.0_11]
>>>>
>>>> ... 73 common frames omitted
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Ricky Saltzer
>>> http://www.cloudera.com <http://www.cloudera.com/>
>>
>
>
>
> --
> Ricky Saltzer
> http://www.cloudera.com
Re: Secure Cluster Mode Issues
Posted by Ricky Saltzer <ri...@cloudera.com>.
Hey guys -
I went ahead and uploaded the boostrap log. I took a look at it and it
seems to be the same error [1]
[1]:
https://gist.githubusercontent.com/rickysaltzer/b156594f92066873f80928eddf84e7bb/raw/4d0e018038b332f4fdf14644910dfa9e70c57e49/gistfile1.txt
On Fri, Nov 4, 2016 at 2:14 PM, Mark Payne <ma...@hotmail.com> wrote:
> Hey Ricky,
>
> When you enable debug logging for SSL, it writes to StdErr (or StdOut?) so
> it will end up in your logs/nifi-bootstrap.log instead of nifi-app.log.
> Can you give that a look?
>
> Thanks
> -Mark
>
> > On Nov 4, 2016, at 2:07 PM, Ricky Saltzer <ri...@cloudera.com> wrote:
> >
> > Hey Andy -
> >
> > Thanks for the response. I'm currently just trying to get one node in
> > clustered mode before adding a second. The keystore is stored locally and
> > I've confirmed it's readable, as it was able to start once I took it out
> of
> > clustered mode. I added that line to the bootstrap.conf, but I don't
> > believe any additional logging was produced in regards to troubleshooting
> > this problem. Just in case, I've attached the entire log [1].
> >
> > [1]:
> > https://gist.githubusercontent.com/rickysaltzer/
> ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0
> fedabc88bb/gistfile1.txt <https://gist.githubusercontent.com/rickysaltzer/
> ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0
> fedabc88bb/gistfile1.txt>
> >
> > On Wed, Nov 2, 2016 at 7:08 PM, Andy LoPresto <alopresto@apache.org
> <ma...@apache.org>> wrote:
> >
> >> Hi Ricky,
> >>
> >> Sorry to hear you are having this issue. Is the keystore available on
> all
> >> nodes of the cluster? It appears from the log message that the keystore
> is
> >> not found during startup. To further debug, you can add the following
> line
> >> in bootstrap.conf to provide additional logging:
> >>
> >> java.arg.15=-Djavax.net.debug=ssl,handshake
> >>
> >> Andy LoPresto
> >> alopresto@apache.org <ma...@apache.org>
> >> *alopresto.apache@gmail.com <ma...@gmail.com> <
> alopresto.apache@gmail.com <ma...@gmail.com>>*
> >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> >>
> >> On Nov 2, 2016, at 2:25 PM, Ricky Saltzer <ri...@cloudera.com> wrote:
> >>
> >> Hey all -
> >>
> >> I'm using NiFi 1.0 and I'm having an issue using secure mode with a
> local
> >> key store while in clustered mode. If I set the node in clustered mode,
> and
> >> also provide a valid keystore, I receive a KeyStoreException [1]. If I
> set
> >> the configuration to not use clustered mode, NiFi will start up fine
> with
> >> the provided key store. Am I supposed to be storing this key store in
> >> Zookeeper somewhere?
> >>
> >>
> >> [1]
> >>
> >>
> >> Caused by: java.security.KeyStoreException: not found
> >>
> >>
> >> at java.security.KeyStore.getInstance(KeyStore.java:839)
> >> ~[na:1.8.0_11]
> >>
> >> at
> >> org.apache.nifi.io.socket.SSLContextFactory.<init>(
> >> SSLContextFactory.java:61)
> >> ~[nifi-socket-utils-1.0.0.jar:1.0.0]
> >>
> >> at
> >> org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
> >> ryBean.getObject(ServerSocketConfigurationFactoryBean.java:45)
> >> ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
> >>
> >> at
> >> org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
> >> ryBean.getObject(ServerSocketConfigurationFactoryBean.java:30)
> >> ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
> >>
> >> at
> >> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.
> >> doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
> >> ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
> >>
> >> ... 69 common frames omitted
> >>
> >> Caused by: java.security.NoSuchAlgorithmException: KeyStore not
> available
> >>
> >> at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
> >> ~[na:1.8.0_11]
> >>
> >> at java.security.Security.getImpl(Security.java:695)
> ~[na:1.8.0_11]
> >>
> >> at java.security.KeyStore.getInstance(KeyStore.java:836)
> >> ~[na:1.8.0_11]
> >>
> >> ... 73 common frames omitted
> >>
> >>
> >>
> >
> >
> > --
> > Ricky Saltzer
> > http://www.cloudera.com <http://www.cloudera.com/>
>
--
Ricky Saltzer
http://www.cloudera.com
Re: Secure Cluster Mode Issues
Posted by Mark Payne <ma...@hotmail.com>.
Hey Ricky,
When you enable debug logging for SSL, it writes to StdErr (or StdOut?) so it will end up in your logs/nifi-bootstrap.log instead of nifi-app.log.
Can you give that a look?
Thanks
-Mark
> On Nov 4, 2016, at 2:07 PM, Ricky Saltzer <ri...@cloudera.com> wrote:
>
> Hey Andy -
>
> Thanks for the response. I'm currently just trying to get one node in
> clustered mode before adding a second. The keystore is stored locally and
> I've confirmed it's readable, as it was able to start once I took it out of
> clustered mode. I added that line to the bootstrap.conf, but I don't
> believe any additional logging was produced in regards to troubleshooting
> this problem. Just in case, I've attached the entire log [1].
>
> [1]:
> https://gist.githubusercontent.com/rickysaltzer/ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0fedabc88bb/gistfile1.txt <https://gist.githubusercontent.com/rickysaltzer/ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0fedabc88bb/gistfile1.txt>
>
> On Wed, Nov 2, 2016 at 7:08 PM, Andy LoPresto <alopresto@apache.org <ma...@apache.org>> wrote:
>
>> Hi Ricky,
>>
>> Sorry to hear you are having this issue. Is the keystore available on all
>> nodes of the cluster? It appears from the log message that the keystore is
>> not found during startup. To further debug, you can add the following line
>> in bootstrap.conf to provide additional logging:
>>
>> java.arg.15=-Djavax.net.debug=ssl,handshake
>>
>> Andy LoPresto
>> alopresto@apache.org <ma...@apache.org>
>> *alopresto.apache@gmail.com <ma...@gmail.com> <alopresto.apache@gmail.com <ma...@gmail.com>>*
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>>
>> On Nov 2, 2016, at 2:25 PM, Ricky Saltzer <ri...@cloudera.com> wrote:
>>
>> Hey all -
>>
>> I'm using NiFi 1.0 and I'm having an issue using secure mode with a local
>> key store while in clustered mode. If I set the node in clustered mode, and
>> also provide a valid keystore, I receive a KeyStoreException [1]. If I set
>> the configuration to not use clustered mode, NiFi will start up fine with
>> the provided key store. Am I supposed to be storing this key store in
>> Zookeeper somewhere?
>>
>>
>> [1]
>>
>>
>> Caused by: java.security.KeyStoreException: not found
>>
>>
>> at java.security.KeyStore.getInstance(KeyStore.java:839)
>> ~[na:1.8.0_11]
>>
>> at
>> org.apache.nifi.io.socket.SSLContextFactory.<init>(
>> SSLContextFactory.java:61)
>> ~[nifi-socket-utils-1.0.0.jar:1.0.0]
>>
>> at
>> org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
>> ryBean.getObject(ServerSocketConfigurationFactoryBean.java:45)
>> ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
>>
>> at
>> org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
>> ryBean.getObject(ServerSocketConfigurationFactoryBean.java:30)
>> ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
>>
>> at
>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.
>> doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
>> ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
>>
>> ... 69 common frames omitted
>>
>> Caused by: java.security.NoSuchAlgorithmException: KeyStore not available
>>
>> at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
>> ~[na:1.8.0_11]
>>
>> at java.security.Security.getImpl(Security.java:695) ~[na:1.8.0_11]
>>
>> at java.security.KeyStore.getInstance(KeyStore.java:836)
>> ~[na:1.8.0_11]
>>
>> ... 73 common frames omitted
>>
>>
>>
>
>
> --
> Ricky Saltzer
> http://www.cloudera.com <http://www.cloudera.com/>
Re: Secure Cluster Mode Issues
Posted by Ricky Saltzer <ri...@cloudera.com>.
Hey Andy -
Thanks for the response. I'm currently just trying to get one node in
clustered mode before adding a second. The keystore is stored locally and
I've confirmed it's readable, as it was able to start once I took it out of
clustered mode. I added that line to the bootstrap.conf, but I don't
believe any additional logging was produced in regards to troubleshooting
this problem. Just in case, I've attached the entire log [1].
[1]:
https://gist.githubusercontent.com/rickysaltzer/ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0fedabc88bb/gistfile1.txt
On Wed, Nov 2, 2016 at 7:08 PM, Andy LoPresto <al...@apache.org> wrote:
> Hi Ricky,
>
> Sorry to hear you are having this issue. Is the keystore available on all
> nodes of the cluster? It appears from the log message that the keystore is
> not found during startup. To further debug, you can add the following line
> in bootstrap.conf to provide additional logging:
>
> java.arg.15=-Djavax.net.debug=ssl,handshake
>
> Andy LoPresto
> alopresto@apache.org
> *alopresto.apache@gmail.com <al...@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> On Nov 2, 2016, at 2:25 PM, Ricky Saltzer <ri...@cloudera.com> wrote:
>
> Hey all -
>
> I'm using NiFi 1.0 and I'm having an issue using secure mode with a local
> key store while in clustered mode. If I set the node in clustered mode, and
> also provide a valid keystore, I receive a KeyStoreException [1]. If I set
> the configuration to not use clustered mode, NiFi will start up fine with
> the provided key store. Am I supposed to be storing this key store in
> Zookeeper somewhere?
>
>
> [1]
>
>
> Caused by: java.security.KeyStoreException: not found
>
>
> at java.security.KeyStore.getInstance(KeyStore.java:839)
> ~[na:1.8.0_11]
>
> at
> org.apache.nifi.io.socket.SSLContextFactory.<init>(
> SSLContextFactory.java:61)
> ~[nifi-socket-utils-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
> ryBean.getObject(ServerSocketConfigurationFactoryBean.java:45)
> ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
> ryBean.getObject(ServerSocketConfigurationFactoryBean.java:30)
> ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
>
> at
> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.
> doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
> ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
>
> ... 69 common frames omitted
>
> Caused by: java.security.NoSuchAlgorithmException: KeyStore not available
>
> at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
> ~[na:1.8.0_11]
>
> at java.security.Security.getImpl(Security.java:695) ~[na:1.8.0_11]
>
> at java.security.KeyStore.getInstance(KeyStore.java:836)
> ~[na:1.8.0_11]
>
> ... 73 common frames omitted
>
>
>
--
Ricky Saltzer
http://www.cloudera.com
Re: Secure Cluster Mode Issues
Posted by Andy LoPresto <al...@apache.org>.
Hi Ricky,
Sorry to hear you are having this issue. Is the keystore available on all nodes of the cluster? It appears from the log message that the keystore is not found during startup. To further debug, you can add the following line in bootstrap.conf to provide additional logging:
java.arg.15=-Djavax.net.debug=ssl,handshake
Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Nov 2, 2016, at 2:25 PM, Ricky Saltzer <ri...@cloudera.com> wrote:
>
> Hey all -
>
> I'm using NiFi 1.0 and I'm having an issue using secure mode with a local
> key store while in clustered mode. If I set the node in clustered mode, and
> also provide a valid keystore, I receive a KeyStoreException [1]. If I set
> the configuration to not use clustered mode, NiFi will start up fine with
> the provided key store. Am I supposed to be storing this key store in
> Zookeeper somewhere?
>
>
> [1]
>
>
> Caused by: java.security.KeyStoreException: not found
>
>
> at java.security.KeyStore.getInstance(KeyStore.java:839)
> ~[na:1.8.0_11]
>
> at
> org.apache.nifi.io.socket.SSLContextFactory.<init>(SSLContextFactory.java:61)
> ~[nifi-socket-utils-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFactoryBean.getObject(ServerSocketConfigurationFactoryBean.java:45)
> ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFactoryBean.getObject(ServerSocketConfigurationFactoryBean.java:30)
> ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
>
> at
> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
> ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
>
> ... 69 common frames omitted
>
> Caused by: java.security.NoSuchAlgorithmException: KeyStore not available
>
> at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
> ~[na:1.8.0_11]
>
> at java.security.Security.getImpl(Security.java:695) ~[na:1.8.0_11]
>
> at java.security.KeyStore.getInstance(KeyStore.java:836)
> ~[na:1.8.0_11]
>
> ... 73 common frames omitted