You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by alopresto <gi...@git.apache.org> on 2018/08/03 04:38:40 UTC
[GitHub] nifi pull request #2935: NIFI-5476 Allow TLS toolkit to use externally-signe...
GitHub user alopresto opened a pull request:
https://github.com/apache/nifi/pull/2935
NIFI-5476 Allow TLS toolkit to use externally-signed CA in standalone mode
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [x] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
- [ ] Is your initial contribution a single, squashed commit?
### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
- [x] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?
### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/alopresto/nifi NIFI-5476
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/2935.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2935
----
commit 5e6c43f83ed818308bdd04d84037bdce6d13206a
Author: Andy LoPresto <al...@...>
Date: 2018-07-31T03:47:14Z
NIFI-5473 Added section on using external signed CA for TLS Toolkit.
Increased TOC level to 3 for ASCIIDOC generated documentation.
This closes #2927.
Signed-off-by: Andrew Lim <an...@gmail.com>
commit 36054d5f54bb6ec5a4b0fd4b219a2e33e26cbf16
Author: Andy LoPresto <al...@...>
Date: 2018-08-01T04:26:16Z
NIFI-5476 Added logic to check CA certificate signature against additional certificates.
Moved utility code to TlsHelper.
Added unit tests.
commit 0b4b0f894ec5807137c0117458317173ce30d9ca
Author: Andy LoPresto <al...@...>
Date: 2018-08-01T06:25:02Z
NIFI-5476 Added command-line parsing for additional CA certificate path.
Added unit tests.
commit 4a9f87634e9330203b7146903f0aa399fad1b1ac
Author: Andy LoPresto <al...@...>
Date: 2018-08-03T03:51:56Z
NIFI-5473 Added documentation on using the TLS Toolkit to generate and sign certificates using an externally-signed CA.
commit 1c5629e5970580b33f0732582b5d55bdd2b1292e
Author: Andy LoPresto <al...@...>
Date: 2018-08-03T04:19:17Z
NIFI-5476 Updated toolkit external CA documentation to be inline with additional context from NIFI-5473.
commit 475ba66c9c4e0c33ab212eac0e99c79e6a53dbcf
Author: Andy LoPresto <al...@...>
Date: 2018-08-03T04:27:10Z
NIFI-5476 Cleaned up toolkit documentation.
----
---
[GitHub] nifi issue #2935: NIFI-5476 Allow TLS toolkit to use externally-signed CA in...
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on the issue:
https://github.com/apache/nifi/pull/2935
@pepov I've incorporated your PKCS #8 to PKCS #1 code and made some refactorings. Thank you.
---
[GitHub] nifi issue #2935: NIFI-5476 Allow TLS toolkit to use externally-signed CA in...
Posted by pepov <gi...@git.apache.org>.
Github user pepov commented on the issue:
https://github.com/apache/nifi/pull/2935
I've tested it manually and ran the unit tests as well. I wanted to ask whether we could make the error message more user friendly but then found myself trying to find out whether we can make life easier by making the conversion automatically. I would find this useful, because for me it wasn't trivial at all to find out the problem with the PKCS#8 unencrypted key file, it just looks very much the same as the PKCS#1 format (except the RSA word of course).
If you agree to add this, here is my attempt to do the conversion: https://github.com/pepov/nifi/commit/77256af2bb3178f19b71ab82d398cd7288f7bb55
Otherwise LGTM
---
[GitHub] nifi issue #2935: NIFI-5476 Allow TLS toolkit to use externally-signed CA in...
Posted by mcgilman <gi...@git.apache.org>.
Github user mcgilman commented on the issue:
https://github.com/apache/nifi/pull/2935
Thanks for the contributions @alopresto and @pepov! I'm also a +1 on the changes proposed here.
---
[GitHub] nifi issue #2935: NIFI-5476 Allow TLS toolkit to use externally-signed CA in...
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on the issue:
https://github.com/apache/nifi/pull/2935
@pepov I was able to reproduce the error you had with TinyCert certificates on PR 2927 and added unit tests and ran through it explicitly to make sure this fixes it (`standalone` mode only). Here are the steps to reproduce for anyone interested (for the purposes of this example, all downloaded files are stored in `~/Workspace/scratch/certs/tinycertCA/originalTinyCerts/` and the toolkit binaries are in `/Users/alopresto/Workspace/nifi/nifi-toolkit/nifi-toolkit-assembly/target/nifi-toolkit-1.8.0-SNAPSHOT-bin/nifi-toolkit-1.8.0-SNAPSHOT`):
1. Use [TinyCert](https://tinycert.org) to generate a *root CA* and then generate a *certificate*. Mine are *root CA*: `C=US, ST=CA, L=Santa Monica, O=Apache NiFi, OU=Secure Digital Certificate Signing, CN=Apache NiFi CA` and *certificate*: `C=US, ST=CA, L=Santa Monica, O=Apache NiFi , OU=Intermediate CA, CN=intermediateca.nifi.apache.org`.
1. You will need to download three (3) files:
* the *root CA* certificate. This is available as *[Your CA] > Download > CA Certificate* which results in `cacert.pem`
* the *certificate* public certificate. This is available as *[Your certificate] > Download > Certificate* which results in `cert.pem`
* the *certificate* private key (unencrypted). This is available as *[Your certificate] > Download > Private key (clear)* which results in `key.dec.pem`
1. Convert the downloaded private key from PKCS #8 format to PKCS #1 format via `openssl rsa -in key.dec.pem -out nifi-key.key`
1. Run the toolkit **without** providing the TinyCert CA certificate to verify that the tool will fail (see *Output 1* below)
* `🔓 0s @ 21:34:09 $ ./bin/tls-toolkit.sh standalone -n 'signed_by_tinycert.nifi.apache.org' \
> -S passwordpassword \
> -P passwordpassword \
> -o ~/Workspace/scratch/certs/tinycertCA/originalTinyCerts/ \
> -O`
1. Run the toolkit **with** the TinyCert CA certificate (see *Output 2* below)
* `$ ./bin/tls-toolkit.sh standalone -n 'signed_by_tinycert.nifi.apache.org' -S passwordpassword -P passwordpassword -o ~/Workspace/scratch/certs/tinycertCA/originalTinyCerts/ -O --additionalCACertificate ~/Workspace/scratch/certs/tinycertCA/originalTinyCerts/cacert.pem`
1. Verify the output, a signed node certificate (see *Output 3* below)
* `$ keytool -list -v -keystore signed_by_tinycert.nifi.apache.org/keystore.jks -storepass passwordpassword`
== Output 1
```
2018/08/02 21:34:20 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandaloneCommandLine: No nifiPropertiesFile specified, using embedded one.
2018/08/02 21:34:20 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone: Running standalone certificate generation with output directory /Users/alopresto/Workspace/scratch/certs/tinycertCA/originalTinyCerts
2018/08/02 21:34:20 INFO [main] org.apache.nifi.toolkit.tls.util.TlsHelper: Verifying the certificate signature for CN=intermediateca.nifi.apache.org,OU=Intermediate CA,O=Apache NiFi\ ,L=Santa Monica,ST=CA,C=US
2018/08/02 21:34:20 INFO [main] org.apache.nifi.toolkit.tls.util.TlsHelper: Attempting to verify certificate CN=intermediateca.nifi.apache.org,OU=Intermediate CA,O=Apache NiFi\ ,L=Santa Monica,ST=CA,C=US signature with CN=intermediateca.nifi.apache.org,OU=Intermediate CA,O=Apache NiFi\ ,L=Santa Monica,ST=CA,C=US
2018/08/02 21:34:20 WARN [main] org.apache.nifi.toolkit.tls.util.TlsHelper: Certificate CN=intermediateca.nifi.apache.org,OU=Intermediate CA,O=Apache NiFi\ ,L=Santa Monica,ST=CA,C=US not signed by CN=intermediateca.nifi.apache.org,OU=Intermediate CA,O=Apache NiFi\ ,L=Santa Monica,ST=CA,C=US [certificate does not verify with supplied key]
Error generating TLS configuration. (The signing certificate was not signed by any known certificates)
usage: org.apache.nifi.toolkit.tls.TlsToolkitMain [-a <arg>] [--additionalCACertificate <arg>] [-B <arg>] [-c <arg>] [-C <arg>] [-d <arg>] [-f <arg>] [-g] [-G
<arg>] [-h] [-k <arg>] [-K <arg>] [-n <arg>] [--nifiDnPrefix <arg>] [--nifiDnSuffix <arg>] [-o <arg>] [-O] [-P <arg>] [-s <arg>] [-S <arg>]
[--subjectAlternativeNames <arg>] [-T <arg>]
Creates certificates and config files for nifi cluster.
-a,--keyAlgorithm <arg> Algorithm to use for generated keys. (default: RSA)
--additionalCACertificate <arg> Path to additional CA certificate (used to sign toolkit CA certificate) in PEM format if necessary
-B,--clientCertPassword <arg> Password for client certificate. Must either be one value or one for each client DN. (autogenerate if not specified)
-c,--certificateAuthorityHostname <arg> Hostname of NiFi Certificate Authority (default: localhost)
-C,--clientCertDn <arg> Generate client certificate suitable for use in browser with specified DN. (Can be specified multiple times.)
-d,--days <arg> Number of days issued certificate should be valid for. (default: 1095)
-f,--nifiPropertiesFile <arg> Base nifi.properties file to update. (Embedded file identical to the one in a default NiFi install will be used if
not specified.)
-g,--differentKeyAndKeystorePasswords Use different generated password for the key and the keyStore.
-G,--globalPortSequence <arg> Use sequential ports that are calculated for all hosts according to the provided hostname expressions. (Can be
specified multiple times, MUST BE SAME FROM RUN TO RUN.)
-h,--help Print help and exit.
-k,--keySize <arg> Number of bits for generated keys. (default: 2048)
-K,--keyPassword <arg> Key password to use. Must either be one value or one for each host. (autogenerate if not specified)
-n,--hostnames <arg> Comma separated list of hostnames.
--nifiDnPrefix <arg> String to prepend to hostname(s) when determining DN. (default: CN=)
--nifiDnSuffix <arg> String to append to hostname(s) when determining DN. (default: , OU=NIFI)
-o,--outputDirectory <arg> The directory to output keystores, truststore, config files. (default: ../nifi-toolkit-1.8.0-SNAPSHOT)
-O,--isOverwrite Overwrite existing host output.
-P,--trustStorePassword <arg> Keystore password to use. Must either be one value or one for each host. (autogenerate if not specified)
-s,--signingAlgorithm <arg> Algorithm to use for signing certificates. (default: SHA256WITHRSA)
-S,--keyStorePassword <arg> Keystore password to use. Must either be one value or one for each host. (autogenerate if not specified)
--subjectAlternativeNames <arg> Comma-separated list of domains to use as Subject Alternative Names in the certificate
-T,--keyStoreType <arg> The type of keyStores to generate. (default: jks)
Java home: /Users/alopresto/.jenv/versions/1.8
NiFi Toolkit home: /Users/alopresto/Workspace/nifi/nifi-toolkit/nifi-toolkit-assembly/target/nifi-toolkit-1.8.0-SNAPSHOT-bin/nifi-toolkit-1.8.0-SNAPSHOT
```
== Output 2
```
2018/08/02 21:37:04 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandaloneCommandLine: No nifiPropertiesFile specified, using embedded one.
2018/08/02 21:37:04 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone: Running standalone certificate generation with output directory /Users/alopresto/Workspace/scratch/certs/tinycertCA/originalTinyCerts
2018/08/02 21:37:04 INFO [main] org.apache.nifi.toolkit.tls.util.TlsHelper: Verifying the certificate signature for CN=intermediateca.nifi.apache.org,OU=Intermediate CA,O=Apache NiFi\ ,L=Santa Monica,ST=CA,C=US
2018/08/02 21:37:04 INFO [main] org.apache.nifi.toolkit.tls.util.TlsHelper: Attempting to verify certificate CN=intermediateca.nifi.apache.org,OU=Intermediate CA,O=Apache NiFi\ ,L=Santa Monica,ST=CA,C=US signature with CN=Apache NiFi CA,OU=Secure Digital Certificate Signing,O=Apache NiFi,L=Santa Monica,ST=CA,C=US
2018/08/02 21:37:04 INFO [main] org.apache.nifi.toolkit.tls.util.TlsHelper: Certificate was signed by CN=Apache NiFi CA,OU=Secure Digital Certificate Signing,O=Apache NiFi,L=Santa Monica,ST=CA,C=US
2018/08/02 21:37:04 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone: Using existing CA certificate /Users/alopresto/Workspace/scratch/certs/tinycertCA/originalTinyCerts/nifi-cert.pem and key /Users/alopresto/Workspace/scratch/certs/tinycertCA/originalTinyCerts/nifi-key.key
2018/08/02 21:37:04 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone: Writing new ssl configuration to /Users/alopresto/Workspace/scratch/certs/tinycertCA/originalTinyCerts/signed_by_tinycert.nifi.apache.org
2018/08/02 21:37:05 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone: Successfully generated TLS configuration for signed_by_tinycert.nifi.apache.org 1 in /Users/alopresto/Workspace/scratch/certs/tinycertCA/originalTinyCerts/signed_by_tinycert.nifi.apache.org
2018/08/02 21:37:05 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone: No clientCertDn specified, not generating any client certificates.
2018/08/02 21:37:05 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone: tls-toolkit standalone completed successfully
```
== Output 3
```
Keystore type: JKS
Keystore provider: SUN
Your keystore contains 1 entry
Alias name: nifi-key
Creation date: Aug 2, 2018
Entry type: PrivateKeyEntry
Certificate chain length: 2
Certificate[1]:
Owner: CN=signed_by_tinycert.nifi.apache.org, OU=NIFI
Issuer: CN=intermediateca.nifi.apache.org, OU=Intermediate CA, O="Apache NiFi ", L=Santa Monica, ST=CA, C=US
Serial number: 164fe13268f00000000
Valid from: Thu Aug 02 21:37:05 PDT 2018 until: Sun Aug 01 21:37:05 PDT 2021
Certificate fingerprints:
MD5: 08:5D:54:43:E2:42:5C:D6:5F:C3:4C:ED:F2:62:E8:F6
SHA1: 77:EA:FF:FA:D9:70:34:28:AB:E3:C1:0D:53:DE:D3:5E:97:37:31:E2
SHA256: 28:04:6A:9E:C7:5B:56:63:4C:E7:72:35:C5:9C:F8:7B:2F:41:27:F2:1B:C5:42:DF:17:B0:5A:60:F7:0E:B7:32
Signature algorithm name: SHA256withRSA
Version: 3
Extensions:
#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: BE 9B BF 9B 39 1D 51 4F DA 73 A1 A6 44 3A 72 BC ....9.QO.s..D:r.
0010: 03 58 9E C9 .X..
]
]
#2: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
CA:false
PathLen: undefined
]
#3: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
clientAuth
serverAuth
]
#4: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
DigitalSignature
Non_repudiation
Key_Encipherment
Data_Encipherment
Key_Agreement
]
#5: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
DNSName: signed_by_tinycert.nifi.apache.org
]
#6: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 19 B0 19 97 52 8C 73 D6 CF DE BE DA C8 45 73 3F ....R.s......Es?
0010: E0 79 CE 51 .y.Q
]
]
Certificate[2]:
Owner: CN=intermediateca.nifi.apache.org, OU=Intermediate CA, O="Apache NiFi ", L=Santa Monica, ST=CA, C=US
Issuer: CN=Apache NiFi CA, OU=Secure Digital Certificate Signing, O="Apache NiFi", L=Santa Monica, ST=CA, C=US
Serial number: 3e8c
Valid from: Tue Jul 31 18:09:01 PDT 2018 until: Wed Jul 31 18:09:01 PDT 2019
Certificate fingerprints:
MD5: D7:0E:DB:52:BE:0B:07:1F:46:B9:38:C6:EA:81:06:D2
SHA1: 42:1B:0A:8A:09:D2:E9:10:8C:2C:C4:92:C3:E6:45:FE:E1:2E:D7:42
SHA256: 5E:6C:AC:E2:ED:E4:3B:EA:37:80:31:F7:30:D8:6E:A8:D2:54:86:F3:CA:36:49:C8:4B:78:E6:CA:4F:63:E0:4A
Signature algorithm name: SHA256withRSA
Version: 3
Extensions:
#1: ObjectId: 1.3.6.1.5.5.7.1.1 Criticality=false
AuthorityInfoAccess [
[
accessMethod: ocsp
accessLocation: URIName: http://ocsp.tinycert.org/ca-2032
,
accessMethod: caIssuers
accessLocation: URIName: http://aia.tinycert.org/ca-2032.crt
]
]
#2: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: 10 FD 64 B7 56 7D C7 C4 AE 05 DF 3D A7 10 CB D7 ..d.V......=....
0010: 7F EB 5B E3 ..[.
]
]
#3: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
CA:false
PathLen: undefined
]
#4: ObjectId: 2.5.29.31 Criticality=false
CRLDistributionPoints [
[DistributionPoint:
[URIName: http://crl.tinycert.org/ca-2032.crl]
]]
#5: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
serverAuth
clientAuth
codeSigning
emailProtection
timeStamping
]
#6: ObjectId: 2.5.29.15 Criticality=false
KeyUsage [
DigitalSignature
Non_repudiation
Key_Encipherment
Data_Encipherment
Key_Agreement
]
#7: ObjectId: 2.16.840.1.113730.1.1 Criticality=false
NetscapeCertType [
SSL client
SSL server
S/MIME
Object Signing
]
#8: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: BE 9B BF 9B 39 1D 51 4F DA 73 A1 A6 44 3A 72 BC ....9.QO.s..D:r.
0010: 03 58 9E C9 .X..
]
]
*******************************************
*******************************************
```
---
[GitHub] nifi issue #2935: NIFI-5476 Allow TLS toolkit to use externally-signed CA in...
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on the issue:
https://github.com/apache/nifi/pull/2935
This PR includes commits from NIFI-5473 because GitHub hasn't synced with the `apache/master` branch yet. That should get cleaned up automatically in a few hours.
---
[GitHub] nifi pull request #2935: NIFI-5476 Allow TLS toolkit to use externally-signe...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/nifi/pull/2935
---