You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by alopresto <gi...@git.apache.org> on 2018/04/11 18:49:36 UTC
[GitHub] nifi issue #2628: NIFI-4942 Add capability for encrypt-config tool to use se...
Github user alopresto commented on the issue:
https://github.com/apache/nifi/pull/2628
Here are some instructions and expected outputs to demonstrate that the tool works as intended:
```
# CD to $NIFI_HOME/conf because secure_hash.key must be written to immediate directory
# Populate sensitive properties in nifi.properties in order for something to be encrypted
sed 's/asswd=/&thisIsABadPassword/' nifi.properties >nifi-sensitive.properties
# Initial encryption of nifi.properties
../../../../../nifi-toolkit/nifi-toolkit-assembly/target/nifi-toolkit-1.7.0-SNAPSHOT-bin/nifi-toolkit-1.7.0-SNAPSHOT/bin/encrypt-config.sh -v \
-b bootstrap.conf \
-n nifi-sensitive.properties \
-o nifi-encrypted.properties \
-p passwordpassword
# Example hashes for "passwordpassword"
# secureHashKey=$s0$100801$H8N5sEErC9hOVpQLxUt+oA$RrwImM1uWD59KuA1AxFamK7oPHlnI1uBXEN2lt4CpbM
# secureHashPassword=$s0$100801$dZ04VTEBHxTR8tb6j29q/w$mXsXKxvd3nYXXOSoxobO7gkLaLAdz2dZRqAvPNfOzWE
# Verify secure_hash.key file generated and populated w/ both key and password hash
more secure_hash.key
# Derived key for "passwordpassword"
# nifi.bootstrap.sensitive.key=A2EA52795B33AB2F21C93E7E820D08369F1448478C877F4C710D6E85FD904AE6
# Verify bootstrap.conf file updated with master key value
more bootstrap.conf
# Verify encryption of sensitive properties occurred
more nifi-sensitive.properties | grep 'assw'
more nifi-encrypted.properties | grep 'assw'
# Migration using raw password
../../../../../nifi-toolkit/nifi-toolkit-assembly/target/nifi-toolkit-1.7.0-SNAPSHOT-bin/nifi-toolkit-1.7.0-SNAPSHOT/bin/encrypt-config.sh -v -m \
-b bootstrap.conf \
-n nifi-encrypted.properties \
-o nifi-migrated.properties \
-p thisIsABadPassword \
-w passwordpassword
# Example hashes for "thisIsABadPassword"
# secureHashKey=$s0$100801$Y5rcY+pECpOBw5JBT1esMw$OEfnR/cze9u6ZjHMbd6NzvQltz2cC0qskSH8XeiXcp4
# secureHashPassword=$s0$100801$rxjtgO5m859l6aI1xHIjpA$jAqTpGrJNiTkcIei6HtbCuZmhkPnqDlC3G4RjxRtf18
# Migration using hashed password (single quote escape hash to avoid dollar-sign variable evaluation)
../../../../../nifi-toolkit/nifi-toolkit-assembly/target/nifi-toolkit-1.7.0-SNAPSHOT-bin/nifi-toolkit-1.7.0-SNAPSHOT/bin/encrypt-config.sh -v -m \
-b bootstrap.conf \
-n nifi-migrated.properties \
-o nifi-migrated-from-hash.properties \
-p thisIsABadPassword2 \
-z '$s0$100801$rxjtgO5m859l6aI1xHIjpA$jAqTpGrJNiTkcIei6HtbCuZmhkPnqDlC3G4RjxRtf18'
# Example output
hw12203:.../nifi/nifi-assembly/target/nifi-1.7.0-SNAPSHOT-bin/nifi-1.7.0-SNAPSHOT/conf (NIFI-4942) alopresto
🔓 174714s @ 14:37:45 $ ../../../../../nifi-toolkit/nifi-toolkit-assembly/target/nifi-toolkit-1.7.0-SNAPSHOT-bin/nifi-toolkit-1.7.0-SNAPSHOT/bin/encrypt-config.sh -v \
> -m -b bootstrap.conf \
> -n nifi-migrated.properties \
> -o nifi-migrated-from-hash.properties \
> -p thisIsABadPassword2 \
> -z '$s0$100801$rxjtgO5m859l6aI1xHIjpA$jAqTpGrJNiTkcIei6HtbCuZmhkPnqDlC3G4RjxRtf18'
Listening for transport dt_socket at address: 8000
2018/04/11 14:38:28 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of nifi.properties
2018/04/11 14:38:28 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: bootstrap.conf: bootstrap.conf
2018/04/11 14:38:28 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) nifi.properties: nifi-migrated.properties
2018/04/11 14:38:28 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) nifi.properties: nifi-migrated-from-hash.properties
2018/04/11 14:38:28 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) login-identity-providers.xml: null
2018/04/11 14:38:28 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) login-identity-providers.xml: null
2018/04/11 14:38:28 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) authorizers.xml: null
2018/04/11 14:38:28 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) authorizers.xml: null
2018/04/11 14:38:28 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) flow.xml.gz: null
2018/04/11 14:38:28 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) flow.xml.gz: null
2018/04/11 14:38:28 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Key migration mode activated
2018/04/11 14:38:28 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Secure hash argument present
2018/04/11 14:38:39 INFO [main] org.apache.nifi.properties.NiFiPropertiesLoader: Loaded 151 properties from /Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.7.0-SNAPSHOT-bin/nifi-1.7.0-SNAPSHOT/conf/nifi-migrated.properties
2018/04/11 14:39:03 INFO [main] org.apache.nifi.properties.NiFiPropertiesLoader: Loaded 151 properties from /Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.7.0-SNAPSHOT-bin/nifi-1.7.0-SNAPSHOT/conf/nifi-migrated.properties
2018/04/11 14:39:03 INFO [main] org.apache.nifi.properties.ProtectedNiFiProperties: There are 3 protected properties of 5 sensitive properties (100%)
2018/04/11 14:39:03 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Loaded NiFiProperties instance with 147 properties
2018/04/11 14:39:03 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Protected nifi.security.keyPasswd with aes/gcm/256 -> RDHo9b1qNOhGsvke||lOPiNxqlII0l6V07KsKQKSXgaZPmphdKLnL9QQB9LsGLLQ
2018/04/11 14:39:03 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Updated protection key nifi.security.keyPasswd.protected
2018/04/11 14:39:03 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Protected nifi.security.keystorePasswd with aes/gcm/256 -> /IefK+XgVTpqGSFt||5O8oPFVXMYc6T1WK99++EWJ92V1TIlF9M7+H7xDnno6+mw
2018/04/11 14:39:03 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Updated protection key nifi.security.keystorePasswd.protected
2018/04/11 14:39:03 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Protected nifi.security.truststorePasswd with aes/gcm/256 -> NyF6m6sg7ZH+eZF1||/ex+7LMFmD9iZ964k/BwlzA+Gd9z6Uuv5S5xv2oHQc9ynQ
2018/04/11 14:39:03 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Updated protection key nifi.security.truststorePasswd.protected
2018/04/11 14:39:03 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Final result: 150 keys including 3 protected keys
# Example hashes for "thisIsABadPassword2"
# secureHashKey=$s0$100801$HdGObf10x+ou+Vru5kns4A$U7kDSAwpoyvrWg7P9iRZxersUPln51TQ2IyKJYkiuTI
# secureHashPassword=$s0$100801$h+9JoIzRU8ENNrKLECVPSQ$rZoWTxMe0er7EwW+ZtaWc1TF5N1oV19y1LXK4AjgZ/A
# Migration using hashed key
../../../../../nifi-toolkit/nifi-toolkit-assembly/target/nifi-toolkit-1.7.0-SNAPSHOT-bin/nifi-toolkit-1.7.0-SNAPSHOT/bin/encrypt-config.sh -v -m \
-b bootstrap.conf \
-n nifi-migrated-from-hash.properties \
-o nifi-migrated-from-hash-using-key.properties \
-p thisIsABadPassword3 \
-y '$s0$100801$HdGObf10x+ou+Vru5kns4A$U7kDSAwpoyvrWg7P9iRZxersUPln51TQ2IyKJYkiuTI'
# Example hashes for "thisIsABadPassword3"
# secureHashKey=$s0$100801$t7L8nq9Bc2gtbCnodjKG+A$i4z4dFHnV7CL3LlC5yt8N56me4pQUMx1Z0buZw0MsHA
# secureHashPassword=$s0$100801$SDL+y7LI1HHxeBo4SEe1ig$OtfQWc4cHtTd0msxbgYytVjS0mB2VDm/z8S78sBjcGI
# Migration using "correct" hashed password but with incorrect master key
../../../../../nifi-toolkit/nifi-toolkit-assembly/target/nifi-toolkit-1.7.0-SNAPSHOT-bin/nifi-toolkit-1.7.0-SNAPSHOT/bin/encrypt-config.sh -v -m \
-b bootstrap.conf \
-n nifi-migrated.properties \
-o nifi-migrated-from-hash.properties \
-p thisIsABadPassword2 \
-z '$s0$100801$t7L8nq9Bc2gtbCnodjKG+A$i4z4dFHnV7CL3LlC5yt8N56me4pQUMx1Z0buZw0MsHA'
# Example output
hw12203:.../nifi/nifi-assembly/target/nifi-1.7.0-SNAPSHOT-bin/nifi-1.7.0-SNAPSHOT/conf (NIFI-4942) alopresto
🔓 175013s @ 14:42:44 $ ../../../../../nifi-toolkit/nifi-toolkit-assembly/target/nifi-toolkit-1.7.0-SNAPSHOT-bin/nifi-toolkit-1.7.0-SNAPSHOT/bin/encrypt-config.sh -v -m \
> -b bootstrap.conf \
> -n nifi-migrated.properties \
> -o nifi-migrated-from-hash.properties \
> -p thisIsABadPassword2 \
> -z '$s0$100801$t7L8nq9Bc2gtbCnodjKG+A$i4z4dFHnV7CL3LlC5yt8N56me4pQUMx1Z0buZw0MsHA'
Listening for transport dt_socket at address: 8000
2018/04/11 14:45:58 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of nifi.properties
2018/04/11 14:45:58 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: bootstrap.conf: bootstrap.conf
2018/04/11 14:45:58 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) nifi.properties: nifi-migrated.properties
2018/04/11 14:45:58 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) nifi.properties: nifi-migrated-from-hash.properties
2018/04/11 14:45:58 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) login-identity-providers.xml: null
2018/04/11 14:45:58 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) login-identity-providers.xml: null
2018/04/11 14:45:58 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) authorizers.xml: null
2018/04/11 14:45:58 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) authorizers.xml: null
2018/04/11 14:45:58 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) flow.xml.gz: null
2018/04/11 14:45:58 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) flow.xml.gz: null
2018/04/11 14:45:58 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Key migration mode activated
2018/04/11 14:45:58 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Secure hash argument present
2018/04/11 14:46:15 INFO [main] org.apache.nifi.properties.NiFiPropertiesLoader: Loaded 151 properties from /Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.7.0-SNAPSHOT-bin/nifi-1.7.0-SNAPSHOT/conf/nifi-migrated.properties
2018/04/11 14:46:21 ERROR [main] org.apache.nifi.properties.ConfigEncryptionTool: Encountered an error
java.security.InvalidKeyException: The provided hashed key/password is not correct
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:83)
at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:105)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:60)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:235)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:247)
at org.apache.nifi.properties.ConfigEncryptionTool.getMigrationKey(ConfigEncryptionTool.groovy:553)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:210)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:71)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:117)
at org.apache.nifi.properties.ConfigEncryptionTool.main(ConfigEncryptionTool.groovy:1677)
at org.apache.nifi.properties.ConfigEncryptionTool$main.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
at org.apache.nifi.toolkit.encryptconfig.LegacyMode.run(LegacyMode.groovy:30)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSite.invoke(PogoMetaMethodSite.java:169)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:71)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
at org.apache.nifi.toolkit.encryptconfig.EncryptConfigMain.main(EncryptConfigMain.groovy:109)
The provided hashed key/password is not correct
usage: org.apache.nifi.properties.ConfigEncryptionTool [-h] [-v] [-n <file>] [-o <file>] [-l <file>] [-i <file>] [-a <file>] [-u <file>] [-f <file>] [-g <file>]
[-b <file>] [-k <keyhex>] [-e <keyhex>] [-p <password>] [-w <password>] [-y <hashed_keyhex>] [-z <hashed_password>] [-r] [-m] [-x] [-s <password|keyhex>]
[-A <algorithm>] [-P <algorithm>] [--currentHashParams]
This tool reads from a nifi.properties and/or login-identity-providers.xml file with plain sensitive configuration values, prompts the user for a master key,
and encrypts each value. It will replace the plain value with the protected value in the same file (or write to a new file if specified). It can also be used to
migrate already-encrypted values in those files or in flow.xml.gz to be encrypted with a new key.
-h,--help Show usage information (this message)
-v,--verbose Sets verbose mode (default false)
-n,--niFiProperties <file> The nifi.properties file containing unprotected config values (will be overwritten unless -o is specified)
-o,--outputNiFiProperties <file> The destination nifi.properties file containing protected config values (will not modify input nifi.properties)
-l,--loginIdentityProviders <file> The login-identity-providers.xml file containing unprotected config values (will be overwritten unless -i is
specified)
-i,--outputLoginIdentityProviders <file> The destination login-identity-providers.xml file containing protected config values (will not modify input
login-identity-providers.xml)
-a,--authorizers <file> The authorizers.xml file containing unprotected config values (will be overwritten unless -u is specified)
-u,--outputAuthorizers <file> The destination authorizers.xml file containing protected config values (will not modify input authorizers.xml)
-f,--flowXml <file> The flow.xml.gz file currently protected with old password (will be overwritten unless -g is specified)
-g,--outputFlowXml <file> The destination flow.xml.gz file containing protected config values (will not modify input flow.xml.gz)
-b,--bootstrapConf <file> The bootstrap.conf file to persist master key
-k,--key <keyhex> The raw hexadecimal key to use to encrypt the sensitive properties
-e,--oldKey <keyhex> The old raw hexadecimal key to use during key migration
-p,--password <password> The password from which to derive the key to use to encrypt the sensitive properties
-w,--oldPassword <password> The old password from which to derive the key during migration
-y,--secureHashKey <hashed_keyhex> The old securely-hashed hexadecimal key to authenticate during key migration (see NiFi Admin Guide)
-z,--secureHashPassword <hashed_password> The old securely-hashed password to authenticate during key migration (see NiFi Admin Guide)
-r,--useRawKey If provided, the secure console will prompt for the raw key value in hexadecimal form
-m,--migrate If provided, the nifi.properties and/or login-identity-providers.xml sensitive properties will be re-encrypted with
a new key
-x,--encryptFlowXmlOnly If provided, the properties in flow.xml.gz will be re-encrypted with a new key but the nifi.properties and/or
login-identity-providers.xml files will not be modified
-s,--propsKey <password|keyhex> The password or key to use to encrypt the sensitive processor properties in flow.xml.gz
-A,--newFlowAlgorithm <algorithm> The algorithm to use to encrypt the sensitive processor properties in flow.xml.gz
-P,--newFlowProvider <algorithm> The security provider to use to encrypt the sensitive processor properties in flow.xml.gz
--currentHashParams Returns the current salt and cost params used to store the hashed key/password
Java home: /Users/alopresto/.jenv/versions/1.8
NiFi Toolkit home: /Users/alopresto/Workspace/nifi/nifi-toolkit/nifi-toolkit-assembly/target/nifi-toolkit-1.7.0-SNAPSHOT-bin/nifi-toolkit-1.7.0-SNAPSHOT
hw12203:.../nifi/nifi-assembly/target/nifi-1.7.0-SNAPSHOT-bin/nifi-1.7.0-SNAPSHOT/conf (NIFI-4942) alopresto
🔓 175232s @ 14:46:23 $
```
---