You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by alopresto <gi...@git.apache.org> on 2018/04/11 18:48:55 UTC
[GitHub] nifi pull request #2628: NIFI-4942 Add capability for encrypt-config tool to...
GitHub user alopresto opened a pull request:
https://github.com/apache/nifi/pull/2628
NIFI-4942 Add capability for encrypt-config tool to use securely hashed key/password for demonstration of previous knowledge
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [x] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
- [ ] Is your initial contribution a single, squashed commit?
### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
- [x] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/alopresto/nifi NIFI-4942
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/2628.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2628
----
commit b122f9192394b1d50eb319eb2b6e1999aefafd30
Author: Andy LoPresto <al...@...>
Date: 2018-03-23T03:47:53Z
NIFI-4942 [WIP] Added skeleton for secure hash handling in encrypt-config toolkit.
Added test resource for Python scrypt implementation/verifier.
Added unit tests.
commit 4f7bd03d8730ec50babb66850d6dbdc05c1a4834
Author: Andy LoPresto <al...@...>
Date: 2018-03-29T02:11:42Z
NIFI-4942 [WIP] More unit tests passing.
commit 5d43edfac6fa99a8f4cba4174628b9133ff5a6f2
Author: Andy LoPresto <al...@...>
Date: 2018-04-02T23:18:38Z
NIFI-4942 All unit tests pass and test artifacts are cleaned up.
commit c2fc9555704b3df68a63200ff8bc850b2c3730fd
Author: Andy LoPresto <al...@...>
Date: 2018-04-05T00:25:33Z
NIFI-4942 Added RAT exclusions.
commit 513dadfb29595d9755d220678b6233c9fb2a8c66
Author: Andy LoPresto <al...@...>
Date: 2018-04-05T00:26:00Z
NIFI-4942 Added Scrypt hash format checker.
Added unit tests.
commit 411b54f15871227d8128446cc131b85da00a56ef
Author: Andy LoPresto <al...@...>
Date: 2018-04-05T00:26:22Z
NIFI-4942 Added NiFi hash format checker.
Added unit tests.
commit 1b2d6406b94c08b99b434bdc7b47d1cf1eb7319c
Author: Andy LoPresto <al...@...>
Date: 2018-04-05T23:45:24Z
NIFI-4942 Added check for simultaneous use of -z/-y.
Added logic to check hashed password/key.
Added logic to retrieve secure hash from file to compare.
Added unit tests (125/125).
commit 706015ce3fa745a7de0485e68e7cc96efe529d62
Author: Andy LoPresto <al...@...>
Date: 2018-04-10T02:48:44Z
NIFI-4942 Added new ExitCode.
Added logic to return current hash params in JSON for Ambari to consume.
Fixed typos in error messages.
Added unit tests (129/129).
commit 6308fd65b994bad637a205a5e661db4712cde811
Author: Andy LoPresto <al...@...>
Date: 2018-04-10T22:28:49Z
NIFI-4942 Added Scrypt hash format verification for hash check.
Added unit tests.
commit 0b2d12f9a440d207f1d21c06566b839a18efd089
Author: Andy LoPresto <al...@...>
Date: 2018-04-11T17:54:40Z
NIFI-4942 Fixed RAT checks.
----
---
[GitHub] nifi issue #2628: NIFI-4942 Add capability for encrypt-config tool to use se...
Posted by YolandaMDavis <gi...@git.apache.org>.
Github user YolandaMDavis commented on the issue:
https://github.com/apache/nifi/pull/2628
Spoke with @alopresto offline. He highlighted that is it important to maintain control of location of secure-hash.key file in order to prevent calling application from piping into a file that is controlled externally. We want to keep it as secured as possible.
Concerning the documentation update recommendation that can be addressed in a separate item.
+1
Will merge shortly
---
[GitHub] nifi issue #2628: NIFI-4942 Add capability for encrypt-config tool to use se...
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on the issue:
https://github.com/apache/nifi/pull/2628
Here are some instructions and expected outputs to demonstrate that the tool works as intended:
```
# CD to $NIFI_HOME/conf because secure_hash.key must be written to immediate directory
# Populate sensitive properties in nifi.properties in order for something to be encrypted
sed 's/asswd=/&thisIsABadPassword/' nifi.properties >nifi-sensitive.properties
# Initial encryption of nifi.properties
../../../../../nifi-toolkit/nifi-toolkit-assembly/target/nifi-toolkit-1.7.0-SNAPSHOT-bin/nifi-toolkit-1.7.0-SNAPSHOT/bin/encrypt-config.sh -v \
-b bootstrap.conf \
-n nifi-sensitive.properties \
-o nifi-encrypted.properties \
-p passwordpassword
# Example hashes for "passwordpassword"
# secureHashKey=$s0$100801$H8N5sEErC9hOVpQLxUt+oA$RrwImM1uWD59KuA1AxFamK7oPHlnI1uBXEN2lt4CpbM
# secureHashPassword=$s0$100801$dZ04VTEBHxTR8tb6j29q/w$mXsXKxvd3nYXXOSoxobO7gkLaLAdz2dZRqAvPNfOzWE
# Verify secure_hash.key file generated and populated w/ both key and password hash
more secure_hash.key
# Derived key for "passwordpassword"
# nifi.bootstrap.sensitive.key=A2EA52795B33AB2F21C93E7E820D08369F1448478C877F4C710D6E85FD904AE6
# Verify bootstrap.conf file updated with master key value
more bootstrap.conf
# Verify encryption of sensitive properties occurred
more nifi-sensitive.properties | grep 'assw'
more nifi-encrypted.properties | grep 'assw'
# Migration using raw password
../../../../../nifi-toolkit/nifi-toolkit-assembly/target/nifi-toolkit-1.7.0-SNAPSHOT-bin/nifi-toolkit-1.7.0-SNAPSHOT/bin/encrypt-config.sh -v -m \
-b bootstrap.conf \
-n nifi-encrypted.properties \
-o nifi-migrated.properties \
-p thisIsABadPassword \
-w passwordpassword
# Example hashes for "thisIsABadPassword"
# secureHashKey=$s0$100801$Y5rcY+pECpOBw5JBT1esMw$OEfnR/cze9u6ZjHMbd6NzvQltz2cC0qskSH8XeiXcp4
# secureHashPassword=$s0$100801$rxjtgO5m859l6aI1xHIjpA$jAqTpGrJNiTkcIei6HtbCuZmhkPnqDlC3G4RjxRtf18
# Migration using hashed password (single quote escape hash to avoid dollar-sign variable evaluation)
../../../../../nifi-toolkit/nifi-toolkit-assembly/target/nifi-toolkit-1.7.0-SNAPSHOT-bin/nifi-toolkit-1.7.0-SNAPSHOT/bin/encrypt-config.sh -v -m \
-b bootstrap.conf \
-n nifi-migrated.properties \
-o nifi-migrated-from-hash.properties \
-p thisIsABadPassword2 \
-z '$s0$100801$rxjtgO5m859l6aI1xHIjpA$jAqTpGrJNiTkcIei6HtbCuZmhkPnqDlC3G4RjxRtf18'
# Example output
hw12203:.../nifi/nifi-assembly/target/nifi-1.7.0-SNAPSHOT-bin/nifi-1.7.0-SNAPSHOT/conf (NIFI-4942) alopresto
🔓 174714s @ 14:37:45 $ ../../../../../nifi-toolkit/nifi-toolkit-assembly/target/nifi-toolkit-1.7.0-SNAPSHOT-bin/nifi-toolkit-1.7.0-SNAPSHOT/bin/encrypt-config.sh -v \
> -m -b bootstrap.conf \
> -n nifi-migrated.properties \
> -o nifi-migrated-from-hash.properties \
> -p thisIsABadPassword2 \
> -z '$s0$100801$rxjtgO5m859l6aI1xHIjpA$jAqTpGrJNiTkcIei6HtbCuZmhkPnqDlC3G4RjxRtf18'
Listening for transport dt_socket at address: 8000
2018/04/11 14:38:28 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of nifi.properties
2018/04/11 14:38:28 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: bootstrap.conf: bootstrap.conf
2018/04/11 14:38:28 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) nifi.properties: nifi-migrated.properties
2018/04/11 14:38:28 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) nifi.properties: nifi-migrated-from-hash.properties
2018/04/11 14:38:28 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) login-identity-providers.xml: null
2018/04/11 14:38:28 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) login-identity-providers.xml: null
2018/04/11 14:38:28 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) authorizers.xml: null
2018/04/11 14:38:28 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) authorizers.xml: null
2018/04/11 14:38:28 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) flow.xml.gz: null
2018/04/11 14:38:28 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) flow.xml.gz: null
2018/04/11 14:38:28 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Key migration mode activated
2018/04/11 14:38:28 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Secure hash argument present
2018/04/11 14:38:39 INFO [main] org.apache.nifi.properties.NiFiPropertiesLoader: Loaded 151 properties from /Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.7.0-SNAPSHOT-bin/nifi-1.7.0-SNAPSHOT/conf/nifi-migrated.properties
2018/04/11 14:39:03 INFO [main] org.apache.nifi.properties.NiFiPropertiesLoader: Loaded 151 properties from /Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.7.0-SNAPSHOT-bin/nifi-1.7.0-SNAPSHOT/conf/nifi-migrated.properties
2018/04/11 14:39:03 INFO [main] org.apache.nifi.properties.ProtectedNiFiProperties: There are 3 protected properties of 5 sensitive properties (100%)
2018/04/11 14:39:03 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Loaded NiFiProperties instance with 147 properties
2018/04/11 14:39:03 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Protected nifi.security.keyPasswd with aes/gcm/256 -> RDHo9b1qNOhGsvke||lOPiNxqlII0l6V07KsKQKSXgaZPmphdKLnL9QQB9LsGLLQ
2018/04/11 14:39:03 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Updated protection key nifi.security.keyPasswd.protected
2018/04/11 14:39:03 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Protected nifi.security.keystorePasswd with aes/gcm/256 -> /IefK+XgVTpqGSFt||5O8oPFVXMYc6T1WK99++EWJ92V1TIlF9M7+H7xDnno6+mw
2018/04/11 14:39:03 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Updated protection key nifi.security.keystorePasswd.protected
2018/04/11 14:39:03 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Protected nifi.security.truststorePasswd with aes/gcm/256 -> NyF6m6sg7ZH+eZF1||/ex+7LMFmD9iZ964k/BwlzA+Gd9z6Uuv5S5xv2oHQc9ynQ
2018/04/11 14:39:03 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Updated protection key nifi.security.truststorePasswd.protected
2018/04/11 14:39:03 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Final result: 150 keys including 3 protected keys
# Example hashes for "thisIsABadPassword2"
# secureHashKey=$s0$100801$HdGObf10x+ou+Vru5kns4A$U7kDSAwpoyvrWg7P9iRZxersUPln51TQ2IyKJYkiuTI
# secureHashPassword=$s0$100801$h+9JoIzRU8ENNrKLECVPSQ$rZoWTxMe0er7EwW+ZtaWc1TF5N1oV19y1LXK4AjgZ/A
# Migration using hashed key
../../../../../nifi-toolkit/nifi-toolkit-assembly/target/nifi-toolkit-1.7.0-SNAPSHOT-bin/nifi-toolkit-1.7.0-SNAPSHOT/bin/encrypt-config.sh -v -m \
-b bootstrap.conf \
-n nifi-migrated-from-hash.properties \
-o nifi-migrated-from-hash-using-key.properties \
-p thisIsABadPassword3 \
-y '$s0$100801$HdGObf10x+ou+Vru5kns4A$U7kDSAwpoyvrWg7P9iRZxersUPln51TQ2IyKJYkiuTI'
# Example hashes for "thisIsABadPassword3"
# secureHashKey=$s0$100801$t7L8nq9Bc2gtbCnodjKG+A$i4z4dFHnV7CL3LlC5yt8N56me4pQUMx1Z0buZw0MsHA
# secureHashPassword=$s0$100801$SDL+y7LI1HHxeBo4SEe1ig$OtfQWc4cHtTd0msxbgYytVjS0mB2VDm/z8S78sBjcGI
# Migration using "correct" hashed password but with incorrect master key
../../../../../nifi-toolkit/nifi-toolkit-assembly/target/nifi-toolkit-1.7.0-SNAPSHOT-bin/nifi-toolkit-1.7.0-SNAPSHOT/bin/encrypt-config.sh -v -m \
-b bootstrap.conf \
-n nifi-migrated.properties \
-o nifi-migrated-from-hash.properties \
-p thisIsABadPassword2 \
-z '$s0$100801$t7L8nq9Bc2gtbCnodjKG+A$i4z4dFHnV7CL3LlC5yt8N56me4pQUMx1Z0buZw0MsHA'
# Example output
hw12203:.../nifi/nifi-assembly/target/nifi-1.7.0-SNAPSHOT-bin/nifi-1.7.0-SNAPSHOT/conf (NIFI-4942) alopresto
🔓 175013s @ 14:42:44 $ ../../../../../nifi-toolkit/nifi-toolkit-assembly/target/nifi-toolkit-1.7.0-SNAPSHOT-bin/nifi-toolkit-1.7.0-SNAPSHOT/bin/encrypt-config.sh -v -m \
> -b bootstrap.conf \
> -n nifi-migrated.properties \
> -o nifi-migrated-from-hash.properties \
> -p thisIsABadPassword2 \
> -z '$s0$100801$t7L8nq9Bc2gtbCnodjKG+A$i4z4dFHnV7CL3LlC5yt8N56me4pQUMx1Z0buZw0MsHA'
Listening for transport dt_socket at address: 8000
2018/04/11 14:45:58 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of nifi.properties
2018/04/11 14:45:58 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: bootstrap.conf: bootstrap.conf
2018/04/11 14:45:58 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) nifi.properties: nifi-migrated.properties
2018/04/11 14:45:58 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) nifi.properties: nifi-migrated-from-hash.properties
2018/04/11 14:45:58 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) login-identity-providers.xml: null
2018/04/11 14:45:58 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) login-identity-providers.xml: null
2018/04/11 14:45:58 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) authorizers.xml: null
2018/04/11 14:45:58 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) authorizers.xml: null
2018/04/11 14:45:58 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) flow.xml.gz: null
2018/04/11 14:45:58 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) flow.xml.gz: null
2018/04/11 14:45:58 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Key migration mode activated
2018/04/11 14:45:58 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Secure hash argument present
2018/04/11 14:46:15 INFO [main] org.apache.nifi.properties.NiFiPropertiesLoader: Loaded 151 properties from /Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.7.0-SNAPSHOT-bin/nifi-1.7.0-SNAPSHOT/conf/nifi-migrated.properties
2018/04/11 14:46:21 ERROR [main] org.apache.nifi.properties.ConfigEncryptionTool: Encountered an error
java.security.InvalidKeyException: The provided hashed key/password is not correct
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:83)
at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:105)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:60)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:235)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:247)
at org.apache.nifi.properties.ConfigEncryptionTool.getMigrationKey(ConfigEncryptionTool.groovy:553)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:210)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:71)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:117)
at org.apache.nifi.properties.ConfigEncryptionTool.main(ConfigEncryptionTool.groovy:1677)
at org.apache.nifi.properties.ConfigEncryptionTool$main.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
at org.apache.nifi.toolkit.encryptconfig.LegacyMode.run(LegacyMode.groovy:30)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSite.invoke(PogoMetaMethodSite.java:169)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:71)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
at org.apache.nifi.toolkit.encryptconfig.EncryptConfigMain.main(EncryptConfigMain.groovy:109)
The provided hashed key/password is not correct
usage: org.apache.nifi.properties.ConfigEncryptionTool [-h] [-v] [-n <file>] [-o <file>] [-l <file>] [-i <file>] [-a <file>] [-u <file>] [-f <file>] [-g <file>]
[-b <file>] [-k <keyhex>] [-e <keyhex>] [-p <password>] [-w <password>] [-y <hashed_keyhex>] [-z <hashed_password>] [-r] [-m] [-x] [-s <password|keyhex>]
[-A <algorithm>] [-P <algorithm>] [--currentHashParams]
This tool reads from a nifi.properties and/or login-identity-providers.xml file with plain sensitive configuration values, prompts the user for a master key,
and encrypts each value. It will replace the plain value with the protected value in the same file (or write to a new file if specified). It can also be used to
migrate already-encrypted values in those files or in flow.xml.gz to be encrypted with a new key.
-h,--help Show usage information (this message)
-v,--verbose Sets verbose mode (default false)
-n,--niFiProperties <file> The nifi.properties file containing unprotected config values (will be overwritten unless -o is specified)
-o,--outputNiFiProperties <file> The destination nifi.properties file containing protected config values (will not modify input nifi.properties)
-l,--loginIdentityProviders <file> The login-identity-providers.xml file containing unprotected config values (will be overwritten unless -i is
specified)
-i,--outputLoginIdentityProviders <file> The destination login-identity-providers.xml file containing protected config values (will not modify input
login-identity-providers.xml)
-a,--authorizers <file> The authorizers.xml file containing unprotected config values (will be overwritten unless -u is specified)
-u,--outputAuthorizers <file> The destination authorizers.xml file containing protected config values (will not modify input authorizers.xml)
-f,--flowXml <file> The flow.xml.gz file currently protected with old password (will be overwritten unless -g is specified)
-g,--outputFlowXml <file> The destination flow.xml.gz file containing protected config values (will not modify input flow.xml.gz)
-b,--bootstrapConf <file> The bootstrap.conf file to persist master key
-k,--key <keyhex> The raw hexadecimal key to use to encrypt the sensitive properties
-e,--oldKey <keyhex> The old raw hexadecimal key to use during key migration
-p,--password <password> The password from which to derive the key to use to encrypt the sensitive properties
-w,--oldPassword <password> The old password from which to derive the key during migration
-y,--secureHashKey <hashed_keyhex> The old securely-hashed hexadecimal key to authenticate during key migration (see NiFi Admin Guide)
-z,--secureHashPassword <hashed_password> The old securely-hashed password to authenticate during key migration (see NiFi Admin Guide)
-r,--useRawKey If provided, the secure console will prompt for the raw key value in hexadecimal form
-m,--migrate If provided, the nifi.properties and/or login-identity-providers.xml sensitive properties will be re-encrypted with
a new key
-x,--encryptFlowXmlOnly If provided, the properties in flow.xml.gz will be re-encrypted with a new key but the nifi.properties and/or
login-identity-providers.xml files will not be modified
-s,--propsKey <password|keyhex> The password or key to use to encrypt the sensitive processor properties in flow.xml.gz
-A,--newFlowAlgorithm <algorithm> The algorithm to use to encrypt the sensitive processor properties in flow.xml.gz
-P,--newFlowProvider <algorithm> The security provider to use to encrypt the sensitive processor properties in flow.xml.gz
--currentHashParams Returns the current salt and cost params used to store the hashed key/password
Java home: /Users/alopresto/.jenv/versions/1.8
NiFi Toolkit home: /Users/alopresto/Workspace/nifi/nifi-toolkit/nifi-toolkit-assembly/target/nifi-toolkit-1.7.0-SNAPSHOT-bin/nifi-toolkit-1.7.0-SNAPSHOT
hw12203:.../nifi/nifi-assembly/target/nifi-1.7.0-SNAPSHOT-bin/nifi-1.7.0-SNAPSHOT/conf (NIFI-4942) alopresto
🔓 175232s @ 14:46:23 $
```
---
[GitHub] nifi pull request #2628: NIFI-4942 Add capability for encrypt-config tool to...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/nifi/pull/2628
---
[GitHub] nifi issue #2628: NIFI-4942 Add capability for encrypt-config tool to use se...
Posted by YolandaMDavis <gi...@git.apache.org>.
Github user YolandaMDavis commented on the issue:
https://github.com/apache/nifi/pull/2628
@alopresto ran through test cases and this works as expected. One question for you is there a way to designate an output location for the secure-hash.key file? Such as if I want to pipe it to stdin or just to another location?
Also as a side note I tested with -p (password) input where it may contain certain characters (@, -, and &). The '&' caused the script to fail but also to stall, needed to do a Control-C to break out of it. The below is resolved by simply including quotes around the password but may be good to document for users:
`ydavis$ /Users/ydavis/dev/projects/nifi/nifi-toolkit/nifi-toolkit-assembly/target/nifi-toolkit-1.7.0-SNAPSHOT-bin/nifi-toolkit-1.7.0-SNAPSHOT/bin/encrypt-config.sh -v -m -b bootstrap.conf -n nifi-migrated-from-hash-key-break-2.properties -o nifi-migrated-from-hash-key-break-3.properties -p thisIs&ABadPassword4 -y '$s0$100801$j8z9NeI9DZEBTbCzOaQJbA$MI0iN/ZPQ5bk4YxcgJ2H95gCToQy3ZbIr7B6OMxB3oA'
[1] 5576
-bash: ABadPassword4: command not found
HW13535:conf ydavis$ 2018/04/12 17:30:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of nifi.properties
2018/04/12 17:30:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: bootstrap.conf: bootstrap.conf
2018/04/12 17:30:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) nifi.properties: nifi-migrated-from-hash-key-break-2.properties
2018/04/12 17:30:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) nifi.properties: nifi-migrated-from-hash-key-break-3.properties
2018/04/12 17:30:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) login-identity-providers.xml: null
2018/04/12 17:30:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) login-identity-providers.xml: null
2018/04/12 17:30:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) authorizers.xml: null
2018/04/12 17:30:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) authorizers.xml: null
2018/04/12 17:30:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) flow.xml.gz: null
2018/04/12 17:30:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) flow.xml.gz: null
2018/04/12 17:30:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Key migration mode activated
2018/04/12 17:30:13 INFO [main] org.apache.nifi.properties.NiFiPropertiesLoader: Loaded 151 properties from /Users/ydavis/dev/tools/nifi-1.7.0-SNAPSHOT/conf/nifi-migrated-from-hash-key-break-2.properties
2018/04/12 17:30:13 ERROR [main] org.apache.nifi.properties.ConfigEncryptionTool: Encountered an error
java.security.KeyException: Cannot derive key from empty/short password -- password must be at least 12 characters
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:83)
at org.codehaus.groovy.reflection.CachedConstructor.doConstructorInvoke(CachedConstructor.java:77)
at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrap.callConstructor(ConstructorSite.java:84)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:60)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:235)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:247)
at org.apache.nifi.properties.ConfigEncryptionTool.deriveKeyFromPassword(ConfigEncryptionTool.groovy:1493)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite$StaticMetaMethodSiteNoUnwrapNoCoerce.invoke(StaticMetaMethodSite.java:151)
at org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite.callStatic(StaticMetaMethodSite.java:102)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallStatic(CallSiteArray.java:56)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callStatic(AbstractCallSite.java:194)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callStatic(AbstractCallSite.java:206)
at org.apache.nifi.properties.ConfigEncryptionTool.getKeyInternal(ConfigEncryptionTool.groovy:527)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:210)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:59)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:52)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:154)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:190)
at org.apache.nifi.properties.ConfigEncryptionTool.getKey(ConfigEncryptionTool.groovy:542)
at org.apache.nifi.properties.ConfigEncryptionTool.getKey(ConfigEncryptionTool.groovy:541)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:210)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:71)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:117)
at org.apache.nifi.properties.ConfigEncryptionTool.main(ConfigEncryptionTool.groovy:1659)
at org.apache.nifi.properties.ConfigEncryptionTool$main.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
at org.apache.nifi.toolkit.encryptconfig.LegacyMode.run(LegacyMode.groovy:30)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSite.invoke(PogoMetaMethodSite.java:169)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:71)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
at org.apache.nifi.toolkit.encryptconfig.EncryptConfigMain.main(EncryptConfigMain.groovy:109)
Cannot derive key from empty/short password -- password must be at least 12 characters
usage: org.apache.nifi.properties.ConfigEncryptionTool [-h] [-v] [-n <file>] [-o <file>] [-l <file>] [-i <file>] [-a <file>] [-u <file>] [-f <file>] [-g <file>]
[-b <file>] [-k <keyhex>] [-e <keyhex>] [-p <password>] [-w <password>] [-y <hashed_keyhex>] [-z <hashed_password>] [-r] [-m] [-x] [-s <password|keyhex>]
[-A <algorithm>] [-P <algorithm>] [--currentHashParams]
This tool reads from a nifi.properties and/or login-identity-providers.xml file with plain sensitive configuration values, prompts the user for a master key,
and encrypts each value. It will replace the plain value with the protected value in the same file (or write to a new file if specified). It can also be used to
migrate already-encrypted values in those files or in flow.xml.gz to be encrypted with a new key.
-h,--help Show usage information (this message)
-v,--verbose Sets verbose mode (default false)
-n,--niFiProperties <file> The nifi.properties file containing unprotected config values (will be overwritten unless -o is specified)
-o,--outputNiFiProperties <file> The destination nifi.properties file containing protected config values (will not modify input nifi.properties)
-l,--loginIdentityProviders <file> The login-identity-providers.xml file containing unprotected config values (will be overwritten unless -i is
specified)
-i,--outputLoginIdentityProviders <file> The destination login-identity-providers.xml file containing protected config values (will not modify input
login-identity-providers.xml)
-a,--authorizers <file> The authorizers.xml file containing unprotected config values (will be overwritten unless -u is specified)
-u,--outputAuthorizers <file> The destination authorizers.xml file containing protected config values (will not modify input authorizers.xml)
-f,--flowXml <file> The flow.xml.gz file currently protected with old password (will be overwritten unless -g is specified)
-g,--outputFlowXml <file> The destination flow.xml.gz file containing protected config values (will not modify input flow.xml.gz)
-b,--bootstrapConf <file> The bootstrap.conf file to persist master key
-k,--key <keyhex> The raw hexadecimal key to use to encrypt the sensitive properties
-e,--oldKey <keyhex> The old raw hexadecimal key to use during key migration
-p,--password <password> The password from which to derive the key to use to encrypt the sensitive properties
-w,--oldPassword <password> The old password from which to derive the key during migration
-y,--secureHashKey <hashed_keyhex> The old securely-hashed hexadecimal key to authenticate during key migration (see NiFi Admin Guide)
-z,--secureHashPassword <hashed_password> The old securely-hashed password to authenticate during key migration (see NiFi Admin Guide)
-r,--useRawKey If provided, the secure console will prompt for the raw key value in hexadecimal form
-m,--migrate If provided, the nifi.properties and/or login-identity-providers.xml sensitive properties will be re-encrypted with
a new key
-x,--encryptFlowXmlOnly If provided, the properties in flow.xml.gz will be re-encrypted with a new key but the nifi.properties and/or
login-identity-providers.xml files will not be modified
-s,--propsKey <password|keyhex> The password or key to use to encrypt the sensitive processor properties in flow.xml.gz
-A,--newFlowAlgorithm <algorithm> The algorithm to use to encrypt the sensitive processor properties in flow.xml.gz
-P,--newFlowProvider <algorithm> The security provider to use to encrypt the sensitive processor properties in flow.xml.gz
--currentHashParams Returns the current salt and cost params used to store the hashed key/password
Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home
NiFi Toolkit home: /Users/ydavis/dev/projects/nifi/nifi-toolkit/nifi-toolkit-assembly/target/nifi-toolkit-1.7.0-SNAPSHOT-bin/nifi-toolkit-1.7.0-SNAPSHOT`
---
[GitHub] nifi issue #2628: NIFI-4942 Add capability for encrypt-config tool to use se...
Posted by YolandaMDavis <gi...@git.apache.org>.
Github user YolandaMDavis commented on the issue:
https://github.com/apache/nifi/pull/2628
@alopresto thanks for addressing this, happy to review. It does look like Travis is failing on a ratcheck related error in nifi-toolkit-encrypt-config
---