You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by alopresto <gi...@git.apache.org> on 2016/11/22 22:27:34 UTC
[GitHub] nifi pull request #1261: NIFI-3024 Added key migration for sensitive process...
GitHub user alopresto opened a pull request:
https://github.com/apache/nifi/pull/1261
NIFI-3024 Added key migration for sensitive processor properties cont\u2026
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [x] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
- [x] Is your initial contribution a single, squashed commit?
### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
- [x] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?
### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
\u2026ained in flow.xml.gz. (nifi.sensitive.props.key)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/alopresto/nifi NIFI-3024-rebased-squashed
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/1261.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1261
----
commit 1b2d79010ef50d54ec01fad4318c2889acee0b42
Author: Andy LoPresto <al...@apache.org>
Date: 2016-11-22T05:19:18Z
NIFI-3024 Added key migration for sensitive processor properties contained in flow.xml.gz. (nifi.sensitive.props.key)
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1261: NIFI-3024 Added key migration for sensitive process...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/nifi/pull/1261
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1261: NIFI-3024 Added key migration for sensitive processor prop...
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on the issue:
https://github.com/apache/nifi/pull/1261
@YolandaMDavis I have good news and bad news -- I am able to reproduce the `pad block corrupted` error on the resources you provided me, but I am not able to reproduce getting the resources to that state when running multiple invocations of the tool in standalone mode. I therefore think it is likely a conflict with cluster synchronization of the flow definition (even though you said each node has the same `nifi.sensitive.props.key` value).
I added two unit tests to the PR:
* `testShouldMigrateFlowXmlContentMultipleTimes` performs flow.xml.gz migration (contained) between 7 different passwords to verify that the flow XML can be decrypted and encrypted multiple times
* `testShouldPerformFullOperationOnFlowXmlMultipleTimes` performs the entire tool invocation between 7 flow passwords (#main invocation with caught System.exit()).
I've included sample output below:
```
...
16/11/23 12:40:05 INFO properties.ConfigEncryptionToolTest: Migrating from thisIsABadPassword4 to thisIsABadPassword5
16/11/23 12:40:05 INFO properties.ConfigEncryptionToolTest: Invoked #main with -n target/tmp/tmp-nifi.properties -f target/tmp/tmp-flow.xml.gz -b target/tmp/tmp_bootstrap.conf -x -v -s thisIsABadPassword5
16/11/23 12:40:05 WARN properties.ConfigEncryptionTool: The source nifi.properties and destination nifi.properties are identical [target/tmp/tmp-nifi.properties] so the original will be overwritten
16/11/23 12:40:05 INFO properties.ConfigEncryptionTool: Handling encryption of flow.xml.gz
16/11/23 12:40:05 WARN properties.ConfigEncryptionTool: The source flow.xml.gz and destination flow.xml.gz are identical [target/tmp/tmp-flow.xml.gz] so the original will be overwritten
16/11/23 12:40:05 INFO properties.ConfigEncryptionTool: bootstrap.conf: target/tmp/tmp_bootstrap.conf
16/11/23 12:40:05 INFO properties.ConfigEncryptionTool: (src) nifi.properties: target/tmp/tmp-nifi.properties
16/11/23 12:40:05 INFO properties.ConfigEncryptionTool: (dest) nifi.properties: target/tmp/tmp-nifi.properties
16/11/23 12:40:05 INFO properties.ConfigEncryptionTool: (src) login-identity-providers.xml: null
16/11/23 12:40:05 INFO properties.ConfigEncryptionTool: (dest) login-identity-providers.xml: null
16/11/23 12:40:05 INFO properties.ConfigEncryptionTool: (src) flow.xml.gz: target/tmp/tmp-flow.xml.gz
16/11/23 12:40:05 INFO properties.ConfigEncryptionTool: (dest) flow.xml.gz: target/tmp/tmp-flow.xml.gz
16/11/23 12:40:05 INFO properties.NiFiPropertiesLoader: Loaded 15 properties from /Users/alopresto/Workspace/nifi/nifi-toolkit/nifi-toolkit-encrypt-config/target/tmp/tmp-nifi.properties
16/11/23 12:40:05 DEBUG properties.ProtectedNiFiProperties: Loaded 15 properties (including 3 protection schemes) into ProtectedNiFiProperties
16/11/23 12:40:05 INFO properties.NiFiPropertiesLoader: Loaded 15 properties from /Users/alopresto/Workspace/nifi/nifi-toolkit/nifi-toolkit-encrypt-config/target/tmp/tmp-nifi.properties
16/11/23 12:40:05 DEBUG properties.ProtectedNiFiProperties: Loaded 15 properties (including 3 protection schemes) into ProtectedNiFiProperties
16/11/23 12:40:05 INFO properties.ProtectedNiFiProperties: There are 3 protected properties of 4 sensitive properties (75%)
16/11/23 12:40:05 INFO properties.AESSensitivePropertyProvider: AES Sensitive Property Provider decrypted a sensitive value successfully
16/11/23 12:40:05 INFO properties.AESSensitivePropertyProvider: AES Sensitive Property Provider decrypted a sensitive value successfully
16/11/23 12:40:05 INFO properties.AESSensitivePropertyProvider: AES Sensitive Property Provider decrypted a sensitive value successfully
16/11/23 12:40:05 INFO properties.ConfigEncryptionTool: Loaded NiFiProperties instance with 12 properties
16/11/23 12:40:05 INFO properties.ConfigEncryptionTool: Decrypted and re-encrypted 2 elements for flow.xml.gz
16/11/23 12:40:05 INFO properties.AESSensitivePropertyProvider: AES Sensitive Property Provider encrypted a sensitive value successfully
16/11/23 12:40:05 INFO properties.ConfigEncryptionTool: Tool is not configured to encrypt nifi.properties, but the existing nifi.properties is encrypted and flow.xml.gz was migrated, so manually persisting the new encrypted value to nifi.properties
16/11/23 12:40:05 DEBUG properties.ProtectedNiFiProperties: Loaded 13 properties (including 1 protection schemes) into ProtectedNiFiProperties
16/11/23 12:40:05 INFO properties.ConfigEncryptionToolTest: [EXPECTED] Tried to exit with status 0.
16/11/23 12:40:05 INFO properties.ConfigEncryptionToolTest: Updated key line: nifi.sensitive.props.key=ufsXsiPb0WNDfJLv||EkMx8/CtHeBtQIezmvONWavw/2y4mAZKbuxOGYWQwQR9F0Y
16/11/23 12:40:05 INFO properties.NiFiPropertiesLoader: Loaded 15 properties from /Users/alopresto/Workspace/nifi/nifi-toolkit/nifi-toolkit-encrypt-config/target/tmp/tmp-nifi.properties
16/11/23 12:40:05 DEBUG properties.ProtectedNiFiProperties: Loaded 15 properties (including 3 protection schemes) into ProtectedNiFiProperties
16/11/23 12:40:05 INFO properties.AESSensitivePropertyProvider: AES Sensitive Property Provider decrypted a sensitive value successfully
16/11/23 12:40:05 INFO properties.AESSensitivePropertyProvider: AES Sensitive Property Provider decrypted a sensitive value successfully
16/11/23 12:40:05 INFO properties.ConfigEncryptionToolTest: Sensitive property key currently protected with aes/gcm/128
16/11/23 12:40:05 INFO properties.ConfigEncryptionToolTest: Updated key line: nifi.bootstrap.sensitive.key=2C576A9585DB862F5ECBEE5B4FFFCCA1
16/11/23 12:40:05 INFO properties.ConfigEncryptionToolTest: Original flow.xml.gz cipher texts: [enc{5bd4893252c3e11255a56cbdfac83976af3d4953f80f6447dd9c5c51a96bdf1af468aa80ab7e521586d496d81b277629}, enc{5bd4893252c3e11255a56cbdfac83976af3d4953f80f6447dd9c5c51a96bdf1af468aa80ab7e521586d496d81b277629}]
16/11/23 12:40:05 INFO properties.ConfigEncryptionToolTest: Updated flow.xml.gz cipher texts: [enc{3151548439c7d34ddd6e1ca40c33a32f87b424fa9f5daf7dc206818eccbfff73f67c8e14d0b952e6b4097a37eafadfea}, enc{3151548439c7d34ddd6e1ca40c33a32f87b424fa9f5daf7dc206818eccbfff73f67c8e14d0b952e6b4097a37eafadfea}]
```
At this point, I propose opening a separate Jira to investigate the cluster multiple-migration issue, and merge this PR to facilitate the upcoming release.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1261: NIFI-3024 Added key migration for sensitive processor prop...
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on the issue:
https://github.com/apache/nifi/pull/1261
@YolandaMDavis caught a tricky one. When performing the migration of `flow.xml.gz` with an already-encrypted `nifi.properties` but using the `-x`/`--encryptFlowXmlOnly` flag, the new `nifi.sensitive.props.key` value is manually encrypted and updated in the `NiFiProperties` object before being re-serialized to the file. However, because this was not going through the normal "encrypt the entire object" logic, the protection scheme in `nifi.sensitive.props.key.protected` was being erased. This resulted in cipher text being stored as the key without an indicator of how to decrypt it.
I added an assertion in the test covering this scenario and was able to reproduce immediately. I applied the fix and pushed. Thanks Yolanda.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1261: NIFI-3024 Added key migration for sensitive processor prop...
Posted by YolandaMDavis <gi...@git.apache.org>.
Github user YolandaMDavis commented on the issue:
https://github.com/apache/nifi/pull/1261
@alopresto I think I isolated part of the issue. Please confirm my use of this command:
/Users/ydavis/dev/tools/nifi-1.1.0/toolkit/nifi-toolkit-1.1.0-pr-1261/bin/encrypt-config.sh -b /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-1/conf/bootstrap.conf -n /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-1/conf/nifi.properties -x -f /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-1/conf/flow.xml.gz -s thisIsADifferentPassword
Please confirm my use of -x in this context. I thought this would prevent updating nifi.properties? The end result did show an update to nifi-properties that excluded the nifi.senstivei.props.key.protected value:
# security properties#
nifi.sensitive.props.key=9ASs7Dpzqm1pOZVR||tJbvXzRFOwGxabcssIkA1l3p3tHnOdo2fPkPXOL3TJgZNkzURMO3qw
nifi.sensitive.props.key.protected=
nifi.sensitive.props.algorithm=PBEWITHMD5AND128BITAES-CBC-OPENSSL
nifi.sensitive.props.provider=BC
nifi.sensitive.props.additional.keys=
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1261: NIFI-3024 Added key migration for sensitive processor prop...
Posted by YolandaMDavis <gi...@git.apache.org>.
Github user YolandaMDavis commented on the issue:
https://github.com/apache/nifi/pull/1261
@alopresto reviewing
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1261: NIFI-3024 Added key migration for sensitive processor prop...
Posted by YolandaMDavis <gi...@git.apache.org>.
Github user YolandaMDavis commented on the issue:
https://github.com/apache/nifi/pull/1261
Encountered issue while attempting the below test cases (3 node cluster):
#initial encryption
/Users/ydavis/dev/tools/nifi-1.1.0/toolkit/nifi-toolkit-1.1.0-pr-1261/bin/encrypt-config.sh -b /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-1/conf/bootstrap.conf -n /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-1/conf/nifi.properties -f /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-1/conf/flow.xml.gz -s thisIsABadPassword -p whomever12345! -v
/Users/ydavis/dev/tools/nifi-1.1.0/toolkit/nifi-toolkit-1.1.0-pr-1261/bin/encrypt-config.sh -b /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-2/conf/bootstrap.conf -n /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-2/conf/nifi.properties -f /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-2/conf/flow.xml.gz -s thisIsABadPassword -p whomever12345! -v
/Users/ydavis/dev/tools/nifi-1.1.0/toolkit/nifi-toolkit-1.1.0-pr-1261/bin/encrypt-config.sh -b /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-3/conf/bootstrap.conf -n /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-3/conf/nifi.properties -f /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-3/conf/flow.xml.gz -s thisIsABadPassword -p whomever12345! -v
#Migration
/Users/ydavis/dev/tools/nifi-1.1.0/toolkit/nifi-toolkit-1.1.0-pr-1261/bin/encrypt-config.sh -b /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-1/conf/bootstrap.conf -n /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-1/conf/nifi.properties -f /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-1/conf/flow.xml.gz -s thisIsABadPassword -m -w whomever12345! -p whatever12345! -v
/Users/ydavis/dev/tools/nifi-1.1.0/toolkit/nifi-toolkit-1.1.0-pr-1261/bin/encrypt-config.sh -b /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-2/conf/bootstrap.conf -n /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-2/conf/nifi.properties -f /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-2/conf/flow.xml.gz -s thisIsABadPassword -m -w whomever12345! -p whatever12345! -v
/Users/ydavis/dev/tools/nifi-1.1.0/toolkit/nifi-toolkit-1.1.0-pr-1261/bin/encrypt-config.sh -b /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-3/conf/bootstrap.conf -n /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-3/conf/nifi.properties -f /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-3/conf/flow.xml.gz -s thisIsABadPassword -m -w whomever12345! -p whatever12345! -v
#Update all encrypt passwords exclude others
/Users/ydavis/dev/tools/nifi-1.1.0/toolkit/nifi-toolkit-1.1.0-pr-1261/bin/encrypt-config.sh -b /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-1/conf/bootstrap.conf -n /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-1/conf/nifi.properties -x -f /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-1/conf/flow.xml.gz -s thisIsADifferentPassword
/Users/ydavis/dev/tools/nifi-1.1.0/toolkit/nifi-toolkit-1.1.0-pr-1261/bin/encrypt-config.sh -b /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-2/conf/bootstrap.conf -n /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-2/conf/nifi.properties -x -f /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-2/conf/flow.xml.gz -s thisIsADifferentPassword
/Users/ydavis/dev/tools/nifi-1.1.0/toolkit/nifi-toolkit-1.1.0-pr-1261/bin/encrypt-config.sh -b /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-3/conf/bootstrap.conf -n /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-3/conf/nifi.properties -x -f /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-3/conf/flow.xml.gz -s thisIsADifferentPassword
All 3 above worked successfully and cluster was able to start and stop each time as well as run flow.
I attempted my fourth test case to change 1 node's senstive key using the command below:
/Users/ydavis/dev/tools/nifi-1.1.0/toolkit/nifi-toolkit-1.1.0-pr-1261/bin/encrypt-config.sh -b /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-2/conf/bootstrap.conf -n /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-2/conf/nifi.properties -x -f /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-2/conf/flow.xml.gz -s thisIsASpecialPassword
On this run config tool reported the following error:
HW11205:nifi-1.1.0 ydavis$ /Users/ydavis/dev/tools/nifi-1.1.0/toolkit/nifi-toolkit-1.1.0-pr-1261/bin/encrypt-config.sh -b /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-2/conf/bootstrap.conf -n /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-2/conf/nifi.properties -x -f /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-2/conf/flow.xml.gz -s thisIsASpecialPassword
2016/11/23 11:06:40 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source nifi.properties and destination nifi.properties are identical [/Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-2/conf/nifi.properties] so the original will be overwritten
2016/11/23 11:06:40 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source flow.xml.gz and destination flow.xml.gz are identical [/Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-2/conf/flow.xml.gz] so the original will be overwritten
2016/11/23 11:06:40 INFO [main] org.apache.nifi.properties.NiFiPropertiesLoader: Loaded 121 properties from /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-2/conf/nifi.properties
2016/11/23 11:06:40 INFO [main] org.apache.nifi.properties.NiFiPropertiesLoader: Loaded 121 properties from /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-2/conf/nifi.properties
2016/11/23 11:06:40 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Loaded NiFiProperties instance with 121 properties
pad block corrupted
Attempting to try this on the other nodes resulted in the same error. Bootstrap/Properties files appeared unchanged however cluster now fails startup.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1261: NIFI-3024 Added key migration for sensitive processor prop...
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on the issue:
https://github.com/apache/nifi/pull/1261
Thanks @brosander . I will open the Jira and link it to [NIFI-3024].
Squashing and merging this PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1261: NIFI-3024 Added key migration for sensitive processor prop...
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on the issue:
https://github.com/apache/nifi/pull/1261
@YolandaMDavis found test errors when running in JCE **limited** mode, which I always forget to check for (seriously, I had a `TODO` in the test). I have resolved the issue (known issue documented in [NIFI-1465](https://issues.apache.org/jira/browse/NIFI-1465) and [NIFI-1255](https://issues.apache.org/jira/browse/NIFI-1255)).
```
hw12203:/Users/alopresto/Workspace/nifi/nifi-toolkit/nifi-toolkit-encrypt-config (NIFI-3024-rebased-squashed) alopresto
\U0001f512 1680s @ 19:00:43 $ mci
[INFO] Scanning for projects...
[INFO] Inspecting build with total of 1 modules...
[INFO] Installing Nexus Staging features:
[INFO] ... total of 1 executions of maven-deploy-plugin replaced with nexus-staging-maven-plugin
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building nifi-toolkit-encrypt-config 1.1.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
...
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.apache.nifi.properties.ConfigEncryptionToolTest
Tests run: 89, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 6.637 sec - in org.apache.nifi.properties.ConfigEncryptionToolTest
Results :
Tests run: 89, Failures: 0, Errors: 0, Skipped: 6
[INFO]
...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 15.805 s
[INFO] Finished at: 2016-11-22T19:01:01-08:00
[INFO] Final Memory: 33M/1062M
[INFO] ------------------------------------------------------------------------
hw12203:/Users/alopresto/Workspace/nifi/nifi-toolkit/nifi-toolkit-encrypt-config (NIFI-3024-rebased-squashed) alopresto
\U0001f512 18s @ 19:01:02 $ jce_unlimited
Enabling JCE unlimited strength crypto policy
/Users/alopresto/Desktop/security/unlimited/US_export_policy.jar -> /Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/security/./US_export_policy.jar
/Users/alopresto/Desktop/security/unlimited/local_policy.jar -> /Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/security/./local_policy.jar
hw12203:/Users/alopresto/Workspace/nifi/nifi-toolkit/nifi-toolkit-encrypt-config (NIFI-3024-rebased-squashed) alopresto
\U0001f513 19s @ 19:01:22 $ mci
[INFO] Scanning for projects...
[INFO] Inspecting build with total of 1 modules...
[INFO] Installing Nexus Staging features:
[INFO] ... total of 1 executions of maven-deploy-plugin replaced with nexus-staging-maven-plugin
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building nifi-toolkit-encrypt-config 1.1.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
...
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.apache.nifi.properties.ConfigEncryptionToolTest
Tests run: 89, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 6.668 sec - in org.apache.nifi.properties.ConfigEncryptionToolTest
Results :
Tests run: 89, Failures: 0, Errors: 0, Skipped: 6
[INFO]
...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 15.041 s
[INFO] Finished at: 2016-11-22T19:01:43-08:00
[INFO] Final Memory: 33M/1068M
[INFO] ------------------------------------------------------------------------
hw12203:/Users/alopresto/Workspace/nifi/nifi-toolkit/nifi-toolkit-encrypt-config (NIFI-3024-rebased-squashed) alopresto
\U0001f513 21s @ 19:01:44 $
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1261: NIFI-3024 Added key migration for sensitive processor prop...
Posted by brosander <gi...@git.apache.org>.
Github user brosander commented on the issue:
https://github.com/apache/nifi/pull/1261
I was unable to repro the issue with my own flow and a standalone nifi node. Since it worked for the first migration and not the second and we have only repro'd so far in clustered mode, I'm inclined to agree with @alopresto
+1 to merging this PR and opening a Jira to investigate the observed behavior further.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1261: NIFI-3024 Added key migration for sensitive processor prop...
Posted by YolandaMDavis <gi...@git.apache.org>.
Github user YolandaMDavis commented on the issue:
https://github.com/apache/nifi/pull/1261
made it passed that issue however on subsequent test encoutered the following (includes command executed):
HW11205:nifi-1.1.0 ydavis$ /Users/ydavis/dev/tools/nifi-1.1.0/toolkit/nifi-toolkit-1.1.0-pr-1261/bin/encrypt-config.sh -b /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-2/conf/bootstrap.conf -n /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-2/conf/nifi.properties -x -f /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-2/conf/flow.xml.gz -s thisIsASpecialPassword
2016/11/23 13:27:59 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source nifi.properties and destination nifi.properties are identical [/Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-2/conf/nifi.properties] so the original will be overwritten
2016/11/23 13:27:59 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source flow.xml.gz and destination flow.xml.gz are identical [/Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-2/conf/flow.xml.gz] so the original will be overwritten
2016/11/23 13:27:59 INFO [main] org.apache.nifi.properties.NiFiPropertiesLoader: Loaded 121 properties from /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-2/conf/nifi.properties
2016/11/23 13:27:59 INFO [main] org.apache.nifi.properties.NiFiPropertiesLoader: Loaded 121 properties from /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-2/conf/nifi.properties
2016/11/23 13:28:00 INFO [main] org.apache.nifi.properties.ProtectedNiFiProperties: There are 1 protected properties of 4 sensitive properties (25%)
2016/11/23 13:28:00 INFO [main] org.apache.nifi.properties.AESSensitivePropertyProvider: AES Sensitive Property Provider decrypted a sensitive value successfully
2016/11/23 13:28:00 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Loaded NiFiProperties instance with 120 properties
pad block corrupted
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1261: NIFI-3024 Added key migration for sensitive processor prop...
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on the issue:
https://github.com/apache/nifi/pull/1261
Reported [NIFI-3098](https://issues.apache.org/jira/browse/NIFI-3098) for further investigation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1261: NIFI-3024 Added key migration for sensitive processor prop...
Posted by YolandaMDavis <gi...@git.apache.org>.
Github user YolandaMDavis commented on the issue:
https://github.com/apache/nifi/pull/1261
@alopresto agreed with this assessment and the move to open to a separate PR. All other tests ran successfully for me beyond this issue
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1261: NIFI-3024 Added key migration for sensitive processor prop...
Posted by YolandaMDavis <gi...@git.apache.org>.
Github user YolandaMDavis commented on the issue:
https://github.com/apache/nifi/pull/1261
@alopresto thanks Andy will confirm
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1261: NIFI-3024 Added key migration for sensitive processor prop...
Posted by YolandaMDavis <gi...@git.apache.org>.
Github user YolandaMDavis commented on the issue:
https://github.com/apache/nifi/pull/1261
was able to get more info I think on the problem in an attempt to recover with migrating:
HW11205:nifi-1.1.0 ydavis$ /Users/ydavis/dev/tools/nifi-1.1.0/toolkit/nifi-toolkit-1.1.0-pr-1261/bin/encrypt-config.sh -b /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-1/conf/bootstrap.conf -n /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-1/conf/nifi.properties -f /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-1/conf/flow.xml.gz -s thisIsABadPassword -m -w whatever12345! -p whomever12345! -v
2016/11/23 11:22:25 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of nifi.properties
2016/11/23 11:22:25 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source nifi.properties and destination nifi.properties are identical [/Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-1/conf/nifi.properties] so the original will be overwritten
2016/11/23 11:22:25 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of flow.xml.gz
2016/11/23 11:22:25 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source flow.xml.gz and destination flow.xml.gz are identical [/Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-1/conf/flow.xml.gz] so the original will be overwritten
2016/11/23 11:22:25 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: bootstrap.conf: /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-1/conf/bootstrap.conf
2016/11/23 11:22:25 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) nifi.properties: /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-1/conf/nifi.properties
2016/11/23 11:22:25 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) nifi.properties: /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-1/conf/nifi.properties
2016/11/23 11:22:25 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) login-identity-providers.xml: null
2016/11/23 11:22:25 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) login-identity-providers.xml: null
2016/11/23 11:22:25 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) flow.xml.gz: /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-1/conf/flow.xml.gz
2016/11/23 11:22:25 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) flow.xml.gz: /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-1/conf/flow.xml.gz
2016/11/23 11:22:25 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Key migration mode activated
2016/11/23 11:22:25 INFO [main] org.apache.nifi.properties.NiFiPropertiesLoader: Loaded 121 properties from /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-1/conf/nifi.properties
2016/11/23 11:22:26 INFO [main] org.apache.nifi.properties.NiFiPropertiesLoader: Loaded 121 properties from /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-1/conf/nifi.properties
2016/11/23 11:22:26 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Loaded NiFiProperties instance with 121 properties
2016/11/23 11:22:26 ERROR [main] org.apache.nifi.properties.ConfigEncryptionTool: Encountered an error
javax.crypto.BadPaddingException: pad block corrupted
at org.bouncycastle.jcajce.provider.symmetric.util.BaseBlockCipher$BufferedGenericBlockCipher.doFinal(Unknown Source)
at org.bouncycastle.jcajce.provider.symmetric.util.BaseBlockCipher.engineDoFinal(Unknown Source)
at javax.crypto.Cipher.doFinal(Cipher.java:2165)
at javax.crypto.Cipher$doFinal$2.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
at org.apache.nifi.properties.ConfigEncryptionTool.decryptFlowElement(ConfigEncryptionTool.groovy:542)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:384)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1019)
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:69)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:52)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:154)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:190)
at org.apache.nifi.properties.ConfigEncryptionTool$_migrateFlowXmlContent_closure4.doCall(ConfigEncryptionTool.groovy:637)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1019)
at groovy.lang.Closure.call(Closure.java:426)
at groovy.lang.Closure.call(Closure.java:442)
at org.codehaus.groovy.runtime.StringGroovyMethods.getReplacement(StringGroovyMethods.java:1543)
at org.codehaus.groovy.runtime.StringGroovyMethods.replaceAll(StringGroovyMethods.java:2580)
at org.codehaus.groovy.runtime.StringGroovyMethods.replaceAll(StringGroovyMethods.java:2506)
at org.codehaus.groovy.runtime.dgm$1127.invoke(Unknown Source)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:133)
at org.apache.nifi.properties.ConfigEncryptionTool.migrateFlowXmlContent(ConfigEncryptionTool.groovy:636)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:210)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:71)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.apache.nifi.properties.ConfigEncryptionTool.main(ConfigEncryptionTool.groovy:1200)
pad block corrupted
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---