You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by kevdoran <gi...@git.apache.org> on 2018/01/05 10:19:41 UTC
[GitHub] nifi pull request #2376: NIFI-4708 Add Registry support to encrypt-config
GitHub user kevdoran opened a pull request:
https://github.com/apache/nifi/pull/2376
NIFI-4708 Add Registry support to encrypt-config
Adds support for NiFI Registry config files to the encrypt-config tool in NiFi Toolkit.
Also adds decryption capability to encrypt-config tool.
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [ ] Has your PR been rebased against the latest commit within the target branch (typically master)?
- [ ] Is your initial contribution a single, squashed commit?
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kevdoran/nifi NIFI-4708
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/2376.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2376
----
commit 7b46d1ad55eea4067e155f1ab819049949ad900b
Author: Kevin Doran <kd...@...>
Date: 2017-12-30T13:54:18Z
NIFI-4708 Add Registry support to encrypt-config
Adds support for NiFI Registry config files to the encrypt-config tool
in NiFi Toolkit.
Also adds decryption capability to encrypt-config tool.
commit 0bbd968d0a64a58e2e33b368ed97b499b3c5d754
Author: Andy LoPresto <al...@...>
Date: 2018-01-03T23:34:38Z
NIFI-4708 [WIP] Added skeleton of new CLI parsing logic to remove "modes" (aka subcommands) and determine which mode logic to delegate to.
commit c75abdbc7af8ca449b6bd8a144fc52a1638f51e3
Author: Kevin Doran <kd...@...>
Date: 2018-01-04T23:23:53Z
NIFI-4708 Remaps updated CLI logic to impl
Remaps the updated CLI parsing logic (which removes modes/subcommands)
to the implementation, adding necessary bridging class for DecryptMode.
commit ab44bbd7f9495ae1923d608bb0dce3254ecba2fe
Author: Kevin Doran <kd...@...>
Date: 2018-01-05T05:11:18Z
NIFI-4708 Add test cases for encrypt-config
----
---
[GitHub] nifi pull request #2376: NIFI-4708 Add Registry support to encrypt-config
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2376#discussion_r159976301
--- Diff: nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/toolkit/encryptconfig/DecryptMode.groovy ---
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.toolkit.encryptconfig
+
+import org.apache.commons.cli.HelpFormatter
+import org.apache.nifi.properties.AESSensitivePropertyProvider
+import org.apache.nifi.properties.SensitivePropertyProvider
+import org.apache.nifi.toolkit.encryptconfig.util.BootstrapUtil
+import org.apache.nifi.toolkit.encryptconfig.util.PropertiesEncryptor
+import org.apache.nifi.toolkit.encryptconfig.util.ToolUtilities
+import org.apache.nifi.toolkit.encryptconfig.util.XmlEncryptor
+import org.apache.nifi.util.console.TextDevices
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+class DecryptMode implements ToolMode {
+
+ private static final Logger logger = LoggerFactory.getLogger(DecryptMode.class)
+
+ enum FileType {
+ properties,
+ xml
+ }
+
+ CliBuilder cli
+
+ DecryptMode() {
+ cli = cliBuilder()
+ }
+
+ void printUsage(String message = "") {
+ if (message) {
+ System.out.println(message)
+ System.out.println()
+ }
+ cli.usage()
+ }
+
+ void printUsageAndExit(String message = "", int exitStatusCode) {
+ printUsage(message)
+ System.exit(exitStatusCode)
+ }
+
+ @Override
+ void run(String[] args) {
+ logger.warn("The decryption capability of this tool is still considered experimental. The results should be manually verified.")
+ try {
+
+ def options = cli.parse(args)
+
+ if (!options || options.h) {
+ printUsageAndExit("", EncryptConfigMain.EXIT_STATUS_OTHER)
+ }
+
+ EncryptConfigLogger.configureLogger(options.v)
+
+ DecryptConfiguration config = new DecryptConfiguration(options)
+
+ run(config)
+
+ } catch (Exception e) {
+ logger.error("Encountered an error: ${e.getMessage()}")
+ logger.debug("", e) // stack trace only when verbose enabled
+ printUsageAndExit(e.getMessage(), EncryptConfigMain.EXIT_STATUS_FAILURE)
+ }
+ }
+
+ void run(DecryptConfiguration config) throws Exception {
+
+ if (!config.fileType) {
+
+ // Try to load the input file to auto-detect the file type
+ boolean isPropertiesFile = PropertiesEncryptor.supportsFile(config.inputFilePath)
+
+ boolean isXmlFile = XmlEncryptor.supportsFile(config.inputFilePath)
+
+ if (ToolUtilities.isExactlyOneTrue(isPropertiesFile, isXmlFile)) {
+ if (isPropertiesFile) {
+ config.fileType = FileType.properties
+ logger.debug("Auto-detection of input file type determined the type to be: ${FileType.properties}")
+ }
+ if (isXmlFile) {
+ config.fileType = FileType.xml
+ logger.debug("Auto-detection of input file type determined the type to be: ${FileType.xml}")
+ }
+ }
+
+ // Could we successfully auto-detect?
+ if (!config.fileType) {
+ throw new RuntimeException("Auto-detection of input file type failed. Please re-run the tool specifying the file type with the -t/--fileType flag.")
+ }
+ }
+
+ String decryptedSerializedContent = null
+ switch (config.fileType) {
+
+ case FileType.properties:
+ PropertiesEncryptor propertiesEncryptor = new PropertiesEncryptor(null, config.decryptionProvider)
+ Properties properties = propertiesEncryptor.loadFile(config.inputFilePath)
+ properties = propertiesEncryptor.decrypt(properties)
+ decryptedSerializedContent = propertiesEncryptor.serializePropertiesAndPreserveFormatIfPossible(properties, config.inputFilePath)
+ break
+
+ case FileType.xml:
+ XmlEncryptor xmlEncryptor = new XmlEncryptor(null, config.decryptionProvider) {
+ @Override
+ List<String> serializeXmlContentAndPreserveFormat(String updatedXmlContent, File originalInputXmlFile) {
+ // For decrypting unknown, generic XML, this tool will not support preserving the format
+ return updatedXmlContent.split("\n")
+ }
+ }
+
+ String xmlContent = xmlEncryptor.loadXmlFile(config.inputFilePath)
+ xmlContent = xmlEncryptor.decrypt(xmlContent)
+ decryptedSerializedContent = xmlEncryptor.serializeXmlContentAndPreserveFormatIfPossible(xmlContent, config.inputFilePath)
+ break
+
+ default:
+ throw new RuntimeException("Unsupported file type '${config.fileType}'")
+ }
+
+ if (!decryptedSerializedContent) {
+ throw new RuntimeException("Failed to load and decrypt input file.")
+ }
+
+ if (config.outputToFile) {
+ try {
+ File outputFile = new File(config.outputFilePath)
+ if (ToolUtilities.isSafeToWrite(outputFile)) {
+ outputFile.text = decryptedSerializedContent
+ logger.info("Wrote decrypted file contents to '${config.outputFilePath}'")
+ }
+ } catch (IOException e) {
+ throw new RuntimeException("Encountered an exception writing the decrypted content to '${config.outputFilePath}': ${e.getMessage()}", e)
+ }
+ } else {
+ System.out.println(decryptedSerializedContent)
+ }
+
+ }
+
+ private CliBuilder cliBuilder() {
+
+ String usage = "${EncryptConfigMain.class.getCanonicalName()} decrypt [options] file"
+
+ int formatWidth = EncryptConfigMain.HELP_FORMAT_WIDTH
+ HelpFormatter formatter = new HelpFormatter()
+ formatter.setWidth(formatWidth)
+ formatter.setOptionComparator(null) // preserve order of options below in help text
+
+ CliBuilder cli = new CliBuilder(
+ usage: usage,
+ width: formatWidth,
+ formatter: formatter,
+ stopAtNonOption: false)
+
+ cli.h(longOpt: 'help', 'Show usage information (this message)')
+ cli.v(longOpt: 'verbose', 'Enables verbose mode (off by default)')
+
+ // Options for the password or key or bootstrap.conf
+ cli.p(longOpt: 'password',
+ args: 1,
+ argName: 'password',
+ optionalArg: true,
+ 'Use a password to derive the key to decrypt the input file. If an argument is not provided to this flag, interactive mode will be triggered to prompt the user to enter the password.')
+ cli.k(longOpt: 'key',
+ args: 1,
+ argName: 'keyhex',
+ optionalArg: true,
+ 'Use a raw hexadecimal key to decrypt the input file. If an argument is not provided to this flag, interactive mode will be triggered to prompt the user to enter the key.')
+ cli.b(longOpt: 'bootstrapConf',
+ args: 1,
+ argName: 'file',
+ 'Use a bootstrap.conf file containing the master key to decrypt the input file (as an alternative to -p or -k)')
+
+ cli.o(longOpt: 'output',
+ args: 1,
+ argName: 'file',
+ 'Specify an output file. If omitted, Standard Out is used. Output file can be set to the input file to decrypt the file in-place.')
+
+ return cli
+
+ }
+
+ static class DecryptConfiguration {
+
+ OptionAccessor rawOptions
+
+ Configuration.KeySource keySource
+ String key
+ SensitivePropertyProvider decryptionProvider
+ String inputBootstrapPath
+
+ FileType fileType
+
+ String inputFilePath
+
+ boolean outputToFile = false
+ String outputFilePath
+
+ DecryptConfiguration() {
+ }
+
+ DecryptConfiguration(OptionAccessor options) {
+ this.rawOptions = options
+
+ validateOptions()
+ determineInputFileFromRemainingArgs()
+
+ determineKey()
+ if (!key) {
+ throw new RuntimeException("Failed to configure tool, could not determine key.")
+ }
+ decryptionProvider = new AESSensitivePropertyProvider(key)
+
+ if (rawOptions.t) {
+ fileType = FileType.valueOf(rawOptions.t)
+ }
+
+ if (rawOptions.o) {
--- End diff --
Yep, as explained above, will remove the -R option from `NiFiRegistryDecryptMode`
---
[GitHub] nifi pull request #2376: NIFI-4708 Add Registry support to encrypt-config
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2376#discussion_r159981266
--- Diff: nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/toolkit/encryptconfig/NiFiRegistryDecryptMode.groovy ---
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.toolkit.encryptconfig
+
+import org.apache.nifi.properties.AESSensitivePropertyProvider
+import org.apache.nifi.toolkit.encryptconfig.util.BootstrapUtil
+import org.apache.nifi.toolkit.encryptconfig.util.ToolUtilities
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+/**
+ * A special DecryptMode that can run using NiFiRegistry CLI Options
+ */
+class NiFiRegistryDecryptMode extends DecryptMode {
+
+ private static final Logger logger = LoggerFactory.getLogger(NiFiRegistryDecryptMode.class)
+
+ CliBuilder cli
+
+ NiFiRegistryDecryptMode() {
+ cli = NiFiRegistryMode.cliBuilder()
+ }
+
+ @Override
+ void run(String[] args) {
+ logger.warn("The decryption capability of this tool is still considered experimental. The results should be manually verified.")
+ try {
+
+ def options = cli.parse(args)
+
+ if (!options || options.h) {
+ EncryptConfigMain.printUsageAndExit("", EncryptConfigMain.EXIT_STATUS_OTHER)
+ }
+
+ EncryptConfigLogger.configureLogger(options.v)
+
+ DecryptConfiguration config = new DecryptConfiguration()
+
+ /* Invalid fields when used with --decrypt: */
+ def invalidDecryptOptions = ["i", "a"]
+ def presentInvalidOptions = Arrays.stream(options.getInner().getOptions()).findAll {
+ invalidDecryptOptions.contains(it.getOpt())
+ }
+ if (presentInvalidOptions.size() > 0) {
+ throw new RuntimeException("Invalid options: ${EncryptConfigMain.DECRYPT_OPT} cannot be used with [${presentInvalidOptions.join(", ")}]. It should only be used with [-r].")
+ }
+
+ /* Required fields when using --decrypt */
+ // registryPropertiesFile (-r)
+ if (!options.r) {
+ throw new RuntimeException("Invalid options: Input nifiRegistryProperties (-r) is required when using --decrypt")
+ }
+ config.inputFilePath = options.r
+ config.fileType = FileType.properties // disables auto-detection, which is still experimental
+
+ // one of [--oldPassword, --oldKey] or [-p, -k, -b <file]
+ String keyHex = null
+ String password = null
+ config.keySource = null
+ if (options.oldPassword) {
+ if (config.keySource != null) {
+ throw new RuntimeException("Invalid options: Only one of [--oldPassword, --oldKey, -b, -p, -k] is allowed for specifying the decryption password/key.")
+ }
+ config.keySource = Configuration.KeySource.PASSWORD
+ keyHex = options.getInner().getOptionValue("oldPassword")
+ }
+ if (options.oldKey) {
+ if (config.keySource != null) {
+ throw new RuntimeException("Invalid options: Only one of [--oldPassword, --oldKey, -b, -p, -k] is allowed for specifying the decryption password/key.")
+ }
+ config.keySource = Configuration.KeySource.KEY_HEX
+ keyHex = options.getInner().getOptionValue("oldKey")
+ }
+ if (options.p) {
+ if (config.keySource != null) {
+ throw new RuntimeException("Invalid options: Only one of [--oldPassword, --oldKey, -b, -p, -k] is allowed for specifying the decryption password/key.")
+ }
+ config.keySource = Configuration.KeySource.PASSWORD
+ password = options.getInner().getOptionValue("p")
+ }
+ if (options.k) {
+ if (config.keySource != null) {
+ throw new RuntimeException("Invalid options: Only one of [--oldPassword, --oldKey, -b, -p, -k] is allowed for specifying the decryption password/key.")
+ }
+ config.keySource = Configuration.KeySource.KEY_HEX
+ keyHex = options.getInner().getOptionValue("k")
+ }
+
+ if (config.keySource) {
+ config.key = ToolUtilities.determineKey(keyHex, password, Configuration.KeySource.PASSWORD.equals(config.keySource))
+ }
+
+ if (options.b) {
+ if (config.keySource != null) {
+ throw new RuntimeException("Invalid options: Only one of [--oldPassword, --oldKey, -b, -p, -k] is allowed for specifying the decryption password/key.")
+ }
+ config.keySource = Configuration.KeySource.BOOTSTRAP_FILE
+ config.inputBootstrapPath = options.b
+
+ logger.debug("Checking expected NiFi Registry bootstrap.conf format")
+ config.key = BootstrapUtil.extractKeyFromBootstrapFile(config.inputBootstrapPath, BootstrapUtil.REGISTRY_BOOTSTRAP_KEY_PROPERTY)
+
+ // check we have found the key
+ if (config.key) {
+ logger.debug("Master key found in ${config.inputBootstrapPath}. This key will be used for decryption operations.")
+ } else {
+ logger.warn("Bootstrap Conf flag present, but master key could not be found in ${config.inputBootstrapPath}.")
+ }
+ }
+
+ config.decryptionProvider = new AESSensitivePropertyProvider(config.key)
+
+ /* Optional fields when using --decrypt */
+ // -R outputRegistryPropertiesFile (-R)
+ if (options.R) {
+ config.outputToFile = true
+ config.outputFilePath = options.R
+ }
+
+ run(config)
+
+ } catch (Exception e) {
+ logger.error("Encountered an error: ${e.getMessage()}")
+ logger.debug("", e) // only print stack trace when verbose is enabled
--- End diff --
- [ ] `logger.error()`
---
[GitHub] nifi pull request #2376: NIFI-4708 Add Registry support to encrypt-config
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2376#discussion_r159926265
--- Diff: nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/toolkit/encryptconfig/DecryptMode.groovy ---
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.toolkit.encryptconfig
+
+import org.apache.commons.cli.HelpFormatter
+import org.apache.nifi.properties.AESSensitivePropertyProvider
+import org.apache.nifi.properties.SensitivePropertyProvider
+import org.apache.nifi.toolkit.encryptconfig.util.BootstrapUtil
+import org.apache.nifi.toolkit.encryptconfig.util.PropertiesEncryptor
+import org.apache.nifi.toolkit.encryptconfig.util.ToolUtilities
+import org.apache.nifi.toolkit.encryptconfig.util.XmlEncryptor
+import org.apache.nifi.util.console.TextDevices
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+class DecryptMode implements ToolMode {
+
+ private static final Logger logger = LoggerFactory.getLogger(DecryptMode.class)
+
+ enum FileType {
+ properties,
+ xml
+ }
+
+ CliBuilder cli
+
+ DecryptMode() {
+ cli = cliBuilder()
+ }
+
+ void printUsage(String message = "") {
+ if (message) {
+ System.out.println(message)
+ System.out.println()
+ }
+ cli.usage()
+ }
+
+ void printUsageAndExit(String message = "", int exitStatusCode) {
+ printUsage(message)
+ System.exit(exitStatusCode)
+ }
+
+ @Override
+ void run(String[] args) {
+ logger.warn("The decryption capability of this tool is still considered experimental. The results should be manually verified.")
+ try {
+
+ def options = cli.parse(args)
+
+ if (!options || options.h) {
+ printUsageAndExit("", EncryptConfigMain.EXIT_STATUS_OTHER)
+ }
+
+ EncryptConfigLogger.configureLogger(options.v)
+
+ DecryptConfiguration config = new DecryptConfiguration(options)
+
+ run(config)
+
+ } catch (Exception e) {
+ logger.error("Encountered an error: ${e.getMessage()}")
+ logger.debug("", e) // stack trace only when verbose enabled
+ printUsageAndExit(e.getMessage(), EncryptConfigMain.EXIT_STATUS_FAILURE)
+ }
+ }
+
+ void run(DecryptConfiguration config) throws Exception {
+
+ if (!config.fileType) {
+
+ // Try to load the input file to auto-detect the file type
+ boolean isPropertiesFile = PropertiesEncryptor.supportsFile(config.inputFilePath)
+
+ boolean isXmlFile = XmlEncryptor.supportsFile(config.inputFilePath)
+
+ if (ToolUtilities.isExactlyOneTrue(isPropertiesFile, isXmlFile)) {
+ if (isPropertiesFile) {
+ config.fileType = FileType.properties
+ logger.debug("Auto-detection of input file type determined the type to be: ${FileType.properties}")
+ }
+ if (isXmlFile) {
+ config.fileType = FileType.xml
+ logger.debug("Auto-detection of input file type determined the type to be: ${FileType.xml}")
+ }
+ }
+
+ // Could we successfully auto-detect?
+ if (!config.fileType) {
+ throw new RuntimeException("Auto-detection of input file type failed. Please re-run the tool specifying the file type with the -t/--fileType flag.")
+ }
+ }
+
+ String decryptedSerializedContent = null
+ switch (config.fileType) {
+
+ case FileType.properties:
+ PropertiesEncryptor propertiesEncryptor = new PropertiesEncryptor(null, config.decryptionProvider)
+ Properties properties = propertiesEncryptor.loadFile(config.inputFilePath)
+ properties = propertiesEncryptor.decrypt(properties)
+ decryptedSerializedContent = propertiesEncryptor.serializePropertiesAndPreserveFormatIfPossible(properties, config.inputFilePath)
+ break
+
+ case FileType.xml:
+ XmlEncryptor xmlEncryptor = new XmlEncryptor(null, config.decryptionProvider) {
+ @Override
+ List<String> serializeXmlContentAndPreserveFormat(String updatedXmlContent, File originalInputXmlFile) {
+ // For decrypting unknown, generic XML, this tool will not support preserving the format
+ return updatedXmlContent.split("\n")
+ }
+ }
+
+ String xmlContent = xmlEncryptor.loadXmlFile(config.inputFilePath)
+ xmlContent = xmlEncryptor.decrypt(xmlContent)
+ decryptedSerializedContent = xmlEncryptor.serializeXmlContentAndPreserveFormatIfPossible(xmlContent, config.inputFilePath)
+ break
+
+ default:
+ throw new RuntimeException("Unsupported file type '${config.fileType}'")
+ }
+
+ if (!decryptedSerializedContent) {
+ throw new RuntimeException("Failed to load and decrypt input file.")
+ }
+
+ if (config.outputToFile) {
+ try {
+ File outputFile = new File(config.outputFilePath)
+ if (ToolUtilities.isSafeToWrite(outputFile)) {
+ outputFile.text = decryptedSerializedContent
+ logger.info("Wrote decrypted file contents to '${config.outputFilePath}'")
+ }
+ } catch (IOException e) {
+ throw new RuntimeException("Encountered an exception writing the decrypted content to '${config.outputFilePath}': ${e.getMessage()}", e)
+ }
+ } else {
+ System.out.println(decryptedSerializedContent)
+ }
+
+ }
+
+ private CliBuilder cliBuilder() {
+
+ String usage = "${EncryptConfigMain.class.getCanonicalName()} decrypt [options] file"
+
+ int formatWidth = EncryptConfigMain.HELP_FORMAT_WIDTH
+ HelpFormatter formatter = new HelpFormatter()
+ formatter.setWidth(formatWidth)
+ formatter.setOptionComparator(null) // preserve order of options below in help text
+
+ CliBuilder cli = new CliBuilder(
+ usage: usage,
+ width: formatWidth,
+ formatter: formatter,
+ stopAtNonOption: false)
+
+ cli.h(longOpt: 'help', 'Show usage information (this message)')
+ cli.v(longOpt: 'verbose', 'Enables verbose mode (off by default)')
+
+ // Options for the password or key or bootstrap.conf
+ cli.p(longOpt: 'password',
+ args: 1,
+ argName: 'password',
+ optionalArg: true,
+ 'Use a password to derive the key to decrypt the input file. If an argument is not provided to this flag, interactive mode will be triggered to prompt the user to enter the password.')
+ cli.k(longOpt: 'key',
+ args: 1,
+ argName: 'keyhex',
+ optionalArg: true,
+ 'Use a raw hexadecimal key to decrypt the input file. If an argument is not provided to this flag, interactive mode will be triggered to prompt the user to enter the key.')
+ cli.b(longOpt: 'bootstrapConf',
+ args: 1,
+ argName: 'file',
+ 'Use a bootstrap.conf file containing the master key to decrypt the input file (as an alternative to -p or -k)')
+
+ cli.o(longOpt: 'output',
+ args: 1,
+ argName: 'file',
+ 'Specify an output file. If omitted, Standard Out is used. Output file can be set to the input file to decrypt the file in-place.')
+
+ return cli
+
+ }
+
+ static class DecryptConfiguration {
--- End diff --
Should this implement the `Configuration` interface?
---
[GitHub] nifi pull request #2376: NIFI-4708 Add Registry support to encrypt-config
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2376#discussion_r159977391
--- Diff: nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/toolkit/encryptconfig/EncryptConfigMain.groovy ---
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.toolkit.encryptconfig
+
+import org.apache.commons.cli.HelpFormatter
+import org.apache.commons.cli.Options
+import org.apache.nifi.properties.ConfigEncryptionTool
+import org.bouncycastle.jce.provider.BouncyCastleProvider
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+import java.security.Security
+
+class EncryptConfigMain {
+
+ private static final Logger logger = LoggerFactory.getLogger(EncryptConfigMain.class)
+
+ static final int EXIT_STATUS_SUCCESS = 0
+ static final int EXIT_STATUS_FAILURE = -1
+ static final int EXIT_STATUS_OTHER = 1
+
+ static final String NIFI_REGISTRY_OPT = "nifiRegistry"
+ static final String NIFI_REGISTRY_FLAG = "--${NIFI_REGISTRY_OPT}".toString()
+ static final String DECRYPT_OPT = "decrypt"
+ static final String DECRYPT_FLAG = "--${DECRYPT_OPT}".toString()
+
+ static final int HELP_FORMAT_WIDTH = 160
+
+ // Access should only be through static methods
+ private EncryptConfigMain() {
+ }
+
+ static printUsage(String message = "") {
+
+ if (message) {
+ System.out.println(message)
+ System.out.println()
+ }
+
+ String header = "\nThis tool enables easy encryption and decryption of configuration files for NiFi and its sub-projects. " +
+ "Unprotected files can be input to this tool to be protected by a key in a manner that is understood by NiFi. " +
+ "Protected files, along with a key, can be input to this tool to be unprotected, for troubleshooting or automation purposes.\n\n"
+
+ def options = new Options()
+ options.addOption("h", "help", false, "Show usage information (this message)")
+ options.addOption(null, NIFI_REGISTRY_OPT, false, "Specifies to target NiFi Registry. When this flag is not included, NiFi is the target.")
+
+ HelpFormatter helpFormatter = new HelpFormatter()
+ helpFormatter.setWidth(160)
+ helpFormatter.setOptionComparator(null)
+ helpFormatter.printHelp("${EncryptConfigMain.class.getCanonicalName()} [-h] [options]", header, options, "\n")
+ System.out.println()
+
+ helpFormatter.setSyntaxPrefix("") // disable "usage: " prefix for the following outputs
+
+ Options nifiModeOptions = ConfigEncryptionTool.getCliOptions()
+ helpFormatter.printHelp(
+ "When targeting NiFi:",
+ nifiModeOptions,
+ false)
+ System.out.println()
+
+ Options nifiRegistryModeOptions = NiFiRegistryMode.getCliOptions()
+ nifiRegistryModeOptions.addOption(null, DECRYPT_OPT, false, "Can be used with -r to decrypt a previously encrypted NiFi Registry Properties file. Decrypted content is printed to STDOUT.")
+ helpFormatter.printHelp(
+ "When targeting NiFi Registry using the ${NIFI_REGISTRY_FLAG} flag:",
+ nifiRegistryModeOptions,
+ false)
+ System.out.println()
+
+// String footer = """
--- End diff --
- [ ] remove dead code in EMC
---
[GitHub] nifi pull request #2376: NIFI-4708 Add Registry support to encrypt-config
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2376#discussion_r159978565
--- Diff: nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/toolkit/encryptconfig/EncryptConfigMain.groovy ---
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.toolkit.encryptconfig
+
+import org.apache.commons.cli.HelpFormatter
+import org.apache.commons.cli.Options
+import org.apache.nifi.properties.ConfigEncryptionTool
+import org.bouncycastle.jce.provider.BouncyCastleProvider
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+import java.security.Security
+
+class EncryptConfigMain {
+
+ private static final Logger logger = LoggerFactory.getLogger(EncryptConfigMain.class)
+
+ static final int EXIT_STATUS_SUCCESS = 0
+ static final int EXIT_STATUS_FAILURE = -1
+ static final int EXIT_STATUS_OTHER = 1
+
+ static final String NIFI_REGISTRY_OPT = "nifiRegistry"
+ static final String NIFI_REGISTRY_FLAG = "--${NIFI_REGISTRY_OPT}".toString()
+ static final String DECRYPT_OPT = "decrypt"
+ static final String DECRYPT_FLAG = "--${DECRYPT_OPT}".toString()
+
+ static final int HELP_FORMAT_WIDTH = 160
+
+ // Access should only be through static methods
+ private EncryptConfigMain() {
+ }
+
+ static printUsage(String message = "") {
+
+ if (message) {
+ System.out.println(message)
+ System.out.println()
+ }
+
+ String header = "\nThis tool enables easy encryption and decryption of configuration files for NiFi and its sub-projects. " +
+ "Unprotected files can be input to this tool to be protected by a key in a manner that is understood by NiFi. " +
+ "Protected files, along with a key, can be input to this tool to be unprotected, for troubleshooting or automation purposes.\n\n"
+
+ def options = new Options()
+ options.addOption("h", "help", false, "Show usage information (this message)")
+ options.addOption(null, NIFI_REGISTRY_OPT, false, "Specifies to target NiFi Registry. When this flag is not included, NiFi is the target.")
+
+ HelpFormatter helpFormatter = new HelpFormatter()
+ helpFormatter.setWidth(160)
+ helpFormatter.setOptionComparator(null)
+ helpFormatter.printHelp("${EncryptConfigMain.class.getCanonicalName()} [-h] [options]", header, options, "\n")
+ System.out.println()
+
+ helpFormatter.setSyntaxPrefix("") // disable "usage: " prefix for the following outputs
+
+ Options nifiModeOptions = ConfigEncryptionTool.getCliOptions()
+ helpFormatter.printHelp(
+ "When targeting NiFi:",
+ nifiModeOptions,
+ false)
+ System.out.println()
+
+ Options nifiRegistryModeOptions = NiFiRegistryMode.getCliOptions()
+ nifiRegistryModeOptions.addOption(null, DECRYPT_OPT, false, "Can be used with -r to decrypt a previously encrypted NiFi Registry Properties file. Decrypted content is printed to STDOUT.")
+ helpFormatter.printHelp(
+ "When targeting NiFi Registry using the ${NIFI_REGISTRY_FLAG} flag:",
+ nifiRegistryModeOptions,
+ false)
+ System.out.println()
+
+// String footer = """
+// |
+// |Encrypt a NiFi Registry properties using a password:
+// | encrypt-config -p <password> -b /path/to/nifi/conf/bootstrap.conf -r /path/to/nifi/conf/nifi.properties
+// |
+// |""".stripMargin()
+ //helpFormatter.printHelp("Examples:", "", new Options(), footer)
+ }
+
+ static void printUsageAndExit(String message = "", int exitStatusCode) {
+ printUsage(message)
+ System.exit(exitStatusCode)
+ }
+
+ static void main(String[] args) {
+ Security.addProvider(new BouncyCastleProvider())
+
+ if (args.length < 1) {
+ printUsageAndExit(EXIT_STATUS_FAILURE)
+ }
+
+ String firstArg = args[0]
+
+ if (["-h", "--help"].contains(firstArg)) {
+ printUsageAndExit(EXIT_STATUS_OTHER)
+ }
+
+ try {
+ List<String> argsList = args
+ ToolMode toolMode = determineModeFromArgs(argsList)
+ if (toolMode) {
+ toolMode.run((String[])argsList.toArray())
+ System.exit(EXIT_STATUS_SUCCESS)
+ } else {
+ printUsageAndExit(EXIT_STATUS_FAILURE)
+ }
+ } catch (Throwable t) {
+ logger.debug("", t)
--- End diff --
- [ ] use `logger.error()` in ECM
---
[GitHub] nifi pull request #2376: NIFI-4708 Add Registry support to encrypt-config
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2376#discussion_r159982577
--- Diff: nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/toolkit/encryptconfig/NiFiRegistryMode.groovy ---
@@ -0,0 +1,383 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.toolkit.encryptconfig
+
+import org.apache.commons.cli.HelpFormatter
+import org.apache.commons.cli.Options
+import org.apache.http.annotation.Experimental
+import org.apache.nifi.properties.AESSensitivePropertyProvider
+import org.apache.nifi.properties.SensitivePropertyProvider
+import org.apache.nifi.toolkit.encryptconfig.util.BootstrapUtil
+import org.apache.nifi.toolkit.encryptconfig.util.NiFiRegistryAuthorizersXmlEncryptor
+import org.apache.nifi.toolkit.encryptconfig.util.NiFiRegistryIdentityProvidersXmlEncryptor
+import org.apache.nifi.toolkit.encryptconfig.util.NiFiRegistryPropertiesEncryptor
+import org.apache.nifi.toolkit.encryptconfig.util.ToolUtilities
+import org.apache.nifi.util.console.TextDevices
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+@Experimental
+class NiFiRegistryMode implements ToolMode {
+
+ private static final Logger logger = LoggerFactory.getLogger(NiFiRegistryMode.class)
+
+ CliBuilder cli
+
+ NiFiRegistryMode() {
+ cli = cliBuilder()
+ }
+
+// private void printUsage(String message = "") {
+// if (message) {
+// System.out.println(message)
+// System.out.println()
+// }
+// cli.usage()
+// }
+
+ @Override
+ void run(String[] args) {
+ logger.warn("The NiFi Registry capabilities of this tool is still considered experimental. The results should be manually verified.")
--- End diff --
- [ ] fix typo
---
[GitHub] nifi pull request #2376: NIFI-4708 Add Registry support to encrypt-config
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2376#discussion_r159925148
--- Diff: nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/toolkit/encryptconfig/DecryptMode.groovy ---
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.toolkit.encryptconfig
+
+import org.apache.commons.cli.HelpFormatter
+import org.apache.nifi.properties.AESSensitivePropertyProvider
+import org.apache.nifi.properties.SensitivePropertyProvider
+import org.apache.nifi.toolkit.encryptconfig.util.BootstrapUtil
+import org.apache.nifi.toolkit.encryptconfig.util.PropertiesEncryptor
+import org.apache.nifi.toolkit.encryptconfig.util.ToolUtilities
+import org.apache.nifi.toolkit.encryptconfig.util.XmlEncryptor
+import org.apache.nifi.util.console.TextDevices
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+class DecryptMode implements ToolMode {
+
+ private static final Logger logger = LoggerFactory.getLogger(DecryptMode.class)
+
+ enum FileType {
+ properties,
+ xml
+ }
+
+ CliBuilder cli
+
+ DecryptMode() {
+ cli = cliBuilder()
+ }
+
+ void printUsage(String message = "") {
+ if (message) {
+ System.out.println(message)
+ System.out.println()
+ }
+ cli.usage()
+ }
+
+ void printUsageAndExit(String message = "", int exitStatusCode) {
+ printUsage(message)
+ System.exit(exitStatusCode)
+ }
+
+ @Override
+ void run(String[] args) {
+ logger.warn("The decryption capability of this tool is still considered experimental. The results should be manually verified.")
+ try {
+
+ def options = cli.parse(args)
+
+ if (!options || options.h) {
+ printUsageAndExit("", EncryptConfigMain.EXIT_STATUS_OTHER)
+ }
+
+ EncryptConfigLogger.configureLogger(options.v)
+
+ DecryptConfiguration config = new DecryptConfiguration(options)
+
+ run(config)
+
+ } catch (Exception e) {
+ logger.error("Encountered an error: ${e.getMessage()}")
+ logger.debug("", e) // stack trace only when verbose enabled
+ printUsageAndExit(e.getMessage(), EncryptConfigMain.EXIT_STATUS_FAILURE)
+ }
+ }
+
+ void run(DecryptConfiguration config) throws Exception {
+
+ if (!config.fileType) {
+
+ // Try to load the input file to auto-detect the file type
+ boolean isPropertiesFile = PropertiesEncryptor.supportsFile(config.inputFilePath)
+
+ boolean isXmlFile = XmlEncryptor.supportsFile(config.inputFilePath)
+
+ if (ToolUtilities.isExactlyOneTrue(isPropertiesFile, isXmlFile)) {
+ if (isPropertiesFile) {
+ config.fileType = FileType.properties
+ logger.debug("Auto-detection of input file type determined the type to be: ${FileType.properties}")
+ }
+ if (isXmlFile) {
+ config.fileType = FileType.xml
+ logger.debug("Auto-detection of input file type determined the type to be: ${FileType.xml}")
+ }
+ }
+
+ // Could we successfully auto-detect?
+ if (!config.fileType) {
+ throw new RuntimeException("Auto-detection of input file type failed. Please re-run the tool specifying the file type with the -t/--fileType flag.")
+ }
+ }
+
+ String decryptedSerializedContent = null
+ switch (config.fileType) {
+
+ case FileType.properties:
+ PropertiesEncryptor propertiesEncryptor = new PropertiesEncryptor(null, config.decryptionProvider)
+ Properties properties = propertiesEncryptor.loadFile(config.inputFilePath)
+ properties = propertiesEncryptor.decrypt(properties)
+ decryptedSerializedContent = propertiesEncryptor.serializePropertiesAndPreserveFormatIfPossible(properties, config.inputFilePath)
+ break
+
+ case FileType.xml:
+ XmlEncryptor xmlEncryptor = new XmlEncryptor(null, config.decryptionProvider) {
+ @Override
+ List<String> serializeXmlContentAndPreserveFormat(String updatedXmlContent, File originalInputXmlFile) {
+ // For decrypting unknown, generic XML, this tool will not support preserving the format
+ return updatedXmlContent.split("\n")
+ }
+ }
+
+ String xmlContent = xmlEncryptor.loadXmlFile(config.inputFilePath)
+ xmlContent = xmlEncryptor.decrypt(xmlContent)
+ decryptedSerializedContent = xmlEncryptor.serializeXmlContentAndPreserveFormatIfPossible(xmlContent, config.inputFilePath)
+ break
+
+ default:
+ throw new RuntimeException("Unsupported file type '${config.fileType}'")
+ }
+
+ if (!decryptedSerializedContent) {
+ throw new RuntimeException("Failed to load and decrypt input file.")
+ }
+
+ if (config.outputToFile) {
--- End diff --
Now that I think about it, I agree with you. That would be the better/safer approach for this initial offering of the decrypt capability. Will update
---
[GitHub] nifi pull request #2376: NIFI-4708 Add Registry support to encrypt-config
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2376#discussion_r159982520
--- Diff: nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/toolkit/encryptconfig/NiFiRegistryDecryptMode.groovy ---
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.toolkit.encryptconfig
+
+import org.apache.nifi.properties.AESSensitivePropertyProvider
+import org.apache.nifi.toolkit.encryptconfig.util.BootstrapUtil
+import org.apache.nifi.toolkit.encryptconfig.util.ToolUtilities
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+/**
+ * A special DecryptMode that can run using NiFiRegistry CLI Options
+ */
+class NiFiRegistryDecryptMode extends DecryptMode {
+
+ private static final Logger logger = LoggerFactory.getLogger(NiFiRegistryDecryptMode.class)
+
+ CliBuilder cli
+
+ NiFiRegistryDecryptMode() {
+ cli = NiFiRegistryMode.cliBuilder()
+ }
+
+ @Override
+ void run(String[] args) {
+ logger.warn("The decryption capability of this tool is still considered experimental. The results should be manually verified.")
+ try {
+
+ def options = cli.parse(args)
+
+ if (!options || options.h) {
+ EncryptConfigMain.printUsageAndExit("", EncryptConfigMain.EXIT_STATUS_OTHER)
+ }
+
+ EncryptConfigLogger.configureLogger(options.v)
+
+ DecryptConfiguration config = new DecryptConfiguration()
+
+ /* Invalid fields when used with --decrypt: */
+ def invalidDecryptOptions = ["i", "a"]
+ def presentInvalidOptions = Arrays.stream(options.getInner().getOptions()).findAll {
+ invalidDecryptOptions.contains(it.getOpt())
+ }
+ if (presentInvalidOptions.size() > 0) {
+ throw new RuntimeException("Invalid options: ${EncryptConfigMain.DECRYPT_OPT} cannot be used with [${presentInvalidOptions.join(", ")}]. It should only be used with [-r].")
+ }
+
+ /* Required fields when using --decrypt */
+ // registryPropertiesFile (-r)
+ if (!options.r) {
+ throw new RuntimeException("Invalid options: Input nifiRegistryProperties (-r) is required when using --decrypt")
+ }
+ config.inputFilePath = options.r
+ config.fileType = FileType.properties // disables auto-detection, which is still experimental
+
+ // one of [--oldPassword, --oldKey] or [-p, -k, -b <file]
+ String keyHex = null
--- End diff --
- [ ] apply patch and remove options for --oldPassword and --oldKey as discussed
---
[GitHub] nifi pull request #2376: NIFI-4708 Add Registry support to encrypt-config
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2376#discussion_r159953548
--- Diff: nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/toolkit/encryptconfig/NiFiRegistryDecryptMode.groovy ---
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.toolkit.encryptconfig
+
+import org.apache.nifi.properties.AESSensitivePropertyProvider
+import org.apache.nifi.toolkit.encryptconfig.util.BootstrapUtil
+import org.apache.nifi.toolkit.encryptconfig.util.ToolUtilities
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+/**
+ * A special DecryptMode that can run using NiFiRegistry CLI Options
+ */
+class NiFiRegistryDecryptMode extends DecryptMode {
+
+ private static final Logger logger = LoggerFactory.getLogger(NiFiRegistryDecryptMode.class)
+
+ CliBuilder cli
+
+ NiFiRegistryDecryptMode() {
+ cli = NiFiRegistryMode.cliBuilder()
+ }
+
+ @Override
+ void run(String[] args) {
+ logger.warn("The decryption capability of this tool is still considered experimental. The results should be manually verified.")
+ try {
+
+ def options = cli.parse(args)
+
+ if (!options || options.h) {
+ EncryptConfigMain.printUsageAndExit("", EncryptConfigMain.EXIT_STATUS_OTHER)
+ }
+
+ EncryptConfigLogger.configureLogger(options.v)
+
+ DecryptConfiguration config = new DecryptConfiguration()
+
+ /* Invalid fields when used with --decrypt: */
+ def invalidDecryptOptions = ["i", "a"]
+ def presentInvalidOptions = Arrays.stream(options.getInner().getOptions()).findAll {
+ invalidDecryptOptions.contains(it.getOpt())
+ }
+ if (presentInvalidOptions.size() > 0) {
+ throw new RuntimeException("Invalid options: ${EncryptConfigMain.DECRYPT_OPT} cannot be used with [${presentInvalidOptions.join(", ")}]. It should only be used with [-r].")
+ }
+
+ /* Required fields when using --decrypt */
+ // registryPropertiesFile (-r)
+ if (!options.r) {
+ throw new RuntimeException("Invalid options: Input nifiRegistryProperties (-r) is required when using --decrypt")
+ }
+ config.inputFilePath = options.r
+ config.fileType = FileType.properties // disables auto-detection, which is still experimental
+
+ // one of [--oldPassword, --oldKey] or [-p, -k, -b <file]
--- End diff --
Typo in comment -- `[-p, -k, -b <file>]`.
---
[GitHub] nifi issue #2376: NIFI-4708 Add Registry support to encrypt-config
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on the issue:
https://github.com/apache/nifi/pull/2376
@alopresto thanks for the thorough review! I've pushed an update that addresses your comments. It also adds a lot of test cases (see NiFiRegistryModeSpec and NiFiRegistryDecryptModeSpec), which cover all the functionality that I intend to expose in this version (ie, [--nifiRegistry [--decrypt] [options]]. Given that I had time to do more testing, I also removed the "experimental" warning output when the new modes are run. Let me know what you think and if you see anything else to improve in this PR.
---
[GitHub] nifi pull request #2376: NIFI-4708 Add Registry support to encrypt-config
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2376#discussion_r159975116
--- Diff: nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/toolkit/encryptconfig/DecryptMode.groovy ---
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.toolkit.encryptconfig
+
+import org.apache.commons.cli.HelpFormatter
+import org.apache.nifi.properties.AESSensitivePropertyProvider
+import org.apache.nifi.properties.SensitivePropertyProvider
+import org.apache.nifi.toolkit.encryptconfig.util.BootstrapUtil
+import org.apache.nifi.toolkit.encryptconfig.util.PropertiesEncryptor
+import org.apache.nifi.toolkit.encryptconfig.util.ToolUtilities
+import org.apache.nifi.toolkit.encryptconfig.util.XmlEncryptor
+import org.apache.nifi.util.console.TextDevices
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+class DecryptMode implements ToolMode {
+
+ private static final Logger logger = LoggerFactory.getLogger(DecryptMode.class)
+
+ enum FileType {
+ properties,
+ xml
+ }
+
+ CliBuilder cli
+
+ DecryptMode() {
+ cli = cliBuilder()
+ }
+
+ void printUsage(String message = "") {
+ if (message) {
+ System.out.println(message)
+ System.out.println()
+ }
+ cli.usage()
+ }
+
+ void printUsageAndExit(String message = "", int exitStatusCode) {
+ printUsage(message)
+ System.exit(exitStatusCode)
+ }
+
+ @Override
+ void run(String[] args) {
+ logger.warn("The decryption capability of this tool is still considered experimental. The results should be manually verified.")
+ try {
+
+ def options = cli.parse(args)
+
+ if (!options || options.h) {
+ printUsageAndExit("", EncryptConfigMain.EXIT_STATUS_OTHER)
+ }
+
+ EncryptConfigLogger.configureLogger(options.v)
+
+ DecryptConfiguration config = new DecryptConfiguration(options)
+
+ run(config)
+
+ } catch (Exception e) {
+ logger.error("Encountered an error: ${e.getMessage()}")
+ logger.debug("", e) // stack trace only when verbose enabled
--- End diff --
Good call.
- [ ] Change error logging to got to error, wrap in conditional if it depends on verbose
---
[GitHub] nifi pull request #2376: NIFI-4708 Add Registry support to encrypt-config
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2376#discussion_r159977131
--- Diff: nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/toolkit/encryptconfig/EncryptConfigLogger.groovy ---
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.toolkit.encryptconfig
+
+import org.apache.log4j.LogManager
+import org.apache.log4j.PropertyConfigurator
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+class EncryptConfigLogger {
+
+ private static final Logger logger = LoggerFactory.getLogger(EncryptConfigLogger.class)
+
+ /**
+ * Configures the logger.
+ *
+ * The nifi-toolkit module uses log4j, which will be configured to append all
+ * log output to the system STDERR. The log level can be specified using the verboseEnabled
+ * argument. A value of <code>true</code> will set the log level to DEBUG, a value of
+ * <code>false</code> will set the log level to INFO.
+ *
+ * @param verboseEnabled flag to indicate if verbose mode is enabled, which sets the log level to DEBUG
+ */
+ static configureLogger(boolean verboseEnabled) {
+
+ Properties log4jProps = null
+ URL log4jPropsPath = this.getClass().getResource("log4j.properties")
+ if (log4jPropsPath) {
+ try {
+ log4jPropsPath.withReader { reader ->
+ log4jProps = new Properties()
+ log4jProps.load(reader)
+ }
+ } catch (IOException e) {
+ // do nothing, we will fallback to hardcoded defaults below
+ }
+ }
+
+ if (!log4jProps) {
+ log4jProps = defaultProperties()
+ }
+
+ if (verboseEnabled) {
+ // Override the log level for this package. For this to work as intended, this class must belong
+ // to the same package (or a parent package) of all the encrypt-config classes
+ log4jProps.put("log4j.logger." + EncryptConfigLogger.class.package.name, "DEBUG")
+ }
+
+ LogManager.resetConfiguration()
+ PropertyConfigurator.configure(log4jProps)
+
+ if (verboseEnabled) {
+ logger.debug("Verbose mode is enabled (goes to stderr by default).")
+ }
+ }
+
+ /**
+ * A copy of the settings in /src/main/resources/log4j.properties, in case that is not on the classpath at runtime
+ * @return Properties containing the default properties for Log4j
+ */
+ static Properties defaultProperties() {
+ Properties defaultProperties = new Properties()
+
+ defaultProperties.setProperty("log4j.rootLogger", "INFO,console")
+
+ defaultProperties.setProperty("log4j.appender.console", "org.apache.log4j.ConsoleAppender")
+ defaultProperties.setProperty("log4j.appender.console.Target", "System.err")
+ defaultProperties.setProperty("log4j.appender.console.layout", "org.apache.log4j.PatternLayout")
+ defaultProperties.setProperty("log4j.appender.console.layout.ConversionPattern", "%d{yyyy-mm-dd HH:mm:ss} %p %c{1}: %m%n")
+
+ return defaultProperties
+ }
+
--- End diff --
- [ ] Format code
---
[GitHub] nifi pull request #2376: NIFI-4708 Add Registry support to encrypt-config
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2376#discussion_r159978704
--- Diff: nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/toolkit/encryptconfig/NiFiRegistryDecryptMode.groovy ---
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.toolkit.encryptconfig
+
+import org.apache.nifi.properties.AESSensitivePropertyProvider
+import org.apache.nifi.toolkit.encryptconfig.util.BootstrapUtil
+import org.apache.nifi.toolkit.encryptconfig.util.ToolUtilities
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+/**
+ * A special DecryptMode that can run using NiFiRegistry CLI Options
+ */
+class NiFiRegistryDecryptMode extends DecryptMode {
+
+ private static final Logger logger = LoggerFactory.getLogger(NiFiRegistryDecryptMode.class)
+
+ CliBuilder cli
+
+ NiFiRegistryDecryptMode() {
+ cli = NiFiRegistryMode.cliBuilder()
+ }
+
+ @Override
+ void run(String[] args) {
+ logger.warn("The decryption capability of this tool is still considered experimental. The results should be manually verified.")
+ try {
+
+ def options = cli.parse(args)
+
+ if (!options || options.h) {
+ EncryptConfigMain.printUsageAndExit("", EncryptConfigMain.EXIT_STATUS_OTHER)
+ }
+
+ EncryptConfigLogger.configureLogger(options.v)
+
+ DecryptConfiguration config = new DecryptConfiguration()
+
+ /* Invalid fields when used with --decrypt: */
+ def invalidDecryptOptions = ["i", "a"]
+ def presentInvalidOptions = Arrays.stream(options.getInner().getOptions()).findAll {
+ invalidDecryptOptions.contains(it.getOpt())
+ }
+ if (presentInvalidOptions.size() > 0) {
+ throw new RuntimeException("Invalid options: ${EncryptConfigMain.DECRYPT_OPT} cannot be used with [${presentInvalidOptions.join(", ")}]. It should only be used with [-r].")
+ }
+
+ /* Required fields when using --decrypt */
+ // registryPropertiesFile (-r)
+ if (!options.r) {
+ throw new RuntimeException("Invalid options: Input nifiRegistryProperties (-r) is required when using --decrypt")
+ }
+ config.inputFilePath = options.r
+ config.fileType = FileType.properties // disables auto-detection, which is still experimental
+
+ // one of [--oldPassword, --oldKey] or [-p, -k, -b <file]
--- End diff --
- [ ] fix typo
---
[GitHub] nifi pull request #2376: NIFI-4708 Add Registry support to encrypt-config
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2376#discussion_r159920353
--- Diff: nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/toolkit/encryptconfig/DecryptMode.groovy ---
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.toolkit.encryptconfig
+
+import org.apache.commons.cli.HelpFormatter
+import org.apache.nifi.properties.AESSensitivePropertyProvider
+import org.apache.nifi.properties.SensitivePropertyProvider
+import org.apache.nifi.toolkit.encryptconfig.util.BootstrapUtil
+import org.apache.nifi.toolkit.encryptconfig.util.PropertiesEncryptor
+import org.apache.nifi.toolkit.encryptconfig.util.ToolUtilities
+import org.apache.nifi.toolkit.encryptconfig.util.XmlEncryptor
+import org.apache.nifi.util.console.TextDevices
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+class DecryptMode implements ToolMode {
+
+ private static final Logger logger = LoggerFactory.getLogger(DecryptMode.class)
+
+ enum FileType {
+ properties,
+ xml
+ }
+
+ CliBuilder cli
+
+ DecryptMode() {
+ cli = cliBuilder()
+ }
+
+ void printUsage(String message = "") {
+ if (message) {
+ System.out.println(message)
+ System.out.println()
+ }
+ cli.usage()
+ }
+
+ void printUsageAndExit(String message = "", int exitStatusCode) {
+ printUsage(message)
+ System.exit(exitStatusCode)
+ }
+
+ @Override
+ void run(String[] args) {
+ logger.warn("The decryption capability of this tool is still considered experimental. The results should be manually verified.")
+ try {
+
+ def options = cli.parse(args)
+
+ if (!options || options.h) {
+ printUsageAndExit("", EncryptConfigMain.EXIT_STATUS_OTHER)
+ }
+
+ EncryptConfigLogger.configureLogger(options.v)
+
+ DecryptConfiguration config = new DecryptConfiguration(options)
+
+ run(config)
+
+ } catch (Exception e) {
+ logger.error("Encountered an error: ${e.getMessage()}")
+ logger.debug("", e) // stack trace only when verbose enabled
--- End diff --
I understand why this was done but I think the better logic is:
```
if (isVerboseEnabled()) {
logger.error("", e)
}
```
Some people/tools are set up to extract messages based on levels, and this is semantically an error.
---
[GitHub] nifi pull request #2376: NIFI-4708 Add Registry support to encrypt-config
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2376#discussion_r159972582
--- Diff: nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/toolkit/encryptconfig/util/BootstrapUtil.groovy ---
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.toolkit.encryptconfig.util
+
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+class BootstrapUtil {
+
+ static final String NIFI_BOOTSTRAP_KEY_PROPERTY = "nifi.bootstrap.sensitive.key";
+ static final String REGISTRY_BOOTSTRAP_KEY_PROPERTY = "nifi.registry.bootstrap.sensitive.key";
+
+ private static final Logger logger = LoggerFactory.getLogger(BootstrapUtil.class)
+
+ private static final String BOOTSTRAP_KEY_COMMENT = "# Master key in hexadecimal format for encrypted sensitive configuration values"
+
+ /**
+ * Tries to load keyHex from input bootstrap.conf
+ *
+ * @return keyHex, if present in input bootstrap file; otherwise, null
+ */
+ static String extractKeyFromBootstrapFile(String inputBootstrapPath, String bootstrapKeyPropertyName) throws IOException {
+
+ File inputBootstrapConfFile
+ if (!(inputBootstrapPath && (inputBootstrapConfFile = new File(inputBootstrapPath)).exists() && inputBootstrapConfFile.canRead())) {
--- End diff --
Not required for this PR, but in the future this is a good anti-pattern for extracting the control logic to a boolean checker method like `isInputBootstrapConfValid()`.
---
[GitHub] nifi pull request #2376: NIFI-4708 Add Registry support to encrypt-config
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/nifi/pull/2376
---
[GitHub] nifi issue #2376: NIFI-4708 Add Registry support to encrypt-config
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on the issue:
https://github.com/apache/nifi/pull/2376
FYI @alopresto and @bbende - This is ready to be reviewed for merge to master.
There are still a few test cases I would like to add, time permitting, for xml files, but that could be addressed in a follow up PR, along with additional refinement and a refactoring that @alopresto and I have discussed.
---
[GitHub] nifi pull request #2376: NIFI-4708 Add Registry support to encrypt-config
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2376#discussion_r159959488
--- Diff: nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/toolkit/encryptconfig/NiFiRegistryDecryptMode.groovy ---
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.toolkit.encryptconfig
+
+import org.apache.nifi.properties.AESSensitivePropertyProvider
+import org.apache.nifi.toolkit.encryptconfig.util.BootstrapUtil
+import org.apache.nifi.toolkit.encryptconfig.util.ToolUtilities
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+/**
+ * A special DecryptMode that can run using NiFiRegistry CLI Options
+ */
+class NiFiRegistryDecryptMode extends DecryptMode {
+
+ private static final Logger logger = LoggerFactory.getLogger(NiFiRegistryDecryptMode.class)
+
+ CliBuilder cli
+
+ NiFiRegistryDecryptMode() {
+ cli = NiFiRegistryMode.cliBuilder()
+ }
+
+ @Override
+ void run(String[] args) {
+ logger.warn("The decryption capability of this tool is still considered experimental. The results should be manually verified.")
+ try {
+
+ def options = cli.parse(args)
+
+ if (!options || options.h) {
+ EncryptConfigMain.printUsageAndExit("", EncryptConfigMain.EXIT_STATUS_OTHER)
+ }
+
+ EncryptConfigLogger.configureLogger(options.v)
+
+ DecryptConfiguration config = new DecryptConfiguration()
+
+ /* Invalid fields when used with --decrypt: */
+ def invalidDecryptOptions = ["i", "a"]
+ def presentInvalidOptions = Arrays.stream(options.getInner().getOptions()).findAll {
+ invalidDecryptOptions.contains(it.getOpt())
+ }
+ if (presentInvalidOptions.size() > 0) {
+ throw new RuntimeException("Invalid options: ${EncryptConfigMain.DECRYPT_OPT} cannot be used with [${presentInvalidOptions.join(", ")}]. It should only be used with [-r].")
+ }
+
+ /* Required fields when using --decrypt */
+ // registryPropertiesFile (-r)
+ if (!options.r) {
+ throw new RuntimeException("Invalid options: Input nifiRegistryProperties (-r) is required when using --decrypt")
+ }
+ config.inputFilePath = options.r
+ config.fileType = FileType.properties // disables auto-detection, which is still experimental
+
+ // one of [--oldPassword, --oldKey] or [-p, -k, -b <file]
+ String keyHex = null
+ String password = null
+ config.keySource = null
+ if (options.oldPassword) {
+ if (config.keySource != null) {
+ throw new RuntimeException("Invalid options: Only one of [--oldPassword, --oldKey, -b, -p, -k] is allowed for specifying the decryption password/key.")
+ }
+ config.keySource = Configuration.KeySource.PASSWORD
+ keyHex = options.getInner().getOptionValue("oldPassword")
+ }
+ if (options.oldKey) {
+ if (config.keySource != null) {
+ throw new RuntimeException("Invalid options: Only one of [--oldPassword, --oldKey, -b, -p, -k] is allowed for specifying the decryption password/key.")
+ }
+ config.keySource = Configuration.KeySource.KEY_HEX
+ keyHex = options.getInner().getOptionValue("oldKey")
+ }
+ if (options.p) {
+ if (config.keySource != null) {
+ throw new RuntimeException("Invalid options: Only one of [--oldPassword, --oldKey, -b, -p, -k] is allowed for specifying the decryption password/key.")
+ }
+ config.keySource = Configuration.KeySource.PASSWORD
+ password = options.getInner().getOptionValue("p")
+ }
+ if (options.k) {
+ if (config.keySource != null) {
+ throw new RuntimeException("Invalid options: Only one of [--oldPassword, --oldKey, -b, -p, -k] is allowed for specifying the decryption password/key.")
+ }
+ config.keySource = Configuration.KeySource.KEY_HEX
+ keyHex = options.getInner().getOptionValue("k")
+ }
+
+ if (config.keySource) {
+ config.key = ToolUtilities.determineKey(keyHex, password, Configuration.KeySource.PASSWORD.equals(config.keySource))
+ }
+
+ if (options.b) {
+ if (config.keySource != null) {
+ throw new RuntimeException("Invalid options: Only one of [--oldPassword, --oldKey, -b, -p, -k] is allowed for specifying the decryption password/key.")
+ }
+ config.keySource = Configuration.KeySource.BOOTSTRAP_FILE
+ config.inputBootstrapPath = options.b
+
+ logger.debug("Checking expected NiFi Registry bootstrap.conf format")
+ config.key = BootstrapUtil.extractKeyFromBootstrapFile(config.inputBootstrapPath, BootstrapUtil.REGISTRY_BOOTSTRAP_KEY_PROPERTY)
+
+ // check we have found the key
+ if (config.key) {
+ logger.debug("Master key found in ${config.inputBootstrapPath}. This key will be used for decryption operations.")
+ } else {
+ logger.warn("Bootstrap Conf flag present, but master key could not be found in ${config.inputBootstrapPath}.")
+ }
+ }
+
+ config.decryptionProvider = new AESSensitivePropertyProvider(config.key)
+
+ /* Optional fields when using --decrypt */
+ // -R outputRegistryPropertiesFile (-R)
--- End diff --
As per discussion, remove this option.
---
[GitHub] nifi pull request #2376: NIFI-4708 Add Registry support to encrypt-config
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2376#discussion_r159983212
--- Diff: nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/toolkit/encryptconfig/util/NiFiRegistryIdentityProvidersXmlEncryptor.groovy ---
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.toolkit.encryptconfig.util
+
+import groovy.xml.XmlUtil
+import org.apache.nifi.properties.SensitivePropertyProvider
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+import org.xml.sax.SAXException
+
+class NiFiRegistryIdentityProvidersXmlEncryptor extends XmlEncryptor {
+
+ private static final Logger logger = LoggerFactory.getLogger(NiFiRegistryIdentityProvidersXmlEncryptor.class)
+
+ private static final String LDAP_PROVIDER_CLASS = "org.apache.nifi.registry.security.ldap.LdapIdentityProvider"
+ private static final String LDAP_PROVIDER_REGEX = /(?s)<provider>(?:(?!<provider>).)*?<class>\s*org\.apache\.nifi\.registry\.security\.ldap\.LdapIdentityProvider.*?<\/provider>/
+ /* Explanation of LDAP_PROVIDER_REGEX:
+ * (?s) -> single-line mode (i.e., `.` in regex matches newlines)
+ * <provider> -> find occurrence of `<provider>` literally (case-sensitive)
+ * (?: ... ) -> group but do not capture submatch
+ * (?! ... ) -> negative lookahead
+ * (?:(?!<provider>).)*? -> find everything until a new `<provider>` starts. This is for not selecting multiple providers in one match
+ * <class> -> find occurrence of `<class>` literally (case-sensitive)
+ * \s* -> find any whitespace
+ * org\.apache\.nifi\.registry\.security\.ldap\.LdapIdentityProvider
+ * -> find occurrence of `org.apache.nifi.registry.security.ldap.LdapIdentityProvider` literally (case-sensitive)
+ * .*?</provider> -> find everything as needed up until and including occurrence of `</provider>`
+ */
+
+ NiFiRegistryIdentityProvidersXmlEncryptor(SensitivePropertyProvider encryptionProvider, SensitivePropertyProvider decryptionProvider) {
+ super(encryptionProvider, decryptionProvider)
+ }
+
+ @Override
+ String encrypt(String plainXmlContent) {
--- End diff --
- [ ] Add Javadoc to sub-classes of XMLEncryptor
---
[GitHub] nifi pull request #2376: NIFI-4708 Add Registry support to encrypt-config
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2376#discussion_r159957605
--- Diff: nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/toolkit/encryptconfig/NiFiRegistryDecryptMode.groovy ---
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.toolkit.encryptconfig
+
+import org.apache.nifi.properties.AESSensitivePropertyProvider
+import org.apache.nifi.toolkit.encryptconfig.util.BootstrapUtil
+import org.apache.nifi.toolkit.encryptconfig.util.ToolUtilities
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+/**
+ * A special DecryptMode that can run using NiFiRegistry CLI Options
+ */
+class NiFiRegistryDecryptMode extends DecryptMode {
+
+ private static final Logger logger = LoggerFactory.getLogger(NiFiRegistryDecryptMode.class)
+
+ CliBuilder cli
+
+ NiFiRegistryDecryptMode() {
+ cli = NiFiRegistryMode.cliBuilder()
+ }
+
+ @Override
+ void run(String[] args) {
+ logger.warn("The decryption capability of this tool is still considered experimental. The results should be manually verified.")
+ try {
+
+ def options = cli.parse(args)
+
+ if (!options || options.h) {
+ EncryptConfigMain.printUsageAndExit("", EncryptConfigMain.EXIT_STATUS_OTHER)
+ }
+
+ EncryptConfigLogger.configureLogger(options.v)
+
+ DecryptConfiguration config = new DecryptConfiguration()
+
+ /* Invalid fields when used with --decrypt: */
+ def invalidDecryptOptions = ["i", "a"]
+ def presentInvalidOptions = Arrays.stream(options.getInner().getOptions()).findAll {
+ invalidDecryptOptions.contains(it.getOpt())
+ }
+ if (presentInvalidOptions.size() > 0) {
+ throw new RuntimeException("Invalid options: ${EncryptConfigMain.DECRYPT_OPT} cannot be used with [${presentInvalidOptions.join(", ")}]. It should only be used with [-r].")
+ }
+
+ /* Required fields when using --decrypt */
+ // registryPropertiesFile (-r)
+ if (!options.r) {
+ throw new RuntimeException("Invalid options: Input nifiRegistryProperties (-r) is required when using --decrypt")
+ }
+ config.inputFilePath = options.r
+ config.fileType = FileType.properties // disables auto-detection, which is still experimental
+
+ // one of [--oldPassword, --oldKey] or [-p, -k, -b <file]
+ String keyHex = null
--- End diff --
I think this logic should be extracted to a method and condensed. Something like this (note: should `oldPassword` and `oldKey` be accepted for decryption?):
```
Index: nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/toolkit/encryptconfig/NiFiRegistryDecryptMode.groovy
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
--- nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/toolkit/encryptconfig/NiFiRegistryDecryptMode.groovy (date 1515148235000)
+++ nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/toolkit/encryptconfig/NiFiRegistryDecryptMode.groovy (revision )
@@ -67,77 +67,112 @@
config.inputFilePath = options.r
config.fileType = FileType.properties // disables auto-detection, which is still experimental
- // one of [--oldPassword, --oldKey] or [-p, -k, -b <file]
- String keyHex = null
- String password = null
- config.keySource = null
- if (options.oldPassword) {
- if (config.keySource != null) {
+
+ config.key = retrieveKey(config, options)
+
+ config.decryptionProvider = new AESSensitivePropertyProvider(config.key)
+
+ /* Optional fields when using --decrypt */
+ // -R outputRegistryPropertiesFile (-R)
+ if (options.R) {
+ config.outputToFile = true
+ config.outputFilePath = options.R
+ }
+
+ run(config)
+
+ } catch (Exception e) {
+ logger.error("Encountered an error: ${e.getMessage()}")
+ logger.debug("", e) // only print stack trace when verbose is enabled
+ EncryptConfigMain.printUsageAndExit(EncryptConfigMain.EXIT_STATUS_FAILURE)
+ }
+ }
+
+ private String retrieveKey(DecryptConfiguration config, OptionAccessor options) {
+ // one of [--oldPassword, --oldKey] or [-p, -k, -b <file]
+ String keyHex = null
+ String password = null
+ config.keySource = null
+
+ Map<String, Configuration.KeySource> keySourceMap = [
+ oldPassword: Configuration.KeySource.PASSWORD,
+ oldKey : Configuration.KeySource.KEY_HEX,
+ p : Configuration.KeySource.PASSWORD,
+ k : Configuration.KeySource.KEY_HEX,
+// b : Configuration.KeySource.BOOTSTRAP_FILE
+ ]
+
+ keySourceMap.each { String name, Configuration.KeySource ks ->
+ if (options."$name") {
+ if (config.keySource) {
throw new RuntimeException("Invalid options: Only one of [--oldPassword, --oldKey, -b, -p, -k] is allowed for specifying the decryption password/key.")
}
- config.keySource = Configuration.KeySource.PASSWORD
- keyHex = options.getInner().getOptionValue("oldPassword")
- }
- if (options.oldKey) {
- if (config.keySource != null) {
- throw new RuntimeException("Invalid options: Only one of [--oldPassword, --oldKey, -b, -p, -k] is allowed for specifying the decryption password/key.")
+ config.keySource = ks
+ if (ks == Configuration.KeySource.PASSWORD) {
+ password = options.getInner().getOptionValue(name)
+ } else {
+ keyHex = options.getInner().getOptionValue(name)
}
- config.keySource = Configuration.KeySource.KEY_HEX
- keyHex = options.getInner().getOptionValue("oldKey")
}
- if (options.p) {
- if (config.keySource != null) {
- throw new RuntimeException("Invalid options: Only one of [--oldPassword, --oldKey, -b, -p, -k] is allowed for specifying the decryption password/key.")
- }
- config.keySource = Configuration.KeySource.PASSWORD
- password = options.getInner().getOptionValue("p")
- }
- if (options.k) {
- if (config.keySource != null) {
- throw new RuntimeException("Invalid options: Only one of [--oldPassword, --oldKey, -b, -p, -k] is allowed for specifying the decryption password/key.")
- }
- config.keySource = Configuration.KeySource.KEY_HEX
- keyHex = options.getInner().getOptionValue("k")
- }
+ }
- if (config.keySource) {
- config.key = ToolUtilities.determineKey(keyHex, password, Configuration.KeySource.PASSWORD.equals(config.keySource))
- }
+ // Convert the password to a key (if necessary) and store in the config object
+ if (config.keySource) {
+ config.key = ToolUtilities.determineKey(keyHex, password, config.keySource == Configuration.KeySource.PASSWORD)
+ }
+
+// if (options.oldPassword) {
+// if (config.keySource != null) {
+// throw new RuntimeException("Invalid options: Only one of [--oldPassword, --oldKey, -b, -p, -k] is allowed for specifying the decryption password/key.")
+// }
+// config.keySource = Configuration.KeySource.PASSWORD
+// keyHex = options.getInner().getOptionValue("oldPassword")
+// }
+// if (options.oldKey) {
+// if (config.keySource != null) {
+// throw new RuntimeException("Invalid options: Only one of [--oldPassword, --oldKey, -b, -p, -k] is allowed for specifying the decryption password/key.")
+// }
+// config.keySource = Configuration.KeySource.KEY_HEX
+// keyHex = options.getInner().getOptionValue("oldKey")
+// }
+// if (options.p) {
+// if (config.keySource != null) {
+// throw new RuntimeException("Invalid options: Only one of [--oldPassword, --oldKey, -b, -p, -k] is allowed for specifying the decryption password/key.")
+// }
+// config.keySource = Configuration.KeySource.PASSWORD
+// password = options.getInner().getOptionValue("p")
+// }
+// if (options.k) {
+// if (config.keySource != null) {
+// throw new RuntimeException("Invalid options: Only one of [--oldPassword, --oldKey, -b, -p, -k] is allowed for specifying the decryption password/key.")
+// }
+// config.keySource = Configuration.KeySource.KEY_HEX
+// keyHex = options.getInner().getOptionValue("k")
+// }
+
+// if (config.keySource) {
+// config.key = ToolUtilities.determineKey(keyHex, password, Configuration.KeySource.PASSWORD.equals(config.keySource))
+// }
- if (options.b) {
- if (config.keySource != null) {
- throw new RuntimeException("Invalid options: Only one of [--oldPassword, --oldKey, -b, -p, -k] is allowed for specifying the decryption password/key.")
- }
- config.keySource = Configuration.KeySource.BOOTSTRAP_FILE
- config.inputBootstrapPath = options.b
+ if (options.b) {
+ if (config.keySource) {
+ throw new RuntimeException("Invalid options: Only one of [--oldPassword, --oldKey, -b, -p, -k] is allowed for specifying the decryption password/key.")
+ }
+ config.keySource = Configuration.KeySource.BOOTSTRAP_FILE
+ config.inputBootstrapPath = options.b
- logger.debug("Checking expected NiFi Registry bootstrap.conf format")
- config.key = BootstrapUtil.extractKeyFromBootstrapFile(config.inputBootstrapPath, BootstrapUtil.REGISTRY_BOOTSTRAP_KEY_PROPERTY)
+ logger.debug("Checking expected NiFi Registry bootstrap.conf format")
+ config.key = BootstrapUtil.extractKeyFromBootstrapFile(config.inputBootstrapPath, BootstrapUtil.REGISTRY_BOOTSTRAP_KEY_PROPERTY)
- // check we have found the key
- if (config.key) {
- logger.debug("Master key found in ${config.inputBootstrapPath}. This key will be used for decryption operations.")
- } else {
- logger.warn("Bootstrap Conf flag present, but master key could not be found in ${config.inputBootstrapPath}.")
- }
- }
+ // check we have found the key
+ if (config.key) {
+ logger.debug("Master key found in ${config.inputBootstrapPath}. This key will be used for decryption operations.")
+ } else {
+ logger.warn("Bootstrap conf flag present, but master key could not be found in ${config.inputBootstrapPath}.")
+ }
+ }
- config.decryptionProvider = new AESSensitivePropertyProvider(config.key)
-
- /* Optional fields when using --decrypt */
- // -R outputRegistryPropertiesFile (-R)
- if (options.R) {
- config.outputToFile = true
- config.outputFilePath = options.R
- }
-
- run(config)
-
- } catch (Exception e) {
- logger.error("Encountered an error: ${e.getMessage()}")
- logger.debug("", e) // only print stack trace when verbose is enabled
- EncryptConfigMain.printUsageAndExit(EncryptConfigMain.EXIT_STATUS_FAILURE)
- }
+ config.key
}
}
```
---
[GitHub] nifi pull request #2376: NIFI-4708 Add Registry support to encrypt-config
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2376#discussion_r159959534
--- Diff: nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/toolkit/encryptconfig/NiFiRegistryDecryptMode.groovy ---
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.toolkit.encryptconfig
+
+import org.apache.nifi.properties.AESSensitivePropertyProvider
+import org.apache.nifi.toolkit.encryptconfig.util.BootstrapUtil
+import org.apache.nifi.toolkit.encryptconfig.util.ToolUtilities
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+/**
+ * A special DecryptMode that can run using NiFiRegistry CLI Options
+ */
+class NiFiRegistryDecryptMode extends DecryptMode {
+
+ private static final Logger logger = LoggerFactory.getLogger(NiFiRegistryDecryptMode.class)
+
+ CliBuilder cli
+
+ NiFiRegistryDecryptMode() {
+ cli = NiFiRegistryMode.cliBuilder()
+ }
+
+ @Override
+ void run(String[] args) {
+ logger.warn("The decryption capability of this tool is still considered experimental. The results should be manually verified.")
+ try {
+
+ def options = cli.parse(args)
+
+ if (!options || options.h) {
+ EncryptConfigMain.printUsageAndExit("", EncryptConfigMain.EXIT_STATUS_OTHER)
+ }
+
+ EncryptConfigLogger.configureLogger(options.v)
+
+ DecryptConfiguration config = new DecryptConfiguration()
+
+ /* Invalid fields when used with --decrypt: */
+ def invalidDecryptOptions = ["i", "a"]
+ def presentInvalidOptions = Arrays.stream(options.getInner().getOptions()).findAll {
+ invalidDecryptOptions.contains(it.getOpt())
+ }
+ if (presentInvalidOptions.size() > 0) {
+ throw new RuntimeException("Invalid options: ${EncryptConfigMain.DECRYPT_OPT} cannot be used with [${presentInvalidOptions.join(", ")}]. It should only be used with [-r].")
+ }
+
+ /* Required fields when using --decrypt */
+ // registryPropertiesFile (-r)
+ if (!options.r) {
+ throw new RuntimeException("Invalid options: Input nifiRegistryProperties (-r) is required when using --decrypt")
+ }
+ config.inputFilePath = options.r
+ config.fileType = FileType.properties // disables auto-detection, which is still experimental
+
+ // one of [--oldPassword, --oldKey] or [-p, -k, -b <file]
+ String keyHex = null
+ String password = null
+ config.keySource = null
+ if (options.oldPassword) {
+ if (config.keySource != null) {
+ throw new RuntimeException("Invalid options: Only one of [--oldPassword, --oldKey, -b, -p, -k] is allowed for specifying the decryption password/key.")
+ }
+ config.keySource = Configuration.KeySource.PASSWORD
+ keyHex = options.getInner().getOptionValue("oldPassword")
+ }
+ if (options.oldKey) {
+ if (config.keySource != null) {
+ throw new RuntimeException("Invalid options: Only one of [--oldPassword, --oldKey, -b, -p, -k] is allowed for specifying the decryption password/key.")
+ }
+ config.keySource = Configuration.KeySource.KEY_HEX
+ keyHex = options.getInner().getOptionValue("oldKey")
+ }
+ if (options.p) {
+ if (config.keySource != null) {
+ throw new RuntimeException("Invalid options: Only one of [--oldPassword, --oldKey, -b, -p, -k] is allowed for specifying the decryption password/key.")
+ }
+ config.keySource = Configuration.KeySource.PASSWORD
+ password = options.getInner().getOptionValue("p")
+ }
+ if (options.k) {
+ if (config.keySource != null) {
+ throw new RuntimeException("Invalid options: Only one of [--oldPassword, --oldKey, -b, -p, -k] is allowed for specifying the decryption password/key.")
+ }
+ config.keySource = Configuration.KeySource.KEY_HEX
+ keyHex = options.getInner().getOptionValue("k")
+ }
+
+ if (config.keySource) {
+ config.key = ToolUtilities.determineKey(keyHex, password, Configuration.KeySource.PASSWORD.equals(config.keySource))
+ }
+
+ if (options.b) {
+ if (config.keySource != null) {
+ throw new RuntimeException("Invalid options: Only one of [--oldPassword, --oldKey, -b, -p, -k] is allowed for specifying the decryption password/key.")
+ }
+ config.keySource = Configuration.KeySource.BOOTSTRAP_FILE
+ config.inputBootstrapPath = options.b
+
+ logger.debug("Checking expected NiFi Registry bootstrap.conf format")
+ config.key = BootstrapUtil.extractKeyFromBootstrapFile(config.inputBootstrapPath, BootstrapUtil.REGISTRY_BOOTSTRAP_KEY_PROPERTY)
+
+ // check we have found the key
+ if (config.key) {
+ logger.debug("Master key found in ${config.inputBootstrapPath}. This key will be used for decryption operations.")
+ } else {
+ logger.warn("Bootstrap Conf flag present, but master key could not be found in ${config.inputBootstrapPath}.")
+ }
+ }
+
+ config.decryptionProvider = new AESSensitivePropertyProvider(config.key)
+
+ /* Optional fields when using --decrypt */
+ // -R outputRegistryPropertiesFile (-R)
+ if (options.R) {
+ config.outputToFile = true
+ config.outputFilePath = options.R
+ }
+
+ run(config)
+
+ } catch (Exception e) {
+ logger.error("Encountered an error: ${e.getMessage()}")
+ logger.debug("", e) // only print stack trace when verbose is enabled
--- End diff --
Change to `logger.error()`.
---
[GitHub] nifi pull request #2376: NIFI-4708 Add Registry support to encrypt-config
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2376#discussion_r159976358
--- Diff: nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/toolkit/encryptconfig/DecryptMode.groovy ---
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.toolkit.encryptconfig
+
+import org.apache.commons.cli.HelpFormatter
+import org.apache.nifi.properties.AESSensitivePropertyProvider
+import org.apache.nifi.properties.SensitivePropertyProvider
+import org.apache.nifi.toolkit.encryptconfig.util.BootstrapUtil
+import org.apache.nifi.toolkit.encryptconfig.util.PropertiesEncryptor
+import org.apache.nifi.toolkit.encryptconfig.util.ToolUtilities
+import org.apache.nifi.toolkit.encryptconfig.util.XmlEncryptor
+import org.apache.nifi.util.console.TextDevices
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+class DecryptMode implements ToolMode {
+
+ private static final Logger logger = LoggerFactory.getLogger(DecryptMode.class)
+
+ enum FileType {
+ properties,
+ xml
+ }
+
+ CliBuilder cli
+
+ DecryptMode() {
+ cli = cliBuilder()
+ }
+
+ void printUsage(String message = "") {
+ if (message) {
+ System.out.println(message)
+ System.out.println()
+ }
+ cli.usage()
+ }
+
+ void printUsageAndExit(String message = "", int exitStatusCode) {
+ printUsage(message)
+ System.exit(exitStatusCode)
+ }
+
+ @Override
+ void run(String[] args) {
+ logger.warn("The decryption capability of this tool is still considered experimental. The results should be manually verified.")
+ try {
+
+ def options = cli.parse(args)
+
+ if (!options || options.h) {
+ printUsageAndExit("", EncryptConfigMain.EXIT_STATUS_OTHER)
+ }
+
+ EncryptConfigLogger.configureLogger(options.v)
+
+ DecryptConfiguration config = new DecryptConfiguration(options)
+
+ run(config)
+
+ } catch (Exception e) {
+ logger.error("Encountered an error: ${e.getMessage()}")
+ logger.debug("", e) // stack trace only when verbose enabled
+ printUsageAndExit(e.getMessage(), EncryptConfigMain.EXIT_STATUS_FAILURE)
+ }
+ }
+
+ void run(DecryptConfiguration config) throws Exception {
+
+ if (!config.fileType) {
+
+ // Try to load the input file to auto-detect the file type
+ boolean isPropertiesFile = PropertiesEncryptor.supportsFile(config.inputFilePath)
+
+ boolean isXmlFile = XmlEncryptor.supportsFile(config.inputFilePath)
+
+ if (ToolUtilities.isExactlyOneTrue(isPropertiesFile, isXmlFile)) {
+ if (isPropertiesFile) {
+ config.fileType = FileType.properties
+ logger.debug("Auto-detection of input file type determined the type to be: ${FileType.properties}")
+ }
+ if (isXmlFile) {
+ config.fileType = FileType.xml
+ logger.debug("Auto-detection of input file type determined the type to be: ${FileType.xml}")
+ }
+ }
+
+ // Could we successfully auto-detect?
+ if (!config.fileType) {
+ throw new RuntimeException("Auto-detection of input file type failed. Please re-run the tool specifying the file type with the -t/--fileType flag.")
+ }
+ }
+
+ String decryptedSerializedContent = null
+ switch (config.fileType) {
+
+ case FileType.properties:
+ PropertiesEncryptor propertiesEncryptor = new PropertiesEncryptor(null, config.decryptionProvider)
+ Properties properties = propertiesEncryptor.loadFile(config.inputFilePath)
+ properties = propertiesEncryptor.decrypt(properties)
+ decryptedSerializedContent = propertiesEncryptor.serializePropertiesAndPreserveFormatIfPossible(properties, config.inputFilePath)
+ break
+
+ case FileType.xml:
+ XmlEncryptor xmlEncryptor = new XmlEncryptor(null, config.decryptionProvider) {
+ @Override
+ List<String> serializeXmlContentAndPreserveFormat(String updatedXmlContent, File originalInputXmlFile) {
+ // For decrypting unknown, generic XML, this tool will not support preserving the format
+ return updatedXmlContent.split("\n")
+ }
+ }
+
+ String xmlContent = xmlEncryptor.loadXmlFile(config.inputFilePath)
+ xmlContent = xmlEncryptor.decrypt(xmlContent)
+ decryptedSerializedContent = xmlEncryptor.serializeXmlContentAndPreserveFormatIfPossible(xmlContent, config.inputFilePath)
+ break
+
+ default:
+ throw new RuntimeException("Unsupported file type '${config.fileType}'")
+ }
+
+ if (!decryptedSerializedContent) {
+ throw new RuntimeException("Failed to load and decrypt input file.")
+ }
+
+ if (config.outputToFile) {
+ try {
+ File outputFile = new File(config.outputFilePath)
+ if (ToolUtilities.isSafeToWrite(outputFile)) {
+ outputFile.text = decryptedSerializedContent
+ logger.info("Wrote decrypted file contents to '${config.outputFilePath}'")
+ }
+ } catch (IOException e) {
+ throw new RuntimeException("Encountered an exception writing the decrypted content to '${config.outputFilePath}': ${e.getMessage()}", e)
+ }
+ } else {
+ System.out.println(decryptedSerializedContent)
+ }
+
+ }
+
+ private CliBuilder cliBuilder() {
+
+ String usage = "${EncryptConfigMain.class.getCanonicalName()} decrypt [options] file"
+
+ int formatWidth = EncryptConfigMain.HELP_FORMAT_WIDTH
+ HelpFormatter formatter = new HelpFormatter()
+ formatter.setWidth(formatWidth)
+ formatter.setOptionComparator(null) // preserve order of options below in help text
+
+ CliBuilder cli = new CliBuilder(
+ usage: usage,
+ width: formatWidth,
+ formatter: formatter,
+ stopAtNonOption: false)
+
+ cli.h(longOpt: 'help', 'Show usage information (this message)')
+ cli.v(longOpt: 'verbose', 'Enables verbose mode (off by default)')
+
+ // Options for the password or key or bootstrap.conf
+ cli.p(longOpt: 'password',
+ args: 1,
+ argName: 'password',
+ optionalArg: true,
+ 'Use a password to derive the key to decrypt the input file. If an argument is not provided to this flag, interactive mode will be triggered to prompt the user to enter the password.')
+ cli.k(longOpt: 'key',
+ args: 1,
+ argName: 'keyhex',
+ optionalArg: true,
+ 'Use a raw hexadecimal key to decrypt the input file. If an argument is not provided to this flag, interactive mode will be triggered to prompt the user to enter the key.')
+ cli.b(longOpt: 'bootstrapConf',
+ args: 1,
+ argName: 'file',
+ 'Use a bootstrap.conf file containing the master key to decrypt the input file (as an alternative to -p or -k)')
+
+ cli.o(longOpt: 'output',
+ args: 1,
+ argName: 'file',
+ 'Specify an output file. If omitted, Standard Out is used. Output file can be set to the input file to decrypt the file in-place.')
+
+ return cli
+
+ }
+
+ static class DecryptConfiguration {
+
+ OptionAccessor rawOptions
+
+ Configuration.KeySource keySource
+ String key
+ SensitivePropertyProvider decryptionProvider
+ String inputBootstrapPath
+
+ FileType fileType
+
+ String inputFilePath
+
+ boolean outputToFile = false
+ String outputFilePath
+
+ DecryptConfiguration() {
+ }
+
+ DecryptConfiguration(OptionAccessor options) {
+ this.rawOptions = options
+
+ validateOptions()
+ determineInputFileFromRemainingArgs()
+
+ determineKey()
+ if (!key) {
+ throw new RuntimeException("Failed to configure tool, could not determine key.")
+ }
+ decryptionProvider = new AESSensitivePropertyProvider(key)
+
+ if (rawOptions.t) {
+ fileType = FileType.valueOf(rawOptions.t)
+ }
+
+ if (rawOptions.o) {
+ outputToFile = true
+ outputFilePath = rawOptions.o
+ }
+ }
+
+ private void validateOptions() {
+
+ String validationFailedMessage = null
+
+ if (!rawOptions.b && !rawOptions.p && !rawOptions.k) {
+ validationFailedMessage = "-p, -k, or -b is required in order to determine the master key to use for decryption."
+ }
+
+ if (validationFailedMessage) {
+ throw new RuntimeException("Invalid options: " + validationFailedMessage)
+ }
+
+ }
+
+ private void determineInputFileFromRemainingArgs() {
+ String[] remainingArgs = this.rawOptions.getInner().getArgs()
+ if (remainingArgs.length == 0) {
+ throw new RuntimeException("Missing argument: Input file must provided.")
--- End diff --
- [ ] fix typo
---
[GitHub] nifi pull request #2376: NIFI-4708 Add Registry support to encrypt-config
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2376#discussion_r159980870
--- Diff: nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/toolkit/encryptconfig/NiFiRegistryDecryptMode.groovy ---
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.toolkit.encryptconfig
+
+import org.apache.nifi.properties.AESSensitivePropertyProvider
+import org.apache.nifi.toolkit.encryptconfig.util.BootstrapUtil
+import org.apache.nifi.toolkit.encryptconfig.util.ToolUtilities
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+/**
+ * A special DecryptMode that can run using NiFiRegistry CLI Options
+ */
+class NiFiRegistryDecryptMode extends DecryptMode {
+
+ private static final Logger logger = LoggerFactory.getLogger(NiFiRegistryDecryptMode.class)
+
+ CliBuilder cli
+
+ NiFiRegistryDecryptMode() {
+ cli = NiFiRegistryMode.cliBuilder()
+ }
+
+ @Override
+ void run(String[] args) {
+ logger.warn("The decryption capability of this tool is still considered experimental. The results should be manually verified.")
+ try {
+
+ def options = cli.parse(args)
+
+ if (!options || options.h) {
+ EncryptConfigMain.printUsageAndExit("", EncryptConfigMain.EXIT_STATUS_OTHER)
+ }
+
+ EncryptConfigLogger.configureLogger(options.v)
+
+ DecryptConfiguration config = new DecryptConfiguration()
+
+ /* Invalid fields when used with --decrypt: */
+ def invalidDecryptOptions = ["i", "a"]
+ def presentInvalidOptions = Arrays.stream(options.getInner().getOptions()).findAll {
+ invalidDecryptOptions.contains(it.getOpt())
+ }
+ if (presentInvalidOptions.size() > 0) {
+ throw new RuntimeException("Invalid options: ${EncryptConfigMain.DECRYPT_OPT} cannot be used with [${presentInvalidOptions.join(", ")}]. It should only be used with [-r].")
+ }
+
+ /* Required fields when using --decrypt */
+ // registryPropertiesFile (-r)
+ if (!options.r) {
+ throw new RuntimeException("Invalid options: Input nifiRegistryProperties (-r) is required when using --decrypt")
+ }
+ config.inputFilePath = options.r
+ config.fileType = FileType.properties // disables auto-detection, which is still experimental
+
+ // one of [--oldPassword, --oldKey] or [-p, -k, -b <file]
+ String keyHex = null
+ String password = null
+ config.keySource = null
+ if (options.oldPassword) {
+ if (config.keySource != null) {
+ throw new RuntimeException("Invalid options: Only one of [--oldPassword, --oldKey, -b, -p, -k] is allowed for specifying the decryption password/key.")
+ }
+ config.keySource = Configuration.KeySource.PASSWORD
+ keyHex = options.getInner().getOptionValue("oldPassword")
+ }
+ if (options.oldKey) {
+ if (config.keySource != null) {
+ throw new RuntimeException("Invalid options: Only one of [--oldPassword, --oldKey, -b, -p, -k] is allowed for specifying the decryption password/key.")
+ }
+ config.keySource = Configuration.KeySource.KEY_HEX
+ keyHex = options.getInner().getOptionValue("oldKey")
+ }
+ if (options.p) {
+ if (config.keySource != null) {
+ throw new RuntimeException("Invalid options: Only one of [--oldPassword, --oldKey, -b, -p, -k] is allowed for specifying the decryption password/key.")
+ }
+ config.keySource = Configuration.KeySource.PASSWORD
+ password = options.getInner().getOptionValue("p")
+ }
+ if (options.k) {
+ if (config.keySource != null) {
+ throw new RuntimeException("Invalid options: Only one of [--oldPassword, --oldKey, -b, -p, -k] is allowed for specifying the decryption password/key.")
+ }
+ config.keySource = Configuration.KeySource.KEY_HEX
+ keyHex = options.getInner().getOptionValue("k")
+ }
+
+ if (config.keySource) {
+ config.key = ToolUtilities.determineKey(keyHex, password, Configuration.KeySource.PASSWORD.equals(config.keySource))
+ }
+
+ if (options.b) {
+ if (config.keySource != null) {
+ throw new RuntimeException("Invalid options: Only one of [--oldPassword, --oldKey, -b, -p, -k] is allowed for specifying the decryption password/key.")
+ }
+ config.keySource = Configuration.KeySource.BOOTSTRAP_FILE
+ config.inputBootstrapPath = options.b
+
+ logger.debug("Checking expected NiFi Registry bootstrap.conf format")
+ config.key = BootstrapUtil.extractKeyFromBootstrapFile(config.inputBootstrapPath, BootstrapUtil.REGISTRY_BOOTSTRAP_KEY_PROPERTY)
+
+ // check we have found the key
+ if (config.key) {
+ logger.debug("Master key found in ${config.inputBootstrapPath}. This key will be used for decryption operations.")
+ } else {
+ logger.warn("Bootstrap Conf flag present, but master key could not be found in ${config.inputBootstrapPath}.")
+ }
+ }
+
+ config.decryptionProvider = new AESSensitivePropertyProvider(config.key)
+
+ /* Optional fields when using --decrypt */
+ // -R outputRegistryPropertiesFile (-R)
--- End diff --
- [ ] remove -R
---
[GitHub] nifi pull request #2376: NIFI-4708 Add Registry support to encrypt-config
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2376#discussion_r159930797
--- Diff: nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/toolkit/encryptconfig/EncryptConfigLogger.groovy ---
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.toolkit.encryptconfig
+
+import org.apache.log4j.LogManager
+import org.apache.log4j.PropertyConfigurator
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+class EncryptConfigLogger {
+
+ private static final Logger logger = LoggerFactory.getLogger(EncryptConfigLogger.class)
+
+ /**
+ * Configures the logger.
+ *
+ * The nifi-toolkit module uses log4j, which will be configured to append all
+ * log output to the system STDERR. The log level can be specified using the verboseEnabled
+ * argument. A value of <code>true</code> will set the log level to DEBUG, a value of
+ * <code>false</code> will set the log level to INFO.
+ *
+ * @param verboseEnabled flag to indicate if verbose mode is enabled, which sets the log level to DEBUG
+ */
+ static configureLogger(boolean verboseEnabled) {
+
+ Properties log4jProps = null
+ URL log4jPropsPath = this.getClass().getResource("log4j.properties")
+ if (log4jPropsPath) {
+ try {
+ log4jPropsPath.withReader { reader ->
+ log4jProps = new Properties()
+ log4jProps.load(reader)
+ }
+ } catch (IOException e) {
+ // do nothing, we will fallback to hardcoded defaults below
+ }
+ }
+
+ if (!log4jProps) {
+ log4jProps = defaultProperties()
+ }
+
+ if (verboseEnabled) {
+ // Override the log level for this package. For this to work as intended, this class must belong
+ // to the same package (or a parent package) of all the encrypt-config classes
+ log4jProps.put("log4j.logger." + EncryptConfigLogger.class.package.name, "DEBUG")
+ }
+
+ LogManager.resetConfiguration()
+ PropertyConfigurator.configure(log4jProps)
+
+ if (verboseEnabled) {
+ logger.debug("Verbose mode is enabled (goes to stderr by default).")
+ }
+ }
+
+ /**
+ * A copy of the settings in /src/main/resources/log4j.properties, in case that is not on the classpath at runtime
+ * @return Properties containing the default properties for Log4j
+ */
+ static Properties defaultProperties() {
+ Properties defaultProperties = new Properties()
+
+ defaultProperties.setProperty("log4j.rootLogger", "INFO,console")
+
+ defaultProperties.setProperty("log4j.appender.console", "org.apache.log4j.ConsoleAppender")
+ defaultProperties.setProperty("log4j.appender.console.Target", "System.err")
+ defaultProperties.setProperty("log4j.appender.console.layout", "org.apache.log4j.PatternLayout")
+ defaultProperties.setProperty("log4j.appender.console.layout.ConversionPattern", "%d{yyyy-mm-dd HH:mm:ss} %p %c{1}: %m%n")
+
+ return defaultProperties
+ }
+
--- End diff --
Maybe just run a "Format Code" command on this one to remove extra whitespace.
---
[GitHub] nifi pull request #2376: NIFI-4708 Add Registry support to encrypt-config
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2376#discussion_r159924367
--- Diff: nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/toolkit/encryptconfig/DecryptMode.groovy ---
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.toolkit.encryptconfig
+
+import org.apache.commons.cli.HelpFormatter
+import org.apache.nifi.properties.AESSensitivePropertyProvider
+import org.apache.nifi.properties.SensitivePropertyProvider
+import org.apache.nifi.toolkit.encryptconfig.util.BootstrapUtil
+import org.apache.nifi.toolkit.encryptconfig.util.PropertiesEncryptor
+import org.apache.nifi.toolkit.encryptconfig.util.ToolUtilities
+import org.apache.nifi.toolkit.encryptconfig.util.XmlEncryptor
+import org.apache.nifi.util.console.TextDevices
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+class DecryptMode implements ToolMode {
+
+ private static final Logger logger = LoggerFactory.getLogger(DecryptMode.class)
+
+ enum FileType {
+ properties,
+ xml
+ }
+
+ CliBuilder cli
+
+ DecryptMode() {
+ cli = cliBuilder()
+ }
+
+ void printUsage(String message = "") {
+ if (message) {
+ System.out.println(message)
+ System.out.println()
+ }
+ cli.usage()
+ }
+
+ void printUsageAndExit(String message = "", int exitStatusCode) {
+ printUsage(message)
+ System.exit(exitStatusCode)
+ }
+
+ @Override
+ void run(String[] args) {
+ logger.warn("The decryption capability of this tool is still considered experimental. The results should be manually verified.")
+ try {
+
+ def options = cli.parse(args)
+
+ if (!options || options.h) {
+ printUsageAndExit("", EncryptConfigMain.EXIT_STATUS_OTHER)
+ }
+
+ EncryptConfigLogger.configureLogger(options.v)
+
+ DecryptConfiguration config = new DecryptConfiguration(options)
+
+ run(config)
+
+ } catch (Exception e) {
+ logger.error("Encountered an error: ${e.getMessage()}")
+ logger.debug("", e) // stack trace only when verbose enabled
+ printUsageAndExit(e.getMessage(), EncryptConfigMain.EXIT_STATUS_FAILURE)
+ }
+ }
+
+ void run(DecryptConfiguration config) throws Exception {
+
+ if (!config.fileType) {
+
+ // Try to load the input file to auto-detect the file type
+ boolean isPropertiesFile = PropertiesEncryptor.supportsFile(config.inputFilePath)
+
+ boolean isXmlFile = XmlEncryptor.supportsFile(config.inputFilePath)
+
+ if (ToolUtilities.isExactlyOneTrue(isPropertiesFile, isXmlFile)) {
+ if (isPropertiesFile) {
+ config.fileType = FileType.properties
+ logger.debug("Auto-detection of input file type determined the type to be: ${FileType.properties}")
+ }
+ if (isXmlFile) {
+ config.fileType = FileType.xml
+ logger.debug("Auto-detection of input file type determined the type to be: ${FileType.xml}")
+ }
+ }
+
+ // Could we successfully auto-detect?
+ if (!config.fileType) {
+ throw new RuntimeException("Auto-detection of input file type failed. Please re-run the tool specifying the file type with the -t/--fileType flag.")
+ }
+ }
+
+ String decryptedSerializedContent = null
+ switch (config.fileType) {
+
+ case FileType.properties:
+ PropertiesEncryptor propertiesEncryptor = new PropertiesEncryptor(null, config.decryptionProvider)
+ Properties properties = propertiesEncryptor.loadFile(config.inputFilePath)
+ properties = propertiesEncryptor.decrypt(properties)
+ decryptedSerializedContent = propertiesEncryptor.serializePropertiesAndPreserveFormatIfPossible(properties, config.inputFilePath)
+ break
+
+ case FileType.xml:
+ XmlEncryptor xmlEncryptor = new XmlEncryptor(null, config.decryptionProvider) {
+ @Override
+ List<String> serializeXmlContentAndPreserveFormat(String updatedXmlContent, File originalInputXmlFile) {
+ // For decrypting unknown, generic XML, this tool will not support preserving the format
+ return updatedXmlContent.split("\n")
+ }
+ }
+
+ String xmlContent = xmlEncryptor.loadXmlFile(config.inputFilePath)
+ xmlContent = xmlEncryptor.decrypt(xmlContent)
+ decryptedSerializedContent = xmlEncryptor.serializeXmlContentAndPreserveFormatIfPossible(xmlContent, config.inputFilePath)
+ break
+
+ default:
+ throw new RuntimeException("Unsupported file type '${config.fileType}'")
+ }
+
+ if (!decryptedSerializedContent) {
+ throw new RuntimeException("Failed to load and decrypt input file.")
+ }
+
+ if (config.outputToFile) {
--- End diff --
I don't think `decrypt` should allow output to a file. At this stage, it is safer to require the output goes to STDOUT to avoid decrypting files in place.
---
[GitHub] nifi pull request #2376: NIFI-4708 Add Registry support to encrypt-config
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2376#discussion_r159961459
--- Diff: nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/toolkit/encryptconfig/NiFiRegistryMode.groovy ---
@@ -0,0 +1,383 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.toolkit.encryptconfig
+
+import org.apache.commons.cli.HelpFormatter
+import org.apache.commons.cli.Options
+import org.apache.http.annotation.Experimental
+import org.apache.nifi.properties.AESSensitivePropertyProvider
+import org.apache.nifi.properties.SensitivePropertyProvider
+import org.apache.nifi.toolkit.encryptconfig.util.BootstrapUtil
+import org.apache.nifi.toolkit.encryptconfig.util.NiFiRegistryAuthorizersXmlEncryptor
+import org.apache.nifi.toolkit.encryptconfig.util.NiFiRegistryIdentityProvidersXmlEncryptor
+import org.apache.nifi.toolkit.encryptconfig.util.NiFiRegistryPropertiesEncryptor
+import org.apache.nifi.toolkit.encryptconfig.util.ToolUtilities
+import org.apache.nifi.util.console.TextDevices
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+@Experimental
+class NiFiRegistryMode implements ToolMode {
+
+ private static final Logger logger = LoggerFactory.getLogger(NiFiRegistryMode.class)
+
+ CliBuilder cli
+
+ NiFiRegistryMode() {
+ cli = cliBuilder()
+ }
+
+// private void printUsage(String message = "") {
+// if (message) {
+// System.out.println(message)
+// System.out.println()
+// }
+// cli.usage()
+// }
+
+ @Override
+ void run(String[] args) {
+ logger.warn("The NiFi Registry capabilities of this tool is still considered experimental. The results should be manually verified.")
+ try {
+
+ def options = cli.parse(args)
+
+ if (!options || options.h) {
+ EncryptConfigMain.printUsageAndExit("", EncryptConfigMain.EXIT_STATUS_OTHER)
+ }
+
+ EncryptConfigLogger.configureLogger(options.v)
+
+ Configuration config = new Configuration(options)
+ run(config)
+
+ } catch (Exception e) {
+ logger.error("Encountered an error: ${e.getMessage()}")
+ logger.debug("", e) // stack trace only when verbose enabled
+ EncryptConfigMain.printUsageAndExit(e.getMessage(), EncryptConfigMain.EXIT_STATUS_FAILURE)
+ }
+ }
+
+ void run(Configuration config) throws Exception {
+
+ if (config.usingPassword) {
+ logger.info("Using encryption key derived from password.")
+ } else if (config.usingRawKeyHex) {
+ logger.info("Using encryption key provided.")
+ } else if (config.usingBootstrapKey) {
+ logger.info("Using encryption key from input bootstrap.conf.")
+ }
+
+ logger.debug("(src) bootstrap.conf: ${config.inputBootstrapPath}")
+ logger.debug("(dest) bootstrap.conf: ${config.outputBootstrapPath}")
+ logger.debug("(src) nifi.properties: ${config.inputNiFiRegistryPropertiesPath}")
--- End diff --
This line and the next should print `nifi-registry.properties`.
---
[GitHub] nifi pull request #2376: NIFI-4708 Add Registry support to encrypt-config
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2376#discussion_r159927195
--- Diff: nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/toolkit/encryptconfig/DecryptMode.groovy ---
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.toolkit.encryptconfig
+
+import org.apache.commons.cli.HelpFormatter
+import org.apache.nifi.properties.AESSensitivePropertyProvider
+import org.apache.nifi.properties.SensitivePropertyProvider
+import org.apache.nifi.toolkit.encryptconfig.util.BootstrapUtil
+import org.apache.nifi.toolkit.encryptconfig.util.PropertiesEncryptor
+import org.apache.nifi.toolkit.encryptconfig.util.ToolUtilities
+import org.apache.nifi.toolkit.encryptconfig.util.XmlEncryptor
+import org.apache.nifi.util.console.TextDevices
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+class DecryptMode implements ToolMode {
+
+ private static final Logger logger = LoggerFactory.getLogger(DecryptMode.class)
+
+ enum FileType {
+ properties,
+ xml
+ }
+
+ CliBuilder cli
+
+ DecryptMode() {
+ cli = cliBuilder()
+ }
+
+ void printUsage(String message = "") {
+ if (message) {
+ System.out.println(message)
+ System.out.println()
+ }
+ cli.usage()
+ }
+
+ void printUsageAndExit(String message = "", int exitStatusCode) {
+ printUsage(message)
+ System.exit(exitStatusCode)
+ }
+
+ @Override
+ void run(String[] args) {
+ logger.warn("The decryption capability of this tool is still considered experimental. The results should be manually verified.")
+ try {
+
+ def options = cli.parse(args)
+
+ if (!options || options.h) {
+ printUsageAndExit("", EncryptConfigMain.EXIT_STATUS_OTHER)
+ }
+
+ EncryptConfigLogger.configureLogger(options.v)
+
+ DecryptConfiguration config = new DecryptConfiguration(options)
+
+ run(config)
+
+ } catch (Exception e) {
+ logger.error("Encountered an error: ${e.getMessage()}")
+ logger.debug("", e) // stack trace only when verbose enabled
+ printUsageAndExit(e.getMessage(), EncryptConfigMain.EXIT_STATUS_FAILURE)
+ }
+ }
+
+ void run(DecryptConfiguration config) throws Exception {
+
+ if (!config.fileType) {
+
+ // Try to load the input file to auto-detect the file type
+ boolean isPropertiesFile = PropertiesEncryptor.supportsFile(config.inputFilePath)
+
+ boolean isXmlFile = XmlEncryptor.supportsFile(config.inputFilePath)
+
+ if (ToolUtilities.isExactlyOneTrue(isPropertiesFile, isXmlFile)) {
+ if (isPropertiesFile) {
+ config.fileType = FileType.properties
+ logger.debug("Auto-detection of input file type determined the type to be: ${FileType.properties}")
+ }
+ if (isXmlFile) {
+ config.fileType = FileType.xml
+ logger.debug("Auto-detection of input file type determined the type to be: ${FileType.xml}")
+ }
+ }
+
+ // Could we successfully auto-detect?
+ if (!config.fileType) {
+ throw new RuntimeException("Auto-detection of input file type failed. Please re-run the tool specifying the file type with the -t/--fileType flag.")
+ }
+ }
+
+ String decryptedSerializedContent = null
+ switch (config.fileType) {
+
+ case FileType.properties:
+ PropertiesEncryptor propertiesEncryptor = new PropertiesEncryptor(null, config.decryptionProvider)
+ Properties properties = propertiesEncryptor.loadFile(config.inputFilePath)
+ properties = propertiesEncryptor.decrypt(properties)
+ decryptedSerializedContent = propertiesEncryptor.serializePropertiesAndPreserveFormatIfPossible(properties, config.inputFilePath)
+ break
+
+ case FileType.xml:
+ XmlEncryptor xmlEncryptor = new XmlEncryptor(null, config.decryptionProvider) {
+ @Override
+ List<String> serializeXmlContentAndPreserveFormat(String updatedXmlContent, File originalInputXmlFile) {
+ // For decrypting unknown, generic XML, this tool will not support preserving the format
+ return updatedXmlContent.split("\n")
+ }
+ }
+
+ String xmlContent = xmlEncryptor.loadXmlFile(config.inputFilePath)
+ xmlContent = xmlEncryptor.decrypt(xmlContent)
+ decryptedSerializedContent = xmlEncryptor.serializeXmlContentAndPreserveFormatIfPossible(xmlContent, config.inputFilePath)
+ break
+
+ default:
+ throw new RuntimeException("Unsupported file type '${config.fileType}'")
+ }
+
+ if (!decryptedSerializedContent) {
+ throw new RuntimeException("Failed to load and decrypt input file.")
+ }
+
+ if (config.outputToFile) {
+ try {
+ File outputFile = new File(config.outputFilePath)
+ if (ToolUtilities.isSafeToWrite(outputFile)) {
+ outputFile.text = decryptedSerializedContent
+ logger.info("Wrote decrypted file contents to '${config.outputFilePath}'")
+ }
+ } catch (IOException e) {
+ throw new RuntimeException("Encountered an exception writing the decrypted content to '${config.outputFilePath}': ${e.getMessage()}", e)
+ }
+ } else {
+ System.out.println(decryptedSerializedContent)
+ }
+
+ }
+
+ private CliBuilder cliBuilder() {
+
+ String usage = "${EncryptConfigMain.class.getCanonicalName()} decrypt [options] file"
+
+ int formatWidth = EncryptConfigMain.HELP_FORMAT_WIDTH
+ HelpFormatter formatter = new HelpFormatter()
+ formatter.setWidth(formatWidth)
+ formatter.setOptionComparator(null) // preserve order of options below in help text
+
+ CliBuilder cli = new CliBuilder(
+ usage: usage,
+ width: formatWidth,
+ formatter: formatter,
+ stopAtNonOption: false)
+
+ cli.h(longOpt: 'help', 'Show usage information (this message)')
+ cli.v(longOpt: 'verbose', 'Enables verbose mode (off by default)')
+
+ // Options for the password or key or bootstrap.conf
+ cli.p(longOpt: 'password',
+ args: 1,
+ argName: 'password',
+ optionalArg: true,
+ 'Use a password to derive the key to decrypt the input file. If an argument is not provided to this flag, interactive mode will be triggered to prompt the user to enter the password.')
+ cli.k(longOpt: 'key',
+ args: 1,
+ argName: 'keyhex',
+ optionalArg: true,
+ 'Use a raw hexadecimal key to decrypt the input file. If an argument is not provided to this flag, interactive mode will be triggered to prompt the user to enter the key.')
+ cli.b(longOpt: 'bootstrapConf',
+ args: 1,
+ argName: 'file',
+ 'Use a bootstrap.conf file containing the master key to decrypt the input file (as an alternative to -p or -k)')
+
+ cli.o(longOpt: 'output',
+ args: 1,
+ argName: 'file',
+ 'Specify an output file. If omitted, Standard Out is used. Output file can be set to the input file to decrypt the file in-place.')
+
+ return cli
+
+ }
+
+ static class DecryptConfiguration {
+
+ OptionAccessor rawOptions
+
+ Configuration.KeySource keySource
+ String key
+ SensitivePropertyProvider decryptionProvider
+ String inputBootstrapPath
+
+ FileType fileType
+
+ String inputFilePath
+
+ boolean outputToFile = false
+ String outputFilePath
+
+ DecryptConfiguration() {
+ }
+
+ DecryptConfiguration(OptionAccessor options) {
+ this.rawOptions = options
+
+ validateOptions()
+ determineInputFileFromRemainingArgs()
+
+ determineKey()
+ if (!key) {
+ throw new RuntimeException("Failed to configure tool, could not determine key.")
+ }
+ decryptionProvider = new AESSensitivePropertyProvider(key)
+
+ if (rawOptions.t) {
+ fileType = FileType.valueOf(rawOptions.t)
+ }
+
+ if (rawOptions.o) {
+ outputToFile = true
+ outputFilePath = rawOptions.o
+ }
+ }
+
+ private void validateOptions() {
+
+ String validationFailedMessage = null
+
+ if (!rawOptions.b && !rawOptions.p && !rawOptions.k) {
+ validationFailedMessage = "-p, -k, or -b is required in order to determine the master key to use for decryption."
+ }
+
+ if (validationFailedMessage) {
+ throw new RuntimeException("Invalid options: " + validationFailedMessage)
+ }
+
+ }
+
+ private void determineInputFileFromRemainingArgs() {
+ String[] remainingArgs = this.rawOptions.getInner().getArgs()
+ if (remainingArgs.length == 0) {
+ throw new RuntimeException("Missing argument: Input file must provided.")
--- End diff --
Input file must *be* provided.
---
[GitHub] nifi issue #2376: NIFI-4708 Add Registry support to encrypt-config
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on the issue:
https://github.com/apache/nifi/pull/2376
Ran `contrib-check` and all tests pass.
Ran a few scenarios using the tool. Everything looks good. The only slight issue was when I intentionally ran with the wrong arguments and it looks like the error message contains the whole option object and stacktrace in addition to the flag, but it's clear enough for a user to understand the issue.
```
hw12203:...assembly/target/nifi-toolkit-1.5.0-SNAPSHOT-bin/nifi-toolkit-1.5.0-SNAPSHOT (pr2376) alopresto
🔓 271505s @ 11:01:23 $ ./bin/encrypt-config.sh --nifiRegistry -b /Users/alopresto/Workspace/registry/nifi-registry-assembly/target/nifi-registry-0.0.1-SNAPSHOT-bin/nifi-registry-0.0.1-SNAPSHOT/conf/bootstrap.conf -r /Users/alopresto/Workspace/registry/nifi-registry-assembly/target/nifi-registry-0.0.1-SNAPSHOT-bin/nifi-registry-0.0.1-SNAPSHOT/conf/nifi-registry.properties -R /Users/alopresto/Workspace/registry/nifi-registry-assembly/target/nifi-registry-0.0.1-SNAPSHOT-bin/nifi-registry-0.0.1-SNAPSHOT/conf/nifi-registry-encrypted.properties -v --decrypt
2018-01-08 11:01:40 DEBUG EncryptConfigLogger: Verbose mode is enabled (goes to stderr by default).
2018-01-08 11:01:40 ERROR NiFiRegistryDecryptMode: Encountered an error: Invalid options: --decrypt cannot be used with [[ option: R outputNifiRegistryProperties [ARG] :: The destination nifi-registry.properties file containing protected config values. :: class java.lang.String ]]. It should only be used with -r and one of [-p, -k, -b].
java.lang.RuntimeException: Invalid options: --decrypt cannot be used with [[ option: R outputNifiRegistryProperties [ARG] :: The destination nifi-registry.properties file containing protected config values. :: class java.lang.String ]]. It should only be used with -r and one of [-p, -k, -b].
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:80)
at org.codehaus.groovy.reflection.CachedConstructor.doConstructorInvoke(CachedConstructor.java:74)
at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrap.callConstructor(ConstructorSite.java:84)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:60)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:235)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:247)
at org.apache.nifi.toolkit.encryptconfig.NiFiRegistryDecryptMode.run(NiFiRegistryDecryptMode.groovy:63)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSite.invoke(PogoMetaMethodSite.java:169)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:71)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
at org.apache.nifi.toolkit.encryptconfig.EncryptConfigMain.main(EncryptConfigMain.groovy:109)
Invalid options: --decrypt cannot be used with [[ option: R outputNifiRegistryProperties [ARG] :: The destination nifi-registry.properties file containing protected config values. :: class java.lang.String ]]. It should only be used with -r and one of [-p, -k, -b].
```
+1, merging.
---
[GitHub] nifi pull request #2376: NIFI-4708 Add Registry support to encrypt-config
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2376#discussion_r159961206
--- Diff: nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/toolkit/encryptconfig/NiFiRegistryMode.groovy ---
@@ -0,0 +1,383 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.toolkit.encryptconfig
+
+import org.apache.commons.cli.HelpFormatter
+import org.apache.commons.cli.Options
+import org.apache.http.annotation.Experimental
+import org.apache.nifi.properties.AESSensitivePropertyProvider
+import org.apache.nifi.properties.SensitivePropertyProvider
+import org.apache.nifi.toolkit.encryptconfig.util.BootstrapUtil
+import org.apache.nifi.toolkit.encryptconfig.util.NiFiRegistryAuthorizersXmlEncryptor
+import org.apache.nifi.toolkit.encryptconfig.util.NiFiRegistryIdentityProvidersXmlEncryptor
+import org.apache.nifi.toolkit.encryptconfig.util.NiFiRegistryPropertiesEncryptor
+import org.apache.nifi.toolkit.encryptconfig.util.ToolUtilities
+import org.apache.nifi.util.console.TextDevices
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+@Experimental
+class NiFiRegistryMode implements ToolMode {
+
+ private static final Logger logger = LoggerFactory.getLogger(NiFiRegistryMode.class)
+
+ CliBuilder cli
+
+ NiFiRegistryMode() {
+ cli = cliBuilder()
+ }
+
+// private void printUsage(String message = "") {
+// if (message) {
+// System.out.println(message)
+// System.out.println()
+// }
+// cli.usage()
+// }
+
+ @Override
+ void run(String[] args) {
+ logger.warn("The NiFi Registry capabilities of this tool is still considered experimental. The results should be manually verified.")
--- End diff --
...capabilities of this tool *are* still considered experimental.
---
[GitHub] nifi pull request #2376: NIFI-4708 Add Registry support to encrypt-config
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2376#discussion_r159931910
--- Diff: nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/toolkit/encryptconfig/EncryptConfigMain.groovy ---
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.toolkit.encryptconfig
+
+import org.apache.commons.cli.HelpFormatter
+import org.apache.commons.cli.Options
+import org.apache.nifi.properties.ConfigEncryptionTool
+import org.bouncycastle.jce.provider.BouncyCastleProvider
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+import java.security.Security
+
+class EncryptConfigMain {
+
+ private static final Logger logger = LoggerFactory.getLogger(EncryptConfigMain.class)
+
+ static final int EXIT_STATUS_SUCCESS = 0
+ static final int EXIT_STATUS_FAILURE = -1
+ static final int EXIT_STATUS_OTHER = 1
+
+ static final String NIFI_REGISTRY_OPT = "nifiRegistry"
+ static final String NIFI_REGISTRY_FLAG = "--${NIFI_REGISTRY_OPT}".toString()
+ static final String DECRYPT_OPT = "decrypt"
+ static final String DECRYPT_FLAG = "--${DECRYPT_OPT}".toString()
+
+ static final int HELP_FORMAT_WIDTH = 160
+
+ // Access should only be through static methods
+ private EncryptConfigMain() {
+ }
+
+ static printUsage(String message = "") {
+
+ if (message) {
+ System.out.println(message)
+ System.out.println()
+ }
+
+ String header = "\nThis tool enables easy encryption and decryption of configuration files for NiFi and its sub-projects. " +
+ "Unprotected files can be input to this tool to be protected by a key in a manner that is understood by NiFi. " +
+ "Protected files, along with a key, can be input to this tool to be unprotected, for troubleshooting or automation purposes.\n\n"
+
+ def options = new Options()
+ options.addOption("h", "help", false, "Show usage information (this message)")
+ options.addOption(null, NIFI_REGISTRY_OPT, false, "Specifies to target NiFi Registry. When this flag is not included, NiFi is the target.")
+
+ HelpFormatter helpFormatter = new HelpFormatter()
+ helpFormatter.setWidth(160)
+ helpFormatter.setOptionComparator(null)
+ helpFormatter.printHelp("${EncryptConfigMain.class.getCanonicalName()} [-h] [options]", header, options, "\n")
+ System.out.println()
+
+ helpFormatter.setSyntaxPrefix("") // disable "usage: " prefix for the following outputs
+
+ Options nifiModeOptions = ConfigEncryptionTool.getCliOptions()
+ helpFormatter.printHelp(
+ "When targeting NiFi:",
+ nifiModeOptions,
+ false)
+ System.out.println()
+
+ Options nifiRegistryModeOptions = NiFiRegistryMode.getCliOptions()
+ nifiRegistryModeOptions.addOption(null, DECRYPT_OPT, false, "Can be used with -r to decrypt a previously encrypted NiFi Registry Properties file. Decrypted content is printed to STDOUT.")
+ helpFormatter.printHelp(
+ "When targeting NiFi Registry using the ${NIFI_REGISTRY_FLAG} flag:",
+ nifiRegistryModeOptions,
+ false)
+ System.out.println()
+
+// String footer = """
--- End diff --
Remove dead code.
---
[GitHub] nifi pull request #2376: NIFI-4708 Add Registry support to encrypt-config
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2376#discussion_r159977055
--- Diff: nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/toolkit/encryptconfig/DecryptMode.groovy ---
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.toolkit.encryptconfig
+
+import org.apache.commons.cli.HelpFormatter
+import org.apache.nifi.properties.AESSensitivePropertyProvider
+import org.apache.nifi.properties.SensitivePropertyProvider
+import org.apache.nifi.toolkit.encryptconfig.util.BootstrapUtil
+import org.apache.nifi.toolkit.encryptconfig.util.PropertiesEncryptor
+import org.apache.nifi.toolkit.encryptconfig.util.ToolUtilities
+import org.apache.nifi.toolkit.encryptconfig.util.XmlEncryptor
+import org.apache.nifi.util.console.TextDevices
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+class DecryptMode implements ToolMode {
+
+ private static final Logger logger = LoggerFactory.getLogger(DecryptMode.class)
+
+ enum FileType {
+ properties,
+ xml
+ }
+
+ CliBuilder cli
+
+ DecryptMode() {
+ cli = cliBuilder()
+ }
+
+ void printUsage(String message = "") {
+ if (message) {
+ System.out.println(message)
+ System.out.println()
+ }
+ cli.usage()
+ }
+
+ void printUsageAndExit(String message = "", int exitStatusCode) {
+ printUsage(message)
+ System.exit(exitStatusCode)
+ }
+
+ @Override
+ void run(String[] args) {
+ logger.warn("The decryption capability of this tool is still considered experimental. The results should be manually verified.")
+ try {
+
+ def options = cli.parse(args)
+
+ if (!options || options.h) {
+ printUsageAndExit("", EncryptConfigMain.EXIT_STATUS_OTHER)
+ }
+
+ EncryptConfigLogger.configureLogger(options.v)
+
+ DecryptConfiguration config = new DecryptConfiguration(options)
+
+ run(config)
+
+ } catch (Exception e) {
+ logger.error("Encountered an error: ${e.getMessage()}")
+ logger.debug("", e) // stack trace only when verbose enabled
+ printUsageAndExit(e.getMessage(), EncryptConfigMain.EXIT_STATUS_FAILURE)
+ }
+ }
+
+ void run(DecryptConfiguration config) throws Exception {
+
+ if (!config.fileType) {
+
+ // Try to load the input file to auto-detect the file type
+ boolean isPropertiesFile = PropertiesEncryptor.supportsFile(config.inputFilePath)
+
+ boolean isXmlFile = XmlEncryptor.supportsFile(config.inputFilePath)
+
+ if (ToolUtilities.isExactlyOneTrue(isPropertiesFile, isXmlFile)) {
+ if (isPropertiesFile) {
+ config.fileType = FileType.properties
+ logger.debug("Auto-detection of input file type determined the type to be: ${FileType.properties}")
+ }
+ if (isXmlFile) {
+ config.fileType = FileType.xml
+ logger.debug("Auto-detection of input file type determined the type to be: ${FileType.xml}")
+ }
+ }
+
+ // Could we successfully auto-detect?
+ if (!config.fileType) {
+ throw new RuntimeException("Auto-detection of input file type failed. Please re-run the tool specifying the file type with the -t/--fileType flag.")
+ }
+ }
+
+ String decryptedSerializedContent = null
+ switch (config.fileType) {
+
+ case FileType.properties:
+ PropertiesEncryptor propertiesEncryptor = new PropertiesEncryptor(null, config.decryptionProvider)
+ Properties properties = propertiesEncryptor.loadFile(config.inputFilePath)
+ properties = propertiesEncryptor.decrypt(properties)
+ decryptedSerializedContent = propertiesEncryptor.serializePropertiesAndPreserveFormatIfPossible(properties, config.inputFilePath)
+ break
+
+ case FileType.xml:
+ XmlEncryptor xmlEncryptor = new XmlEncryptor(null, config.decryptionProvider) {
+ @Override
+ List<String> serializeXmlContentAndPreserveFormat(String updatedXmlContent, File originalInputXmlFile) {
+ // For decrypting unknown, generic XML, this tool will not support preserving the format
+ return updatedXmlContent.split("\n")
+ }
+ }
+
+ String xmlContent = xmlEncryptor.loadXmlFile(config.inputFilePath)
+ xmlContent = xmlEncryptor.decrypt(xmlContent)
+ decryptedSerializedContent = xmlEncryptor.serializeXmlContentAndPreserveFormatIfPossible(xmlContent, config.inputFilePath)
+ break
+
+ default:
+ throw new RuntimeException("Unsupported file type '${config.fileType}'")
+ }
+
+ if (!decryptedSerializedContent) {
+ throw new RuntimeException("Failed to load and decrypt input file.")
+ }
+
+ if (config.outputToFile) {
+ try {
+ File outputFile = new File(config.outputFilePath)
+ if (ToolUtilities.isSafeToWrite(outputFile)) {
+ outputFile.text = decryptedSerializedContent
+ logger.info("Wrote decrypted file contents to '${config.outputFilePath}'")
+ }
+ } catch (IOException e) {
+ throw new RuntimeException("Encountered an exception writing the decrypted content to '${config.outputFilePath}': ${e.getMessage()}", e)
+ }
+ } else {
+ System.out.println(decryptedSerializedContent)
+ }
+
+ }
+
+ private CliBuilder cliBuilder() {
+
+ String usage = "${EncryptConfigMain.class.getCanonicalName()} decrypt [options] file"
+
+ int formatWidth = EncryptConfigMain.HELP_FORMAT_WIDTH
+ HelpFormatter formatter = new HelpFormatter()
+ formatter.setWidth(formatWidth)
+ formatter.setOptionComparator(null) // preserve order of options below in help text
+
+ CliBuilder cli = new CliBuilder(
+ usage: usage,
+ width: formatWidth,
+ formatter: formatter,
+ stopAtNonOption: false)
+
+ cli.h(longOpt: 'help', 'Show usage information (this message)')
+ cli.v(longOpt: 'verbose', 'Enables verbose mode (off by default)')
+
+ // Options for the password or key or bootstrap.conf
+ cli.p(longOpt: 'password',
+ args: 1,
+ argName: 'password',
+ optionalArg: true,
+ 'Use a password to derive the key to decrypt the input file. If an argument is not provided to this flag, interactive mode will be triggered to prompt the user to enter the password.')
+ cli.k(longOpt: 'key',
+ args: 1,
+ argName: 'keyhex',
+ optionalArg: true,
+ 'Use a raw hexadecimal key to decrypt the input file. If an argument is not provided to this flag, interactive mode will be triggered to prompt the user to enter the key.')
+ cli.b(longOpt: 'bootstrapConf',
+ args: 1,
+ argName: 'file',
+ 'Use a bootstrap.conf file containing the master key to decrypt the input file (as an alternative to -p or -k)')
+
+ cli.o(longOpt: 'output',
+ args: 1,
+ argName: 'file',
+ 'Specify an output file. If omitted, Standard Out is used. Output file can be set to the input file to decrypt the file in-place.')
+
+ return cli
+
+ }
+
+ static class DecryptConfiguration {
+
+ OptionAccessor rawOptions
+
+ Configuration.KeySource keySource
+ String key
+ SensitivePropertyProvider decryptionProvider
+ String inputBootstrapPath
+
+ FileType fileType
+
+ String inputFilePath
+
+ boolean outputToFile = false
+ String outputFilePath
+
+ DecryptConfiguration() {
+ }
+
+ DecryptConfiguration(OptionAccessor options) {
+ this.rawOptions = options
+
+ validateOptions()
+ determineInputFileFromRemainingArgs()
+
+ determineKey()
+ if (!key) {
+ throw new RuntimeException("Failed to configure tool, could not determine key.")
+ }
+ decryptionProvider = new AESSensitivePropertyProvider(key)
+
+ if (rawOptions.t) {
+ fileType = FileType.valueOf(rawOptions.t)
+ }
+
+ if (rawOptions.o) {
+ outputToFile = true
+ outputFilePath = rawOptions.o
+ }
+ }
+
+ private void validateOptions() {
+
+ String validationFailedMessage = null
+
+ if (!rawOptions.b && !rawOptions.p && !rawOptions.k) {
+ validationFailedMessage = "-p, -k, or -b is required in order to determine the master key to use for decryption."
+ }
+
+ if (validationFailedMessage) {
+ throw new RuntimeException("Invalid options: " + validationFailedMessage)
+ }
+
+ }
+
+ private void determineInputFileFromRemainingArgs() {
+ String[] remainingArgs = this.rawOptions.getInner().getArgs()
+ if (remainingArgs.length == 0) {
+ throw new RuntimeException("Missing argument: Input file must provided.")
+ } else if (remainingArgs.length > 1) {
+ throw new RuntimeException("Too many arguments: Please specify exactly one input file in addition to the options.")
--- End diff --
This is actually correct if you call `DecryptMode.run(String[])` (not exposed to end users in this commit). The usage you're thinking of is through `NiFiRegistryDecryptMode.run(String[])`, which does its own argument parsing and calls `DecryptMode.run(DecryptConfiguration)`
---
[GitHub] nifi pull request #2376: NIFI-4708 Add Registry support to encrypt-config
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2376#discussion_r159926943
--- Diff: nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/toolkit/encryptconfig/DecryptMode.groovy ---
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.toolkit.encryptconfig
+
+import org.apache.commons.cli.HelpFormatter
+import org.apache.nifi.properties.AESSensitivePropertyProvider
+import org.apache.nifi.properties.SensitivePropertyProvider
+import org.apache.nifi.toolkit.encryptconfig.util.BootstrapUtil
+import org.apache.nifi.toolkit.encryptconfig.util.PropertiesEncryptor
+import org.apache.nifi.toolkit.encryptconfig.util.ToolUtilities
+import org.apache.nifi.toolkit.encryptconfig.util.XmlEncryptor
+import org.apache.nifi.util.console.TextDevices
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+class DecryptMode implements ToolMode {
+
+ private static final Logger logger = LoggerFactory.getLogger(DecryptMode.class)
+
+ enum FileType {
+ properties,
+ xml
+ }
+
+ CliBuilder cli
+
+ DecryptMode() {
+ cli = cliBuilder()
+ }
+
+ void printUsage(String message = "") {
+ if (message) {
+ System.out.println(message)
+ System.out.println()
+ }
+ cli.usage()
+ }
+
+ void printUsageAndExit(String message = "", int exitStatusCode) {
+ printUsage(message)
+ System.exit(exitStatusCode)
+ }
+
+ @Override
+ void run(String[] args) {
+ logger.warn("The decryption capability of this tool is still considered experimental. The results should be manually verified.")
+ try {
+
+ def options = cli.parse(args)
+
+ if (!options || options.h) {
+ printUsageAndExit("", EncryptConfigMain.EXIT_STATUS_OTHER)
+ }
+
+ EncryptConfigLogger.configureLogger(options.v)
+
+ DecryptConfiguration config = new DecryptConfiguration(options)
+
+ run(config)
+
+ } catch (Exception e) {
+ logger.error("Encountered an error: ${e.getMessage()}")
+ logger.debug("", e) // stack trace only when verbose enabled
+ printUsageAndExit(e.getMessage(), EncryptConfigMain.EXIT_STATUS_FAILURE)
+ }
+ }
+
+ void run(DecryptConfiguration config) throws Exception {
+
+ if (!config.fileType) {
+
+ // Try to load the input file to auto-detect the file type
+ boolean isPropertiesFile = PropertiesEncryptor.supportsFile(config.inputFilePath)
+
+ boolean isXmlFile = XmlEncryptor.supportsFile(config.inputFilePath)
+
+ if (ToolUtilities.isExactlyOneTrue(isPropertiesFile, isXmlFile)) {
+ if (isPropertiesFile) {
+ config.fileType = FileType.properties
+ logger.debug("Auto-detection of input file type determined the type to be: ${FileType.properties}")
+ }
+ if (isXmlFile) {
+ config.fileType = FileType.xml
+ logger.debug("Auto-detection of input file type determined the type to be: ${FileType.xml}")
+ }
+ }
+
+ // Could we successfully auto-detect?
+ if (!config.fileType) {
+ throw new RuntimeException("Auto-detection of input file type failed. Please re-run the tool specifying the file type with the -t/--fileType flag.")
+ }
+ }
+
+ String decryptedSerializedContent = null
+ switch (config.fileType) {
+
+ case FileType.properties:
+ PropertiesEncryptor propertiesEncryptor = new PropertiesEncryptor(null, config.decryptionProvider)
+ Properties properties = propertiesEncryptor.loadFile(config.inputFilePath)
+ properties = propertiesEncryptor.decrypt(properties)
+ decryptedSerializedContent = propertiesEncryptor.serializePropertiesAndPreserveFormatIfPossible(properties, config.inputFilePath)
+ break
+
+ case FileType.xml:
+ XmlEncryptor xmlEncryptor = new XmlEncryptor(null, config.decryptionProvider) {
+ @Override
+ List<String> serializeXmlContentAndPreserveFormat(String updatedXmlContent, File originalInputXmlFile) {
+ // For decrypting unknown, generic XML, this tool will not support preserving the format
+ return updatedXmlContent.split("\n")
+ }
+ }
+
+ String xmlContent = xmlEncryptor.loadXmlFile(config.inputFilePath)
+ xmlContent = xmlEncryptor.decrypt(xmlContent)
+ decryptedSerializedContent = xmlEncryptor.serializeXmlContentAndPreserveFormatIfPossible(xmlContent, config.inputFilePath)
+ break
+
+ default:
+ throw new RuntimeException("Unsupported file type '${config.fileType}'")
+ }
+
+ if (!decryptedSerializedContent) {
+ throw new RuntimeException("Failed to load and decrypt input file.")
+ }
+
+ if (config.outputToFile) {
+ try {
+ File outputFile = new File(config.outputFilePath)
+ if (ToolUtilities.isSafeToWrite(outputFile)) {
+ outputFile.text = decryptedSerializedContent
+ logger.info("Wrote decrypted file contents to '${config.outputFilePath}'")
+ }
+ } catch (IOException e) {
+ throw new RuntimeException("Encountered an exception writing the decrypted content to '${config.outputFilePath}': ${e.getMessage()}", e)
+ }
+ } else {
+ System.out.println(decryptedSerializedContent)
+ }
+
+ }
+
+ private CliBuilder cliBuilder() {
+
+ String usage = "${EncryptConfigMain.class.getCanonicalName()} decrypt [options] file"
+
+ int formatWidth = EncryptConfigMain.HELP_FORMAT_WIDTH
+ HelpFormatter formatter = new HelpFormatter()
+ formatter.setWidth(formatWidth)
+ formatter.setOptionComparator(null) // preserve order of options below in help text
+
+ CliBuilder cli = new CliBuilder(
+ usage: usage,
+ width: formatWidth,
+ formatter: formatter,
+ stopAtNonOption: false)
+
+ cli.h(longOpt: 'help', 'Show usage information (this message)')
+ cli.v(longOpt: 'verbose', 'Enables verbose mode (off by default)')
+
+ // Options for the password or key or bootstrap.conf
+ cli.p(longOpt: 'password',
+ args: 1,
+ argName: 'password',
+ optionalArg: true,
+ 'Use a password to derive the key to decrypt the input file. If an argument is not provided to this flag, interactive mode will be triggered to prompt the user to enter the password.')
+ cli.k(longOpt: 'key',
+ args: 1,
+ argName: 'keyhex',
+ optionalArg: true,
+ 'Use a raw hexadecimal key to decrypt the input file. If an argument is not provided to this flag, interactive mode will be triggered to prompt the user to enter the key.')
+ cli.b(longOpt: 'bootstrapConf',
+ args: 1,
+ argName: 'file',
+ 'Use a bootstrap.conf file containing the master key to decrypt the input file (as an alternative to -p or -k)')
+
+ cli.o(longOpt: 'output',
+ args: 1,
+ argName: 'file',
+ 'Specify an output file. If omitted, Standard Out is used. Output file can be set to the input file to decrypt the file in-place.')
+
+ return cli
+
+ }
+
+ static class DecryptConfiguration {
+
+ OptionAccessor rawOptions
+
+ Configuration.KeySource keySource
+ String key
+ SensitivePropertyProvider decryptionProvider
+ String inputBootstrapPath
+
+ FileType fileType
+
+ String inputFilePath
+
+ boolean outputToFile = false
+ String outputFilePath
+
+ DecryptConfiguration() {
+ }
+
+ DecryptConfiguration(OptionAccessor options) {
+ this.rawOptions = options
+
+ validateOptions()
+ determineInputFileFromRemainingArgs()
+
+ determineKey()
+ if (!key) {
+ throw new RuntimeException("Failed to configure tool, could not determine key.")
+ }
+ decryptionProvider = new AESSensitivePropertyProvider(key)
+
+ if (rawOptions.t) {
+ fileType = FileType.valueOf(rawOptions.t)
+ }
+
+ if (rawOptions.o) {
--- End diff --
As per discussion, this option can be removed.
---
[GitHub] nifi pull request #2376: NIFI-4708 Add Registry support to encrypt-config
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2376#discussion_r159982754
--- Diff: nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/toolkit/encryptconfig/util/NiFiRegistryAuthorizersXmlEncryptor.groovy ---
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.toolkit.encryptconfig.util
+
+import groovy.xml.XmlUtil
+import org.apache.nifi.properties.SensitivePropertyProvider
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+import org.xml.sax.SAXException
+
+class NiFiRegistryAuthorizersXmlEncryptor extends XmlEncryptor {
+
+ private static final Logger logger = LoggerFactory.getLogger(NiFiRegistryAuthorizersXmlEncryptor.class)
+
+ private static final String LDAP_USER_GROUP_PROVIDER_CLASS = "org.apache.nifi.registry.security.ldap.tenants.LdapUserGroupProvider"
+ private static final String LDAP_USER_GROUP_PROVIDER_REGEX =
+ /(?s)<userGroupProvider>(?:(?!<userGroupProvider>).)*?<class>\s*org\.apache\.nifi\.registry\.security\.ldap\.tenants\.LdapUserGroupProvider.*?<\/userGroupProvider>/
+ /* Explanation of LDAP_USER_GROUP_PROVIDER_REGEX:
+ * (?s) -> single-line mode (i.e., `.` in regex matches newlines)
+ * <userGroupProvider> -> find occurrence of `<userGroupProvider>` literally (case-sensitive)
+ * (?: ... ) -> group but do not capture submatch
+ * (?! ... ) -> negative lookahead
+ * (?:(?!<userGroupProvider>).)*? -> find everything until a new `<userGroupProvider>` starts. This is for not selecting multiple userGroupProviders in one match
+ * <class> -> find occurrence of `<class>` literally (case-sensitive)
+ * \s* -> find any whitespace
+ * org\.apache\.nifi\.registry\.security\.ldap\.tenants\.LdapUserGroupProvider
+ * -> find occurrence of `org.apache.nifi.registry.security.ldap.tenants.LdapUserGroupProvider` literally (case-sensitive)
+ * .*?</userGroupProvider> -> find everything as needed up until and including occurrence of '</userGroupProvider>'
+ */
+
+ NiFiRegistryAuthorizersXmlEncryptor(SensitivePropertyProvider encryptionProvider, SensitivePropertyProvider decryptionProvider) {
+ super(encryptionProvider, decryptionProvider)
+ }
+
+ @Override
--- End diff --
Add Javadoc explaining operation of overriding method and why custom implementation is necessary.
---
[GitHub] nifi pull request #2376: NIFI-4708 Add Registry support to encrypt-config
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2376#discussion_r159982745
--- Diff: nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/toolkit/encryptconfig/NiFiRegistryMode.groovy ---
@@ -0,0 +1,383 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.toolkit.encryptconfig
+
+import org.apache.commons.cli.HelpFormatter
+import org.apache.commons.cli.Options
+import org.apache.http.annotation.Experimental
+import org.apache.nifi.properties.AESSensitivePropertyProvider
+import org.apache.nifi.properties.SensitivePropertyProvider
+import org.apache.nifi.toolkit.encryptconfig.util.BootstrapUtil
+import org.apache.nifi.toolkit.encryptconfig.util.NiFiRegistryAuthorizersXmlEncryptor
+import org.apache.nifi.toolkit.encryptconfig.util.NiFiRegistryIdentityProvidersXmlEncryptor
+import org.apache.nifi.toolkit.encryptconfig.util.NiFiRegistryPropertiesEncryptor
+import org.apache.nifi.toolkit.encryptconfig.util.ToolUtilities
+import org.apache.nifi.util.console.TextDevices
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+@Experimental
+class NiFiRegistryMode implements ToolMode {
+
+ private static final Logger logger = LoggerFactory.getLogger(NiFiRegistryMode.class)
+
+ CliBuilder cli
+
+ NiFiRegistryMode() {
+ cli = cliBuilder()
+ }
+
+// private void printUsage(String message = "") {
+// if (message) {
+// System.out.println(message)
+// System.out.println()
+// }
+// cli.usage()
+// }
+
+ @Override
+ void run(String[] args) {
+ logger.warn("The NiFi Registry capabilities of this tool is still considered experimental. The results should be manually verified.")
+ try {
+
+ def options = cli.parse(args)
+
+ if (!options || options.h) {
+ EncryptConfigMain.printUsageAndExit("", EncryptConfigMain.EXIT_STATUS_OTHER)
+ }
+
+ EncryptConfigLogger.configureLogger(options.v)
+
+ Configuration config = new Configuration(options)
+ run(config)
+
+ } catch (Exception e) {
+ logger.error("Encountered an error: ${e.getMessage()}")
+ logger.debug("", e) // stack trace only when verbose enabled
+ EncryptConfigMain.printUsageAndExit(e.getMessage(), EncryptConfigMain.EXIT_STATUS_FAILURE)
+ }
+ }
+
+ void run(Configuration config) throws Exception {
+
+ if (config.usingPassword) {
+ logger.info("Using encryption key derived from password.")
+ } else if (config.usingRawKeyHex) {
+ logger.info("Using encryption key provided.")
+ } else if (config.usingBootstrapKey) {
+ logger.info("Using encryption key from input bootstrap.conf.")
+ }
+
+ logger.debug("(src) bootstrap.conf: ${config.inputBootstrapPath}")
+ logger.debug("(dest) bootstrap.conf: ${config.outputBootstrapPath}")
+ logger.debug("(src) nifi.properties: ${config.inputNiFiRegistryPropertiesPath}")
--- End diff --
- [ ] fix file label
---
[GitHub] nifi pull request #2376: NIFI-4708 Add Registry support to encrypt-config
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2376#discussion_r159976150
--- Diff: nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/toolkit/encryptconfig/DecryptMode.groovy ---
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.toolkit.encryptconfig
+
+import org.apache.commons.cli.HelpFormatter
+import org.apache.nifi.properties.AESSensitivePropertyProvider
+import org.apache.nifi.properties.SensitivePropertyProvider
+import org.apache.nifi.toolkit.encryptconfig.util.BootstrapUtil
+import org.apache.nifi.toolkit.encryptconfig.util.PropertiesEncryptor
+import org.apache.nifi.toolkit.encryptconfig.util.ToolUtilities
+import org.apache.nifi.toolkit.encryptconfig.util.XmlEncryptor
+import org.apache.nifi.util.console.TextDevices
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+class DecryptMode implements ToolMode {
+
+ private static final Logger logger = LoggerFactory.getLogger(DecryptMode.class)
+
+ enum FileType {
+ properties,
+ xml
+ }
+
+ CliBuilder cli
+
+ DecryptMode() {
+ cli = cliBuilder()
+ }
+
+ void printUsage(String message = "") {
+ if (message) {
+ System.out.println(message)
+ System.out.println()
+ }
+ cli.usage()
+ }
+
+ void printUsageAndExit(String message = "", int exitStatusCode) {
+ printUsage(message)
+ System.exit(exitStatusCode)
+ }
+
+ @Override
+ void run(String[] args) {
+ logger.warn("The decryption capability of this tool is still considered experimental. The results should be manually verified.")
+ try {
+
+ def options = cli.parse(args)
+
+ if (!options || options.h) {
+ printUsageAndExit("", EncryptConfigMain.EXIT_STATUS_OTHER)
+ }
+
+ EncryptConfigLogger.configureLogger(options.v)
+
+ DecryptConfiguration config = new DecryptConfiguration(options)
+
+ run(config)
+
+ } catch (Exception e) {
+ logger.error("Encountered an error: ${e.getMessage()}")
+ logger.debug("", e) // stack trace only when verbose enabled
+ printUsageAndExit(e.getMessage(), EncryptConfigMain.EXIT_STATUS_FAILURE)
+ }
+ }
+
+ void run(DecryptConfiguration config) throws Exception {
+
+ if (!config.fileType) {
+
+ // Try to load the input file to auto-detect the file type
+ boolean isPropertiesFile = PropertiesEncryptor.supportsFile(config.inputFilePath)
+
+ boolean isXmlFile = XmlEncryptor.supportsFile(config.inputFilePath)
+
+ if (ToolUtilities.isExactlyOneTrue(isPropertiesFile, isXmlFile)) {
+ if (isPropertiesFile) {
+ config.fileType = FileType.properties
+ logger.debug("Auto-detection of input file type determined the type to be: ${FileType.properties}")
+ }
+ if (isXmlFile) {
+ config.fileType = FileType.xml
+ logger.debug("Auto-detection of input file type determined the type to be: ${FileType.xml}")
+ }
+ }
+
+ // Could we successfully auto-detect?
+ if (!config.fileType) {
+ throw new RuntimeException("Auto-detection of input file type failed. Please re-run the tool specifying the file type with the -t/--fileType flag.")
+ }
+ }
+
+ String decryptedSerializedContent = null
+ switch (config.fileType) {
+
+ case FileType.properties:
+ PropertiesEncryptor propertiesEncryptor = new PropertiesEncryptor(null, config.decryptionProvider)
+ Properties properties = propertiesEncryptor.loadFile(config.inputFilePath)
+ properties = propertiesEncryptor.decrypt(properties)
+ decryptedSerializedContent = propertiesEncryptor.serializePropertiesAndPreserveFormatIfPossible(properties, config.inputFilePath)
+ break
+
+ case FileType.xml:
+ XmlEncryptor xmlEncryptor = new XmlEncryptor(null, config.decryptionProvider) {
+ @Override
+ List<String> serializeXmlContentAndPreserveFormat(String updatedXmlContent, File originalInputXmlFile) {
+ // For decrypting unknown, generic XML, this tool will not support preserving the format
+ return updatedXmlContent.split("\n")
+ }
+ }
+
+ String xmlContent = xmlEncryptor.loadXmlFile(config.inputFilePath)
+ xmlContent = xmlEncryptor.decrypt(xmlContent)
+ decryptedSerializedContent = xmlEncryptor.serializeXmlContentAndPreserveFormatIfPossible(xmlContent, config.inputFilePath)
+ break
+
+ default:
+ throw new RuntimeException("Unsupported file type '${config.fileType}'")
+ }
+
+ if (!decryptedSerializedContent) {
+ throw new RuntimeException("Failed to load and decrypt input file.")
+ }
+
+ if (config.outputToFile) {
+ try {
+ File outputFile = new File(config.outputFilePath)
+ if (ToolUtilities.isSafeToWrite(outputFile)) {
+ outputFile.text = decryptedSerializedContent
+ logger.info("Wrote decrypted file contents to '${config.outputFilePath}'")
+ }
+ } catch (IOException e) {
+ throw new RuntimeException("Encountered an exception writing the decrypted content to '${config.outputFilePath}': ${e.getMessage()}", e)
+ }
+ } else {
+ System.out.println(decryptedSerializedContent)
+ }
+
+ }
+
+ private CliBuilder cliBuilder() {
+
+ String usage = "${EncryptConfigMain.class.getCanonicalName()} decrypt [options] file"
+
+ int formatWidth = EncryptConfigMain.HELP_FORMAT_WIDTH
+ HelpFormatter formatter = new HelpFormatter()
+ formatter.setWidth(formatWidth)
+ formatter.setOptionComparator(null) // preserve order of options below in help text
+
+ CliBuilder cli = new CliBuilder(
+ usage: usage,
+ width: formatWidth,
+ formatter: formatter,
+ stopAtNonOption: false)
+
+ cli.h(longOpt: 'help', 'Show usage information (this message)')
+ cli.v(longOpt: 'verbose', 'Enables verbose mode (off by default)')
+
+ // Options for the password or key or bootstrap.conf
+ cli.p(longOpt: 'password',
+ args: 1,
+ argName: 'password',
+ optionalArg: true,
+ 'Use a password to derive the key to decrypt the input file. If an argument is not provided to this flag, interactive mode will be triggered to prompt the user to enter the password.')
+ cli.k(longOpt: 'key',
+ args: 1,
+ argName: 'keyhex',
+ optionalArg: true,
+ 'Use a raw hexadecimal key to decrypt the input file. If an argument is not provided to this flag, interactive mode will be triggered to prompt the user to enter the key.')
+ cli.b(longOpt: 'bootstrapConf',
+ args: 1,
+ argName: 'file',
+ 'Use a bootstrap.conf file containing the master key to decrypt the input file (as an alternative to -p or -k)')
+
+ cli.o(longOpt: 'output',
+ args: 1,
+ argName: 'file',
+ 'Specify an output file. If omitted, Standard Out is used. Output file can be set to the input file to decrypt the file in-place.')
+
+ return cli
+
+ }
+
+ static class DecryptConfiguration {
--- End diff --
Yes.
- [ ] Adopt the Configuration interface where applicable to convey intended class structure to future developers as the Configuration interface is filled in more.
---
[GitHub] nifi pull request #2376: NIFI-4708 Add Registry support to encrypt-config
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2376#discussion_r159973528
--- Diff: nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/toolkit/encryptconfig/util/BootstrapUtil.groovy ---
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.toolkit.encryptconfig.util
+
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+class BootstrapUtil {
+
+ static final String NIFI_BOOTSTRAP_KEY_PROPERTY = "nifi.bootstrap.sensitive.key";
+ static final String REGISTRY_BOOTSTRAP_KEY_PROPERTY = "nifi.registry.bootstrap.sensitive.key";
+
+ private static final Logger logger = LoggerFactory.getLogger(BootstrapUtil.class)
+
+ private static final String BOOTSTRAP_KEY_COMMENT = "# Master key in hexadecimal format for encrypted sensitive configuration values"
+
+ /**
+ * Tries to load keyHex from input bootstrap.conf
+ *
+ * @return keyHex, if present in input bootstrap file; otherwise, null
+ */
+ static String extractKeyFromBootstrapFile(String inputBootstrapPath, String bootstrapKeyPropertyName) throws IOException {
+
+ File inputBootstrapConfFile
+ if (!(inputBootstrapPath && (inputBootstrapConfFile = new File(inputBootstrapPath)).exists() && inputBootstrapConfFile.canRead())) {
--- End diff --
Good catch, will change this to use the utility method `ToolUtilities.canRead(File)` method used elsewhere
---
[GitHub] nifi pull request #2376: NIFI-4708 Add Registry support to encrypt-config
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2376#discussion_r159966668
--- Diff: nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/toolkit/encryptconfig/NiFiRegistryMode.groovy ---
@@ -0,0 +1,383 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.toolkit.encryptconfig
+
+import org.apache.commons.cli.HelpFormatter
+import org.apache.commons.cli.Options
+import org.apache.http.annotation.Experimental
+import org.apache.nifi.properties.AESSensitivePropertyProvider
+import org.apache.nifi.properties.SensitivePropertyProvider
+import org.apache.nifi.toolkit.encryptconfig.util.BootstrapUtil
+import org.apache.nifi.toolkit.encryptconfig.util.NiFiRegistryAuthorizersXmlEncryptor
+import org.apache.nifi.toolkit.encryptconfig.util.NiFiRegistryIdentityProvidersXmlEncryptor
+import org.apache.nifi.toolkit.encryptconfig.util.NiFiRegistryPropertiesEncryptor
+import org.apache.nifi.toolkit.encryptconfig.util.ToolUtilities
+import org.apache.nifi.util.console.TextDevices
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+@Experimental
+class NiFiRegistryMode implements ToolMode {
+
+ private static final Logger logger = LoggerFactory.getLogger(NiFiRegistryMode.class)
+
+ CliBuilder cli
+
+ NiFiRegistryMode() {
+ cli = cliBuilder()
+ }
+
+// private void printUsage(String message = "") {
+// if (message) {
+// System.out.println(message)
+// System.out.println()
+// }
+// cli.usage()
+// }
+
+ @Override
+ void run(String[] args) {
+ logger.warn("The NiFi Registry capabilities of this tool is still considered experimental. The results should be manually verified.")
+ try {
+
+ def options = cli.parse(args)
+
+ if (!options || options.h) {
+ EncryptConfigMain.printUsageAndExit("", EncryptConfigMain.EXIT_STATUS_OTHER)
+ }
+
+ EncryptConfigLogger.configureLogger(options.v)
+
+ Configuration config = new Configuration(options)
+ run(config)
+
+ } catch (Exception e) {
+ logger.error("Encountered an error: ${e.getMessage()}")
+ logger.debug("", e) // stack trace only when verbose enabled
+ EncryptConfigMain.printUsageAndExit(e.getMessage(), EncryptConfigMain.EXIT_STATUS_FAILURE)
+ }
+ }
+
+ void run(Configuration config) throws Exception {
+
+ if (config.usingPassword) {
+ logger.info("Using encryption key derived from password.")
+ } else if (config.usingRawKeyHex) {
+ logger.info("Using encryption key provided.")
+ } else if (config.usingBootstrapKey) {
+ logger.info("Using encryption key from input bootstrap.conf.")
+ }
+
+ logger.debug("(src) bootstrap.conf: ${config.inputBootstrapPath}")
+ logger.debug("(dest) bootstrap.conf: ${config.outputBootstrapPath}")
+ logger.debug("(src) nifi.properties: ${config.inputNiFiRegistryPropertiesPath}")
+ logger.debug("(dest) nifi.properties: ${config.outputNiFiRegistryPropertiesPath}")
+ logger.debug("(src) identity-providers.xml: ${config.inputIdentityProvidersPath}")
+ logger.debug("(dest) identity-providers.xml: ${config.outputIdentityProvidersPath}")
+ logger.debug("(src) authorizers.xml: ${config.inputAuthorizersPath}")
+ logger.debug("(dest) authorizers.xml: ${config.outputAuthorizersPath}")
+
+ Properties niFiRegistryProperties = null
+ if (config.handlingNiFiRegistryProperties) {
+ try {
+ niFiRegistryProperties = config.propertiesEncryptor.loadFile(config.inputNiFiRegistryPropertiesPath)
+ // if properties are not protected, then the call to decrypt is a no-op
+ niFiRegistryProperties = config.propertiesEncryptor.decrypt(niFiRegistryProperties)
+ niFiRegistryProperties = config.propertiesEncryptor.encrypt(niFiRegistryProperties)
+ } catch (Exception e) {
+ throw new RuntimeException("Encountered error trying to load and encrypt NiFi Registry Properties in ${config.inputNiFiRegistryPropertiesPath}: ${e.getMessage()}", e)
+ }
+ }
+
+ String identityProvidersXml = null
+ if (config.handlingIdentityProviders) {
+ try {
+ identityProvidersXml = config.identityProvidersXmlEncryptor.loadXmlFile(config.inputIdentityProvidersPath)
+ // if xml is not protected, then the call to decrypt is a no-op
+ identityProvidersXml = config.identityProvidersXmlEncryptor.decrypt(identityProvidersXml)
+ identityProvidersXml = config.identityProvidersXmlEncryptor.encrypt(identityProvidersXml)
+ } catch (Exception e) {
+ throw new RuntimeException("Encountered error trying to load and encrypt Identity Providers XML in ${config.inputIdentityProvidersPath}: ${e.getMessage()}", e)
+ }
+ }
+
+ String authorizersXml = null
+ if (config.handlingAuthorizers) {
+ try {
+ authorizersXml = config.authorizersXmlEncryptor.loadXmlFile(config.inputAuthorizersPath)
+ // if xml is not protected, then the call to decrypt is a no-op
+ authorizersXml = config.authorizersXmlEncryptor.decrypt(authorizersXml)
+ authorizersXml = config.authorizersXmlEncryptor.encrypt(authorizersXml)
+ } catch (Exception e) {
+ throw new RuntimeException("Encountered error trying to load and encrypt Authorizers XML in ${config.inputAuthorizersPath}: ${e.getMessage()}", e)
+ }
+ }
+
+ try {
+ // Do this as part of a transaction?
+ synchronized (this) {
+
+ if (config.writingKeyToBootstrap) {
+ BootstrapUtil.writeKeyToBootstrapFile(config.encryptionKey, BootstrapUtil.REGISTRY_BOOTSTRAP_KEY_PROPERTY, config.outputBootstrapPath, config.inputBootstrapPath)
+ logger.info("Updated bootstrap config file with master key: ${config.outputBootstrapPath}")
+ }
+
+ if (config.handlingNiFiRegistryProperties) {
+ config.propertiesEncryptor.write(niFiRegistryProperties, config.outputNiFiRegistryPropertiesPath, config.inputNiFiRegistryPropertiesPath)
+ logger.info("Updated NiFi Registry Properties file with protected values: ${config.outputNiFiRegistryPropertiesPath}")
+ }
+ if (config.handlingIdentityProviders) {
+ config.identityProvidersXmlEncryptor.writeXmlFile(identityProvidersXml, config.outputIdentityProvidersPath, config.inputIdentityProvidersPath)
+ logger.info("Updated Identity Providers XML file with protected values: ${config.outputIdentityProvidersPath}")
+ }
+ if (config.handlingAuthorizers) {
+ config.authorizersXmlEncryptor.writeXmlFile(authorizersXml, config.outputAuthorizersPath, config.inputAuthorizersPath)
+ logger.info("Updated Authorizers XML file with protected values: ${config.outputAuthorizersPath}")
+ }
+ }
+ } catch (Exception e) {
+ throw new RuntimeException("Encountered error while writing the output files: ${e.getMessage()}", e)
+ }
+ }
+
+ static Options getCliOptions() {
+ return cliBuilder().options
+ }
+
+ static CliBuilder cliBuilder() {
+
+ String usage = "${NiFiRegistryMode.class.getCanonicalName()} [options]"
+
+ int formatWidth = EncryptConfigMain.HELP_FORMAT_WIDTH
+ HelpFormatter formatter = new HelpFormatter()
+ formatter.setWidth(formatWidth)
+ formatter.setOptionComparator(null) // preserve order of options below in help text
+
+ CliBuilder cli = new CliBuilder(
+ usage: usage,
+ width: formatWidth,
+ formatter: formatter)
+
+ cli.h(longOpt: 'help', 'Show usage information (this message)')
+ cli.v(longOpt: 'verbose', 'Sets verbose mode (default false)')
+
+ // Options for the new password or key
+ cli.p(longOpt: 'password',
+ args: 1,
+ argName: 'password',
+ optionalArg: true,
+ 'Protect the files using a password-derived key. If an argument is not provided to this flag, interactive mode will be triggered to prompt the user to enter the password.')
+ cli.k(longOpt: 'key',
+ args: 1,
+ argName: 'keyhex',
+ optionalArg: true,
+ 'Protect the files using a raw hexadecimal key. If an argument is not provided to this flag, interactive mode will be triggered to prompt the user to enter the key.')
+
+ // Options for the old password or key, if running the tool to migrate keys
+ cli._(longOpt: 'oldPassword',
+ args: 1,
+ argName: 'password',
+ 'If the input files are already protected using a password-derived key, this specifies the old password so that the files can be unprotected before re-protecting.')
+ cli._(longOpt: 'oldKey',
+ args: 1,
+ argName: 'keyhex',
+ 'If the input files are already protected using a key, this specifies the raw hexadecimal key so that the files can be unprotected before re-protecting.')
+
+ // Options for output bootstrap.conf file
+ cli.b(longOpt: 'bootstrapConf',
+ args: 1,
+ argName: 'file',
+ 'The bootstrap.conf file containing no master key or an existing master key. If a new password or key is specified (using -p or -k) and no output bootstrap.conf file is specified, then this file will be overwritten to persist the new master key.')
+ cli.B(longOpt: 'outputBootstrapConf',
+ args: 1,
+ argName: 'file',
+ 'The destination bootstrap.conf file to persist master key. If specified, the input bootstrap.conf will not be modified.')
+
+ // Options for input/output nifi-registry.properties files
+ cli.r(longOpt: 'nifiRegistryProperties',
+ args: 1,
+ argName: 'file',
+ 'The nifi-registry.properties file containing unprotected config values, overwritten if no output file specified.')
+ cli.R(longOpt: 'outputNifiRegistryProperties',
+ args: 1,
+ argName: 'file',
+ 'The destination nifi-registry.properties file containing protected config values.')
+
+ // Options for input/output authorizers.xml files
+ cli.a(longOpt: 'authorizersXml',
+ args: 1,
+ argName: 'file',
+ 'The authorizers.xml file containing unprotected config values, overwritten if no output file specified.')
+ cli.A(longOpt: 'outputAuthorizersXml',
+ args: 1,
+ argName: 'file',
+ 'The destination authorizers.xml file containing protected config values.')
+
+ // Options for input/output identity-providers.xml files
+ cli.i(longOpt: 'identityProvidersXml',
+ args: 1,
+ argName: 'file',
+ 'The identity-providers.xml file containing unprotected config values, overwritten if no output file specified.')
+ cli.I(longOpt: 'outputIdentityProvidersXml',
+ args: 1,
+ argName: 'file',
+ 'The destination identity-providers.xml file containing protected config values.')
+
+ return cli
+
+ }
+
+ class Configuration {
+
+ OptionAccessor rawOptions
+
+ boolean usingRawKeyHex
+ boolean usingPassword
+ boolean usingBootstrapKey
+
+ String encryptionKey
+ String decryptionKey
+
+ SensitivePropertyProvider encryptionProvider
+ SensitivePropertyProvider decryptionProvider
+
+ boolean writingKeyToBootstrap = false
+ String inputBootstrapPath
+ String outputBootstrapPath
+
+ boolean handlingNiFiRegistryProperties = false
+ String inputNiFiRegistryPropertiesPath
+ String outputNiFiRegistryPropertiesPath
+ NiFiRegistryPropertiesEncryptor propertiesEncryptor
+
+ boolean handlingIdentityProviders = false
+ String inputIdentityProvidersPath
+ String outputIdentityProvidersPath
+ NiFiRegistryIdentityProvidersXmlEncryptor identityProvidersXmlEncryptor
+
+ boolean handlingAuthorizers = false
+ String inputAuthorizersPath
+ String outputAuthorizersPath
+ NiFiRegistryAuthorizersXmlEncryptor authorizersXmlEncryptor
+
+ Configuration() {
+ }
+
+ Configuration(OptionAccessor options) {
+ this.rawOptions = options
+
+ validateOptions()
+
+ // Set input bootstrap.conf path
+ inputBootstrapPath = rawOptions.b
+
+ // Determine key for encryption (required)
+ determineEncryptionKey()
+ if (!encryptionKey) {
+ throw new RuntimeException("Failed to configure tool, could not determine encryption key. Must provide -p, -k, or -b. If using -b, bootstrap.conf argument must already contain master key.")
+ }
+ encryptionProvider = new AESSensitivePropertyProvider(encryptionKey)
+
+ // Determine key for decryption (if migrating)
+ determineDecryptionKey()
+ if (!decryptionKey) {
+ logger.debug("No decryption key specified via options, so if any input files require decryption prior to re-encryption (i.e., migration), this tool will fail.")
+ }
+ decryptionProvider = decryptionKey ? new AESSensitivePropertyProvider(decryptionKey) : null
+
+ writingKeyToBootstrap = (usingPassword || usingRawKeyHex || rawOptions.B)
+ if (writingKeyToBootstrap) {
+ outputBootstrapPath = rawOptions.B ? rawOptions.B : inputBootstrapPath
--- End diff --
In the future, the Elvis operator condenses this in Groovy (not necessary to make the change for this PR).
`outputBootstrapPath = rawOptions.B ?: inputBootstrapPath`
---
[GitHub] nifi pull request #2376: NIFI-4708 Add Registry support to encrypt-config
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2376#discussion_r159932352
--- Diff: nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/toolkit/encryptconfig/EncryptConfigMain.groovy ---
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.toolkit.encryptconfig
+
+import org.apache.commons.cli.HelpFormatter
+import org.apache.commons.cli.Options
+import org.apache.nifi.properties.ConfigEncryptionTool
+import org.bouncycastle.jce.provider.BouncyCastleProvider
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+import java.security.Security
+
+class EncryptConfigMain {
+
+ private static final Logger logger = LoggerFactory.getLogger(EncryptConfigMain.class)
+
+ static final int EXIT_STATUS_SUCCESS = 0
+ static final int EXIT_STATUS_FAILURE = -1
+ static final int EXIT_STATUS_OTHER = 1
+
+ static final String NIFI_REGISTRY_OPT = "nifiRegistry"
+ static final String NIFI_REGISTRY_FLAG = "--${NIFI_REGISTRY_OPT}".toString()
+ static final String DECRYPT_OPT = "decrypt"
+ static final String DECRYPT_FLAG = "--${DECRYPT_OPT}".toString()
+
+ static final int HELP_FORMAT_WIDTH = 160
+
+ // Access should only be through static methods
+ private EncryptConfigMain() {
+ }
+
+ static printUsage(String message = "") {
+
+ if (message) {
+ System.out.println(message)
+ System.out.println()
+ }
+
+ String header = "\nThis tool enables easy encryption and decryption of configuration files for NiFi and its sub-projects. " +
+ "Unprotected files can be input to this tool to be protected by a key in a manner that is understood by NiFi. " +
+ "Protected files, along with a key, can be input to this tool to be unprotected, for troubleshooting or automation purposes.\n\n"
+
+ def options = new Options()
+ options.addOption("h", "help", false, "Show usage information (this message)")
+ options.addOption(null, NIFI_REGISTRY_OPT, false, "Specifies to target NiFi Registry. When this flag is not included, NiFi is the target.")
+
+ HelpFormatter helpFormatter = new HelpFormatter()
+ helpFormatter.setWidth(160)
+ helpFormatter.setOptionComparator(null)
+ helpFormatter.printHelp("${EncryptConfigMain.class.getCanonicalName()} [-h] [options]", header, options, "\n")
+ System.out.println()
+
+ helpFormatter.setSyntaxPrefix("") // disable "usage: " prefix for the following outputs
+
+ Options nifiModeOptions = ConfigEncryptionTool.getCliOptions()
+ helpFormatter.printHelp(
+ "When targeting NiFi:",
+ nifiModeOptions,
+ false)
+ System.out.println()
+
+ Options nifiRegistryModeOptions = NiFiRegistryMode.getCliOptions()
+ nifiRegistryModeOptions.addOption(null, DECRYPT_OPT, false, "Can be used with -r to decrypt a previously encrypted NiFi Registry Properties file. Decrypted content is printed to STDOUT.")
+ helpFormatter.printHelp(
+ "When targeting NiFi Registry using the ${NIFI_REGISTRY_FLAG} flag:",
+ nifiRegistryModeOptions,
+ false)
+ System.out.println()
+
+// String footer = """
+// |
+// |Encrypt a NiFi Registry properties using a password:
+// | encrypt-config -p <password> -b /path/to/nifi/conf/bootstrap.conf -r /path/to/nifi/conf/nifi.properties
+// |
+// |""".stripMargin()
+ //helpFormatter.printHelp("Examples:", "", new Options(), footer)
+ }
+
+ static void printUsageAndExit(String message = "", int exitStatusCode) {
+ printUsage(message)
+ System.exit(exitStatusCode)
+ }
+
+ static void main(String[] args) {
+ Security.addProvider(new BouncyCastleProvider())
+
+ if (args.length < 1) {
+ printUsageAndExit(EXIT_STATUS_FAILURE)
+ }
+
+ String firstArg = args[0]
+
+ if (["-h", "--help"].contains(firstArg)) {
+ printUsageAndExit(EXIT_STATUS_OTHER)
+ }
+
+ try {
+ List<String> argsList = args
+ ToolMode toolMode = determineModeFromArgs(argsList)
+ if (toolMode) {
+ toolMode.run((String[])argsList.toArray())
+ System.exit(EXIT_STATUS_SUCCESS)
+ } else {
+ printUsageAndExit(EXIT_STATUS_FAILURE)
+ }
+ } catch (Throwable t) {
+ logger.debug("", t)
--- End diff --
Change this to `logger.error()` as well.
---
[GitHub] nifi pull request #2376: NIFI-4708 Add Registry support to encrypt-config
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2376#discussion_r159927720
--- Diff: nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/toolkit/encryptconfig/DecryptMode.groovy ---
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.toolkit.encryptconfig
+
+import org.apache.commons.cli.HelpFormatter
+import org.apache.nifi.properties.AESSensitivePropertyProvider
+import org.apache.nifi.properties.SensitivePropertyProvider
+import org.apache.nifi.toolkit.encryptconfig.util.BootstrapUtil
+import org.apache.nifi.toolkit.encryptconfig.util.PropertiesEncryptor
+import org.apache.nifi.toolkit.encryptconfig.util.ToolUtilities
+import org.apache.nifi.toolkit.encryptconfig.util.XmlEncryptor
+import org.apache.nifi.util.console.TextDevices
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+class DecryptMode implements ToolMode {
+
+ private static final Logger logger = LoggerFactory.getLogger(DecryptMode.class)
+
+ enum FileType {
+ properties,
+ xml
+ }
+
+ CliBuilder cli
+
+ DecryptMode() {
+ cli = cliBuilder()
+ }
+
+ void printUsage(String message = "") {
+ if (message) {
+ System.out.println(message)
+ System.out.println()
+ }
+ cli.usage()
+ }
+
+ void printUsageAndExit(String message = "", int exitStatusCode) {
+ printUsage(message)
+ System.exit(exitStatusCode)
+ }
+
+ @Override
+ void run(String[] args) {
+ logger.warn("The decryption capability of this tool is still considered experimental. The results should be manually verified.")
+ try {
+
+ def options = cli.parse(args)
+
+ if (!options || options.h) {
+ printUsageAndExit("", EncryptConfigMain.EXIT_STATUS_OTHER)
+ }
+
+ EncryptConfigLogger.configureLogger(options.v)
+
+ DecryptConfiguration config = new DecryptConfiguration(options)
+
+ run(config)
+
+ } catch (Exception e) {
+ logger.error("Encountered an error: ${e.getMessage()}")
+ logger.debug("", e) // stack trace only when verbose enabled
+ printUsageAndExit(e.getMessage(), EncryptConfigMain.EXIT_STATUS_FAILURE)
+ }
+ }
+
+ void run(DecryptConfiguration config) throws Exception {
+
+ if (!config.fileType) {
+
+ // Try to load the input file to auto-detect the file type
+ boolean isPropertiesFile = PropertiesEncryptor.supportsFile(config.inputFilePath)
+
+ boolean isXmlFile = XmlEncryptor.supportsFile(config.inputFilePath)
+
+ if (ToolUtilities.isExactlyOneTrue(isPropertiesFile, isXmlFile)) {
+ if (isPropertiesFile) {
+ config.fileType = FileType.properties
+ logger.debug("Auto-detection of input file type determined the type to be: ${FileType.properties}")
+ }
+ if (isXmlFile) {
+ config.fileType = FileType.xml
+ logger.debug("Auto-detection of input file type determined the type to be: ${FileType.xml}")
+ }
+ }
+
+ // Could we successfully auto-detect?
+ if (!config.fileType) {
+ throw new RuntimeException("Auto-detection of input file type failed. Please re-run the tool specifying the file type with the -t/--fileType flag.")
+ }
+ }
+
+ String decryptedSerializedContent = null
+ switch (config.fileType) {
+
+ case FileType.properties:
+ PropertiesEncryptor propertiesEncryptor = new PropertiesEncryptor(null, config.decryptionProvider)
+ Properties properties = propertiesEncryptor.loadFile(config.inputFilePath)
+ properties = propertiesEncryptor.decrypt(properties)
+ decryptedSerializedContent = propertiesEncryptor.serializePropertiesAndPreserveFormatIfPossible(properties, config.inputFilePath)
+ break
+
+ case FileType.xml:
+ XmlEncryptor xmlEncryptor = new XmlEncryptor(null, config.decryptionProvider) {
+ @Override
+ List<String> serializeXmlContentAndPreserveFormat(String updatedXmlContent, File originalInputXmlFile) {
+ // For decrypting unknown, generic XML, this tool will not support preserving the format
+ return updatedXmlContent.split("\n")
+ }
+ }
+
+ String xmlContent = xmlEncryptor.loadXmlFile(config.inputFilePath)
+ xmlContent = xmlEncryptor.decrypt(xmlContent)
+ decryptedSerializedContent = xmlEncryptor.serializeXmlContentAndPreserveFormatIfPossible(xmlContent, config.inputFilePath)
+ break
+
+ default:
+ throw new RuntimeException("Unsupported file type '${config.fileType}'")
+ }
+
+ if (!decryptedSerializedContent) {
+ throw new RuntimeException("Failed to load and decrypt input file.")
+ }
+
+ if (config.outputToFile) {
+ try {
+ File outputFile = new File(config.outputFilePath)
+ if (ToolUtilities.isSafeToWrite(outputFile)) {
+ outputFile.text = decryptedSerializedContent
+ logger.info("Wrote decrypted file contents to '${config.outputFilePath}'")
+ }
+ } catch (IOException e) {
+ throw new RuntimeException("Encountered an exception writing the decrypted content to '${config.outputFilePath}': ${e.getMessage()}", e)
+ }
+ } else {
+ System.out.println(decryptedSerializedContent)
+ }
+
+ }
+
+ private CliBuilder cliBuilder() {
+
+ String usage = "${EncryptConfigMain.class.getCanonicalName()} decrypt [options] file"
+
+ int formatWidth = EncryptConfigMain.HELP_FORMAT_WIDTH
+ HelpFormatter formatter = new HelpFormatter()
+ formatter.setWidth(formatWidth)
+ formatter.setOptionComparator(null) // preserve order of options below in help text
+
+ CliBuilder cli = new CliBuilder(
+ usage: usage,
+ width: formatWidth,
+ formatter: formatter,
+ stopAtNonOption: false)
+
+ cli.h(longOpt: 'help', 'Show usage information (this message)')
+ cli.v(longOpt: 'verbose', 'Enables verbose mode (off by default)')
+
+ // Options for the password or key or bootstrap.conf
+ cli.p(longOpt: 'password',
+ args: 1,
+ argName: 'password',
+ optionalArg: true,
+ 'Use a password to derive the key to decrypt the input file. If an argument is not provided to this flag, interactive mode will be triggered to prompt the user to enter the password.')
+ cli.k(longOpt: 'key',
+ args: 1,
+ argName: 'keyhex',
+ optionalArg: true,
+ 'Use a raw hexadecimal key to decrypt the input file. If an argument is not provided to this flag, interactive mode will be triggered to prompt the user to enter the key.')
+ cli.b(longOpt: 'bootstrapConf',
+ args: 1,
+ argName: 'file',
+ 'Use a bootstrap.conf file containing the master key to decrypt the input file (as an alternative to -p or -k)')
+
+ cli.o(longOpt: 'output',
+ args: 1,
+ argName: 'file',
+ 'Specify an output file. If omitted, Standard Out is used. Output file can be set to the input file to decrypt the file in-place.')
+
+ return cli
+
+ }
+
+ static class DecryptConfiguration {
+
+ OptionAccessor rawOptions
+
+ Configuration.KeySource keySource
+ String key
+ SensitivePropertyProvider decryptionProvider
+ String inputBootstrapPath
+
+ FileType fileType
+
+ String inputFilePath
+
+ boolean outputToFile = false
+ String outputFilePath
+
+ DecryptConfiguration() {
+ }
+
+ DecryptConfiguration(OptionAccessor options) {
+ this.rawOptions = options
+
+ validateOptions()
+ determineInputFileFromRemainingArgs()
+
+ determineKey()
+ if (!key) {
+ throw new RuntimeException("Failed to configure tool, could not determine key.")
+ }
+ decryptionProvider = new AESSensitivePropertyProvider(key)
+
+ if (rawOptions.t) {
+ fileType = FileType.valueOf(rawOptions.t)
+ }
+
+ if (rawOptions.o) {
+ outputToFile = true
+ outputFilePath = rawOptions.o
+ }
+ }
+
+ private void validateOptions() {
+
+ String validationFailedMessage = null
+
+ if (!rawOptions.b && !rawOptions.p && !rawOptions.k) {
+ validationFailedMessage = "-p, -k, or -b is required in order to determine the master key to use for decryption."
+ }
+
+ if (validationFailedMessage) {
+ throw new RuntimeException("Invalid options: " + validationFailedMessage)
+ }
+
+ }
+
+ private void determineInputFileFromRemainingArgs() {
+ String[] remainingArgs = this.rawOptions.getInner().getArgs()
+ if (remainingArgs.length == 0) {
+ throw new RuntimeException("Missing argument: Input file must provided.")
+ } else if (remainingArgs.length > 1) {
+ throw new RuntimeException("Too many arguments: Please specify exactly one input file in addition to the options.")
--- End diff --
Is order important here? If `$ encrypt-config.sh --decrypt -b bootstrap.conf file_to_decrypt.properties` works but `$ encrypt-config.sh file_to_decrypt.properties --decrypt -b bootstrap.conf` doesn't, I think this message should indicate that.
---
[GitHub] nifi pull request #2376: NIFI-4708 Add Registry support to encrypt-config
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2376#discussion_r159974781
--- Diff: nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/toolkit/encryptconfig/NiFiRegistryMode.groovy ---
@@ -0,0 +1,383 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.toolkit.encryptconfig
+
+import org.apache.commons.cli.HelpFormatter
+import org.apache.commons.cli.Options
+import org.apache.http.annotation.Experimental
+import org.apache.nifi.properties.AESSensitivePropertyProvider
+import org.apache.nifi.properties.SensitivePropertyProvider
+import org.apache.nifi.toolkit.encryptconfig.util.BootstrapUtil
+import org.apache.nifi.toolkit.encryptconfig.util.NiFiRegistryAuthorizersXmlEncryptor
+import org.apache.nifi.toolkit.encryptconfig.util.NiFiRegistryIdentityProvidersXmlEncryptor
+import org.apache.nifi.toolkit.encryptconfig.util.NiFiRegistryPropertiesEncryptor
+import org.apache.nifi.toolkit.encryptconfig.util.ToolUtilities
+import org.apache.nifi.util.console.TextDevices
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+@Experimental
+class NiFiRegistryMode implements ToolMode {
+
+ private static final Logger logger = LoggerFactory.getLogger(NiFiRegistryMode.class)
+
+ CliBuilder cli
+
+ NiFiRegistryMode() {
+ cli = cliBuilder()
+ }
+
+// private void printUsage(String message = "") {
+// if (message) {
+// System.out.println(message)
+// System.out.println()
+// }
+// cli.usage()
+// }
+
+ @Override
+ void run(String[] args) {
+ logger.warn("The NiFi Registry capabilities of this tool is still considered experimental. The results should be manually verified.")
+ try {
+
+ def options = cli.parse(args)
+
+ if (!options || options.h) {
+ EncryptConfigMain.printUsageAndExit("", EncryptConfigMain.EXIT_STATUS_OTHER)
+ }
+
+ EncryptConfigLogger.configureLogger(options.v)
+
+ Configuration config = new Configuration(options)
+ run(config)
+
+ } catch (Exception e) {
+ logger.error("Encountered an error: ${e.getMessage()}")
+ logger.debug("", e) // stack trace only when verbose enabled
+ EncryptConfigMain.printUsageAndExit(e.getMessage(), EncryptConfigMain.EXIT_STATUS_FAILURE)
+ }
+ }
+
+ void run(Configuration config) throws Exception {
+
+ if (config.usingPassword) {
+ logger.info("Using encryption key derived from password.")
+ } else if (config.usingRawKeyHex) {
+ logger.info("Using encryption key provided.")
+ } else if (config.usingBootstrapKey) {
+ logger.info("Using encryption key from input bootstrap.conf.")
+ }
+
+ logger.debug("(src) bootstrap.conf: ${config.inputBootstrapPath}")
+ logger.debug("(dest) bootstrap.conf: ${config.outputBootstrapPath}")
+ logger.debug("(src) nifi.properties: ${config.inputNiFiRegistryPropertiesPath}")
+ logger.debug("(dest) nifi.properties: ${config.outputNiFiRegistryPropertiesPath}")
+ logger.debug("(src) identity-providers.xml: ${config.inputIdentityProvidersPath}")
+ logger.debug("(dest) identity-providers.xml: ${config.outputIdentityProvidersPath}")
+ logger.debug("(src) authorizers.xml: ${config.inputAuthorizersPath}")
+ logger.debug("(dest) authorizers.xml: ${config.outputAuthorizersPath}")
+
+ Properties niFiRegistryProperties = null
+ if (config.handlingNiFiRegistryProperties) {
+ try {
+ niFiRegistryProperties = config.propertiesEncryptor.loadFile(config.inputNiFiRegistryPropertiesPath)
+ // if properties are not protected, then the call to decrypt is a no-op
+ niFiRegistryProperties = config.propertiesEncryptor.decrypt(niFiRegistryProperties)
+ niFiRegistryProperties = config.propertiesEncryptor.encrypt(niFiRegistryProperties)
+ } catch (Exception e) {
+ throw new RuntimeException("Encountered error trying to load and encrypt NiFi Registry Properties in ${config.inputNiFiRegistryPropertiesPath}: ${e.getMessage()}", e)
+ }
+ }
+
+ String identityProvidersXml = null
+ if (config.handlingIdentityProviders) {
+ try {
+ identityProvidersXml = config.identityProvidersXmlEncryptor.loadXmlFile(config.inputIdentityProvidersPath)
+ // if xml is not protected, then the call to decrypt is a no-op
+ identityProvidersXml = config.identityProvidersXmlEncryptor.decrypt(identityProvidersXml)
+ identityProvidersXml = config.identityProvidersXmlEncryptor.encrypt(identityProvidersXml)
+ } catch (Exception e) {
+ throw new RuntimeException("Encountered error trying to load and encrypt Identity Providers XML in ${config.inputIdentityProvidersPath}: ${e.getMessage()}", e)
+ }
+ }
+
+ String authorizersXml = null
+ if (config.handlingAuthorizers) {
+ try {
+ authorizersXml = config.authorizersXmlEncryptor.loadXmlFile(config.inputAuthorizersPath)
+ // if xml is not protected, then the call to decrypt is a no-op
+ authorizersXml = config.authorizersXmlEncryptor.decrypt(authorizersXml)
+ authorizersXml = config.authorizersXmlEncryptor.encrypt(authorizersXml)
+ } catch (Exception e) {
+ throw new RuntimeException("Encountered error trying to load and encrypt Authorizers XML in ${config.inputAuthorizersPath}: ${e.getMessage()}", e)
+ }
+ }
+
+ try {
+ // Do this as part of a transaction?
+ synchronized (this) {
+
+ if (config.writingKeyToBootstrap) {
+ BootstrapUtil.writeKeyToBootstrapFile(config.encryptionKey, BootstrapUtil.REGISTRY_BOOTSTRAP_KEY_PROPERTY, config.outputBootstrapPath, config.inputBootstrapPath)
+ logger.info("Updated bootstrap config file with master key: ${config.outputBootstrapPath}")
+ }
+
+ if (config.handlingNiFiRegistryProperties) {
+ config.propertiesEncryptor.write(niFiRegistryProperties, config.outputNiFiRegistryPropertiesPath, config.inputNiFiRegistryPropertiesPath)
+ logger.info("Updated NiFi Registry Properties file with protected values: ${config.outputNiFiRegistryPropertiesPath}")
+ }
+ if (config.handlingIdentityProviders) {
+ config.identityProvidersXmlEncryptor.writeXmlFile(identityProvidersXml, config.outputIdentityProvidersPath, config.inputIdentityProvidersPath)
+ logger.info("Updated Identity Providers XML file with protected values: ${config.outputIdentityProvidersPath}")
+ }
+ if (config.handlingAuthorizers) {
+ config.authorizersXmlEncryptor.writeXmlFile(authorizersXml, config.outputAuthorizersPath, config.inputAuthorizersPath)
+ logger.info("Updated Authorizers XML file with protected values: ${config.outputAuthorizersPath}")
+ }
+ }
+ } catch (Exception e) {
+ throw new RuntimeException("Encountered error while writing the output files: ${e.getMessage()}", e)
+ }
+ }
+
+ static Options getCliOptions() {
+ return cliBuilder().options
+ }
+
+ static CliBuilder cliBuilder() {
+
+ String usage = "${NiFiRegistryMode.class.getCanonicalName()} [options]"
+
+ int formatWidth = EncryptConfigMain.HELP_FORMAT_WIDTH
+ HelpFormatter formatter = new HelpFormatter()
+ formatter.setWidth(formatWidth)
+ formatter.setOptionComparator(null) // preserve order of options below in help text
+
+ CliBuilder cli = new CliBuilder(
+ usage: usage,
+ width: formatWidth,
+ formatter: formatter)
+
+ cli.h(longOpt: 'help', 'Show usage information (this message)')
+ cli.v(longOpt: 'verbose', 'Sets verbose mode (default false)')
+
+ // Options for the new password or key
+ cli.p(longOpt: 'password',
+ args: 1,
+ argName: 'password',
+ optionalArg: true,
+ 'Protect the files using a password-derived key. If an argument is not provided to this flag, interactive mode will be triggered to prompt the user to enter the password.')
+ cli.k(longOpt: 'key',
+ args: 1,
+ argName: 'keyhex',
+ optionalArg: true,
+ 'Protect the files using a raw hexadecimal key. If an argument is not provided to this flag, interactive mode will be triggered to prompt the user to enter the key.')
+
+ // Options for the old password or key, if running the tool to migrate keys
+ cli._(longOpt: 'oldPassword',
+ args: 1,
+ argName: 'password',
+ 'If the input files are already protected using a password-derived key, this specifies the old password so that the files can be unprotected before re-protecting.')
+ cli._(longOpt: 'oldKey',
+ args: 1,
+ argName: 'keyhex',
+ 'If the input files are already protected using a key, this specifies the raw hexadecimal key so that the files can be unprotected before re-protecting.')
+
+ // Options for output bootstrap.conf file
+ cli.b(longOpt: 'bootstrapConf',
+ args: 1,
+ argName: 'file',
+ 'The bootstrap.conf file containing no master key or an existing master key. If a new password or key is specified (using -p or -k) and no output bootstrap.conf file is specified, then this file will be overwritten to persist the new master key.')
+ cli.B(longOpt: 'outputBootstrapConf',
+ args: 1,
+ argName: 'file',
+ 'The destination bootstrap.conf file to persist master key. If specified, the input bootstrap.conf will not be modified.')
+
+ // Options for input/output nifi-registry.properties files
+ cli.r(longOpt: 'nifiRegistryProperties',
+ args: 1,
+ argName: 'file',
+ 'The nifi-registry.properties file containing unprotected config values, overwritten if no output file specified.')
+ cli.R(longOpt: 'outputNifiRegistryProperties',
+ args: 1,
+ argName: 'file',
+ 'The destination nifi-registry.properties file containing protected config values.')
+
+ // Options for input/output authorizers.xml files
+ cli.a(longOpt: 'authorizersXml',
+ args: 1,
+ argName: 'file',
+ 'The authorizers.xml file containing unprotected config values, overwritten if no output file specified.')
+ cli.A(longOpt: 'outputAuthorizersXml',
+ args: 1,
+ argName: 'file',
+ 'The destination authorizers.xml file containing protected config values.')
+
+ // Options for input/output identity-providers.xml files
+ cli.i(longOpt: 'identityProvidersXml',
+ args: 1,
+ argName: 'file',
+ 'The identity-providers.xml file containing unprotected config values, overwritten if no output file specified.')
+ cli.I(longOpt: 'outputIdentityProvidersXml',
+ args: 1,
+ argName: 'file',
+ 'The destination identity-providers.xml file containing protected config values.')
+
+ return cli
+
+ }
+
+ class Configuration {
+
+ OptionAccessor rawOptions
+
+ boolean usingRawKeyHex
+ boolean usingPassword
+ boolean usingBootstrapKey
+
+ String encryptionKey
+ String decryptionKey
+
+ SensitivePropertyProvider encryptionProvider
+ SensitivePropertyProvider decryptionProvider
+
+ boolean writingKeyToBootstrap = false
+ String inputBootstrapPath
+ String outputBootstrapPath
+
+ boolean handlingNiFiRegistryProperties = false
+ String inputNiFiRegistryPropertiesPath
+ String outputNiFiRegistryPropertiesPath
+ NiFiRegistryPropertiesEncryptor propertiesEncryptor
+
+ boolean handlingIdentityProviders = false
+ String inputIdentityProvidersPath
+ String outputIdentityProvidersPath
+ NiFiRegistryIdentityProvidersXmlEncryptor identityProvidersXmlEncryptor
+
+ boolean handlingAuthorizers = false
+ String inputAuthorizersPath
+ String outputAuthorizersPath
+ NiFiRegistryAuthorizersXmlEncryptor authorizersXmlEncryptor
+
+ Configuration() {
+ }
+
+ Configuration(OptionAccessor options) {
+ this.rawOptions = options
+
+ validateOptions()
+
+ // Set input bootstrap.conf path
+ inputBootstrapPath = rawOptions.b
+
+ // Determine key for encryption (required)
+ determineEncryptionKey()
+ if (!encryptionKey) {
+ throw new RuntimeException("Failed to configure tool, could not determine encryption key. Must provide -p, -k, or -b. If using -b, bootstrap.conf argument must already contain master key.")
+ }
+ encryptionProvider = new AESSensitivePropertyProvider(encryptionKey)
+
+ // Determine key for decryption (if migrating)
+ determineDecryptionKey()
+ if (!decryptionKey) {
+ logger.debug("No decryption key specified via options, so if any input files require decryption prior to re-encryption (i.e., migration), this tool will fail.")
+ }
+ decryptionProvider = decryptionKey ? new AESSensitivePropertyProvider(decryptionKey) : null
+
+ writingKeyToBootstrap = (usingPassword || usingRawKeyHex || rawOptions.B)
+ if (writingKeyToBootstrap) {
+ outputBootstrapPath = rawOptions.B ? rawOptions.B : inputBootstrapPath
--- End diff --
Oh cool, I didn't know about that Groovy idiom - thanks!
- [ ] (optional, time permitting) change to using Elvis operator where applicable
---
[GitHub] nifi pull request #2376: NIFI-4708 Add Registry support to encrypt-config
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2376#discussion_r159982870
--- Diff: nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/toolkit/encryptconfig/util/NiFiRegistryIdentityProvidersXmlEncryptor.groovy ---
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.toolkit.encryptconfig.util
+
+import groovy.xml.XmlUtil
+import org.apache.nifi.properties.SensitivePropertyProvider
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+import org.xml.sax.SAXException
+
+class NiFiRegistryIdentityProvidersXmlEncryptor extends XmlEncryptor {
+
+ private static final Logger logger = LoggerFactory.getLogger(NiFiRegistryIdentityProvidersXmlEncryptor.class)
+
+ private static final String LDAP_PROVIDER_CLASS = "org.apache.nifi.registry.security.ldap.LdapIdentityProvider"
+ private static final String LDAP_PROVIDER_REGEX = /(?s)<provider>(?:(?!<provider>).)*?<class>\s*org\.apache\.nifi\.registry\.security\.ldap\.LdapIdentityProvider.*?<\/provider>/
+ /* Explanation of LDAP_PROVIDER_REGEX:
+ * (?s) -> single-line mode (i.e., `.` in regex matches newlines)
+ * <provider> -> find occurrence of `<provider>` literally (case-sensitive)
+ * (?: ... ) -> group but do not capture submatch
+ * (?! ... ) -> negative lookahead
+ * (?:(?!<provider>).)*? -> find everything until a new `<provider>` starts. This is for not selecting multiple providers in one match
+ * <class> -> find occurrence of `<class>` literally (case-sensitive)
+ * \s* -> find any whitespace
+ * org\.apache\.nifi\.registry\.security\.ldap\.LdapIdentityProvider
+ * -> find occurrence of `org.apache.nifi.registry.security.ldap.LdapIdentityProvider` literally (case-sensitive)
+ * .*?</provider> -> find everything as needed up until and including occurrence of `</provider>`
+ */
+
+ NiFiRegistryIdentityProvidersXmlEncryptor(SensitivePropertyProvider encryptionProvider, SensitivePropertyProvider decryptionProvider) {
+ super(encryptionProvider, decryptionProvider)
+ }
+
+ @Override
+ String encrypt(String plainXmlContent) {
--- End diff --
Add Javadoc explaining operation of overriding method and why custom implementation is necessary.
---