You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by kevdoran <gi...@git.apache.org> on 2017/12/18 15:39:48 UTC
[GitHub] nifi pull request #2350: NIFI-4701 Support encrypted authorizers.xml
GitHub user kevdoran opened a pull request:
https://github.com/apache/nifi/pull/2350
NIFI-4701 Support encrypted authorizers.xml
Enable properties in authorizers.xml to be encrypted by the master key.
Adds authorizers.xml to the files understood by the encrypt-config tool in the NiFi Toolkit. If passed to the tool using the CLI, then the sensitive properties for LdapUserGroupProvider will be encrypted.
Also fixes a bug wherein encrypt-config replaces multiple XML nodes in login-indentity-providers.xml when LdapProvider is not the first provider listed in the file.
---
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [ ] Has your PR been rebased against the latest commit within the target branch (typically master)?
- [ ] Is your initial contribution a single, squashed commit?
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kevdoran/nifi NIFI-4701
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/2350.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2350
----
commit 35d69e06cd878ae22e0de142803d9d0d9112c7a0
Author: Kevin Doran <kd...@gmail.com>
Date: 2017-12-17T16:40:06Z
NIFI-4701 Support encrypted authorizers.xml
Enable properties in authorizers.xml to be encrypted by the master key.
commit a9c496e97a8e3f4fbcee3b7c0ab260e88ecc9b19
Author: Kevin Doran <kd...@gmail.com>
Date: 2017-12-18T04:46:39Z
NIFI-4701 Add authorizers.xml support to toolkit
Adds authorizers.xml to the files understood by the encrypt-config
tool in the NiFi Toolkit. If passed to the tool using the CLI, then
the sensitive properties for LdapUserGroupProvider will be encrypted.
Also fixes a bug wherein encrypt-config replaces multiple XML nodes in
login-indentity-providers.xml when LdapProvider is not the first
provider listed in the file.
----
---
[GitHub] nifi pull request #2350: NIFI-4701 Support encrypted authorizers.xml
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/nifi/pull/2350
---
[GitHub] nifi issue #2350: NIFI-4701 Support encrypted authorizers.xml
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on the issue:
https://github.com/apache/nifi/pull/2350
Made all changes noted in the review comments. Ran sample executions with invalid and valid input files / scenarios to ensure the tooling changes worked.
Ran `contrib-check` and all tests pass. +1, merging.
---
[GitHub] nifi pull request #2350: NIFI-4701 Support encrypted authorizers.xml
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2350#discussion_r159101417
--- Diff: nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/properties/ConfigEncryptionTool.groovy ---
@@ -473,6 +536,34 @@ class ConfigEncryptionTool {
}
}
+ /**
+ * Loads the authorizers configuration from the provided file path.
+ *
+ * @param existingKeyHex the key used to encrypt the configs (defaults to the current key)
+ *
+ * @return the file content
+ * @throw IOException if the authorizers.xml file cannot be read
+ */
+ private String loadAuthorizers(String existingKeyHex = keyHex) throws IOException {
+ File authorizersFile
+ if (authorizersPath && (authorizersFile = new File(authorizersPath)).exists()) {
+ try {
+ String xmlContent = authorizersFile.text
+ List<String> lines = authorizersFile.readLines()
+ logger.info("Loaded Authroizers content (${lines.size()} lines)")
--- End diff --
I think this was copied from the LIP section and should be fixed there too -- this is redundant. In order to capture the number of lines and get all the contents as a single string, we should use the `readLines()` method and then `join` the `List<String>`.
---
[GitHub] nifi pull request #2350: NIFI-4701 Support encrypted authorizers.xml
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2350#discussion_r159130193
--- Diff: nifi-toolkit/nifi-toolkit-encrypt-config/src/test/groovy/org/apache/nifi/properties/ConfigEncryptionToolTest.groovy ---
@@ -2506,92 +2604,789 @@ class ConfigEncryptionToolTest extends GroovyTestCase {
}
@Test
- void testShouldPerformFullOperationForNiFiPropertiesAndLoginIdentityProviders() {
+ void testShouldDecryptAuthorizers() {
// Arrange
- exit.expectSystemExitWithStatus(0)
+ String authorizersPath = "src/test/resources/authorizers-populated-encrypted.xml"
+ File authorizersFile = new File(authorizersPath)
- File tmpDir = setupTmpDir()
+ setupTmpDir()
- File emptyKeyFile = new File("src/test/resources/bootstrap_with_empty_master_key.conf")
- File bootstrapFile = new File("target/tmp/tmp_bootstrap.conf")
- bootstrapFile.delete()
+ File workingFile = new File("target/tmp/tmp-authorizers.xml")
+ workingFile.delete()
+ Files.copy(authorizersFile.toPath(), workingFile.toPath())
+ ConfigEncryptionTool tool = new ConfigEncryptionTool()
- Files.copy(emptyKeyFile.toPath(), bootstrapFile.toPath())
- final List<String> originalBootstrapLines = bootstrapFile.readLines()
- String originalKeyLine = originalBootstrapLines.find {
- it.startsWith(ConfigEncryptionTool.BOOTSTRAP_KEY_PREFIX)
- }
- logger.info("Original key line from bootstrap.conf: ${originalKeyLine}")
- assert originalKeyLine == ConfigEncryptionTool.BOOTSTRAP_KEY_PREFIX
+ // Sanity check for decryption
+ String cipherText = "q4r7WIgN0MaxdAKM||SGgdCTPGSFEcuH4RraMYEdeyVbOx93abdWTVSWvh1w+klA"
+ String EXPECTED_PASSWORD = "thisIsABadPassword"
+ AESSensitivePropertyProvider spp = new AESSensitivePropertyProvider(KEY_HEX_128)
+ assert spp.unprotect(cipherText) == EXPECTED_PASSWORD
- final String EXPECTED_KEY_LINE = ConfigEncryptionTool.BOOTSTRAP_KEY_PREFIX + KEY_HEX
+ tool.keyHex = KEY_HEX_128
- // Set up the NFP file
- File inputPropertiesFile = new File("src/test/resources/nifi_with_sensitive_properties_unprotected.properties")
- File outputPropertiesFile = new File("target/tmp/tmp_nifi.properties")
- outputPropertiesFile.delete()
+ def lines = workingFile.readLines()
+ logger.info("Read lines: \n${lines.join("\n")}")
- NiFiProperties inputProperties = new NiFiPropertiesLoader().load(inputPropertiesFile)
- logger.info("Loaded ${inputProperties.size()} properties from input file")
- ProtectedNiFiProperties protectedInputProperties = new ProtectedNiFiProperties(inputProperties)
- def originalSensitiveValues = protectedInputProperties.getSensitivePropertyKeys().collectEntries { String key -> [(key): protectedInputProperties.getProperty(key)] }
- logger.info("Original sensitive values: ${originalSensitiveValues}")
+ // Act
+ def decryptedLines = tool.decryptAuthorizers(lines.join("\n")).split("\n")
+ logger.info("Decrypted lines: \n${decryptedLines.join("\n")}")
- // Set up the LIP file
- File inputLIPFile = new File("src/test/resources/login-identity-providers-populated.xml")
- File outputLIPFile = new File("target/tmp/tmp-lip.xml")
- outputLIPFile.delete()
+ // Assert
+ def passwordLines = decryptedLines.findAll { it =~ PASSWORD_PROP_REGEX }
+ assert passwordLines.size() == AUTHORIZERS_PASSWORD_LINE_COUNT
+ assert passwordLines.every { it =~ ">thisIsABadPassword<" }
+ // Some lines were not encrypted originally so the encryption attribute would not have been updated
+ assert passwordLines.any { it =~ "encryption=\"none\"" }
+ }
- String originalXmlContent = inputLIPFile.text
- logger.info("Original XML content: ${originalXmlContent}")
+ @Test
+ void testShouldDecryptAuthorizersWithMultilineElements() {
+ // Arrange
+ String authorizersPath = "src/test/resources/authorizers-populated-encrypted-multiline.xml"
+ File authorizersFile = new File(authorizersPath)
+
+ setupTmpDir()
+
+ File workingFile = new File("target/tmp/tmp-authorizers.xml")
+ workingFile.delete()
+ Files.copy(authorizersFile.toPath(), workingFile.toPath())
+ ConfigEncryptionTool tool = new ConfigEncryptionTool()
+ tool.isVerbose = true
+
+ tool.keyHex = KEY_HEX_128
+
+ def lines = workingFile.readLines()
+ logger.info("Read lines: \n${lines.join("\n")}")
+
+ // Act
+ def decryptedLines = tool.decryptAuthorizers(lines.join("\n")).split("\n")
+ logger.info("Decrypted lines: \n${decryptedLines.join("\n")}")
+
+ // Assert
+ def passwordLines = decryptedLines.findAll { it =~ PASSWORD_PROP_REGEX }
+ assert passwordLines.size() == AUTHORIZERS_PASSWORD_LINE_COUNT
+ assert passwordLines.every { it =~ ">thisIsABadPassword<" }
+ // Some lines were not encrypted originally so the encryption attribute would not have been updated
+ assert passwordLines.any { it =~ "encryption=\"none\"" }
+ }
+
+ @Test
+ void testShouldDecryptAuthorizersWithMultipleElementsPerLine() {
+ // Arrange
+ String authorizersPath = "src/test/resources/authorizers-populated-encrypted-multiple-per-line.xml"
+ File authorizersFile = new File(authorizersPath)
+
+ setupTmpDir()
+
+ File workingFile = new File("target/tmp/tmp-authorizers.xml")
+ workingFile.delete()
+ Files.copy(authorizersFile.toPath(), workingFile.toPath())
+ ConfigEncryptionTool tool = new ConfigEncryptionTool()
+ tool.isVerbose = true
+
+ tool.keyHex = KEY_HEX_128
+
+ def lines = workingFile.readLines()
+ logger.info("Read lines: \n${lines.join("\n")}")
+
+ // Act
+ def decryptedLines = tool.decryptAuthorizers(lines.join("\n")).split("\n")
+ logger.info("Decrypted lines: \n${decryptedLines.join("\n")}")
+
+ // Assert
+ def passwordLines = decryptedLines.findAll { it =~ PASSWORD_PROP_REGEX }
+ assert passwordLines.size() == AUTHORIZERS_PASSWORD_LINE_COUNT
+ assert passwordLines.every { it =~ ">thisIsABadPassword<" }
+ // Some lines were not encrypted originally so the encryption attribute would not have been updated
+ assert passwordLines.any { it =~ "encryption=\"none\"" }
+ }
+
+
+ @Test
+ void testDecryptAuthorizersShouldHandleCommentedElements() {
+ // Arrange
+ String authorizersPath = "src/test/resources/authorizers-commented.xml"
+ File authorizersFile = new File(authorizersPath)
+
+ setupTmpDir()
+
+ File workingFile = new File("target/tmp/tmp-authorizers.xml")
+ workingFile.delete()
+ Files.copy(authorizersFile.toPath(), workingFile.toPath())
+ ConfigEncryptionTool tool = new ConfigEncryptionTool()
+ tool.isVerbose = true
+
+ tool.keyHex = KEY_HEX_128
+
+ def lines = workingFile.readLines()
+ logger.info("Read lines: \n${lines.join("\n")}")
+
+ // Act
+ def decryptedLines = tool.decryptAuthorizers(lines.join("\n")).split("\n")
+ logger.info("Decrypted lines: \n${decryptedLines.join("\n")}")
+
+ // Assert
+
+ // If no encrypted properties are found, the original input text is just returned (comments and formatting in tact)
+ assert decryptedLines == lines
+ }
+
+ @Test
+ void testShouldEncryptAuthorizers() {
+ // Arrange
+ String authorizersPath = "src/test/resources/authorizers-populated.xml"
+ File authorizersFile = new File(authorizersPath)
+
+ setupTmpDir()
+
+ File workingFile = new File("target/tmp/tmp-authorizers.xml")
+ workingFile.delete()
+ Files.copy(authorizersFile.toPath(), workingFile.toPath())
+ ConfigEncryptionTool tool = new ConfigEncryptionTool()
+ tool.isVerbose = true
+
+ tool.keyHex = KEY_HEX
+ String encryptionScheme = "encryption=\"aes/gcm/${getKeyLength(KEY_HEX)}\""
- String[] args = ["-n", inputPropertiesFile.path, "-l", inputLIPFile.path, "-b", bootstrapFile.path, "-i", outputLIPFile.path, "-o", outputPropertiesFile.path, "-k", KEY_HEX, "-v"]
+ def lines = workingFile.readLines()
+ logger.info("Read lines: \n${lines.join("\n")}")
AESSensitivePropertyProvider spp = new AESSensitivePropertyProvider(KEY_HEX)
- exit.checkAssertionAfterwards(new Assertion() {
- public void checkAssertion() {
- final List<String> updatedPropertiesLines = outputPropertiesFile.readLines()
- logger.info("Updated nifi.properties:")
- logger.info("\n" * 2 + updatedPropertiesLines.join("\n"))
+ // Act
+ def encryptedLines = tool.encryptAuthorizers(lines.join("\n")).split("\n")
+ logger.info("Encrypted lines: \n${encryptedLines.join("\n")}")
- // Check that the output values for sensitive properties are not the same as the original (i.e. it was encrypted)
- NiFiProperties updatedProperties = new NiFiPropertiesLoader().readProtectedPropertiesFromDisk(outputPropertiesFile)
- assert updatedProperties.size() >= inputProperties.size()
- originalSensitiveValues.every { String key, String originalValue ->
- assert updatedProperties.getProperty(key) != originalValue
- }
+ // Assert
+ def passwordLines = encryptedLines.findAll { it =~ PASSWORD_PROP_REGEX }
+ assert passwordLines.size() == AUTHORIZERS_PASSWORD_LINE_COUNT
+ assert passwordLines.every { !it.contains(">thisIsABadPassword<") }
+ assert passwordLines.every { it.contains(encryptionScheme) }
+ passwordLines.each {
+ String ct = (it =~ ">(.*)</property>")[0][1]
+ logger.info("Cipher text: ${ct}")
+ assert spp.unprotect(ct) == PASSWORD
+ }
+ }
- // Check that the new NiFiProperties instance matches the output file (values still encrypted)
- updatedProperties.getPropertyKeys().every { String key ->
- assert updatedPropertiesLines.contains("${key}=${updatedProperties.getProperty(key)}".toString())
- }
+ @Test
+ void testShouldEncryptAuthorizersWithEmptySensitiveElements() {
+ // Arrange
+ String authorizersPath = "src/test/resources/authorizers-populated-empty.xml"
+ File authorizersFile = new File(authorizersPath)
- final String updatedXmlContent = outputLIPFile.text
- logger.info("Updated XML content: ${updatedXmlContent}")
+ setupTmpDir()
- // Check that the output values for sensitive properties are not the same as the original (i.e. it was encrypted)
- def originalParsedXml = new XmlSlurper().parseText(originalXmlContent)
- def updatedParsedXml = new XmlSlurper().parseText(updatedXmlContent)
- assert originalParsedXml != updatedParsedXml
- assert originalParsedXml.'**'.findAll { it.@encryption } != updatedParsedXml.'**'.findAll {
- it.@encryption
- }
+ File workingFile = new File("target/tmp/tmp-authorizers.xml")
+ workingFile.delete()
+ Files.copy(authorizersFile.toPath(), workingFile.toPath())
+ ConfigEncryptionTool tool = new ConfigEncryptionTool()
+ tool.isVerbose = true
- def encryptedValues = updatedParsedXml.provider.find {
- it.identifier == 'ldap-provider'
- }.property.findAll {
- it.@name =~ "Password" && it.@encryption =~ "aes/gcm/\\d{3}"
- }
+ tool.keyHex = KEY_HEX
+ String encryptionScheme = "encryption=\"aes/gcm/${getKeyLength(KEY_HEX)}\""
- encryptedValues.each {
- assert spp.unprotect(it.text()) == PASSWORD
- }
+ def lines = workingFile.readLines()
+ logger.info("Read lines: \n${lines.join("\n")}")
+ AESSensitivePropertyProvider spp = new AESSensitivePropertyProvider(KEY_HEX)
+
+ // Act
+ def encryptedLines = tool.encryptAuthorizers(lines.join("\n")).split("\n")
+ logger.info("Encrypted lines: \n${encryptedLines.join("\n")}")
+
+ // Assert
+ def passwordLines = encryptedLines.findAll { it =~ PASSWORD_PROP_REGEX }
+ assert passwordLines.size() == AUTHORIZERS_PASSWORD_LINE_COUNT
+ def populatedPasswordLines = passwordLines.findAll { it =~ />.+</ }
+ assert populatedPasswordLines.every { !it.contains(">thisIsABadPassword<") }
+ assert populatedPasswordLines.every { it.contains(encryptionScheme) }
+ populatedPasswordLines.each {
+ String ct = (it =~ ">(.*)</property>")[0][1]
+ logger.info("Cipher text: ${ct}")
+ assert spp.unprotect(ct) == PASSWORD
+ }
+ }
+
+ @Test
+ void testShouldEncryptAuthorizersWithMultilineElements() {
+ // Arrange
+ String authorizersPath = "src/test/resources/authorizers-populated-multiline.xml"
+ File authorizersFile = new File(authorizersPath)
+
+ setupTmpDir()
+
+ File workingFile = new File("target/tmp/tmp-authorizers.xml")
+ workingFile.delete()
+ Files.copy(authorizersFile.toPath(), workingFile.toPath())
+ ConfigEncryptionTool tool = new ConfigEncryptionTool()
+ tool.isVerbose = true
+
+ tool.keyHex = KEY_HEX
+ String encryptionScheme = "encryption=\"aes/gcm/${getKeyLength(KEY_HEX)}\""
+
+ def lines = workingFile.readLines()
+ logger.info("Read lines: \n${lines.join("\n")}")
+
+ AESSensitivePropertyProvider spp = new AESSensitivePropertyProvider(KEY_HEX)
+
+ // Act
+ def encryptedLines = tool.encryptAuthorizers(lines.join("\n")).split("\n")
+ logger.info("Encrypted lines: \n${encryptedLines.join("\n")}")
+
+ // Assert
+ def passwordLines = encryptedLines.findAll { it =~ PASSWORD_PROP_REGEX }
+ assert passwordLines.size() == AUTHORIZERS_PASSWORD_LINE_COUNT
+ assert passwordLines.every { !it.contains(">thisIsABadPassword<") }
+ assert passwordLines.every { it.contains(encryptionScheme) }
+ passwordLines.each {
+ String ct = (it =~ ">(.*)</property>")[0][1]
+ logger.info("Cipher text: ${ct}")
+ assert spp.unprotect(ct) == PASSWORD
+ }
+ }
+
+ @Test
+ void testShouldEncryptAuthorizersWithMultipleElementsPerLine() {
+ // Arrange
+ String authorizersPath = "src/test/resources/authorizers-populated-multiple-per-line.xml"
+ File authorizersFile = new File(authorizersPath)
+
+ setupTmpDir()
+
+ File workingFile = new File("target/tmp/tmp-authorizers.xml")
+ workingFile.delete()
+ Files.copy(authorizersFile.toPath(), workingFile.toPath())
+ ConfigEncryptionTool tool = new ConfigEncryptionTool()
+ tool.isVerbose = true
+
+ tool.keyHex = KEY_HEX
+ String encryptionScheme = "encryption=\"aes/gcm/${getKeyLength(KEY_HEX)}\""
+
+ def lines = workingFile.readLines()
+ logger.info("Read lines: \n${lines.join("\n")}")
+
+ AESSensitivePropertyProvider spp = new AESSensitivePropertyProvider(KEY_HEX)
+
+ // Act
+ def encryptedLines = tool.encryptAuthorizers(lines.join("\n")).split("\n")
+ logger.info("Encrypted lines: \n${encryptedLines.join("\n")}")
+
+ // Assert
+ def passwordLines = encryptedLines.findAll { it =~ PASSWORD_PROP_REGEX }
+ assert passwordLines.size() == AUTHORIZERS_PASSWORD_LINE_COUNT
+ assert passwordLines.every { !it.contains(">thisIsABadPassword<") }
+ assert passwordLines.every { it.contains(encryptionScheme) }
+ passwordLines.each {
+ String ct = (it =~ ">(.*)</property>")[0][1]
+ logger.info("Cipher text: ${ct}")
+ assert spp.unprotect(ct) == PASSWORD
+ }
+ }
+
+ @Test
+ void testShouldEncryptAuthorizersWithRenamedProvider() {
+ // Arrange
+ String authorizersPath = "src/test/resources/authorizers-populated-renamed.xml"
+ File authorizersFile = new File(authorizersPath)
+
+ setupTmpDir()
+
+ File workingFile = new File("target/tmp/tmp-authorizers.xml")
+ workingFile.delete()
+ Files.copy(authorizersFile.toPath(), workingFile.toPath())
+ ConfigEncryptionTool tool = new ConfigEncryptionTool()
+ tool.isVerbose = true
+
+ tool.keyHex = KEY_HEX
+ String encryptionScheme = "encryption=\"aes/gcm/${getKeyLength(KEY_HEX)}\""
+
+ def lines = workingFile.readLines()
+ logger.info("Read lines: \n${lines.join("\n")}")
+ assert lines.findAll { it =~ "ldap-user-group-provider" }.empty
+
+ AESSensitivePropertyProvider spp = new AESSensitivePropertyProvider(KEY_HEX)
+
+ // Act
+ def encryptedLines = tool.encryptAuthorizers(lines.join("\n")).split("\n")
+ logger.info("Encrypted lines: \n${encryptedLines.join("\n")}")
+
+ // Assert
+ def passwordLines = encryptedLines.findAll { it =~ PASSWORD_PROP_REGEX }
+ assert passwordLines.size() == AUTHORIZERS_PASSWORD_LINE_COUNT
+ def populatedPasswordLines = passwordLines.findAll { it =~ />.+</ }
+ assert populatedPasswordLines.every { !it.contains(">thisIsABadPassword<") }
+ assert populatedPasswordLines.every { it.contains(encryptionScheme) }
+ populatedPasswordLines.each {
+ String ct = (it =~ ">(.*)</property>")[0][1]
+ logger.info("Cipher text: ${ct}")
+ assert spp.unprotect(ct) == PASSWORD
+ }
+ }
+
+ @Test
+ void testEncryptAuthorizersShouldHandleCommentedElements() {
+ // Arrange
+ String authorizersPath = "src/test/resources/authorizers-commented.xml"
+ File authorizersFile = new File(authorizersPath)
+
+ setupTmpDir()
+
+ File workingFile = new File("target/tmp/tmp-authorizers.xml")
+ workingFile.delete()
+ Files.copy(authorizersFile.toPath(), workingFile.toPath())
+ ConfigEncryptionTool tool = new ConfigEncryptionTool()
+ tool.isVerbose = true
+
+ tool.keyHex = KEY_HEX_128
--- End diff --
This should just be `KEY_HEX` so the JCE determines which key length to use.
---
[GitHub] nifi pull request #2350: NIFI-4701 Support encrypted authorizers.xml
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2350#discussion_r159102294
--- Diff: nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/properties/ConfigEncryptionTool.groovy ---
@@ -921,6 +1090,39 @@ class ConfigEncryptionTool {
}
}
+ /**
+ * Writes the contents of the authorizers configuration file with encrypted values to the output {@code authorizers.xml} file.
+ *
+ * @throw IOException if there is a problem reading or writing the authorizers.xml file
+ */
+ private void writeAuthorizers() throws IOException {
+ if (!outputAuthorizersPath) {
+ throw new IllegalArgumentException("Cannot write encrypted properties to empty authorizers.xml path")
+ }
+
+ File outputAuthorizersFile = new File(outputAuthorizersPath)
+
+ if (isSafeToWrite(outputAuthorizersFile)) {
+ try {
+ String updatedXmlContent
+ File authorizersFile = new File(authorizersPath)
+ if (authorizersFile.exists() && authorizersFile.canRead()) {
+ // Instead of just writing the XML content to a file, this method attempts to maintain the structure of the original file and preserves comments
+ updatedXmlContent = serializeAuthorizersAndPreserveFormat(authorizers, authorizersFile).join("\n")
+ }
--- End diff --
Due to a possible race condition (`authorizersFile` exists and can be read when the tool execution starts, but has been deleted/made unreadable by an external process before `writeAuthorizers` executes), the value of `updatedXmlContent` will be empty, and it will overwrite `authorizers.xml`. There should be an `else` branch here which simply serializes `authorizers` to XML without the preserved whitespace and comments in order to maintain the content.
This should probably also be done for the LDAP section.
---
[GitHub] nifi pull request #2350: NIFI-4701 Support encrypted authorizers.xml
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2350#discussion_r159101938
--- Diff: nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/properties/ConfigEncryptionTool.groovy ---
@@ -730,6 +821,42 @@ class ConfigEncryptionTool {
}
}
+ String decryptAuthorizers(String encryptedXml, String existingKeyHex = keyHex) {
+ AESSensitivePropertyProvider sensitivePropertyProvider = new AESSensitivePropertyProvider(existingKeyHex)
+
+ try {
+ def doc = new XmlSlurper().parseText(encryptedXml)
+ // Find the provider element by class even if it has been renamed
+ def passwords = doc.userGroupProvider.find { it.'class' as String == LDAP_USER_GROUP_PROVIDER_CLASS }.property.findAll {
+ it.@name =~ "Password" && it.@encryption =~ "aes/gcm/\\d{3}"
+ }
+
+ if (passwords.isEmpty()) {
+ if (isVerbose) {
+ logger.info("No encrypted password property elements found in authorizers.xml")
+ }
+ return encryptedXml
+ }
+
+ passwords.each { password ->
+ if (isVerbose) {
--- End diff --
Informational note: in the event the file is in an unsupported state (perhaps manually decrypted but the `encryption` attribute is still present), this will log the plaintext password to the console output before attempting to decrypt. This is not necessarily a vulnerability of the tool, as the incoming data is not in the expected format. It would take additional effort to capture the "raw" value, compare the attempted decryption and the original value, and output the raw value if the contents are different. This would still allow the tool to print the attempted decryption input value if the attempt throws an exception, but this level of effort is unnecessary for this edge case. Just a note for the future.
---
[GitHub] nifi pull request #2350: NIFI-4701 Support encrypted authorizers.xml
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2350#discussion_r159130240
--- Diff: nifi-toolkit/nifi-toolkit-encrypt-config/src/test/groovy/org/apache/nifi/properties/ConfigEncryptionToolTest.groovy ---
@@ -2506,92 +2604,789 @@ class ConfigEncryptionToolTest extends GroovyTestCase {
}
@Test
- void testShouldPerformFullOperationForNiFiPropertiesAndLoginIdentityProviders() {
+ void testShouldDecryptAuthorizers() {
// Arrange
- exit.expectSystemExitWithStatus(0)
+ String authorizersPath = "src/test/resources/authorizers-populated-encrypted.xml"
+ File authorizersFile = new File(authorizersPath)
- File tmpDir = setupTmpDir()
+ setupTmpDir()
- File emptyKeyFile = new File("src/test/resources/bootstrap_with_empty_master_key.conf")
- File bootstrapFile = new File("target/tmp/tmp_bootstrap.conf")
- bootstrapFile.delete()
+ File workingFile = new File("target/tmp/tmp-authorizers.xml")
+ workingFile.delete()
+ Files.copy(authorizersFile.toPath(), workingFile.toPath())
+ ConfigEncryptionTool tool = new ConfigEncryptionTool()
- Files.copy(emptyKeyFile.toPath(), bootstrapFile.toPath())
- final List<String> originalBootstrapLines = bootstrapFile.readLines()
- String originalKeyLine = originalBootstrapLines.find {
- it.startsWith(ConfigEncryptionTool.BOOTSTRAP_KEY_PREFIX)
- }
- logger.info("Original key line from bootstrap.conf: ${originalKeyLine}")
- assert originalKeyLine == ConfigEncryptionTool.BOOTSTRAP_KEY_PREFIX
+ // Sanity check for decryption
+ String cipherText = "q4r7WIgN0MaxdAKM||SGgdCTPGSFEcuH4RraMYEdeyVbOx93abdWTVSWvh1w+klA"
+ String EXPECTED_PASSWORD = "thisIsABadPassword"
+ AESSensitivePropertyProvider spp = new AESSensitivePropertyProvider(KEY_HEX_128)
+ assert spp.unprotect(cipherText) == EXPECTED_PASSWORD
- final String EXPECTED_KEY_LINE = ConfigEncryptionTool.BOOTSTRAP_KEY_PREFIX + KEY_HEX
+ tool.keyHex = KEY_HEX_128
- // Set up the NFP file
- File inputPropertiesFile = new File("src/test/resources/nifi_with_sensitive_properties_unprotected.properties")
- File outputPropertiesFile = new File("target/tmp/tmp_nifi.properties")
- outputPropertiesFile.delete()
+ def lines = workingFile.readLines()
+ logger.info("Read lines: \n${lines.join("\n")}")
- NiFiProperties inputProperties = new NiFiPropertiesLoader().load(inputPropertiesFile)
- logger.info("Loaded ${inputProperties.size()} properties from input file")
- ProtectedNiFiProperties protectedInputProperties = new ProtectedNiFiProperties(inputProperties)
- def originalSensitiveValues = protectedInputProperties.getSensitivePropertyKeys().collectEntries { String key -> [(key): protectedInputProperties.getProperty(key)] }
- logger.info("Original sensitive values: ${originalSensitiveValues}")
+ // Act
+ def decryptedLines = tool.decryptAuthorizers(lines.join("\n")).split("\n")
+ logger.info("Decrypted lines: \n${decryptedLines.join("\n")}")
- // Set up the LIP file
- File inputLIPFile = new File("src/test/resources/login-identity-providers-populated.xml")
- File outputLIPFile = new File("target/tmp/tmp-lip.xml")
- outputLIPFile.delete()
+ // Assert
+ def passwordLines = decryptedLines.findAll { it =~ PASSWORD_PROP_REGEX }
+ assert passwordLines.size() == AUTHORIZERS_PASSWORD_LINE_COUNT
+ assert passwordLines.every { it =~ ">thisIsABadPassword<" }
+ // Some lines were not encrypted originally so the encryption attribute would not have been updated
+ assert passwordLines.any { it =~ "encryption=\"none\"" }
+ }
- String originalXmlContent = inputLIPFile.text
- logger.info("Original XML content: ${originalXmlContent}")
+ @Test
+ void testShouldDecryptAuthorizersWithMultilineElements() {
+ // Arrange
+ String authorizersPath = "src/test/resources/authorizers-populated-encrypted-multiline.xml"
+ File authorizersFile = new File(authorizersPath)
+
+ setupTmpDir()
+
+ File workingFile = new File("target/tmp/tmp-authorizers.xml")
+ workingFile.delete()
+ Files.copy(authorizersFile.toPath(), workingFile.toPath())
+ ConfigEncryptionTool tool = new ConfigEncryptionTool()
+ tool.isVerbose = true
+
+ tool.keyHex = KEY_HEX_128
+
+ def lines = workingFile.readLines()
+ logger.info("Read lines: \n${lines.join("\n")}")
+
+ // Act
+ def decryptedLines = tool.decryptAuthorizers(lines.join("\n")).split("\n")
+ logger.info("Decrypted lines: \n${decryptedLines.join("\n")}")
+
+ // Assert
+ def passwordLines = decryptedLines.findAll { it =~ PASSWORD_PROP_REGEX }
+ assert passwordLines.size() == AUTHORIZERS_PASSWORD_LINE_COUNT
+ assert passwordLines.every { it =~ ">thisIsABadPassword<" }
+ // Some lines were not encrypted originally so the encryption attribute would not have been updated
+ assert passwordLines.any { it =~ "encryption=\"none\"" }
+ }
+
+ @Test
+ void testShouldDecryptAuthorizersWithMultipleElementsPerLine() {
+ // Arrange
+ String authorizersPath = "src/test/resources/authorizers-populated-encrypted-multiple-per-line.xml"
+ File authorizersFile = new File(authorizersPath)
+
+ setupTmpDir()
+
+ File workingFile = new File("target/tmp/tmp-authorizers.xml")
+ workingFile.delete()
+ Files.copy(authorizersFile.toPath(), workingFile.toPath())
+ ConfigEncryptionTool tool = new ConfigEncryptionTool()
+ tool.isVerbose = true
+
+ tool.keyHex = KEY_HEX_128
+
+ def lines = workingFile.readLines()
+ logger.info("Read lines: \n${lines.join("\n")}")
+
+ // Act
+ def decryptedLines = tool.decryptAuthorizers(lines.join("\n")).split("\n")
+ logger.info("Decrypted lines: \n${decryptedLines.join("\n")}")
+
+ // Assert
+ def passwordLines = decryptedLines.findAll { it =~ PASSWORD_PROP_REGEX }
+ assert passwordLines.size() == AUTHORIZERS_PASSWORD_LINE_COUNT
+ assert passwordLines.every { it =~ ">thisIsABadPassword<" }
+ // Some lines were not encrypted originally so the encryption attribute would not have been updated
+ assert passwordLines.any { it =~ "encryption=\"none\"" }
+ }
+
+
+ @Test
+ void testDecryptAuthorizersShouldHandleCommentedElements() {
+ // Arrange
+ String authorizersPath = "src/test/resources/authorizers-commented.xml"
+ File authorizersFile = new File(authorizersPath)
+
+ setupTmpDir()
+
+ File workingFile = new File("target/tmp/tmp-authorizers.xml")
+ workingFile.delete()
+ Files.copy(authorizersFile.toPath(), workingFile.toPath())
+ ConfigEncryptionTool tool = new ConfigEncryptionTool()
+ tool.isVerbose = true
+
+ tool.keyHex = KEY_HEX_128
+
+ def lines = workingFile.readLines()
+ logger.info("Read lines: \n${lines.join("\n")}")
+
+ // Act
+ def decryptedLines = tool.decryptAuthorizers(lines.join("\n")).split("\n")
+ logger.info("Decrypted lines: \n${decryptedLines.join("\n")}")
+
+ // Assert
+
+ // If no encrypted properties are found, the original input text is just returned (comments and formatting in tact)
+ assert decryptedLines == lines
+ }
+
+ @Test
+ void testShouldEncryptAuthorizers() {
+ // Arrange
+ String authorizersPath = "src/test/resources/authorizers-populated.xml"
+ File authorizersFile = new File(authorizersPath)
+
+ setupTmpDir()
+
+ File workingFile = new File("target/tmp/tmp-authorizers.xml")
+ workingFile.delete()
+ Files.copy(authorizersFile.toPath(), workingFile.toPath())
+ ConfigEncryptionTool tool = new ConfigEncryptionTool()
+ tool.isVerbose = true
+
+ tool.keyHex = KEY_HEX
+ String encryptionScheme = "encryption=\"aes/gcm/${getKeyLength(KEY_HEX)}\""
- String[] args = ["-n", inputPropertiesFile.path, "-l", inputLIPFile.path, "-b", bootstrapFile.path, "-i", outputLIPFile.path, "-o", outputPropertiesFile.path, "-k", KEY_HEX, "-v"]
+ def lines = workingFile.readLines()
+ logger.info("Read lines: \n${lines.join("\n")}")
AESSensitivePropertyProvider spp = new AESSensitivePropertyProvider(KEY_HEX)
- exit.checkAssertionAfterwards(new Assertion() {
- public void checkAssertion() {
- final List<String> updatedPropertiesLines = outputPropertiesFile.readLines()
- logger.info("Updated nifi.properties:")
- logger.info("\n" * 2 + updatedPropertiesLines.join("\n"))
+ // Act
+ def encryptedLines = tool.encryptAuthorizers(lines.join("\n")).split("\n")
+ logger.info("Encrypted lines: \n${encryptedLines.join("\n")}")
- // Check that the output values for sensitive properties are not the same as the original (i.e. it was encrypted)
- NiFiProperties updatedProperties = new NiFiPropertiesLoader().readProtectedPropertiesFromDisk(outputPropertiesFile)
- assert updatedProperties.size() >= inputProperties.size()
- originalSensitiveValues.every { String key, String originalValue ->
- assert updatedProperties.getProperty(key) != originalValue
- }
+ // Assert
+ def passwordLines = encryptedLines.findAll { it =~ PASSWORD_PROP_REGEX }
+ assert passwordLines.size() == AUTHORIZERS_PASSWORD_LINE_COUNT
+ assert passwordLines.every { !it.contains(">thisIsABadPassword<") }
+ assert passwordLines.every { it.contains(encryptionScheme) }
+ passwordLines.each {
+ String ct = (it =~ ">(.*)</property>")[0][1]
+ logger.info("Cipher text: ${ct}")
+ assert spp.unprotect(ct) == PASSWORD
+ }
+ }
- // Check that the new NiFiProperties instance matches the output file (values still encrypted)
- updatedProperties.getPropertyKeys().every { String key ->
- assert updatedPropertiesLines.contains("${key}=${updatedProperties.getProperty(key)}".toString())
- }
+ @Test
+ void testShouldEncryptAuthorizersWithEmptySensitiveElements() {
+ // Arrange
+ String authorizersPath = "src/test/resources/authorizers-populated-empty.xml"
+ File authorizersFile = new File(authorizersPath)
- final String updatedXmlContent = outputLIPFile.text
- logger.info("Updated XML content: ${updatedXmlContent}")
+ setupTmpDir()
- // Check that the output values for sensitive properties are not the same as the original (i.e. it was encrypted)
- def originalParsedXml = new XmlSlurper().parseText(originalXmlContent)
- def updatedParsedXml = new XmlSlurper().parseText(updatedXmlContent)
- assert originalParsedXml != updatedParsedXml
- assert originalParsedXml.'**'.findAll { it.@encryption } != updatedParsedXml.'**'.findAll {
- it.@encryption
- }
+ File workingFile = new File("target/tmp/tmp-authorizers.xml")
+ workingFile.delete()
+ Files.copy(authorizersFile.toPath(), workingFile.toPath())
+ ConfigEncryptionTool tool = new ConfigEncryptionTool()
+ tool.isVerbose = true
- def encryptedValues = updatedParsedXml.provider.find {
- it.identifier == 'ldap-provider'
- }.property.findAll {
- it.@name =~ "Password" && it.@encryption =~ "aes/gcm/\\d{3}"
- }
+ tool.keyHex = KEY_HEX
+ String encryptionScheme = "encryption=\"aes/gcm/${getKeyLength(KEY_HEX)}\""
- encryptedValues.each {
- assert spp.unprotect(it.text()) == PASSWORD
- }
+ def lines = workingFile.readLines()
+ logger.info("Read lines: \n${lines.join("\n")}")
+ AESSensitivePropertyProvider spp = new AESSensitivePropertyProvider(KEY_HEX)
+
+ // Act
+ def encryptedLines = tool.encryptAuthorizers(lines.join("\n")).split("\n")
+ logger.info("Encrypted lines: \n${encryptedLines.join("\n")}")
+
+ // Assert
+ def passwordLines = encryptedLines.findAll { it =~ PASSWORD_PROP_REGEX }
+ assert passwordLines.size() == AUTHORIZERS_PASSWORD_LINE_COUNT
+ def populatedPasswordLines = passwordLines.findAll { it =~ />.+</ }
+ assert populatedPasswordLines.every { !it.contains(">thisIsABadPassword<") }
+ assert populatedPasswordLines.every { it.contains(encryptionScheme) }
+ populatedPasswordLines.each {
+ String ct = (it =~ ">(.*)</property>")[0][1]
+ logger.info("Cipher text: ${ct}")
+ assert spp.unprotect(ct) == PASSWORD
+ }
+ }
+
+ @Test
+ void testShouldEncryptAuthorizersWithMultilineElements() {
+ // Arrange
+ String authorizersPath = "src/test/resources/authorizers-populated-multiline.xml"
+ File authorizersFile = new File(authorizersPath)
+
+ setupTmpDir()
+
+ File workingFile = new File("target/tmp/tmp-authorizers.xml")
+ workingFile.delete()
+ Files.copy(authorizersFile.toPath(), workingFile.toPath())
+ ConfigEncryptionTool tool = new ConfigEncryptionTool()
+ tool.isVerbose = true
+
+ tool.keyHex = KEY_HEX
+ String encryptionScheme = "encryption=\"aes/gcm/${getKeyLength(KEY_HEX)}\""
+
+ def lines = workingFile.readLines()
+ logger.info("Read lines: \n${lines.join("\n")}")
+
+ AESSensitivePropertyProvider spp = new AESSensitivePropertyProvider(KEY_HEX)
+
+ // Act
+ def encryptedLines = tool.encryptAuthorizers(lines.join("\n")).split("\n")
+ logger.info("Encrypted lines: \n${encryptedLines.join("\n")}")
+
+ // Assert
+ def passwordLines = encryptedLines.findAll { it =~ PASSWORD_PROP_REGEX }
+ assert passwordLines.size() == AUTHORIZERS_PASSWORD_LINE_COUNT
+ assert passwordLines.every { !it.contains(">thisIsABadPassword<") }
+ assert passwordLines.every { it.contains(encryptionScheme) }
+ passwordLines.each {
+ String ct = (it =~ ">(.*)</property>")[0][1]
+ logger.info("Cipher text: ${ct}")
+ assert spp.unprotect(ct) == PASSWORD
+ }
+ }
+
+ @Test
+ void testShouldEncryptAuthorizersWithMultipleElementsPerLine() {
+ // Arrange
+ String authorizersPath = "src/test/resources/authorizers-populated-multiple-per-line.xml"
+ File authorizersFile = new File(authorizersPath)
+
+ setupTmpDir()
+
+ File workingFile = new File("target/tmp/tmp-authorizers.xml")
+ workingFile.delete()
+ Files.copy(authorizersFile.toPath(), workingFile.toPath())
+ ConfigEncryptionTool tool = new ConfigEncryptionTool()
+ tool.isVerbose = true
+
+ tool.keyHex = KEY_HEX
+ String encryptionScheme = "encryption=\"aes/gcm/${getKeyLength(KEY_HEX)}\""
+
+ def lines = workingFile.readLines()
+ logger.info("Read lines: \n${lines.join("\n")}")
+
+ AESSensitivePropertyProvider spp = new AESSensitivePropertyProvider(KEY_HEX)
+
+ // Act
+ def encryptedLines = tool.encryptAuthorizers(lines.join("\n")).split("\n")
+ logger.info("Encrypted lines: \n${encryptedLines.join("\n")}")
+
+ // Assert
+ def passwordLines = encryptedLines.findAll { it =~ PASSWORD_PROP_REGEX }
+ assert passwordLines.size() == AUTHORIZERS_PASSWORD_LINE_COUNT
+ assert passwordLines.every { !it.contains(">thisIsABadPassword<") }
+ assert passwordLines.every { it.contains(encryptionScheme) }
+ passwordLines.each {
+ String ct = (it =~ ">(.*)</property>")[0][1]
+ logger.info("Cipher text: ${ct}")
+ assert spp.unprotect(ct) == PASSWORD
+ }
+ }
+
+ @Test
+ void testShouldEncryptAuthorizersWithRenamedProvider() {
+ // Arrange
+ String authorizersPath = "src/test/resources/authorizers-populated-renamed.xml"
+ File authorizersFile = new File(authorizersPath)
+
+ setupTmpDir()
+
+ File workingFile = new File("target/tmp/tmp-authorizers.xml")
+ workingFile.delete()
+ Files.copy(authorizersFile.toPath(), workingFile.toPath())
+ ConfigEncryptionTool tool = new ConfigEncryptionTool()
+ tool.isVerbose = true
+
+ tool.keyHex = KEY_HEX
+ String encryptionScheme = "encryption=\"aes/gcm/${getKeyLength(KEY_HEX)}\""
+
+ def lines = workingFile.readLines()
+ logger.info("Read lines: \n${lines.join("\n")}")
+ assert lines.findAll { it =~ "ldap-user-group-provider" }.empty
+
+ AESSensitivePropertyProvider spp = new AESSensitivePropertyProvider(KEY_HEX)
+
+ // Act
+ def encryptedLines = tool.encryptAuthorizers(lines.join("\n")).split("\n")
+ logger.info("Encrypted lines: \n${encryptedLines.join("\n")}")
+
+ // Assert
+ def passwordLines = encryptedLines.findAll { it =~ PASSWORD_PROP_REGEX }
+ assert passwordLines.size() == AUTHORIZERS_PASSWORD_LINE_COUNT
+ def populatedPasswordLines = passwordLines.findAll { it =~ />.+</ }
+ assert populatedPasswordLines.every { !it.contains(">thisIsABadPassword<") }
+ assert populatedPasswordLines.every { it.contains(encryptionScheme) }
+ populatedPasswordLines.each {
+ String ct = (it =~ ">(.*)</property>")[0][1]
+ logger.info("Cipher text: ${ct}")
+ assert spp.unprotect(ct) == PASSWORD
+ }
+ }
+
+ @Test
+ void testEncryptAuthorizersShouldHandleCommentedElements() {
+ // Arrange
+ String authorizersPath = "src/test/resources/authorizers-commented.xml"
+ File authorizersFile = new File(authorizersPath)
+
+ setupTmpDir()
+
+ File workingFile = new File("target/tmp/tmp-authorizers.xml")
+ workingFile.delete()
+ Files.copy(authorizersFile.toPath(), workingFile.toPath())
+ ConfigEncryptionTool tool = new ConfigEncryptionTool()
+ tool.isVerbose = true
+
+ tool.keyHex = KEY_HEX_128
+
+ def lines = workingFile.readLines()
+ logger.info("Read lines: \n${lines.join("\n")}")
+
+ // Act
+ def encryptedLines = tool.encryptAuthorizers(lines.join("\n")).split("\n")
+ logger.info("Encrypted lines: \n${encryptedLines.join("\n")}")
+
+ // Assert
+
+ // If no sensitive properties are found, the original input text is just returned (comments and formatting in tact)
+ assert encryptedLines == lines
+ }
+
+ @Test
+ void testSerializeAuthorizersAndPreserveFormatShouldRespectComments() {
+ // Arrange
+ String authorizersPath = "src/test/resources/authorizers-populated.xml"
+ File authorizersFile = new File(authorizersPath)
+
+ setupTmpDir()
+
+ File workingFile = new File("target/tmp/tmp-authorizers.xml")
+ workingFile.delete()
+ Files.copy(authorizersFile.toPath(), workingFile.toPath())
+ ConfigEncryptionTool tool = new ConfigEncryptionTool()
+ tool.isVerbose = true
+
+ def lines = workingFile.readLines()
+ logger.info("Read lines: \n${lines.join("\n")}")
+
+ String plainXml = workingFile.text
+ String encryptedXml = tool.encryptAuthorizers(plainXml, KEY_HEX)
+ logger.info("Encrypted XML: \n${encryptedXml}")
+
+ // Act
+ def serializedLines = tool.serializeAuthorizersAndPreserveFormat(encryptedXml, workingFile)
+ logger.info("Serialized lines: \n${serializedLines.join("\n")}")
+
+ // Assert
+
+ // Some empty lines will be removed
+ def trimmedLines = lines.collect { it.trim() }.findAll { it }
+ def trimmedSerializedLines = serializedLines.collect { it.trim() }.findAll { it }
+ assert trimmedLines.size() == trimmedSerializedLines.size()
+
+ // Ensure the replacement actually occurred
+ assert trimmedSerializedLines.findAll { it =~ "encryption=" }.size() == AUTHORIZERS_PASSWORD_LINE_COUNT
+ }
+
+ @Test
+ void testSerializeAuthorizersAndPreserveFormatShouldHandleRenamedProvider() {
+ // Arrange
+ String authorizersPath = "src/test/resources/authorizers-populated-renamed.xml"
+ File authorizersFile = new File(authorizersPath)
+
+ setupTmpDir()
+
+ File workingFile = new File("target/tmp/tmp-authorizers.xml")
+ workingFile.delete()
+ Files.copy(authorizersFile.toPath(), workingFile.toPath())
+ ConfigEncryptionTool tool = new ConfigEncryptionTool()
+ tool.isVerbose = true
+
+ def lines = workingFile.readLines()
+ logger.info("Read lines: \n${lines.join("\n")}")
+ assert lines.findAll { it =~ "ldap-user-group-provider" }.empty
+
+ String plainXml = workingFile.text
+ String encryptedXml = tool.encryptAuthorizers(plainXml, KEY_HEX)
+ logger.info("Encrypted XML: \n${encryptedXml}")
+
+ // Act
+ def serializedLines = tool.serializeAuthorizersAndPreserveFormat(encryptedXml, workingFile)
+ logger.info("Serialized lines: \n${serializedLines.join("\n")}")
+
+ // Assert
+
+ // Some empty lines will be removed
+ def trimmedLines = lines.collect { it.trim() }.findAll { it }
+ def trimmedSerializedLines = serializedLines.collect { it.trim() }.findAll { it }
+ assert trimmedLines.size() == trimmedSerializedLines.size()
+
+ // Ensure the replacement actually occurred
+ assert trimmedSerializedLines.findAll { it =~ "encryption=" }.size() == AUTHORIZERS_PASSWORD_LINE_COUNT
+ }
+
+ @Test
+ void testSerializeAuthorizersAndPreserveFormatShouldHandleCommentedFile() {
+ // Arrange
+ String authorizersPath = "src/test/resources/authorizers-commented.xml"
+ File authorizersFile = new File(authorizersPath)
+
+ File tmpDir = setupTmpDir()
+
+ File workingFile = new File("target/tmp/tmp-authorizers.xml")
+ workingFile.delete()
+ Files.copy(authorizersFile.toPath(), workingFile.toPath())
+ ConfigEncryptionTool tool = new ConfigEncryptionTool()
+ tool.isVerbose = true
+
+ tool.keyHex = KEY_HEX_128
+
+ def lines = workingFile.readLines()
+ logger.info("Read lines: \n${lines.join("\n")}")
+
+ // If no sensitive properties are found, the original input text is just returned (comments and formatting in tact)
+ def encryptedLines = tool.encryptAuthorizers(lines.join("\n")).split("\n")
+ logger.info("Encrypted lines: \n${encryptedLines.join("\n")}")
+ assert encryptedLines == lines
+
+ // Act
+ def serializedLines = ConfigEncryptionTool.serializeAuthorizersAndPreserveFormat(encryptedLines.join("\n"), workingFile)
+ logger.info("Serialized lines: \n${serializedLines.join("\n")}")
+
+ // Assert
+ assert serializedLines == encryptedLines
+ assert TestAppender.events.any {
+ it.renderedMessage =~ "No provider element with class org.apache.nifi.ldap.tenants.LdapUserGroupProvider found in XML content; " +
+ "the file could be empty or the element may be missing or commented out"
+ }
+ }
+
+ @Test
+ void testSerializeAuthorizersAndPreserveFormatShouldHandleEmptyFile() {
+ // Arrange
+ setupTmpDir()
+
+ File workingFile = new File("target/tmp/tmp-authorizers.xml")
+ workingFile.delete()
+ workingFile.createNewFile()
+ ConfigEncryptionTool tool = new ConfigEncryptionTool()
+ tool.isVerbose = true
+
+ tool.keyHex = KEY_HEX_128
--- End diff --
This should just be `KEY_HEX` so the JCE determines which key length to use.
---
[GitHub] nifi pull request #2350: NIFI-4701 Support encrypted authorizers.xml
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2350#discussion_r159102652
--- Diff: nifi-toolkit/nifi-toolkit-encrypt-config/src/test/groovy/org/apache/nifi/properties/ConfigEncryptionToolTest.groovy ---
@@ -319,6 +320,59 @@ class ConfigEncryptionToolTest extends GroovyTestCase {
}
}
+ @Test
+ void testShouldParseAuthorizersArgument() {
+ // Arrange
+ def flags = ["-a", "--authorizers"]
+ String authorizersPath = "src/test/resources/authorizers.xml"
+ ConfigEncryptionTool tool = new ConfigEncryptionTool()
+
+ // Act
+ flags.each { String arg ->
+ tool.parse([arg, authorizersPath] as String[])
+ logger.info("Parsed authorizers.xml location: ${tool.authorizersPath}")
+
+ // Assert
+ assert tool.authorizersPath == authorizersPath
+ assert tool.handlingAuthorizers
+ }
+ }
+
+ @Test
+ void testShouldParseOutputAuthorizersArgument() {
+ // Arrange
+ def flags = ["-u", "--outputAuthorizers"]
+ String authorizersPath = "src/test/resources/authorizers.xml"
+ ConfigEncryptionTool tool = new ConfigEncryptionTool()
+
+ // Act
+ flags.each { String arg ->
+ tool.parse([arg, authorizersPath, "-a", authorizersPath] as String[])
--- End diff --
Change so the `outputAuthorizersPath` is different from `authorizersPath` (just call `authorizersPath.reverse()`; it doesn't have to be a valid file) to ensure from the equality check at the end that the correct value is being read here.
---
[GitHub] nifi pull request #2350: NIFI-4701 Support encrypted authorizers.xml
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2350#discussion_r158322844
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -1455,25 +1455,27 @@ The default encryption algorithm utilized is AES/GCM 128/256-bit. 128-bit is use
You can use the following command line options with the `encrypt-config` tool:
--- End diff --
The ordering is changed here (to match to ordering of the usage output when running the tool), so it looks like a larger change. There are only two new options:
* `-a`,`--authorizers <arg>` The authorizers.xml file containing unprotected config values (will be overwritten)
* `-u`,`--outputAuthorizers <arg>` The destination authorizers.xml file containing protected config values (will not modify input authorizers.xml)
---
[GitHub] nifi pull request #2350: NIFI-4701 Support encrypted authorizers.xml
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2350#discussion_r159130239
--- Diff: nifi-toolkit/nifi-toolkit-encrypt-config/src/test/groovy/org/apache/nifi/properties/ConfigEncryptionToolTest.groovy ---
@@ -2506,92 +2604,789 @@ class ConfigEncryptionToolTest extends GroovyTestCase {
}
@Test
- void testShouldPerformFullOperationForNiFiPropertiesAndLoginIdentityProviders() {
+ void testShouldDecryptAuthorizers() {
// Arrange
- exit.expectSystemExitWithStatus(0)
+ String authorizersPath = "src/test/resources/authorizers-populated-encrypted.xml"
+ File authorizersFile = new File(authorizersPath)
- File tmpDir = setupTmpDir()
+ setupTmpDir()
- File emptyKeyFile = new File("src/test/resources/bootstrap_with_empty_master_key.conf")
- File bootstrapFile = new File("target/tmp/tmp_bootstrap.conf")
- bootstrapFile.delete()
+ File workingFile = new File("target/tmp/tmp-authorizers.xml")
+ workingFile.delete()
+ Files.copy(authorizersFile.toPath(), workingFile.toPath())
+ ConfigEncryptionTool tool = new ConfigEncryptionTool()
- Files.copy(emptyKeyFile.toPath(), bootstrapFile.toPath())
- final List<String> originalBootstrapLines = bootstrapFile.readLines()
- String originalKeyLine = originalBootstrapLines.find {
- it.startsWith(ConfigEncryptionTool.BOOTSTRAP_KEY_PREFIX)
- }
- logger.info("Original key line from bootstrap.conf: ${originalKeyLine}")
- assert originalKeyLine == ConfigEncryptionTool.BOOTSTRAP_KEY_PREFIX
+ // Sanity check for decryption
+ String cipherText = "q4r7WIgN0MaxdAKM||SGgdCTPGSFEcuH4RraMYEdeyVbOx93abdWTVSWvh1w+klA"
+ String EXPECTED_PASSWORD = "thisIsABadPassword"
+ AESSensitivePropertyProvider spp = new AESSensitivePropertyProvider(KEY_HEX_128)
+ assert spp.unprotect(cipherText) == EXPECTED_PASSWORD
- final String EXPECTED_KEY_LINE = ConfigEncryptionTool.BOOTSTRAP_KEY_PREFIX + KEY_HEX
+ tool.keyHex = KEY_HEX_128
- // Set up the NFP file
- File inputPropertiesFile = new File("src/test/resources/nifi_with_sensitive_properties_unprotected.properties")
- File outputPropertiesFile = new File("target/tmp/tmp_nifi.properties")
- outputPropertiesFile.delete()
+ def lines = workingFile.readLines()
+ logger.info("Read lines: \n${lines.join("\n")}")
- NiFiProperties inputProperties = new NiFiPropertiesLoader().load(inputPropertiesFile)
- logger.info("Loaded ${inputProperties.size()} properties from input file")
- ProtectedNiFiProperties protectedInputProperties = new ProtectedNiFiProperties(inputProperties)
- def originalSensitiveValues = protectedInputProperties.getSensitivePropertyKeys().collectEntries { String key -> [(key): protectedInputProperties.getProperty(key)] }
- logger.info("Original sensitive values: ${originalSensitiveValues}")
+ // Act
+ def decryptedLines = tool.decryptAuthorizers(lines.join("\n")).split("\n")
+ logger.info("Decrypted lines: \n${decryptedLines.join("\n")}")
- // Set up the LIP file
- File inputLIPFile = new File("src/test/resources/login-identity-providers-populated.xml")
- File outputLIPFile = new File("target/tmp/tmp-lip.xml")
- outputLIPFile.delete()
+ // Assert
+ def passwordLines = decryptedLines.findAll { it =~ PASSWORD_PROP_REGEX }
+ assert passwordLines.size() == AUTHORIZERS_PASSWORD_LINE_COUNT
+ assert passwordLines.every { it =~ ">thisIsABadPassword<" }
+ // Some lines were not encrypted originally so the encryption attribute would not have been updated
+ assert passwordLines.any { it =~ "encryption=\"none\"" }
+ }
- String originalXmlContent = inputLIPFile.text
- logger.info("Original XML content: ${originalXmlContent}")
+ @Test
+ void testShouldDecryptAuthorizersWithMultilineElements() {
+ // Arrange
+ String authorizersPath = "src/test/resources/authorizers-populated-encrypted-multiline.xml"
+ File authorizersFile = new File(authorizersPath)
+
+ setupTmpDir()
+
+ File workingFile = new File("target/tmp/tmp-authorizers.xml")
+ workingFile.delete()
+ Files.copy(authorizersFile.toPath(), workingFile.toPath())
+ ConfigEncryptionTool tool = new ConfigEncryptionTool()
+ tool.isVerbose = true
+
+ tool.keyHex = KEY_HEX_128
+
+ def lines = workingFile.readLines()
+ logger.info("Read lines: \n${lines.join("\n")}")
+
+ // Act
+ def decryptedLines = tool.decryptAuthorizers(lines.join("\n")).split("\n")
+ logger.info("Decrypted lines: \n${decryptedLines.join("\n")}")
+
+ // Assert
+ def passwordLines = decryptedLines.findAll { it =~ PASSWORD_PROP_REGEX }
+ assert passwordLines.size() == AUTHORIZERS_PASSWORD_LINE_COUNT
+ assert passwordLines.every { it =~ ">thisIsABadPassword<" }
+ // Some lines were not encrypted originally so the encryption attribute would not have been updated
+ assert passwordLines.any { it =~ "encryption=\"none\"" }
+ }
+
+ @Test
+ void testShouldDecryptAuthorizersWithMultipleElementsPerLine() {
+ // Arrange
+ String authorizersPath = "src/test/resources/authorizers-populated-encrypted-multiple-per-line.xml"
+ File authorizersFile = new File(authorizersPath)
+
+ setupTmpDir()
+
+ File workingFile = new File("target/tmp/tmp-authorizers.xml")
+ workingFile.delete()
+ Files.copy(authorizersFile.toPath(), workingFile.toPath())
+ ConfigEncryptionTool tool = new ConfigEncryptionTool()
+ tool.isVerbose = true
+
+ tool.keyHex = KEY_HEX_128
+
+ def lines = workingFile.readLines()
+ logger.info("Read lines: \n${lines.join("\n")}")
+
+ // Act
+ def decryptedLines = tool.decryptAuthorizers(lines.join("\n")).split("\n")
+ logger.info("Decrypted lines: \n${decryptedLines.join("\n")}")
+
+ // Assert
+ def passwordLines = decryptedLines.findAll { it =~ PASSWORD_PROP_REGEX }
+ assert passwordLines.size() == AUTHORIZERS_PASSWORD_LINE_COUNT
+ assert passwordLines.every { it =~ ">thisIsABadPassword<" }
+ // Some lines were not encrypted originally so the encryption attribute would not have been updated
+ assert passwordLines.any { it =~ "encryption=\"none\"" }
+ }
+
+
+ @Test
+ void testDecryptAuthorizersShouldHandleCommentedElements() {
+ // Arrange
+ String authorizersPath = "src/test/resources/authorizers-commented.xml"
+ File authorizersFile = new File(authorizersPath)
+
+ setupTmpDir()
+
+ File workingFile = new File("target/tmp/tmp-authorizers.xml")
+ workingFile.delete()
+ Files.copy(authorizersFile.toPath(), workingFile.toPath())
+ ConfigEncryptionTool tool = new ConfigEncryptionTool()
+ tool.isVerbose = true
+
+ tool.keyHex = KEY_HEX_128
+
+ def lines = workingFile.readLines()
+ logger.info("Read lines: \n${lines.join("\n")}")
+
+ // Act
+ def decryptedLines = tool.decryptAuthorizers(lines.join("\n")).split("\n")
+ logger.info("Decrypted lines: \n${decryptedLines.join("\n")}")
+
+ // Assert
+
+ // If no encrypted properties are found, the original input text is just returned (comments and formatting in tact)
+ assert decryptedLines == lines
+ }
+
+ @Test
+ void testShouldEncryptAuthorizers() {
+ // Arrange
+ String authorizersPath = "src/test/resources/authorizers-populated.xml"
+ File authorizersFile = new File(authorizersPath)
+
+ setupTmpDir()
+
+ File workingFile = new File("target/tmp/tmp-authorizers.xml")
+ workingFile.delete()
+ Files.copy(authorizersFile.toPath(), workingFile.toPath())
+ ConfigEncryptionTool tool = new ConfigEncryptionTool()
+ tool.isVerbose = true
+
+ tool.keyHex = KEY_HEX
+ String encryptionScheme = "encryption=\"aes/gcm/${getKeyLength(KEY_HEX)}\""
- String[] args = ["-n", inputPropertiesFile.path, "-l", inputLIPFile.path, "-b", bootstrapFile.path, "-i", outputLIPFile.path, "-o", outputPropertiesFile.path, "-k", KEY_HEX, "-v"]
+ def lines = workingFile.readLines()
+ logger.info("Read lines: \n${lines.join("\n")}")
AESSensitivePropertyProvider spp = new AESSensitivePropertyProvider(KEY_HEX)
- exit.checkAssertionAfterwards(new Assertion() {
- public void checkAssertion() {
- final List<String> updatedPropertiesLines = outputPropertiesFile.readLines()
- logger.info("Updated nifi.properties:")
- logger.info("\n" * 2 + updatedPropertiesLines.join("\n"))
+ // Act
+ def encryptedLines = tool.encryptAuthorizers(lines.join("\n")).split("\n")
+ logger.info("Encrypted lines: \n${encryptedLines.join("\n")}")
- // Check that the output values for sensitive properties are not the same as the original (i.e. it was encrypted)
- NiFiProperties updatedProperties = new NiFiPropertiesLoader().readProtectedPropertiesFromDisk(outputPropertiesFile)
- assert updatedProperties.size() >= inputProperties.size()
- originalSensitiveValues.every { String key, String originalValue ->
- assert updatedProperties.getProperty(key) != originalValue
- }
+ // Assert
+ def passwordLines = encryptedLines.findAll { it =~ PASSWORD_PROP_REGEX }
+ assert passwordLines.size() == AUTHORIZERS_PASSWORD_LINE_COUNT
+ assert passwordLines.every { !it.contains(">thisIsABadPassword<") }
+ assert passwordLines.every { it.contains(encryptionScheme) }
+ passwordLines.each {
+ String ct = (it =~ ">(.*)</property>")[0][1]
+ logger.info("Cipher text: ${ct}")
+ assert spp.unprotect(ct) == PASSWORD
+ }
+ }
- // Check that the new NiFiProperties instance matches the output file (values still encrypted)
- updatedProperties.getPropertyKeys().every { String key ->
- assert updatedPropertiesLines.contains("${key}=${updatedProperties.getProperty(key)}".toString())
- }
+ @Test
+ void testShouldEncryptAuthorizersWithEmptySensitiveElements() {
+ // Arrange
+ String authorizersPath = "src/test/resources/authorizers-populated-empty.xml"
+ File authorizersFile = new File(authorizersPath)
- final String updatedXmlContent = outputLIPFile.text
- logger.info("Updated XML content: ${updatedXmlContent}")
+ setupTmpDir()
- // Check that the output values for sensitive properties are not the same as the original (i.e. it was encrypted)
- def originalParsedXml = new XmlSlurper().parseText(originalXmlContent)
- def updatedParsedXml = new XmlSlurper().parseText(updatedXmlContent)
- assert originalParsedXml != updatedParsedXml
- assert originalParsedXml.'**'.findAll { it.@encryption } != updatedParsedXml.'**'.findAll {
- it.@encryption
- }
+ File workingFile = new File("target/tmp/tmp-authorizers.xml")
+ workingFile.delete()
+ Files.copy(authorizersFile.toPath(), workingFile.toPath())
+ ConfigEncryptionTool tool = new ConfigEncryptionTool()
+ tool.isVerbose = true
- def encryptedValues = updatedParsedXml.provider.find {
- it.identifier == 'ldap-provider'
- }.property.findAll {
- it.@name =~ "Password" && it.@encryption =~ "aes/gcm/\\d{3}"
- }
+ tool.keyHex = KEY_HEX
+ String encryptionScheme = "encryption=\"aes/gcm/${getKeyLength(KEY_HEX)}\""
- encryptedValues.each {
- assert spp.unprotect(it.text()) == PASSWORD
- }
+ def lines = workingFile.readLines()
+ logger.info("Read lines: \n${lines.join("\n")}")
+ AESSensitivePropertyProvider spp = new AESSensitivePropertyProvider(KEY_HEX)
+
+ // Act
+ def encryptedLines = tool.encryptAuthorizers(lines.join("\n")).split("\n")
+ logger.info("Encrypted lines: \n${encryptedLines.join("\n")}")
+
+ // Assert
+ def passwordLines = encryptedLines.findAll { it =~ PASSWORD_PROP_REGEX }
+ assert passwordLines.size() == AUTHORIZERS_PASSWORD_LINE_COUNT
+ def populatedPasswordLines = passwordLines.findAll { it =~ />.+</ }
+ assert populatedPasswordLines.every { !it.contains(">thisIsABadPassword<") }
+ assert populatedPasswordLines.every { it.contains(encryptionScheme) }
+ populatedPasswordLines.each {
+ String ct = (it =~ ">(.*)</property>")[0][1]
+ logger.info("Cipher text: ${ct}")
+ assert spp.unprotect(ct) == PASSWORD
+ }
+ }
+
+ @Test
+ void testShouldEncryptAuthorizersWithMultilineElements() {
+ // Arrange
+ String authorizersPath = "src/test/resources/authorizers-populated-multiline.xml"
+ File authorizersFile = new File(authorizersPath)
+
+ setupTmpDir()
+
+ File workingFile = new File("target/tmp/tmp-authorizers.xml")
+ workingFile.delete()
+ Files.copy(authorizersFile.toPath(), workingFile.toPath())
+ ConfigEncryptionTool tool = new ConfigEncryptionTool()
+ tool.isVerbose = true
+
+ tool.keyHex = KEY_HEX
+ String encryptionScheme = "encryption=\"aes/gcm/${getKeyLength(KEY_HEX)}\""
+
+ def lines = workingFile.readLines()
+ logger.info("Read lines: \n${lines.join("\n")}")
+
+ AESSensitivePropertyProvider spp = new AESSensitivePropertyProvider(KEY_HEX)
+
+ // Act
+ def encryptedLines = tool.encryptAuthorizers(lines.join("\n")).split("\n")
+ logger.info("Encrypted lines: \n${encryptedLines.join("\n")}")
+
+ // Assert
+ def passwordLines = encryptedLines.findAll { it =~ PASSWORD_PROP_REGEX }
+ assert passwordLines.size() == AUTHORIZERS_PASSWORD_LINE_COUNT
+ assert passwordLines.every { !it.contains(">thisIsABadPassword<") }
+ assert passwordLines.every { it.contains(encryptionScheme) }
+ passwordLines.each {
+ String ct = (it =~ ">(.*)</property>")[0][1]
+ logger.info("Cipher text: ${ct}")
+ assert spp.unprotect(ct) == PASSWORD
+ }
+ }
+
+ @Test
+ void testShouldEncryptAuthorizersWithMultipleElementsPerLine() {
+ // Arrange
+ String authorizersPath = "src/test/resources/authorizers-populated-multiple-per-line.xml"
+ File authorizersFile = new File(authorizersPath)
+
+ setupTmpDir()
+
+ File workingFile = new File("target/tmp/tmp-authorizers.xml")
+ workingFile.delete()
+ Files.copy(authorizersFile.toPath(), workingFile.toPath())
+ ConfigEncryptionTool tool = new ConfigEncryptionTool()
+ tool.isVerbose = true
+
+ tool.keyHex = KEY_HEX
+ String encryptionScheme = "encryption=\"aes/gcm/${getKeyLength(KEY_HEX)}\""
+
+ def lines = workingFile.readLines()
+ logger.info("Read lines: \n${lines.join("\n")}")
+
+ AESSensitivePropertyProvider spp = new AESSensitivePropertyProvider(KEY_HEX)
+
+ // Act
+ def encryptedLines = tool.encryptAuthorizers(lines.join("\n")).split("\n")
+ logger.info("Encrypted lines: \n${encryptedLines.join("\n")}")
+
+ // Assert
+ def passwordLines = encryptedLines.findAll { it =~ PASSWORD_PROP_REGEX }
+ assert passwordLines.size() == AUTHORIZERS_PASSWORD_LINE_COUNT
+ assert passwordLines.every { !it.contains(">thisIsABadPassword<") }
+ assert passwordLines.every { it.contains(encryptionScheme) }
+ passwordLines.each {
+ String ct = (it =~ ">(.*)</property>")[0][1]
+ logger.info("Cipher text: ${ct}")
+ assert spp.unprotect(ct) == PASSWORD
+ }
+ }
+
+ @Test
+ void testShouldEncryptAuthorizersWithRenamedProvider() {
+ // Arrange
+ String authorizersPath = "src/test/resources/authorizers-populated-renamed.xml"
+ File authorizersFile = new File(authorizersPath)
+
+ setupTmpDir()
+
+ File workingFile = new File("target/tmp/tmp-authorizers.xml")
+ workingFile.delete()
+ Files.copy(authorizersFile.toPath(), workingFile.toPath())
+ ConfigEncryptionTool tool = new ConfigEncryptionTool()
+ tool.isVerbose = true
+
+ tool.keyHex = KEY_HEX
+ String encryptionScheme = "encryption=\"aes/gcm/${getKeyLength(KEY_HEX)}\""
+
+ def lines = workingFile.readLines()
+ logger.info("Read lines: \n${lines.join("\n")}")
+ assert lines.findAll { it =~ "ldap-user-group-provider" }.empty
+
+ AESSensitivePropertyProvider spp = new AESSensitivePropertyProvider(KEY_HEX)
+
+ // Act
+ def encryptedLines = tool.encryptAuthorizers(lines.join("\n")).split("\n")
+ logger.info("Encrypted lines: \n${encryptedLines.join("\n")}")
+
+ // Assert
+ def passwordLines = encryptedLines.findAll { it =~ PASSWORD_PROP_REGEX }
+ assert passwordLines.size() == AUTHORIZERS_PASSWORD_LINE_COUNT
+ def populatedPasswordLines = passwordLines.findAll { it =~ />.+</ }
+ assert populatedPasswordLines.every { !it.contains(">thisIsABadPassword<") }
+ assert populatedPasswordLines.every { it.contains(encryptionScheme) }
+ populatedPasswordLines.each {
+ String ct = (it =~ ">(.*)</property>")[0][1]
+ logger.info("Cipher text: ${ct}")
+ assert spp.unprotect(ct) == PASSWORD
+ }
+ }
+
+ @Test
+ void testEncryptAuthorizersShouldHandleCommentedElements() {
+ // Arrange
+ String authorizersPath = "src/test/resources/authorizers-commented.xml"
+ File authorizersFile = new File(authorizersPath)
+
+ setupTmpDir()
+
+ File workingFile = new File("target/tmp/tmp-authorizers.xml")
+ workingFile.delete()
+ Files.copy(authorizersFile.toPath(), workingFile.toPath())
+ ConfigEncryptionTool tool = new ConfigEncryptionTool()
+ tool.isVerbose = true
+
+ tool.keyHex = KEY_HEX_128
+
+ def lines = workingFile.readLines()
+ logger.info("Read lines: \n${lines.join("\n")}")
+
+ // Act
+ def encryptedLines = tool.encryptAuthorizers(lines.join("\n")).split("\n")
+ logger.info("Encrypted lines: \n${encryptedLines.join("\n")}")
+
+ // Assert
+
+ // If no sensitive properties are found, the original input text is just returned (comments and formatting in tact)
+ assert encryptedLines == lines
+ }
+
+ @Test
+ void testSerializeAuthorizersAndPreserveFormatShouldRespectComments() {
+ // Arrange
+ String authorizersPath = "src/test/resources/authorizers-populated.xml"
+ File authorizersFile = new File(authorizersPath)
+
+ setupTmpDir()
+
+ File workingFile = new File("target/tmp/tmp-authorizers.xml")
+ workingFile.delete()
+ Files.copy(authorizersFile.toPath(), workingFile.toPath())
+ ConfigEncryptionTool tool = new ConfigEncryptionTool()
+ tool.isVerbose = true
+
+ def lines = workingFile.readLines()
+ logger.info("Read lines: \n${lines.join("\n")}")
+
+ String plainXml = workingFile.text
+ String encryptedXml = tool.encryptAuthorizers(plainXml, KEY_HEX)
+ logger.info("Encrypted XML: \n${encryptedXml}")
+
+ // Act
+ def serializedLines = tool.serializeAuthorizersAndPreserveFormat(encryptedXml, workingFile)
+ logger.info("Serialized lines: \n${serializedLines.join("\n")}")
+
+ // Assert
+
+ // Some empty lines will be removed
+ def trimmedLines = lines.collect { it.trim() }.findAll { it }
+ def trimmedSerializedLines = serializedLines.collect { it.trim() }.findAll { it }
+ assert trimmedLines.size() == trimmedSerializedLines.size()
+
+ // Ensure the replacement actually occurred
+ assert trimmedSerializedLines.findAll { it =~ "encryption=" }.size() == AUTHORIZERS_PASSWORD_LINE_COUNT
+ }
+
+ @Test
+ void testSerializeAuthorizersAndPreserveFormatShouldHandleRenamedProvider() {
+ // Arrange
+ String authorizersPath = "src/test/resources/authorizers-populated-renamed.xml"
+ File authorizersFile = new File(authorizersPath)
+
+ setupTmpDir()
+
+ File workingFile = new File("target/tmp/tmp-authorizers.xml")
+ workingFile.delete()
+ Files.copy(authorizersFile.toPath(), workingFile.toPath())
+ ConfigEncryptionTool tool = new ConfigEncryptionTool()
+ tool.isVerbose = true
+
+ def lines = workingFile.readLines()
+ logger.info("Read lines: \n${lines.join("\n")}")
+ assert lines.findAll { it =~ "ldap-user-group-provider" }.empty
+
+ String plainXml = workingFile.text
+ String encryptedXml = tool.encryptAuthorizers(plainXml, KEY_HEX)
+ logger.info("Encrypted XML: \n${encryptedXml}")
+
+ // Act
+ def serializedLines = tool.serializeAuthorizersAndPreserveFormat(encryptedXml, workingFile)
+ logger.info("Serialized lines: \n${serializedLines.join("\n")}")
+
+ // Assert
+
+ // Some empty lines will be removed
+ def trimmedLines = lines.collect { it.trim() }.findAll { it }
+ def trimmedSerializedLines = serializedLines.collect { it.trim() }.findAll { it }
+ assert trimmedLines.size() == trimmedSerializedLines.size()
+
+ // Ensure the replacement actually occurred
+ assert trimmedSerializedLines.findAll { it =~ "encryption=" }.size() == AUTHORIZERS_PASSWORD_LINE_COUNT
+ }
+
+ @Test
+ void testSerializeAuthorizersAndPreserveFormatShouldHandleCommentedFile() {
+ // Arrange
+ String authorizersPath = "src/test/resources/authorizers-commented.xml"
+ File authorizersFile = new File(authorizersPath)
+
+ File tmpDir = setupTmpDir()
+
+ File workingFile = new File("target/tmp/tmp-authorizers.xml")
+ workingFile.delete()
+ Files.copy(authorizersFile.toPath(), workingFile.toPath())
+ ConfigEncryptionTool tool = new ConfigEncryptionTool()
+ tool.isVerbose = true
+
+ tool.keyHex = KEY_HEX_128
--- End diff --
This should just be `KEY_HEX` so the JCE determines which key length to use.
---
[GitHub] nifi pull request #2350: NIFI-4701 Support encrypted authorizers.xml
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2350#discussion_r159102075
--- Diff: nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/properties/ConfigEncryptionTool.groovy ---
@@ -772,6 +899,48 @@ class ConfigEncryptionTool {
}
}
+ String encryptAuthorizers(String plainXml, String newKeyHex = keyHex) {
+ AESSensitivePropertyProvider sensitivePropertyProvider = new AESSensitivePropertyProvider(newKeyHex)
+
+ // TODO: Switch to XmlParser & XmlNodePrinter to maintain "empty" element structure
+ try {
+ def doc = new XmlSlurper().parseText(plainXml)
+ // Find the provider element by class even if it has been renamed
+ def passwords = doc.userGroupProvider.find { it.'class' as String == LDAP_USER_GROUP_PROVIDER_CLASS }
+ .property.findAll {
+ // Only operate on un-encrypted passwords
+ it.@name =~ "Password" && (it.@encryption == "none" || it.@encryption == "") && it.text()
+ }
+
+ if (passwords.isEmpty()) {
+ if (isVerbose) {
+ logger.info("No unencrypted password property elements found in login-identity-providers.xml")
+ }
+ return plainXml
+ }
+
+ passwords.each { password ->
+ if (isVerbose) {
+ logger.info("Attempting to encrypt ${password.name()}")
+ }
+ String encryptedValue = sensitivePropertyProvider.protect(password.text().trim())
+ password.replaceNode {
+ property(name: password.@name, encryption: sensitivePropertyProvider.identifierKey, encryptedValue)
+ }
+ }
+
+ // Does not preserve whitespace formatting or comments
+ String updatedXml = XmlUtil.serialize(doc)
+ logger.info("Updated XML content: ${updatedXml}")
+ updatedXml
+ } catch (Exception e) {
+ if (isVerbose) {
+ logger.error("Encountered exception", e)
+ }
+ printUsageAndThrow("Cannot encrypt login identity providers XML content", ExitCode.SERVICE_ERROR)
--- End diff --
This message should also be updated to `authorizers.xml`.
---
[GitHub] nifi pull request #2350: NIFI-4701 Support encrypted authorizers.xml
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2350#discussion_r159102020
--- Diff: nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/properties/ConfigEncryptionTool.groovy ---
@@ -772,6 +899,48 @@ class ConfigEncryptionTool {
}
}
+ String encryptAuthorizers(String plainXml, String newKeyHex = keyHex) {
+ AESSensitivePropertyProvider sensitivePropertyProvider = new AESSensitivePropertyProvider(newKeyHex)
+
+ // TODO: Switch to XmlParser & XmlNodePrinter to maintain "empty" element structure
+ try {
+ def doc = new XmlSlurper().parseText(plainXml)
+ // Find the provider element by class even if it has been renamed
+ def passwords = doc.userGroupProvider.find { it.'class' as String == LDAP_USER_GROUP_PROVIDER_CLASS }
+ .property.findAll {
+ // Only operate on un-encrypted passwords
+ it.@name =~ "Password" && (it.@encryption == "none" || it.@encryption == "") && it.text()
+ }
+
+ if (passwords.isEmpty()) {
+ if (isVerbose) {
+ logger.info("No unencrypted password property elements found in login-identity-providers.xml")
--- End diff --
The message should be updated to `authorizers.xml`.
---