You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by GitBox <gi...@apache.org> on 2020/03/26 18:11:36 UTC
[GitHub] [nifi] thenatog opened a new pull request #4164: NIFI-5481 - Add
new providers of protected sensitive configuration values
thenatog opened a new pull request #4164: NIFI-5481 - Add new providers of protected sensitive configuration values
URL: https://github.com/apache/nifi/pull/4164
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
#### Description of PR
_Enables X functionality; fixes bug NIFI-YYYY._
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [x] Does your PR title start with **NIFI-XXXX** where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target branch (typically `master`)?
- [x] Is your initial contribution a single, squashed commit? _Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not `squash` or use `--force` when pushing to allow for clean monitoring of changes._
### For code changes:
- [x] Have you ensured that the full suite of tests is executed via `mvn -Pcontrib-check clean install` at the root `nifi` folder?
- [x] Have you written or updated unit tests to verify your changes?
- [x] Have you verified that the full build is successful on both JDK 8 and JDK 11?
- [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [] If applicable, have you updated the `LICENSE` file, including the main `LICENSE` file under `nifi-assembly`?
- [] If applicable, have you updated the `NOTICE` file, including the main `NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to .name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] alopresto commented on issue #4164: NIFI-5481 - Add new
providers of protected sensitive configuration values
Posted by GitBox <gi...@apache.org>.
alopresto commented on issue #4164: NIFI-5481 - Add new providers of protected sensitive configuration values
URL: https://github.com/apache/nifi/pull/4164#issuecomment-604608401
This effort is complicated and has been on-going for some time. Please clearly state what elements of the effort are being addressed explicitly with this PR. Thank you.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] bahlulh commented on a change in pull request #4164:
NIFI-5481 - Add new providers of protected sensitive configuration values
Posted by GitBox <gi...@apache.org>.
bahlulh commented on a change in pull request #4164: NIFI-5481 - Add new providers of protected sensitive configuration values
URL: https://github.com/apache/nifi/pull/4164#discussion_r404906870
##########
File path: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/java/org/apache/nifi/properties/sensitive/ProtectedNiFiProperties.java
##########
@@ -359,40 +373,22 @@ public NiFiProperties getUnprotectedProperties() throws SensitivePropertyProtect
}
}
- NiFiProperties unprotected = new StandardNiFiProperties(rawProperties);
-
- return unprotected;
+ return new StandardNiFiProperties(rawProperties);
} else {
logger.debug("No protected properties");
return getInternalNiFiProperties();
}
}
- /**
- * Registers a new {@link SensitivePropertyProvider}. This method will throw a {@link UnsupportedOperationException} if a provider is already registered for the protection scheme.
- *
- * @param sensitivePropertyProvider the provider
- */
- void addSensitivePropertyProvider(SensitivePropertyProvider sensitivePropertyProvider) {
- if (sensitivePropertyProvider == null) {
- throw new IllegalArgumentException("Cannot add null SensitivePropertyProvider");
- }
-
- if (getSensitivePropertyProviders().containsKey(sensitivePropertyProvider.getIdentifierKey())) {
- throw new UnsupportedOperationException("Cannot overwrite existing sensitive property provider registered for " + sensitivePropertyProvider.getIdentifierKey());
- }
-
- getSensitivePropertyProviders().put(sensitivePropertyProvider.getIdentifierKey(), sensitivePropertyProvider);
- }
-
- private String getDefaultProtectionScheme() {
- if (!getSensitivePropertyProviders().isEmpty()) {
- List<String> schemes = new ArrayList<>(getSensitivePropertyProviders().keySet());
- Collections.sort(schemes);
- return schemes.get(0);
- } else {
- throw new IllegalStateException("No registered protection schemes");
- }
+ @Override
+ public String toString() {
Review comment:
Why do you need this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] natural commented on a change in pull request #4164:
NIFI-5481 - Add new providers of protected sensitive configuration values
Posted by GitBox <gi...@apache.org>.
natural commented on a change in pull request #4164: NIFI-5481 - Add new providers of protected sensitive configuration values
URL: https://github.com/apache/nifi/pull/4164#discussion_r409985312
##########
File path: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/java/org/apache/nifi/properties/sensitive/property/provider/keystore/KeyStoreWrappedSensitivePropertyProvider.java
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.properties.sensitive.property.provider.keystore;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.properties.sensitive.ExternalProperties;
+import org.apache.nifi.properties.sensitive.property.provider.aes.AESSensitivePropertyProvider;
+import org.apache.nifi.properties.sensitive.SensitivePropertyConfigurationException;
+import org.apache.nifi.properties.sensitive.SensitivePropertyProtectionException;
+import org.apache.nifi.properties.sensitive.SensitivePropertyProvider;
+import org.apache.nifi.properties.sensitive.StandardExternalPropertyLookup;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.security.Key;
+import java.security.KeyStore;
+import java.security.KeyStoreException;
+import java.security.NoSuchAlgorithmException;
+import java.security.UnrecoverableKeyException;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Sensitive properties using KeyStore keys with an inner AES SPP.
+ */
+public class KeyStoreWrappedSensitivePropertyProvider implements SensitivePropertyProvider {
+ private static final Logger logger = LoggerFactory.getLogger(KeyStoreWrappedSensitivePropertyProvider.class);
+
+ private static final String PROVIDER_NAME = "KeyStore Sensitive Property Provider";
+ private static final String MATERIAL_PREFIX = "keystore";
+ private static final String MATERIAL_SEPARATOR = "/";
+
+ public static final String KEYSTORE_TYPE_JCECKS = "jceks";
+ private static final String KEYSTORE_TYPE_PKCS12 = "pkcs12";
+ private static final String KEYSTORE_TYPE_BKS = "bks";
+
+ static final Set<String> KEYSTORE_TYPES = new HashSet<>(Arrays.asList(
+ KEYSTORE_TYPE_JCECKS,
+ KEYSTORE_TYPE_PKCS12,
+ KEYSTORE_TYPE_BKS));
+
+ private final ExternalProperties externalProperties;
+ private final SensitivePropertyProvider wrappedSensitivePropertyProvider;
+ private final String storeType;
+ private final String keyAlias;
+
+ /**
+ * Constructor, as expected by the standard sensitive property provider implementation.
+ *
+ * @param keyId string in the form "keystore/jcecks/user-key-alias"
+ */
+ public KeyStoreWrappedSensitivePropertyProvider(String keyId) {
+ this(keyId, null, null);
+ }
+
+ public KeyStoreWrappedSensitivePropertyProvider(String keyId, KeyStoreProvider keyStoreProvider, ExternalProperties externalProperties) {
+ if (externalProperties == null) {
+ externalProperties = new StandardExternalPropertyLookup(null, getKeyStorePropertiesMapping() );
+ }
+ this.externalProperties = externalProperties;
+
+ if (StringUtils.isBlank(keyId))
+ throw new SensitivePropertyConfigurationException("The key cannot be empty");
+
+ String storeType;
+ String keyAlias;
+ try {
+ String[] parts = keyId.split(MATERIAL_SEPARATOR);
+ storeType = parts.length > 0 ? parts[1] : "";
+ keyAlias = parts.length > 1 ? parts[2] : "";
+ } catch (final ArrayIndexOutOfBoundsException e) {
+ throw new SensitivePropertyConfigurationException("Invalid Key Store key", e);
+ }
+
+ this.storeType = storeType;
+ this.keyAlias = keyAlias;
+
+ if (keyStoreProvider == null ){
+ keyStoreProvider = new StandardKeyStoreProvider(getStoreUri(), this.storeType, getStorePassword());
+ }
+
+ try {
+ KeyStore store = keyStoreProvider.getKeyStore();
+ Key secretKey = store.getKey(keyAlias, getKeyPassword().toCharArray());
+ this.wrappedSensitivePropertyProvider = new AESSensitivePropertyProvider(secretKey.getEncoded());
+ } catch (final IOException | NoSuchAlgorithmException | UnrecoverableKeyException | KeyStoreException e) {
+ throw new SensitivePropertyConfigurationException(e);
+ }
+ }
+
+ private Map<String, String> getKeyStorePropertiesMapping() {
+ Map<String, String> map = new HashMap<>();
+ map.put("keystore.key-password", "KEYSTORE_KEY_PASSWORD");
+ map.put("keystore.password", "KEYSTORE_PASSWORD");
+ map.put("keystore.file", "KEYSTORE_FILE");
+ return map;
+ }
+
+ private String getKeyPassword() {
+ return externalProperties.get("keystore.key-password", "");
+ }
+
+ private String getStorePassword() {
+ return externalProperties.get("keystore.password", "");
+ }
+
+ private String getStoreUri() {
+ return externalProperties.get("keystore.file", "");
+ }
+
+ public static String formatForType(String storeType, String keyAlias) {
+ return MATERIAL_PREFIX + MATERIAL_SEPARATOR + storeType + MATERIAL_SEPARATOR + keyAlias;
+ }
+
+ /**
+ * Returns the name of the underlying implementation.
+ *
+ * @return the name of this sensitive property provider
+ */
+ @Override
+ public String getName() {
+ return PROVIDER_NAME;
+ }
+
+ /**
+ * Returns the key used to identify the provider implementation in {@code nifi.properties}.
+ *
+ * @return the key to persist in the sibling property
+ */
+ @Override
+ public String getIdentifierKey() {
+ return MATERIAL_PREFIX + MATERIAL_SEPARATOR + storeType + MATERIAL_SEPARATOR + keyAlias;
+ }
+
+ /**
+ * Returns the "protected" form of this value. This is a form which can safely be persisted in the {@code nifi.properties} file without compromising the value.
+ * An encryption-based provider would return a cipher text, while a remote-lookup provider could return a unique ID to retrieve the secured value.
+ *
+ * @param unprotectedValue the sensitive value
+ * @return the value to persist in the {@code nifi.properties} file
+ */
+ @Override
+ public String protect(String unprotectedValue) throws SensitivePropertyProtectionException {
+ return wrappedSensitivePropertyProvider.protect(unprotectedValue);
+ }
+
+ /**
+ * Returns the "unprotected" form of this value. This is the raw sensitive value which is used by the application logic.
+ * An encryption-based provider would decrypt a cipher text and return the plaintext, while a remote-lookup provider could retrieve the secured value.
+ *
+ * @param protectedValue the protected value read from the {@code nifi.properties} file
+ * @return the raw value to be used by the application
+ */
+ @Override
+ public String unprotect(String protectedValue) throws SensitivePropertyProtectionException {
+ try {
+ return wrappedSensitivePropertyProvider.unprotect(protectedValue);
+ } catch (final IllegalArgumentException e) {
+ throw new SensitivePropertyProtectionException(e);
+ }
+ }
+
+ /**
+ * True when the client specifies a key like 'keystore/pkcs12/...'.
+ *
+ * @param material name of encryption or protection scheme
+ * @return true if this class can provide protected values
+ */
+ public static boolean isProviderFor(String material) {
+ if (StringUtils.isBlank(material)) {
+ return false;
+ }
+ String[] parts = material.split(MATERIAL_SEPARATOR, 3);
+ return parts.length == 3 && parts[0].toLowerCase().equals(MATERIAL_PREFIX) && KEYSTORE_TYPES.contains(parts[1].toLowerCase());
+ }
+
+ /**
+ * Returns a printable representation of a key.
+ *
+ * @param key key material or key id
+ * @return printable string
+ */
+ public static String toPrintableString(String key) {
Review comment:
Originally, it was for printing sensitive values contained in keys. I'm not sure if it's currently used.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] bahlulh commented on issue #4164: NIFI-5481 - Add new
providers of protected sensitive configuration values
Posted by GitBox <gi...@apache.org>.
bahlulh commented on issue #4164: NIFI-5481 - Add new providers of protected sensitive configuration values
URL: https://github.com/apache/nifi/pull/4164#issuecomment-614284723
> @bahlulh Thanks for the review Bahlul, the Azure sensitive property provider is on its way (it's already written). https://issues.apache.org/jira/browse/NIFI-6615. We decided to add these sensitive property providers in stages as the original PR to add all providers was even bigger than this one. You can check it out here: https://github.com/apache/nifi/pull/4080/files. You should see the AzureKeyVaultSensitivePropertyProvider.java.
>
> Once this PR is merged, the next PR will add a new NiFi maven module which will eventually contain the other sensitive property providers, including the AKV one. You can help us out by testing the AKV provider once that PR is up.
Thank you @thenatog. I am very interested to test the AKV when the PR is ready.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] bahlulh commented on a change in pull request #4164:
NIFI-5481 - Add new providers of protected sensitive configuration values
Posted by GitBox <gi...@apache.org>.
bahlulh commented on a change in pull request #4164: NIFI-5481 - Add new providers of protected sensitive configuration values
URL: https://github.com/apache/nifi/pull/4164#discussion_r404915334
##########
File path: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/java/org/apache/nifi/properties/sensitive/StandardSensitivePropertyProvider.java
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.properties.sensitive;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.properties.sensitive.property.provider.aes.AESSensitivePropertyProvider;
+import org.apache.nifi.properties.sensitive.property.provider.keystore.KeyStoreWrappedSensitivePropertyProvider;
+import org.bouncycastle.jce.provider.BouncyCastleProvider;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.security.Security;
+
+/**
+ *
+ * This class describes the available sensitive property providers {@link SensitivePropertyProvider} which can be chosen to encrypt/decrypt the sensitive properties in the nifi.properties file.
+ * Most of these providers will require external setup in their respective system (eg. Hashicorp Vault) and then configuration given in the nifi.properties file.
+ *
+ * For further details, review the <a href="https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#sensitive_property_providers">Apache NiFi User Guide - Sensitive Property Providers</a> section.
+ *
+ * As of Apache NiFi 1.12.0 this implementation is considered <a href="https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#experimental_warning">*experimental*</a>.
+ * Available implementations can be found listed in StandardSensitivePropertyProvider.
+ *
+ * This class hides the various SPP subclass construction from clients.
+ *
+ */
+public class StandardSensitivePropertyProvider {
+ private static final Logger logger = LoggerFactory.getLogger(StandardSensitivePropertyProvider.class);
+
+ static {
+ Security.addProvider(new BouncyCastleProvider());
+ }
+
+ /**
+ * Creates a {@link SensitivePropertyProvider} suitable for a given key or key id.
+ *
+ * If an empty or null key/key id is given, this implementation returns null. This is a convenience
+ * for clients using the various Property classes, as those classes allow a null SensitivePropertyProvider.
+ *
+ * If no provider recognizes a key/key id, this implementation throws {@link SensitivePropertyProtectionException}.
+ *
+ * @param key provider encryption key
+ * @return concrete instance of SensitivePropertyProvider, or null when no key/key id is specified
+ * @throws SensitivePropertyProtectionException when a key/key id is not handled by any provider.
+ */
+ public static SensitivePropertyProvider fromKey(String key) {
+ if (StringUtils.isBlank(key)) {
+ return null;
+ } else if (KeyStoreWrappedSensitivePropertyProvider.isProviderFor(key)) {
Review comment:
Can KeyStoreWrappedSensitivePropertyProvider and AESSensitivePropertyProvider be merged into one class?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] thenatog commented on issue #4164: NIFI-5481 - Add new
providers of protected sensitive configuration values
Posted by GitBox <gi...@apache.org>.
thenatog commented on issue #4164: NIFI-5481 - Add new providers of protected sensitive configuration values
URL: https://github.com/apache/nifi/pull/4164#issuecomment-604713583
This PR refactors the Sensitive Property Providers that are used to encrypt the NiFi configuration files as the first step towards adding other KMS providers in future. The future additions can be external Key Management Service (KMS) providers such as cloud services like GCP KMS, Azure Key Vault, AWS KMS, Hashicorp Vault etc by implementing the SensitivePropertyProvider interface. This will allow NiFi to retrieve and use keys from these providers to encrypt/decrypt sensitive properties (passwords) in the nifi.properties, loginIdentityProviders, authorizers files. At the moment, this PR continues to use the original (historical default) AES SPP as the default provider, and also allows for a local keystore file provider with the KeyStoreWrappedSensitivePropertyProvider.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] natural commented on a change in pull request #4164:
NIFI-5481 - Add new providers of protected sensitive configuration values
Posted by GitBox <gi...@apache.org>.
natural commented on a change in pull request #4164: NIFI-5481 - Add new providers of protected sensitive configuration values
URL: https://github.com/apache/nifi/pull/4164#discussion_r409985580
##########
File path: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/java/org/apache/nifi/properties/sensitive/ProtectedNiFiProperties.java
##########
@@ -359,40 +373,22 @@ public NiFiProperties getUnprotectedProperties() throws SensitivePropertyProtect
}
}
- NiFiProperties unprotected = new StandardNiFiProperties(rawProperties);
-
- return unprotected;
+ return new StandardNiFiProperties(rawProperties);
} else {
logger.debug("No protected properties");
return getInternalNiFiProperties();
}
}
- /**
- * Registers a new {@link SensitivePropertyProvider}. This method will throw a {@link UnsupportedOperationException} if a provider is already registered for the protection scheme.
- *
- * @param sensitivePropertyProvider the provider
- */
- void addSensitivePropertyProvider(SensitivePropertyProvider sensitivePropertyProvider) {
- if (sensitivePropertyProvider == null) {
- throw new IllegalArgumentException("Cannot add null SensitivePropertyProvider");
- }
-
- if (getSensitivePropertyProviders().containsKey(sensitivePropertyProvider.getIdentifierKey())) {
- throw new UnsupportedOperationException("Cannot overwrite existing sensitive property provider registered for " + sensitivePropertyProvider.getIdentifierKey());
- }
-
- getSensitivePropertyProviders().put(sensitivePropertyProvider.getIdentifierKey(), sensitivePropertyProvider);
- }
-
- private String getDefaultProtectionScheme() {
- if (!getSensitivePropertyProviders().isEmpty()) {
- List<String> schemes = new ArrayList<>(getSensitivePropertyProviders().keySet());
- Collections.sort(schemes);
- return schemes.get(0);
- } else {
- throw new IllegalStateException("No registered protection schemes");
- }
+ @Override
+ public String toString() {
Review comment:
Super helpful in logs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] natural commented on a change in pull request #4164:
NIFI-5481 - Add new providers of protected sensitive configuration values
Posted by GitBox <gi...@apache.org>.
natural commented on a change in pull request #4164: NIFI-5481 - Add new providers of protected sensitive configuration values
URL: https://github.com/apache/nifi/pull/4164#discussion_r409984829
##########
File path: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/java/org/apache/nifi/properties/sensitive/property/provider/keystore/KeyStoreWrappedSensitivePropertyProvider.java
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.properties.sensitive.property.provider.keystore;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.properties.sensitive.ExternalProperties;
+import org.apache.nifi.properties.sensitive.property.provider.aes.AESSensitivePropertyProvider;
+import org.apache.nifi.properties.sensitive.SensitivePropertyConfigurationException;
+import org.apache.nifi.properties.sensitive.SensitivePropertyProtectionException;
+import org.apache.nifi.properties.sensitive.SensitivePropertyProvider;
+import org.apache.nifi.properties.sensitive.StandardExternalPropertyLookup;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.security.Key;
+import java.security.KeyStore;
+import java.security.KeyStoreException;
+import java.security.NoSuchAlgorithmException;
+import java.security.UnrecoverableKeyException;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Sensitive properties using KeyStore keys with an inner AES SPP.
+ */
+public class KeyStoreWrappedSensitivePropertyProvider implements SensitivePropertyProvider {
+ private static final Logger logger = LoggerFactory.getLogger(KeyStoreWrappedSensitivePropertyProvider.class);
+
+ private static final String PROVIDER_NAME = "KeyStore Sensitive Property Provider";
+ private static final String MATERIAL_PREFIX = "keystore";
+ private static final String MATERIAL_SEPARATOR = "/";
+
+ public static final String KEYSTORE_TYPE_JCECKS = "jceks";
+ private static final String KEYSTORE_TYPE_PKCS12 = "pkcs12";
+ private static final String KEYSTORE_TYPE_BKS = "bks";
+
+ static final Set<String> KEYSTORE_TYPES = new HashSet<>(Arrays.asList(
+ KEYSTORE_TYPE_JCECKS,
+ KEYSTORE_TYPE_PKCS12,
+ KEYSTORE_TYPE_BKS));
+
+ private final ExternalProperties externalProperties;
+ private final SensitivePropertyProvider wrappedSensitivePropertyProvider;
+ private final String storeType;
+ private final String keyAlias;
+
+ /**
+ * Constructor, as expected by the standard sensitive property provider implementation.
+ *
+ * @param keyId string in the form "keystore/jcecks/user-key-alias"
+ */
+ public KeyStoreWrappedSensitivePropertyProvider(String keyId) {
+ this(keyId, null, null);
+ }
+
+ public KeyStoreWrappedSensitivePropertyProvider(String keyId, KeyStoreProvider keyStoreProvider, ExternalProperties externalProperties) {
+ if (externalProperties == null) {
+ externalProperties = new StandardExternalPropertyLookup(null, getKeyStorePropertiesMapping() );
+ }
+ this.externalProperties = externalProperties;
+
+ if (StringUtils.isBlank(keyId))
+ throw new SensitivePropertyConfigurationException("The key cannot be empty");
+
+ String storeType;
+ String keyAlias;
+ try {
+ String[] parts = keyId.split(MATERIAL_SEPARATOR);
+ storeType = parts.length > 0 ? parts[1] : "";
+ keyAlias = parts.length > 1 ? parts[2] : "";
+ } catch (final ArrayIndexOutOfBoundsException e) {
+ throw new SensitivePropertyConfigurationException("Invalid Key Store key", e);
+ }
+
+ this.storeType = storeType;
+ this.keyAlias = keyAlias;
+
+ if (keyStoreProvider == null ){
+ keyStoreProvider = new StandardKeyStoreProvider(getStoreUri(), this.storeType, getStorePassword());
+ }
+
+ try {
+ KeyStore store = keyStoreProvider.getKeyStore();
+ Key secretKey = store.getKey(keyAlias, getKeyPassword().toCharArray());
+ this.wrappedSensitivePropertyProvider = new AESSensitivePropertyProvider(secretKey.getEncoded());
+ } catch (final IOException | NoSuchAlgorithmException | UnrecoverableKeyException | KeyStoreException e) {
+ throw new SensitivePropertyConfigurationException(e);
+ }
+ }
+
+ private Map<String, String> getKeyStorePropertiesMapping() {
+ Map<String, String> map = new HashMap<>();
+ map.put("keystore.key-password", "KEYSTORE_KEY_PASSWORD");
+ map.put("keystore.password", "KEYSTORE_PASSWORD");
+ map.put("keystore.file", "KEYSTORE_FILE");
+ return map;
+ }
+
+ private String getKeyPassword() {
+ return externalProperties.get("keystore.key-password", "");
+ }
+
+ private String getStorePassword() {
+ return externalProperties.get("keystore.password", "");
+ }
+
+ private String getStoreUri() {
+ return externalProperties.get("keystore.file", "");
+ }
+
+ public static String formatForType(String storeType, String keyAlias) {
+ return MATERIAL_PREFIX + MATERIAL_SEPARATOR + storeType + MATERIAL_SEPARATOR + keyAlias;
+ }
+
+ /**
+ * Returns the name of the underlying implementation.
+ *
+ * @return the name of this sensitive property provider
+ */
+ @Override
+ public String getName() {
+ return PROVIDER_NAME;
+ }
+
+ /**
+ * Returns the key used to identify the provider implementation in {@code nifi.properties}.
+ *
+ * @return the key to persist in the sibling property
+ */
+ @Override
+ public String getIdentifierKey() {
+ return MATERIAL_PREFIX + MATERIAL_SEPARATOR + storeType + MATERIAL_SEPARATOR + keyAlias;
+ }
+
+ /**
+ * Returns the "protected" form of this value. This is a form which can safely be persisted in the {@code nifi.properties} file without compromising the value.
+ * An encryption-based provider would return a cipher text, while a remote-lookup provider could return a unique ID to retrieve the secured value.
+ *
+ * @param unprotectedValue the sensitive value
+ * @return the value to persist in the {@code nifi.properties} file
+ */
+ @Override
+ public String protect(String unprotectedValue) throws SensitivePropertyProtectionException {
+ return wrappedSensitivePropertyProvider.protect(unprotectedValue);
+ }
+
+ /**
+ * Returns the "unprotected" form of this value. This is the raw sensitive value which is used by the application logic.
+ * An encryption-based provider would decrypt a cipher text and return the plaintext, while a remote-lookup provider could retrieve the secured value.
+ *
+ * @param protectedValue the protected value read from the {@code nifi.properties} file
+ * @return the raw value to be used by the application
+ */
+ @Override
+ public String unprotect(String protectedValue) throws SensitivePropertyProtectionException {
+ try {
+ return wrappedSensitivePropertyProvider.unprotect(protectedValue);
+ } catch (final IllegalArgumentException e) {
+ throw new SensitivePropertyProtectionException(e);
+ }
+ }
+
+ /**
+ * True when the client specifies a key like 'keystore/pkcs12/...'.
+ *
+ * @param material name of encryption or protection scheme
+ * @return true if this class can provide protected values
+ */
+ public static boolean isProviderFor(String material) {
+ if (StringUtils.isBlank(material)) {
+ return false;
+ }
+ String[] parts = material.split(MATERIAL_SEPARATOR, 3);
+ return parts.length == 3 && parts[0].toLowerCase().equals(MATERIAL_PREFIX) && KEYSTORE_TYPES.contains(parts[1].toLowerCase());
Review comment:
Using `contains` because we're checking for set membership as a way to validate.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] thenatog commented on issue #4164: NIFI-5481 - Add new
providers of protected sensitive configuration values
Posted by GitBox <gi...@apache.org>.
thenatog commented on issue #4164: NIFI-5481 - Add new providers of protected sensitive configuration values
URL: https://github.com/apache/nifi/pull/4164#issuecomment-614185585
@bahlulh Thanks for the review Bahlul, the Azure sensitive property provider is on its way (it's already written). https://issues.apache.org/jira/browse/NIFI-6615. We decided to add these sensitive property providers in stages as the original PR to add all providers was even bigger than this one. You can check it out here: https://github.com/apache/nifi/pull/4080/files.
Once this PR is merged, the next PR will add a new NiFi maven module which will eventually contain the other sensitive property providers, including the AKV one. You can help us out by testing the AKV provider once that PR is up.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] natural removed a comment on issue #4164: NIFI-5481 - Add
new providers of protected sensitive configuration values
Posted by GitBox <gi...@apache.org>.
natural removed a comment on issue #4164: NIFI-5481 - Add new providers of protected sensitive configuration values
URL: https://github.com/apache/nifi/pull/4164#issuecomment-615032288
> Hi @thenatog,
> I am planning to add an Azure Key Vault (AKV) secret provider to Nifi. Can you please give me some idea on how it will fit with this change? If you are available in nifi slack channel, we can also discuss it there.
Hi @bahlulh
There's an Azure provider already written as part of this effort, it's just not in the PR. We're trying to split the changes into this PR (refactoring) and follow-on PRs for the various third-party providers.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] thenatog commented on a change in pull request #4164:
NIFI-5481 - Add new providers of protected sensitive configuration values
Posted by GitBox <gi...@apache.org>.
thenatog commented on a change in pull request #4164: NIFI-5481 - Add new providers of protected sensitive configuration values
URL: https://github.com/apache/nifi/pull/4164#discussion_r409029591
##########
File path: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/java/org/apache/nifi/properties/sensitive/StandardSensitivePropertyProvider.java
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.properties.sensitive;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.properties.sensitive.property.provider.aes.AESSensitivePropertyProvider;
+import org.apache.nifi.properties.sensitive.property.provider.keystore.KeyStoreWrappedSensitivePropertyProvider;
+import org.bouncycastle.jce.provider.BouncyCastleProvider;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.security.Security;
+
+/**
+ *
+ * This class describes the available sensitive property providers {@link SensitivePropertyProvider} which can be chosen to encrypt/decrypt the sensitive properties in the nifi.properties file.
+ * Most of these providers will require external setup in their respective system (eg. Hashicorp Vault) and then configuration given in the nifi.properties file.
+ *
+ * For further details, review the <a href="https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#sensitive_property_providers">Apache NiFi User Guide - Sensitive Property Providers</a> section.
+ *
+ * As of Apache NiFi 1.12.0 this implementation is considered <a href="https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#experimental_warning">*experimental*</a>.
+ * Available implementations can be found listed in StandardSensitivePropertyProvider.
+ *
+ * This class hides the various SPP subclass construction from clients.
+ *
+ */
+public class StandardSensitivePropertyProvider {
+ private static final Logger logger = LoggerFactory.getLogger(StandardSensitivePropertyProvider.class);
+
+ static {
+ Security.addProvider(new BouncyCastleProvider());
+ }
+
+ /**
+ * Creates a {@link SensitivePropertyProvider} suitable for a given key or key id.
+ *
+ * If an empty or null key/key id is given, this implementation returns null. This is a convenience
+ * for clients using the various Property classes, as those classes allow a null SensitivePropertyProvider.
+ *
+ * If no provider recognizes a key/key id, this implementation throws {@link SensitivePropertyProtectionException}.
+ *
+ * @param key provider encryption key
+ * @return concrete instance of SensitivePropertyProvider, or null when no key/key id is specified
+ * @throws SensitivePropertyProtectionException when a key/key id is not handled by any provider.
+ */
+ public static SensitivePropertyProvider fromKey(String key) {
+ if (StringUtils.isBlank(key)) {
+ return null;
+ } else if (KeyStoreWrappedSensitivePropertyProvider.isProviderFor(key)) {
Review comment:
They theoretically could be as they're now both native options for NiFi to use as sensitive property providers. However I think it makes sense to keep them separate as they offer two different sensitive property key provider options: one uses configuration files to store an AES key, one uses a Java keystore to hold a key. This separation of concerns will hold true for all future providers for AWS, GCP, Azure and others.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] bahlulh commented on a change in pull request #4164:
NIFI-5481 - Add new providers of protected sensitive configuration values
Posted by GitBox <gi...@apache.org>.
bahlulh commented on a change in pull request #4164: NIFI-5481 - Add new providers of protected sensitive configuration values
URL: https://github.com/apache/nifi/pull/4164#discussion_r402477690
##########
File path: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/java/org/apache/nifi/properties/sensitive/property/provider/keystore/KeyStoreWrappedSensitivePropertyProvider.java
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.properties.sensitive.property.provider.keystore;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.properties.sensitive.ExternalProperties;
+import org.apache.nifi.properties.sensitive.property.provider.aes.AESSensitivePropertyProvider;
+import org.apache.nifi.properties.sensitive.SensitivePropertyConfigurationException;
+import org.apache.nifi.properties.sensitive.SensitivePropertyProtectionException;
+import org.apache.nifi.properties.sensitive.SensitivePropertyProvider;
+import org.apache.nifi.properties.sensitive.StandardExternalPropertyLookup;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.security.Key;
+import java.security.KeyStore;
+import java.security.KeyStoreException;
+import java.security.NoSuchAlgorithmException;
+import java.security.UnrecoverableKeyException;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Sensitive properties using KeyStore keys with an inner AES SPP.
+ */
+public class KeyStoreWrappedSensitivePropertyProvider implements SensitivePropertyProvider {
+ private static final Logger logger = LoggerFactory.getLogger(KeyStoreWrappedSensitivePropertyProvider.class);
+
+ private static final String PROVIDER_NAME = "KeyStore Sensitive Property Provider";
+ private static final String MATERIAL_PREFIX = "keystore";
+ private static final String MATERIAL_SEPARATOR = "/";
+
+ public static final String KEYSTORE_TYPE_JCECKS = "jceks";
+ private static final String KEYSTORE_TYPE_PKCS12 = "pkcs12";
+ private static final String KEYSTORE_TYPE_BKS = "bks";
+
+ static final Set<String> KEYSTORE_TYPES = new HashSet<>(Arrays.asList(
+ KEYSTORE_TYPE_JCECKS,
+ KEYSTORE_TYPE_PKCS12,
+ KEYSTORE_TYPE_BKS));
+
+ private final ExternalProperties externalProperties;
+ private final SensitivePropertyProvider wrappedSensitivePropertyProvider;
+ private final String storeType;
+ private final String keyAlias;
+
+ /**
+ * Constructor, as expected by the standard sensitive property provider implementation.
+ *
+ * @param keyId string in the form "keystore/jcecks/user-key-alias"
+ */
+ public KeyStoreWrappedSensitivePropertyProvider(String keyId) {
+ this(keyId, null, null);
+ }
+
+ public KeyStoreWrappedSensitivePropertyProvider(String keyId, KeyStoreProvider keyStoreProvider, ExternalProperties externalProperties) {
+ if (externalProperties == null) {
+ externalProperties = new StandardExternalPropertyLookup(null, getKeyStorePropertiesMapping() );
+ }
+ this.externalProperties = externalProperties;
+
+ if (StringUtils.isBlank(keyId))
+ throw new SensitivePropertyConfigurationException("The key cannot be empty");
+
+ String storeType;
+ String keyAlias;
+ try {
+ String[] parts = keyId.split(MATERIAL_SEPARATOR);
+ storeType = parts.length > 0 ? parts[1] : "";
+ keyAlias = parts.length > 1 ? parts[2] : "";
+ } catch (final ArrayIndexOutOfBoundsException e) {
+ throw new SensitivePropertyConfigurationException("Invalid Key Store key", e);
+ }
+
+ this.storeType = storeType;
+ this.keyAlias = keyAlias;
+
+ if (keyStoreProvider == null ){
+ keyStoreProvider = new StandardKeyStoreProvider(getStoreUri(), this.storeType, getStorePassword());
+ }
+
+ try {
+ KeyStore store = keyStoreProvider.getKeyStore();
+ Key secretKey = store.getKey(keyAlias, getKeyPassword().toCharArray());
+ this.wrappedSensitivePropertyProvider = new AESSensitivePropertyProvider(secretKey.getEncoded());
+ } catch (final IOException | NoSuchAlgorithmException | UnrecoverableKeyException | KeyStoreException e) {
+ throw new SensitivePropertyConfigurationException(e);
+ }
+ }
+
+ private Map<String, String> getKeyStorePropertiesMapping() {
+ Map<String, String> map = new HashMap<>();
+ map.put("keystore.key-password", "KEYSTORE_KEY_PASSWORD");
+ map.put("keystore.password", "KEYSTORE_PASSWORD");
+ map.put("keystore.file", "KEYSTORE_FILE");
+ return map;
+ }
+
+ private String getKeyPassword() {
+ return externalProperties.get("keystore.key-password", "");
+ }
+
+ private String getStorePassword() {
+ return externalProperties.get("keystore.password", "");
+ }
+
+ private String getStoreUri() {
Review comment:
The name "getStoreUri" does not match with "keystore.file"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] natural commented on issue #4164: NIFI-5481 - Add new
providers of protected sensitive configuration values
Posted by GitBox <gi...@apache.org>.
natural commented on issue #4164: NIFI-5481 - Add new providers of protected sensitive configuration values
URL: https://github.com/apache/nifi/pull/4164#issuecomment-615032288
> Hi @thenatog,
> I am planning to add an Azure Key Vault (AKV) secret provider to Nifi. Can you please give me some idea on how it will fit with this change? If you are available in nifi slack channel, we can also discuss it there.
Hi @bahlulh
There's an Azure provider already written as part of this effort, it's just not in the PR. We're trying to split the changes into this PR (refactoring) and follow-on PRs for the various third-party providers.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] bahlulh commented on a change in pull request #4164:
NIFI-5481 - Add new providers of protected sensitive configuration values
Posted by GitBox <gi...@apache.org>.
bahlulh commented on a change in pull request #4164: NIFI-5481 - Add new providers of protected sensitive configuration values
URL: https://github.com/apache/nifi/pull/4164#discussion_r402481445
##########
File path: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/java/org/apache/nifi/properties/sensitive/property/provider/keystore/KeyStoreWrappedSensitivePropertyProvider.java
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.properties.sensitive.property.provider.keystore;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.properties.sensitive.ExternalProperties;
+import org.apache.nifi.properties.sensitive.property.provider.aes.AESSensitivePropertyProvider;
+import org.apache.nifi.properties.sensitive.SensitivePropertyConfigurationException;
+import org.apache.nifi.properties.sensitive.SensitivePropertyProtectionException;
+import org.apache.nifi.properties.sensitive.SensitivePropertyProvider;
+import org.apache.nifi.properties.sensitive.StandardExternalPropertyLookup;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.security.Key;
+import java.security.KeyStore;
+import java.security.KeyStoreException;
+import java.security.NoSuchAlgorithmException;
+import java.security.UnrecoverableKeyException;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Sensitive properties using KeyStore keys with an inner AES SPP.
+ */
+public class KeyStoreWrappedSensitivePropertyProvider implements SensitivePropertyProvider {
+ private static final Logger logger = LoggerFactory.getLogger(KeyStoreWrappedSensitivePropertyProvider.class);
+
+ private static final String PROVIDER_NAME = "KeyStore Sensitive Property Provider";
+ private static final String MATERIAL_PREFIX = "keystore";
+ private static final String MATERIAL_SEPARATOR = "/";
+
+ public static final String KEYSTORE_TYPE_JCECKS = "jceks";
+ private static final String KEYSTORE_TYPE_PKCS12 = "pkcs12";
+ private static final String KEYSTORE_TYPE_BKS = "bks";
+
+ static final Set<String> KEYSTORE_TYPES = new HashSet<>(Arrays.asList(
+ KEYSTORE_TYPE_JCECKS,
+ KEYSTORE_TYPE_PKCS12,
+ KEYSTORE_TYPE_BKS));
+
+ private final ExternalProperties externalProperties;
+ private final SensitivePropertyProvider wrappedSensitivePropertyProvider;
+ private final String storeType;
+ private final String keyAlias;
+
+ /**
+ * Constructor, as expected by the standard sensitive property provider implementation.
+ *
+ * @param keyId string in the form "keystore/jcecks/user-key-alias"
+ */
+ public KeyStoreWrappedSensitivePropertyProvider(String keyId) {
+ this(keyId, null, null);
+ }
+
+ public KeyStoreWrappedSensitivePropertyProvider(String keyId, KeyStoreProvider keyStoreProvider, ExternalProperties externalProperties) {
+ if (externalProperties == null) {
+ externalProperties = new StandardExternalPropertyLookup(null, getKeyStorePropertiesMapping() );
+ }
+ this.externalProperties = externalProperties;
+
+ if (StringUtils.isBlank(keyId))
+ throw new SensitivePropertyConfigurationException("The key cannot be empty");
+
+ String storeType;
+ String keyAlias;
+ try {
+ String[] parts = keyId.split(MATERIAL_SEPARATOR);
+ storeType = parts.length > 0 ? parts[1] : "";
+ keyAlias = parts.length > 1 ? parts[2] : "";
+ } catch (final ArrayIndexOutOfBoundsException e) {
+ throw new SensitivePropertyConfigurationException("Invalid Key Store key", e);
+ }
+
+ this.storeType = storeType;
+ this.keyAlias = keyAlias;
+
+ if (keyStoreProvider == null ){
+ keyStoreProvider = new StandardKeyStoreProvider(getStoreUri(), this.storeType, getStorePassword());
+ }
+
+ try {
+ KeyStore store = keyStoreProvider.getKeyStore();
+ Key secretKey = store.getKey(keyAlias, getKeyPassword().toCharArray());
+ this.wrappedSensitivePropertyProvider = new AESSensitivePropertyProvider(secretKey.getEncoded());
+ } catch (final IOException | NoSuchAlgorithmException | UnrecoverableKeyException | KeyStoreException e) {
+ throw new SensitivePropertyConfigurationException(e);
+ }
+ }
+
+ private Map<String, String> getKeyStorePropertiesMapping() {
+ Map<String, String> map = new HashMap<>();
+ map.put("keystore.key-password", "KEYSTORE_KEY_PASSWORD");
+ map.put("keystore.password", "KEYSTORE_PASSWORD");
+ map.put("keystore.file", "KEYSTORE_FILE");
+ return map;
+ }
+
+ private String getKeyPassword() {
+ return externalProperties.get("keystore.key-password", "");
+ }
+
+ private String getStorePassword() {
+ return externalProperties.get("keystore.password", "");
+ }
+
+ private String getStoreUri() {
+ return externalProperties.get("keystore.file", "");
+ }
+
+ public static String formatForType(String storeType, String keyAlias) {
+ return MATERIAL_PREFIX + MATERIAL_SEPARATOR + storeType + MATERIAL_SEPARATOR + keyAlias;
+ }
+
+ /**
+ * Returns the name of the underlying implementation.
+ *
+ * @return the name of this sensitive property provider
+ */
+ @Override
+ public String getName() {
+ return PROVIDER_NAME;
+ }
+
+ /**
+ * Returns the key used to identify the provider implementation in {@code nifi.properties}.
+ *
+ * @return the key to persist in the sibling property
+ */
+ @Override
+ public String getIdentifierKey() {
+ return MATERIAL_PREFIX + MATERIAL_SEPARATOR + storeType + MATERIAL_SEPARATOR + keyAlias;
+ }
+
+ /**
+ * Returns the "protected" form of this value. This is a form which can safely be persisted in the {@code nifi.properties} file without compromising the value.
+ * An encryption-based provider would return a cipher text, while a remote-lookup provider could return a unique ID to retrieve the secured value.
+ *
+ * @param unprotectedValue the sensitive value
+ * @return the value to persist in the {@code nifi.properties} file
+ */
+ @Override
+ public String protect(String unprotectedValue) throws SensitivePropertyProtectionException {
+ return wrappedSensitivePropertyProvider.protect(unprotectedValue);
+ }
+
+ /**
+ * Returns the "unprotected" form of this value. This is the raw sensitive value which is used by the application logic.
+ * An encryption-based provider would decrypt a cipher text and return the plaintext, while a remote-lookup provider could retrieve the secured value.
+ *
+ * @param protectedValue the protected value read from the {@code nifi.properties} file
+ * @return the raw value to be used by the application
+ */
+ @Override
+ public String unprotect(String protectedValue) throws SensitivePropertyProtectionException {
+ try {
+ return wrappedSensitivePropertyProvider.unprotect(protectedValue);
+ } catch (final IllegalArgumentException e) {
+ throw new SensitivePropertyProtectionException(e);
+ }
+ }
+
+ /**
+ * True when the client specifies a key like 'keystore/pkcs12/...'.
+ *
+ * @param material name of encryption or protection scheme
+ * @return true if this class can provide protected values
+ */
+ public static boolean isProviderFor(String material) {
+ if (StringUtils.isBlank(material)) {
+ return false;
+ }
+ String[] parts = material.split(MATERIAL_SEPARATOR, 3);
+ return parts.length == 3 && parts[0].toLowerCase().equals(MATERIAL_PREFIX) && KEYSTORE_TYPES.contains(parts[1].toLowerCase());
+ }
+
+ /**
+ * Returns a printable representation of a key.
+ *
+ * @param key key material or key id
+ * @return printable string
+ */
+ public static String toPrintableString(String key) {
Review comment:
What is the purpose of this function?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] bahlulh commented on a change in pull request #4164:
NIFI-5481 - Add new providers of protected sensitive configuration values
Posted by GitBox <gi...@apache.org>.
bahlulh commented on a change in pull request #4164: NIFI-5481 - Add new providers of protected sensitive configuration values
URL: https://github.com/apache/nifi/pull/4164#discussion_r402480844
##########
File path: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/java/org/apache/nifi/properties/sensitive/property/provider/keystore/KeyStoreWrappedSensitivePropertyProvider.java
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.properties.sensitive.property.provider.keystore;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.properties.sensitive.ExternalProperties;
+import org.apache.nifi.properties.sensitive.property.provider.aes.AESSensitivePropertyProvider;
+import org.apache.nifi.properties.sensitive.SensitivePropertyConfigurationException;
+import org.apache.nifi.properties.sensitive.SensitivePropertyProtectionException;
+import org.apache.nifi.properties.sensitive.SensitivePropertyProvider;
+import org.apache.nifi.properties.sensitive.StandardExternalPropertyLookup;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.security.Key;
+import java.security.KeyStore;
+import java.security.KeyStoreException;
+import java.security.NoSuchAlgorithmException;
+import java.security.UnrecoverableKeyException;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Sensitive properties using KeyStore keys with an inner AES SPP.
+ */
+public class KeyStoreWrappedSensitivePropertyProvider implements SensitivePropertyProvider {
+ private static final Logger logger = LoggerFactory.getLogger(KeyStoreWrappedSensitivePropertyProvider.class);
+
+ private static final String PROVIDER_NAME = "KeyStore Sensitive Property Provider";
+ private static final String MATERIAL_PREFIX = "keystore";
+ private static final String MATERIAL_SEPARATOR = "/";
+
+ public static final String KEYSTORE_TYPE_JCECKS = "jceks";
+ private static final String KEYSTORE_TYPE_PKCS12 = "pkcs12";
+ private static final String KEYSTORE_TYPE_BKS = "bks";
+
+ static final Set<String> KEYSTORE_TYPES = new HashSet<>(Arrays.asList(
+ KEYSTORE_TYPE_JCECKS,
+ KEYSTORE_TYPE_PKCS12,
+ KEYSTORE_TYPE_BKS));
+
+ private final ExternalProperties externalProperties;
+ private final SensitivePropertyProvider wrappedSensitivePropertyProvider;
+ private final String storeType;
+ private final String keyAlias;
+
+ /**
+ * Constructor, as expected by the standard sensitive property provider implementation.
+ *
+ * @param keyId string in the form "keystore/jcecks/user-key-alias"
+ */
+ public KeyStoreWrappedSensitivePropertyProvider(String keyId) {
+ this(keyId, null, null);
+ }
+
+ public KeyStoreWrappedSensitivePropertyProvider(String keyId, KeyStoreProvider keyStoreProvider, ExternalProperties externalProperties) {
+ if (externalProperties == null) {
+ externalProperties = new StandardExternalPropertyLookup(null, getKeyStorePropertiesMapping() );
+ }
+ this.externalProperties = externalProperties;
+
+ if (StringUtils.isBlank(keyId))
+ throw new SensitivePropertyConfigurationException("The key cannot be empty");
+
+ String storeType;
+ String keyAlias;
+ try {
+ String[] parts = keyId.split(MATERIAL_SEPARATOR);
+ storeType = parts.length > 0 ? parts[1] : "";
+ keyAlias = parts.length > 1 ? parts[2] : "";
+ } catch (final ArrayIndexOutOfBoundsException e) {
+ throw new SensitivePropertyConfigurationException("Invalid Key Store key", e);
+ }
+
+ this.storeType = storeType;
+ this.keyAlias = keyAlias;
+
+ if (keyStoreProvider == null ){
+ keyStoreProvider = new StandardKeyStoreProvider(getStoreUri(), this.storeType, getStorePassword());
+ }
+
+ try {
+ KeyStore store = keyStoreProvider.getKeyStore();
+ Key secretKey = store.getKey(keyAlias, getKeyPassword().toCharArray());
+ this.wrappedSensitivePropertyProvider = new AESSensitivePropertyProvider(secretKey.getEncoded());
+ } catch (final IOException | NoSuchAlgorithmException | UnrecoverableKeyException | KeyStoreException e) {
+ throw new SensitivePropertyConfigurationException(e);
+ }
+ }
+
+ private Map<String, String> getKeyStorePropertiesMapping() {
+ Map<String, String> map = new HashMap<>();
+ map.put("keystore.key-password", "KEYSTORE_KEY_PASSWORD");
+ map.put("keystore.password", "KEYSTORE_PASSWORD");
+ map.put("keystore.file", "KEYSTORE_FILE");
+ return map;
+ }
+
+ private String getKeyPassword() {
+ return externalProperties.get("keystore.key-password", "");
+ }
+
+ private String getStorePassword() {
+ return externalProperties.get("keystore.password", "");
+ }
+
+ private String getStoreUri() {
+ return externalProperties.get("keystore.file", "");
+ }
+
+ public static String formatForType(String storeType, String keyAlias) {
+ return MATERIAL_PREFIX + MATERIAL_SEPARATOR + storeType + MATERIAL_SEPARATOR + keyAlias;
+ }
+
+ /**
+ * Returns the name of the underlying implementation.
+ *
+ * @return the name of this sensitive property provider
+ */
+ @Override
+ public String getName() {
+ return PROVIDER_NAME;
+ }
+
+ /**
+ * Returns the key used to identify the provider implementation in {@code nifi.properties}.
+ *
+ * @return the key to persist in the sibling property
+ */
+ @Override
+ public String getIdentifierKey() {
+ return MATERIAL_PREFIX + MATERIAL_SEPARATOR + storeType + MATERIAL_SEPARATOR + keyAlias;
+ }
+
+ /**
+ * Returns the "protected" form of this value. This is a form which can safely be persisted in the {@code nifi.properties} file without compromising the value.
+ * An encryption-based provider would return a cipher text, while a remote-lookup provider could return a unique ID to retrieve the secured value.
+ *
+ * @param unprotectedValue the sensitive value
+ * @return the value to persist in the {@code nifi.properties} file
+ */
+ @Override
+ public String protect(String unprotectedValue) throws SensitivePropertyProtectionException {
+ return wrappedSensitivePropertyProvider.protect(unprotectedValue);
+ }
+
+ /**
+ * Returns the "unprotected" form of this value. This is the raw sensitive value which is used by the application logic.
+ * An encryption-based provider would decrypt a cipher text and return the plaintext, while a remote-lookup provider could retrieve the secured value.
+ *
+ * @param protectedValue the protected value read from the {@code nifi.properties} file
+ * @return the raw value to be used by the application
+ */
+ @Override
+ public String unprotect(String protectedValue) throws SensitivePropertyProtectionException {
+ try {
+ return wrappedSensitivePropertyProvider.unprotect(protectedValue);
+ } catch (final IllegalArgumentException e) {
+ throw new SensitivePropertyProtectionException(e);
+ }
+ }
+
+ /**
+ * True when the client specifies a key like 'keystore/pkcs12/...'.
+ *
+ * @param material name of encryption or protection scheme
+ * @return true if this class can provide protected values
+ */
+ public static boolean isProviderFor(String material) {
+ if (StringUtils.isBlank(material)) {
+ return false;
+ }
+ String[] parts = material.split(MATERIAL_SEPARATOR, 3);
+ return parts.length == 3 && parts[0].toLowerCase().equals(MATERIAL_PREFIX) && KEYSTORE_TYPES.contains(parts[1].toLowerCase());
Review comment:
Why this is "contains" and not "equals"?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] thenatog edited a comment on issue #4164: NIFI-5481 - Add
new providers of protected sensitive configuration values
Posted by GitBox <gi...@apache.org>.
thenatog edited a comment on issue #4164: NIFI-5481 - Add new providers of protected sensitive configuration values
URL: https://github.com/apache/nifi/pull/4164#issuecomment-614185585
@bahlulh Thanks for the review Bahlul, the Azure sensitive property provider is on its way (it's already written). https://issues.apache.org/jira/browse/NIFI-6615. We decided to add these sensitive property providers in stages as the original PR to add all providers was even bigger than this one. You can check it out here: https://github.com/apache/nifi/pull/4080/files. You should see the AzureKeyVaultSensitivePropertyProvider.java.
Once this PR is merged, the next PR will add a new NiFi maven module which will eventually contain the other sensitive property providers, including the AKV one. You can help us out by testing the AKV provider once that PR is up.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services