You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by GitBox <gi...@apache.org> on 2021/09/16 13:43:10 UTC
[GitHub] [nifi] gresockj opened a new pull request #5391: NIFI-9174: Adding AWS SecretsManager ParamValueProvider for Stateless
gresockj opened a new pull request #5391:
URL: https://github.com/apache/nifi/pull/5391
#### Description of PR
_Adds a ParameterValueProvider using AWS SecretsManager_
To test:
```
aws secretsmanager create-secret --name "Context/Param" --secret-string '{ "value": "secretValue" }'
```
Create a flow with a Parameter Context named "Context" and a Parameter named "Param". The flow could demonstrate the value by logging the parameter via `LogAttribute`. Save the flow definition and launch stateless using the following configuration:
```
nifi.stateless.parameter.provider.AWSSecretsManager.name=AWS Secrets Manager Provider
nifi.stateless.parameter.provider.AWSSecretsManager.type=org.apache.nifi.stateless.parameter.aws.SecretsManagerParameterValueProvider
nifi.stateless.parameter.provider.AWSSecretsManager.properties.aws-configuration-file=./conf/bootstrap-aws.conf
```
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with **NIFI-XXXX** where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [ ] Has your PR been rebased against the latest commit within the target branch (typically `main`)?
- [ ] Is your initial contribution a single, squashed commit? _Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not `squash` or use `--force` when pushing to allow for clean monitoring of changes._
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via `mvn -Pcontrib-check clean install` at the root `nifi` folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] Have you verified that the full build is successful on JDK 8?
- [ ] Have you verified that the full build is successful on JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main `LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main `NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to .name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for build issues and submit an update to your PR as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on pull request #5391: NIFI-9174: Adding AWS SecretsManager ParamValueProvider for Stateless
Posted by GitBox <gi...@apache.org>.
exceptionfactory commented on pull request #5391:
URL: https://github.com/apache/nifi/pull/5391#issuecomment-924510416
@gresockj can you rebase this PR to incorporate the recent build improvements to address test failures?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] gresockj commented on a change in pull request #5391: NIFI-9174: Adding AWS SecretsManager ParamValueProvider for Stateless
Posted by GitBox <gi...@apache.org>.
gresockj commented on a change in pull request #5391:
URL: https://github.com/apache/nifi/pull/5391#discussion_r718961733
##########
File path: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-parameter-value-providers/src/main/java/org/apache/nifi/stateless/parameter/aws/SecretsManagerParameterValueProvider.java
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.stateless.parameter.aws;
+
+import com.amazonaws.auth.AWSStaticCredentialsProvider;
+import com.amazonaws.auth.BasicAWSCredentials;
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
+import com.amazonaws.services.secretsmanager.AWSSecretsManager;
+import com.amazonaws.services.secretsmanager.AWSSecretsManagerClientBuilder;
+import com.amazonaws.services.secretsmanager.model.GetSecretValueRequest;
+import com.amazonaws.services.secretsmanager.model.GetSecretValueResult;
+import com.amazonaws.services.secretsmanager.model.ListSecretsRequest;
+import com.amazonaws.services.secretsmanager.model.ListSecretsResult;
+import com.amazonaws.services.secretsmanager.model.SecretListEntry;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stateless.parameter.AbstractParameterValueProvider;
+import org.apache.nifi.stateless.parameter.ParameterValueProvider;
+import org.apache.nifi.stateless.parameter.ParameterValueProviderInitializationContext;
+
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Paths;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+
+/**
+ * Reads secrets from AWS Secrets Manager to provide parameter values. Secrets must be created similar to the following AWS cli command: <br/><br/>
+ * <code>aws secretsmanager create-secret --name "[ParamContextName]/[ParamName]" --secret-string '[ParamValue]'</code> <br/><br/>
+ *
+ * A standard configuration for this provider would be: <br/><br/>
+ *
+ * <code>
+ * nifi.stateless.parameter.provider.AWSSecretsManager.name=AWS Secrets Manager Value Provider
+ * nifi.stateless.parameter.provider.AWSSecretsManager.type=org.apache.nifi.stateless.parameter.aws.SecretsManagerParameterValueProvider
+ * nifi.stateless.parameter.provider.AWSSecretsManager.properties.aws-credentials-file=./conf/bootstrap-aws.conf
+ * </code>
+ */
+public class SecretsManagerParameterValueProvider extends AbstractParameterValueProvider implements ParameterValueProvider {
+ private static final String QUALIFIED_SECRET_FORMAT = "%s/%s";
+ private static final String ACCESS_KEY_PROPS_NAME = "aws.access.key.id";
+ private static final String SECRET_KEY_PROPS_NAME = "aws.secret.access.key";
+ private static final String REGION_KEY_PROPS_NAME = "aws.region";
+
+ public static final PropertyDescriptor AWS_CREDENTIALS_FILE = new PropertyDescriptor.Builder()
+ .displayName("AWS Credentials File")
+ .name("aws-credentials-file")
+ .required(false)
+ .defaultValue("./conf/bootstrap-aws.conf")
+ .description("Location of the bootstrap-aws.conf file that configures the AWS credentials. If not provided, the default AWS credentials will be used.")
+ .addValidator(StandardValidators.FILE_EXISTS_VALIDATOR)
+ .build();
+
+ private final Set<String> supportedParameterNames = new HashSet<>();
+
+ private List<PropertyDescriptor> descriptors;
+
+ private AWSSecretsManager secretsManager;
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ protected void init(final ParameterValueProviderInitializationContext context) {
+ super.init(context);
+
+ this.descriptors = Collections.singletonList(AWS_CREDENTIALS_FILE);
+
+ final String awsCredentialsFilename = context.getProperty(AWS_CREDENTIALS_FILE).getValue();
+ try {
+ this.secretsManager = this.configureClient(awsCredentialsFilename);
+ } catch (final IOException e) {
+ throw new IllegalStateException("Could not configure AWS Secrets Manager Client", e);
+ }
+
+ cacheSupportedParameterNames();
+ }
+
+ @Override
+ public boolean isParameterDefined(final String contextName, final String parameterName) {
+ return supportedParameterNames.contains(getSecretName(contextName, parameterName));
Review comment:
Thanks for the additional context! I'll make that update. (no pun intended)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on pull request #5391: NIFI-9174: Adding AWS SecretsManager ParamValueProvider for Stateless
Posted by GitBox <gi...@apache.org>.
exceptionfactory commented on pull request #5391:
URL: https://github.com/apache/nifi/pull/5391#issuecomment-922323775
Thanks for the reply and reference to HashiCorp Vault documentation @gresockj. With that background, the HashiCorp Vault implementation makes sense as it stands.
Making the adjustments to support a simple string for AWS Secrets Manager sounds good, that should also eliminate the dependency on Jackson.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] markap14 commented on a change in pull request #5391: NIFI-9174: Adding AWS SecretsManager ParamValueProvider for Stateless
Posted by GitBox <gi...@apache.org>.
markap14 commented on a change in pull request #5391:
URL: https://github.com/apache/nifi/pull/5391#discussion_r718956080
##########
File path: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-parameter-value-providers/src/main/java/org/apache/nifi/stateless/parameter/aws/SecretsManagerParameterValueProvider.java
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.stateless.parameter.aws;
+
+import com.amazonaws.auth.AWSStaticCredentialsProvider;
+import com.amazonaws.auth.BasicAWSCredentials;
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
+import com.amazonaws.services.secretsmanager.AWSSecretsManager;
+import com.amazonaws.services.secretsmanager.AWSSecretsManagerClientBuilder;
+import com.amazonaws.services.secretsmanager.model.GetSecretValueRequest;
+import com.amazonaws.services.secretsmanager.model.GetSecretValueResult;
+import com.amazonaws.services.secretsmanager.model.ListSecretsRequest;
+import com.amazonaws.services.secretsmanager.model.ListSecretsResult;
+import com.amazonaws.services.secretsmanager.model.SecretListEntry;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stateless.parameter.AbstractParameterValueProvider;
+import org.apache.nifi.stateless.parameter.ParameterValueProvider;
+import org.apache.nifi.stateless.parameter.ParameterValueProviderInitializationContext;
+
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Paths;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+
+/**
+ * Reads secrets from AWS Secrets Manager to provide parameter values. Secrets must be created similar to the following AWS cli command: <br/><br/>
+ * <code>aws secretsmanager create-secret --name "[ParamContextName]/[ParamName]" --secret-string '[ParamValue]'</code> <br/><br/>
+ *
+ * A standard configuration for this provider would be: <br/><br/>
+ *
+ * <code>
+ * nifi.stateless.parameter.provider.AWSSecretsManager.name=AWS Secrets Manager Value Provider
+ * nifi.stateless.parameter.provider.AWSSecretsManager.type=org.apache.nifi.stateless.parameter.aws.SecretsManagerParameterValueProvider
+ * nifi.stateless.parameter.provider.AWSSecretsManager.properties.aws-credentials-file=./conf/bootstrap-aws.conf
+ * </code>
+ */
+public class SecretsManagerParameterValueProvider extends AbstractParameterValueProvider implements ParameterValueProvider {
+ private static final String QUALIFIED_SECRET_FORMAT = "%s/%s";
+ private static final String ACCESS_KEY_PROPS_NAME = "aws.access.key.id";
+ private static final String SECRET_KEY_PROPS_NAME = "aws.secret.access.key";
+ private static final String REGION_KEY_PROPS_NAME = "aws.region";
+
+ public static final PropertyDescriptor AWS_CREDENTIALS_FILE = new PropertyDescriptor.Builder()
+ .displayName("AWS Credentials File")
+ .name("aws-credentials-file")
+ .required(false)
+ .defaultValue("./conf/bootstrap-aws.conf")
+ .description("Location of the bootstrap-aws.conf file that configures the AWS credentials. If not provided, the default AWS credentials will be used.")
+ .addValidator(StandardValidators.FILE_EXISTS_VALIDATOR)
+ .build();
+
+ private final Set<String> supportedParameterNames = new HashSet<>();
+
+ private List<PropertyDescriptor> descriptors;
+
+ private AWSSecretsManager secretsManager;
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ protected void init(final ParameterValueProviderInitializationContext context) {
+ super.init(context);
+
+ this.descriptors = Collections.singletonList(AWS_CREDENTIALS_FILE);
+
+ final String awsCredentialsFilename = context.getProperty(AWS_CREDENTIALS_FILE).getValue();
+ try {
+ this.secretsManager = this.configureClient(awsCredentialsFilename);
+ } catch (final IOException e) {
+ throw new IllegalStateException("Could not configure AWS Secrets Manager Client", e);
+ }
+
+ cacheSupportedParameterNames();
+ }
+
+ @Override
+ public boolean isParameterDefined(final String contextName, final String parameterName) {
+ return supportedParameterNames.contains(getSecretName(contextName, parameterName));
+ }
+
+ @Override
+ public String getParameterValue(final String contextName, final String parameterName) {
+ final String secretName = getSecretName(contextName, parameterName);
+ final GetSecretValueRequest getSecretValueRequest = new GetSecretValueRequest()
+ .withSecretId(secretName);
+ final GetSecretValueResult getSecretValueResult = secretsManager.getSecretValue(getSecretValueRequest);
+
+ if (getSecretValueResult == null) {
+ throw new IllegalArgumentException(String.format("Secret [%s] not found", secretName));
+ }
+ if (getSecretValueResult.getSecretString() != null) {
+ return getSecretValueResult.getSecretString();
Review comment:
I added a secret in SecretManager named MyContext/MyParameter. I used an "Other type of secret" and I set the key to "MyContext/MyParameter" and the value to "Hello". I expected the parameter to resolve to `Hello`. Instead, it resolved to `{"MyContext/MyParameter":"Hello"}`. Am I using this wrong?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] gresockj commented on a change in pull request #5391: NIFI-9174: Adding AWS SecretsManager ParamValueProvider for Stateless
Posted by GitBox <gi...@apache.org>.
gresockj commented on a change in pull request #5391:
URL: https://github.com/apache/nifi/pull/5391#discussion_r718961645
##########
File path: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-parameter-value-providers/src/main/java/org/apache/nifi/stateless/parameter/aws/SecretsManagerParameterValueProvider.java
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.stateless.parameter.aws;
+
+import com.amazonaws.auth.AWSStaticCredentialsProvider;
+import com.amazonaws.auth.BasicAWSCredentials;
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
+import com.amazonaws.services.secretsmanager.AWSSecretsManager;
+import com.amazonaws.services.secretsmanager.AWSSecretsManagerClientBuilder;
+import com.amazonaws.services.secretsmanager.model.GetSecretValueRequest;
+import com.amazonaws.services.secretsmanager.model.GetSecretValueResult;
+import com.amazonaws.services.secretsmanager.model.ListSecretsRequest;
+import com.amazonaws.services.secretsmanager.model.ListSecretsResult;
+import com.amazonaws.services.secretsmanager.model.SecretListEntry;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stateless.parameter.AbstractParameterValueProvider;
+import org.apache.nifi.stateless.parameter.ParameterValueProvider;
+import org.apache.nifi.stateless.parameter.ParameterValueProviderInitializationContext;
+
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Paths;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+
+/**
+ * Reads secrets from AWS Secrets Manager to provide parameter values. Secrets must be created similar to the following AWS cli command: <br/><br/>
+ * <code>aws secretsmanager create-secret --name "[ParamContextName]/[ParamName]" --secret-string '[ParamValue]'</code> <br/><br/>
+ *
+ * A standard configuration for this provider would be: <br/><br/>
+ *
+ * <code>
+ * nifi.stateless.parameter.provider.AWSSecretsManager.name=AWS Secrets Manager Value Provider
+ * nifi.stateless.parameter.provider.AWSSecretsManager.type=org.apache.nifi.stateless.parameter.aws.SecretsManagerParameterValueProvider
+ * nifi.stateless.parameter.provider.AWSSecretsManager.properties.aws-credentials-file=./conf/bootstrap-aws.conf
+ * </code>
+ */
+public class SecretsManagerParameterValueProvider extends AbstractParameterValueProvider implements ParameterValueProvider {
+ private static final String QUALIFIED_SECRET_FORMAT = "%s/%s";
+ private static final String ACCESS_KEY_PROPS_NAME = "aws.access.key.id";
+ private static final String SECRET_KEY_PROPS_NAME = "aws.secret.access.key";
+ private static final String REGION_KEY_PROPS_NAME = "aws.region";
+
+ public static final PropertyDescriptor AWS_CREDENTIALS_FILE = new PropertyDescriptor.Builder()
+ .displayName("AWS Credentials File")
+ .name("aws-credentials-file")
+ .required(false)
+ .defaultValue("./conf/bootstrap-aws.conf")
+ .description("Location of the bootstrap-aws.conf file that configures the AWS credentials. If not provided, the default AWS credentials will be used.")
+ .addValidator(StandardValidators.FILE_EXISTS_VALIDATOR)
+ .build();
+
+ private final Set<String> supportedParameterNames = new HashSet<>();
+
+ private List<PropertyDescriptor> descriptors;
+
+ private AWSSecretsManager secretsManager;
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ protected void init(final ParameterValueProviderInitializationContext context) {
+ super.init(context);
+
+ this.descriptors = Collections.singletonList(AWS_CREDENTIALS_FILE);
+
+ final String awsCredentialsFilename = context.getProperty(AWS_CREDENTIALS_FILE).getValue();
+ try {
+ this.secretsManager = this.configureClient(awsCredentialsFilename);
+ } catch (final IOException e) {
+ throw new IllegalStateException("Could not configure AWS Secrets Manager Client", e);
+ }
+
+ cacheSupportedParameterNames();
+ }
+
+ @Override
+ public boolean isParameterDefined(final String contextName, final String parameterName) {
+ return supportedParameterNames.contains(getSecretName(contextName, parameterName));
+ }
+
+ @Override
+ public String getParameterValue(final String contextName, final String parameterName) {
+ final String secretName = getSecretName(contextName, parameterName);
+ final GetSecretValueRequest getSecretValueRequest = new GetSecretValueRequest()
+ .withSecretId(secretName);
+ final GetSecretValueResult getSecretValueResult = secretsManager.getSecretValue(getSecretValueRequest);
+
+ if (getSecretValueResult == null) {
+ throw new IllegalArgumentException(String.format("Secret [%s] not found", secretName));
+ }
+ if (getSecretValueResult.getSecretString() != null) {
+ return getSecretValueResult.getSecretString();
Review comment:
Are you adding the secret as a plain text secret? If you use the AWS key/value interface, it creates the JSON you posted instead of simply "Hello". Basically, if you use the command line example in the PR, it should resolve to "Hello". I will also take a look at this again, in case it is a bug in the code, though.
##########
File path: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-parameter-value-providers/src/main/java/org/apache/nifi/stateless/parameter/aws/SecretsManagerParameterValueProvider.java
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.stateless.parameter.aws;
+
+import com.amazonaws.auth.AWSStaticCredentialsProvider;
+import com.amazonaws.auth.BasicAWSCredentials;
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
+import com.amazonaws.services.secretsmanager.AWSSecretsManager;
+import com.amazonaws.services.secretsmanager.AWSSecretsManagerClientBuilder;
+import com.amazonaws.services.secretsmanager.model.GetSecretValueRequest;
+import com.amazonaws.services.secretsmanager.model.GetSecretValueResult;
+import com.amazonaws.services.secretsmanager.model.ListSecretsRequest;
+import com.amazonaws.services.secretsmanager.model.ListSecretsResult;
+import com.amazonaws.services.secretsmanager.model.SecretListEntry;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stateless.parameter.AbstractParameterValueProvider;
+import org.apache.nifi.stateless.parameter.ParameterValueProvider;
+import org.apache.nifi.stateless.parameter.ParameterValueProviderInitializationContext;
+
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Paths;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+
+/**
+ * Reads secrets from AWS Secrets Manager to provide parameter values. Secrets must be created similar to the following AWS cli command: <br/><br/>
+ * <code>aws secretsmanager create-secret --name "[ParamContextName]/[ParamName]" --secret-string '[ParamValue]'</code> <br/><br/>
+ *
+ * A standard configuration for this provider would be: <br/><br/>
+ *
+ * <code>
+ * nifi.stateless.parameter.provider.AWSSecretsManager.name=AWS Secrets Manager Value Provider
+ * nifi.stateless.parameter.provider.AWSSecretsManager.type=org.apache.nifi.stateless.parameter.aws.SecretsManagerParameterValueProvider
+ * nifi.stateless.parameter.provider.AWSSecretsManager.properties.aws-credentials-file=./conf/bootstrap-aws.conf
+ * </code>
+ */
+public class SecretsManagerParameterValueProvider extends AbstractParameterValueProvider implements ParameterValueProvider {
+ private static final String QUALIFIED_SECRET_FORMAT = "%s/%s";
+ private static final String ACCESS_KEY_PROPS_NAME = "aws.access.key.id";
+ private static final String SECRET_KEY_PROPS_NAME = "aws.secret.access.key";
+ private static final String REGION_KEY_PROPS_NAME = "aws.region";
+
+ public static final PropertyDescriptor AWS_CREDENTIALS_FILE = new PropertyDescriptor.Builder()
+ .displayName("AWS Credentials File")
+ .name("aws-credentials-file")
+ .required(false)
+ .defaultValue("./conf/bootstrap-aws.conf")
+ .description("Location of the bootstrap-aws.conf file that configures the AWS credentials. If not provided, the default AWS credentials will be used.")
+ .addValidator(StandardValidators.FILE_EXISTS_VALIDATOR)
+ .build();
+
+ private final Set<String> supportedParameterNames = new HashSet<>();
+
+ private List<PropertyDescriptor> descriptors;
+
+ private AWSSecretsManager secretsManager;
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ protected void init(final ParameterValueProviderInitializationContext context) {
+ super.init(context);
+
+ this.descriptors = Collections.singletonList(AWS_CREDENTIALS_FILE);
+
+ final String awsCredentialsFilename = context.getProperty(AWS_CREDENTIALS_FILE).getValue();
+ try {
+ this.secretsManager = this.configureClient(awsCredentialsFilename);
+ } catch (final IOException e) {
+ throw new IllegalStateException("Could not configure AWS Secrets Manager Client", e);
+ }
+
+ cacheSupportedParameterNames();
+ }
+
+ @Override
+ public boolean isParameterDefined(final String contextName, final String parameterName) {
+ return supportedParameterNames.contains(getSecretName(contextName, parameterName));
Review comment:
Thanks for the additional context! I'll make that update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] markap14 commented on a change in pull request #5391: NIFI-9174: Adding AWS SecretsManager ParamValueProvider for Stateless
Posted by GitBox <gi...@apache.org>.
markap14 commented on a change in pull request #5391:
URL: https://github.com/apache/nifi/pull/5391#discussion_r718953908
##########
File path: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-parameter-value-providers/src/main/java/org/apache/nifi/stateless/parameter/aws/SecretsManagerParameterValueProvider.java
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.stateless.parameter.aws;
+
+import com.amazonaws.auth.AWSStaticCredentialsProvider;
+import com.amazonaws.auth.BasicAWSCredentials;
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
+import com.amazonaws.services.secretsmanager.AWSSecretsManager;
+import com.amazonaws.services.secretsmanager.AWSSecretsManagerClientBuilder;
+import com.amazonaws.services.secretsmanager.model.GetSecretValueRequest;
+import com.amazonaws.services.secretsmanager.model.GetSecretValueResult;
+import com.amazonaws.services.secretsmanager.model.ListSecretsRequest;
+import com.amazonaws.services.secretsmanager.model.ListSecretsResult;
+import com.amazonaws.services.secretsmanager.model.SecretListEntry;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stateless.parameter.AbstractParameterValueProvider;
+import org.apache.nifi.stateless.parameter.ParameterValueProvider;
+import org.apache.nifi.stateless.parameter.ParameterValueProviderInitializationContext;
+
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Paths;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+
+/**
+ * Reads secrets from AWS Secrets Manager to provide parameter values. Secrets must be created similar to the following AWS cli command: <br/><br/>
+ * <code>aws secretsmanager create-secret --name "[ParamContextName]/[ParamName]" --secret-string '[ParamValue]'</code> <br/><br/>
+ *
+ * A standard configuration for this provider would be: <br/><br/>
+ *
+ * <code>
+ * nifi.stateless.parameter.provider.AWSSecretsManager.name=AWS Secrets Manager Value Provider
+ * nifi.stateless.parameter.provider.AWSSecretsManager.type=org.apache.nifi.stateless.parameter.aws.SecretsManagerParameterValueProvider
+ * nifi.stateless.parameter.provider.AWSSecretsManager.properties.aws-credentials-file=./conf/bootstrap-aws.conf
+ * </code>
+ */
+public class SecretsManagerParameterValueProvider extends AbstractParameterValueProvider implements ParameterValueProvider {
+ private static final String QUALIFIED_SECRET_FORMAT = "%s/%s";
+ private static final String ACCESS_KEY_PROPS_NAME = "aws.access.key.id";
+ private static final String SECRET_KEY_PROPS_NAME = "aws.secret.access.key";
+ private static final String REGION_KEY_PROPS_NAME = "aws.region";
+
+ public static final PropertyDescriptor AWS_CREDENTIALS_FILE = new PropertyDescriptor.Builder()
+ .displayName("AWS Credentials File")
+ .name("aws-credentials-file")
+ .required(false)
+ .defaultValue("./conf/bootstrap-aws.conf")
+ .description("Location of the bootstrap-aws.conf file that configures the AWS credentials. If not provided, the default AWS credentials will be used.")
+ .addValidator(StandardValidators.FILE_EXISTS_VALIDATOR)
+ .build();
+
+ private final Set<String> supportedParameterNames = new HashSet<>();
+
+ private List<PropertyDescriptor> descriptors;
+
+ private AWSSecretsManager secretsManager;
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ protected void init(final ParameterValueProviderInitializationContext context) {
+ super.init(context);
+
+ this.descriptors = Collections.singletonList(AWS_CREDENTIALS_FILE);
+
+ final String awsCredentialsFilename = context.getProperty(AWS_CREDENTIALS_FILE).getValue();
+ try {
+ this.secretsManager = this.configureClient(awsCredentialsFilename);
+ } catch (final IOException e) {
+ throw new IllegalStateException("Could not configure AWS Secrets Manager Client", e);
+ }
+
+ cacheSupportedParameterNames();
+ }
+
+ @Override
+ public boolean isParameterDefined(final String contextName, final String parameterName) {
+ return supportedParameterNames.contains(getSecretName(contextName, parameterName));
Review comment:
Generally, the way that the parameters work in stateless, the context name should be considered optional. I.e., if the context name is "MyContext" and the parameter name is "MyParameter", we should return a value for "MyContext/MyParameter" if it exists, but if that doesn't exist, we should return the value for "MyParameter"
##########
File path: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-parameter-value-providers/src/main/java/org/apache/nifi/stateless/parameter/aws/SecretsManagerParameterValueProvider.java
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.stateless.parameter.aws;
+
+import com.amazonaws.auth.AWSStaticCredentialsProvider;
+import com.amazonaws.auth.BasicAWSCredentials;
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
+import com.amazonaws.services.secretsmanager.AWSSecretsManager;
+import com.amazonaws.services.secretsmanager.AWSSecretsManagerClientBuilder;
+import com.amazonaws.services.secretsmanager.model.GetSecretValueRequest;
+import com.amazonaws.services.secretsmanager.model.GetSecretValueResult;
+import com.amazonaws.services.secretsmanager.model.ListSecretsRequest;
+import com.amazonaws.services.secretsmanager.model.ListSecretsResult;
+import com.amazonaws.services.secretsmanager.model.SecretListEntry;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stateless.parameter.AbstractParameterValueProvider;
+import org.apache.nifi.stateless.parameter.ParameterValueProvider;
+import org.apache.nifi.stateless.parameter.ParameterValueProviderInitializationContext;
+
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Paths;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+
+/**
+ * Reads secrets from AWS Secrets Manager to provide parameter values. Secrets must be created similar to the following AWS cli command: <br/><br/>
+ * <code>aws secretsmanager create-secret --name "[ParamContextName]/[ParamName]" --secret-string '[ParamValue]'</code> <br/><br/>
+ *
+ * A standard configuration for this provider would be: <br/><br/>
+ *
+ * <code>
+ * nifi.stateless.parameter.provider.AWSSecretsManager.name=AWS Secrets Manager Value Provider
+ * nifi.stateless.parameter.provider.AWSSecretsManager.type=org.apache.nifi.stateless.parameter.aws.SecretsManagerParameterValueProvider
+ * nifi.stateless.parameter.provider.AWSSecretsManager.properties.aws-credentials-file=./conf/bootstrap-aws.conf
+ * </code>
+ */
+public class SecretsManagerParameterValueProvider extends AbstractParameterValueProvider implements ParameterValueProvider {
+ private static final String QUALIFIED_SECRET_FORMAT = "%s/%s";
+ private static final String ACCESS_KEY_PROPS_NAME = "aws.access.key.id";
+ private static final String SECRET_KEY_PROPS_NAME = "aws.secret.access.key";
+ private static final String REGION_KEY_PROPS_NAME = "aws.region";
+
+ public static final PropertyDescriptor AWS_CREDENTIALS_FILE = new PropertyDescriptor.Builder()
+ .displayName("AWS Credentials File")
+ .name("aws-credentials-file")
+ .required(false)
+ .defaultValue("./conf/bootstrap-aws.conf")
+ .description("Location of the bootstrap-aws.conf file that configures the AWS credentials. If not provided, the default AWS credentials will be used.")
+ .addValidator(StandardValidators.FILE_EXISTS_VALIDATOR)
+ .build();
+
+ private final Set<String> supportedParameterNames = new HashSet<>();
+
+ private List<PropertyDescriptor> descriptors;
+
+ private AWSSecretsManager secretsManager;
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ protected void init(final ParameterValueProviderInitializationContext context) {
+ super.init(context);
+
+ this.descriptors = Collections.singletonList(AWS_CREDENTIALS_FILE);
+
+ final String awsCredentialsFilename = context.getProperty(AWS_CREDENTIALS_FILE).getValue();
+ try {
+ this.secretsManager = this.configureClient(awsCredentialsFilename);
+ } catch (final IOException e) {
+ throw new IllegalStateException("Could not configure AWS Secrets Manager Client", e);
+ }
+
+ cacheSupportedParameterNames();
+ }
+
+ @Override
+ public boolean isParameterDefined(final String contextName, final String parameterName) {
+ return supportedParameterNames.contains(getSecretName(contextName, parameterName));
+ }
+
+ @Override
+ public String getParameterValue(final String contextName, final String parameterName) {
+ final String secretName = getSecretName(contextName, parameterName);
+ final GetSecretValueRequest getSecretValueRequest = new GetSecretValueRequest()
+ .withSecretId(secretName);
+ final GetSecretValueResult getSecretValueResult = secretsManager.getSecretValue(getSecretValueRequest);
+
+ if (getSecretValueResult == null) {
+ throw new IllegalArgumentException(String.format("Secret [%s] not found", secretName));
+ }
+ if (getSecretValueResult.getSecretString() != null) {
+ return getSecretValueResult.getSecretString();
Review comment:
I added a secret in SecretManager named MyContext/MyParameter. I used an "Other type of secret" and I set the key to "MyContext/MyParameter" and the value to "Hello". I expected the parameter to resolve to `Hello`. Instead, it resolved to `{"MyContext/MyParameter":"Hello"}`. Am I using this wrong?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] markap14 commented on a change in pull request #5391: NIFI-9174: Adding AWS SecretsManager ParamValueProvider for Stateless
Posted by GitBox <gi...@apache.org>.
markap14 commented on a change in pull request #5391:
URL: https://github.com/apache/nifi/pull/5391#discussion_r718953908
##########
File path: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-parameter-value-providers/src/main/java/org/apache/nifi/stateless/parameter/aws/SecretsManagerParameterValueProvider.java
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.stateless.parameter.aws;
+
+import com.amazonaws.auth.AWSStaticCredentialsProvider;
+import com.amazonaws.auth.BasicAWSCredentials;
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
+import com.amazonaws.services.secretsmanager.AWSSecretsManager;
+import com.amazonaws.services.secretsmanager.AWSSecretsManagerClientBuilder;
+import com.amazonaws.services.secretsmanager.model.GetSecretValueRequest;
+import com.amazonaws.services.secretsmanager.model.GetSecretValueResult;
+import com.amazonaws.services.secretsmanager.model.ListSecretsRequest;
+import com.amazonaws.services.secretsmanager.model.ListSecretsResult;
+import com.amazonaws.services.secretsmanager.model.SecretListEntry;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stateless.parameter.AbstractParameterValueProvider;
+import org.apache.nifi.stateless.parameter.ParameterValueProvider;
+import org.apache.nifi.stateless.parameter.ParameterValueProviderInitializationContext;
+
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Paths;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+
+/**
+ * Reads secrets from AWS Secrets Manager to provide parameter values. Secrets must be created similar to the following AWS cli command: <br/><br/>
+ * <code>aws secretsmanager create-secret --name "[ParamContextName]/[ParamName]" --secret-string '[ParamValue]'</code> <br/><br/>
+ *
+ * A standard configuration for this provider would be: <br/><br/>
+ *
+ * <code>
+ * nifi.stateless.parameter.provider.AWSSecretsManager.name=AWS Secrets Manager Value Provider
+ * nifi.stateless.parameter.provider.AWSSecretsManager.type=org.apache.nifi.stateless.parameter.aws.SecretsManagerParameterValueProvider
+ * nifi.stateless.parameter.provider.AWSSecretsManager.properties.aws-credentials-file=./conf/bootstrap-aws.conf
+ * </code>
+ */
+public class SecretsManagerParameterValueProvider extends AbstractParameterValueProvider implements ParameterValueProvider {
+ private static final String QUALIFIED_SECRET_FORMAT = "%s/%s";
+ private static final String ACCESS_KEY_PROPS_NAME = "aws.access.key.id";
+ private static final String SECRET_KEY_PROPS_NAME = "aws.secret.access.key";
+ private static final String REGION_KEY_PROPS_NAME = "aws.region";
+
+ public static final PropertyDescriptor AWS_CREDENTIALS_FILE = new PropertyDescriptor.Builder()
+ .displayName("AWS Credentials File")
+ .name("aws-credentials-file")
+ .required(false)
+ .defaultValue("./conf/bootstrap-aws.conf")
+ .description("Location of the bootstrap-aws.conf file that configures the AWS credentials. If not provided, the default AWS credentials will be used.")
+ .addValidator(StandardValidators.FILE_EXISTS_VALIDATOR)
+ .build();
+
+ private final Set<String> supportedParameterNames = new HashSet<>();
+
+ private List<PropertyDescriptor> descriptors;
+
+ private AWSSecretsManager secretsManager;
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ protected void init(final ParameterValueProviderInitializationContext context) {
+ super.init(context);
+
+ this.descriptors = Collections.singletonList(AWS_CREDENTIALS_FILE);
+
+ final String awsCredentialsFilename = context.getProperty(AWS_CREDENTIALS_FILE).getValue();
+ try {
+ this.secretsManager = this.configureClient(awsCredentialsFilename);
+ } catch (final IOException e) {
+ throw new IllegalStateException("Could not configure AWS Secrets Manager Client", e);
+ }
+
+ cacheSupportedParameterNames();
+ }
+
+ @Override
+ public boolean isParameterDefined(final String contextName, final String parameterName) {
+ return supportedParameterNames.contains(getSecretName(contextName, parameterName));
Review comment:
Generally, the way that the parameters work in stateless, the context name should be considered optional. I.e., if the context name is "MyContext" and the parameter name is "MyParameter", we should return a value for "MyContext/MyParameter" if it exists, but if that doesn't exist, we should return the value for "MyParameter"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] markap14 commented on a change in pull request #5391: NIFI-9174: Adding AWS SecretsManager ParamValueProvider for Stateless
Posted by GitBox <gi...@apache.org>.
markap14 commented on a change in pull request #5391:
URL: https://github.com/apache/nifi/pull/5391#discussion_r722268522
##########
File path: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-parameter-value-providers/src/main/java/org/apache/nifi/stateless/parameter/AwsSecretsManagerParameterValueProvider.java
##########
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.stateless.parameter;
+
+import com.amazonaws.auth.AWSStaticCredentialsProvider;
+import com.amazonaws.auth.BasicAWSCredentials;
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
+import com.amazonaws.services.secretsmanager.AWSSecretsManager;
+import com.amazonaws.services.secretsmanager.AWSSecretsManagerClientBuilder;
+import com.amazonaws.services.secretsmanager.model.GetSecretValueRequest;
+import com.amazonaws.services.secretsmanager.model.GetSecretValueResult;
+import com.amazonaws.services.secretsmanager.model.ResourceNotFoundException;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Paths;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+
+/**
+ * Reads secrets from AWS Secrets Manager to provide parameter values. Secrets must be created similar to the following AWS cli command: <br/><br/>
+ * <code>aws secretsmanager create-secret --name "[Context]" --secret-string '{ "[Param]": "[secretValue]", "[Param2]": "[secretValue2]" }'</code> <br/><br/>
+ *
+ * A standard configuration for this provider would be: <br/><br/>
+ *
+ * <code>
+ * nifi.stateless.parameter.provider.AWSSecretsManager.name=AWS Secrets Manager Value Provider
+ * nifi.stateless.parameter.provider.AWSSecretsManager.type=org.apache.nifi.stateless.parameter.AwsSecretsManagerParameterValueProvider
+ * nifi.stateless.parameter.provider.AWSSecretsManager.properties.aws-credentials-file=./conf/bootstrap-aws.conf
+ * </code>
+ */
+public class AwsSecretsManagerParameterValueProvider extends AbstractSecretBasedParameterValueProvider implements ParameterValueProvider {
+ private static final Logger logger = LoggerFactory.getLogger(AwsSecretsManagerParameterValueProvider.class);
+
+ private static final String ACCESS_KEY_PROPS_NAME = "aws.access.key.id";
+ private static final String SECRET_KEY_PROPS_NAME = "aws.secret.access.key";
+ private static final String REGION_KEY_PROPS_NAME = "aws.region";
+
+ public static final PropertyDescriptor AWS_CREDENTIALS_FILE = new PropertyDescriptor.Builder()
+ .displayName("AWS Credentials File")
+ .name("aws-credentials-file")
+ .required(false)
+ .defaultValue("./conf/bootstrap-aws.conf")
+ .description("Location of the bootstrap-aws.conf file that configures the AWS credentials. If not provided, the default AWS credentials will be used.")
+ .addValidator(StandardValidators.FILE_EXISTS_VALIDATOR)
Review comment:
"If not provided, the default AWS credentials will be used." The problem here is that it is always provided, because it has a default value. Probably need to remove the default value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] markap14 commented on a change in pull request #5391: NIFI-9174: Adding AWS SecretsManager ParamValueProvider for Stateless
Posted by GitBox <gi...@apache.org>.
markap14 commented on a change in pull request #5391:
URL: https://github.com/apache/nifi/pull/5391#discussion_r721545479
##########
File path: nifi-stateless/nifi-stateless-api/src/main/java/org/apache/nifi/stateless/parameter/AbstractSecretBasedParameterValueProvider.java
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.stateless.parameter;
+
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * A base class for secret-based <code>ParameterValueProvider</code>s, which map a ParameterContext to a named "Secret" with key/value pairs. This
+ * class allows a default Secret name to be configured for parameters not found in specific ParameterContext Secrets, and uses dynamic user-added
+ * properties to map ParameterContext names to different Secret names. Subclasses must provide the implementation for retrieving the actual
+ * secret values.
+ */
+public abstract class AbstractSecretBasedParameterValueProvider extends AbstractParameterValueProvider implements ParameterValueProvider {
+ private static final Validator NON_EMPTY_VALIDATOR = (subject, value, context) ->
+ new ValidationResult.Builder().subject(subject).input(value).valid(value != null && !value.isEmpty()).explanation(subject + " cannot be empty").build();
+
+ public static final PropertyDescriptor DEFAULT_SECRET_NAME = new PropertyDescriptor.Builder()
+ .displayName("Default Secret Name")
+ .name("default-secret-name")
+ .required(true)
+ .defaultValue("Default")
Review comment:
I think I would make this optional and not specify a default value. As is, the default is to look for some secret named "Default" which is probably not what we want.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] markap14 commented on a change in pull request #5391: NIFI-9174: Adding AWS SecretsManager ParamValueProvider for Stateless
Posted by GitBox <gi...@apache.org>.
markap14 commented on a change in pull request #5391:
URL: https://github.com/apache/nifi/pull/5391#discussion_r719465065
##########
File path: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-parameter-value-providers/src/main/java/org/apache/nifi/stateless/parameter/aws/SecretsManagerParameterValueProvider.java
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.stateless.parameter.aws;
+
+import com.amazonaws.auth.AWSStaticCredentialsProvider;
+import com.amazonaws.auth.BasicAWSCredentials;
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
+import com.amazonaws.services.secretsmanager.AWSSecretsManager;
+import com.amazonaws.services.secretsmanager.AWSSecretsManagerClientBuilder;
+import com.amazonaws.services.secretsmanager.model.GetSecretValueRequest;
+import com.amazonaws.services.secretsmanager.model.GetSecretValueResult;
+import com.amazonaws.services.secretsmanager.model.ListSecretsRequest;
+import com.amazonaws.services.secretsmanager.model.ListSecretsResult;
+import com.amazonaws.services.secretsmanager.model.SecretListEntry;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stateless.parameter.AbstractParameterValueProvider;
+import org.apache.nifi.stateless.parameter.ParameterValueProvider;
+import org.apache.nifi.stateless.parameter.ParameterValueProviderInitializationContext;
+
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Paths;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+
+/**
+ * Reads secrets from AWS Secrets Manager to provide parameter values. Secrets must be created similar to the following AWS cli command: <br/><br/>
+ * <code>aws secretsmanager create-secret --name "[ParamContextName]/[ParamName]" --secret-string '[ParamValue]'</code> <br/><br/>
+ *
+ * A standard configuration for this provider would be: <br/><br/>
+ *
+ * <code>
+ * nifi.stateless.parameter.provider.AWSSecretsManager.name=AWS Secrets Manager Value Provider
+ * nifi.stateless.parameter.provider.AWSSecretsManager.type=org.apache.nifi.stateless.parameter.aws.SecretsManagerParameterValueProvider
+ * nifi.stateless.parameter.provider.AWSSecretsManager.properties.aws-credentials-file=./conf/bootstrap-aws.conf
+ * </code>
+ */
+public class SecretsManagerParameterValueProvider extends AbstractParameterValueProvider implements ParameterValueProvider {
+ private static final String QUALIFIED_SECRET_FORMAT = "%s/%s";
+ private static final String ACCESS_KEY_PROPS_NAME = "aws.access.key.id";
+ private static final String SECRET_KEY_PROPS_NAME = "aws.secret.access.key";
+ private static final String REGION_KEY_PROPS_NAME = "aws.region";
+
+ public static final PropertyDescriptor AWS_CREDENTIALS_FILE = new PropertyDescriptor.Builder()
+ .displayName("AWS Credentials File")
+ .name("aws-credentials-file")
+ .required(false)
+ .defaultValue("./conf/bootstrap-aws.conf")
+ .description("Location of the bootstrap-aws.conf file that configures the AWS credentials. If not provided, the default AWS credentials will be used.")
+ .addValidator(StandardValidators.FILE_EXISTS_VALIDATOR)
+ .build();
+
+ private final Set<String> supportedParameterNames = new HashSet<>();
+
+ private List<PropertyDescriptor> descriptors;
+
+ private AWSSecretsManager secretsManager;
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ protected void init(final ParameterValueProviderInitializationContext context) {
+ super.init(context);
+
+ this.descriptors = Collections.singletonList(AWS_CREDENTIALS_FILE);
+
+ final String awsCredentialsFilename = context.getProperty(AWS_CREDENTIALS_FILE).getValue();
+ try {
+ this.secretsManager = this.configureClient(awsCredentialsFilename);
+ } catch (final IOException e) {
+ throw new IllegalStateException("Could not configure AWS Secrets Manager Client", e);
+ }
+
+ cacheSupportedParameterNames();
+ }
+
+ @Override
+ public boolean isParameterDefined(final String contextName, final String parameterName) {
+ return supportedParameterNames.contains(getSecretName(contextName, parameterName));
+ }
+
+ @Override
+ public String getParameterValue(final String contextName, final String parameterName) {
+ final String secretName = getSecretName(contextName, parameterName);
+ final GetSecretValueRequest getSecretValueRequest = new GetSecretValueRequest()
+ .withSecretId(secretName);
+ final GetSecretValueResult getSecretValueResult = secretsManager.getSecretValue(getSecretValueRequest);
+
+ if (getSecretValueResult == null) {
+ throw new IllegalArgumentException(String.format("Secret [%s] not found", secretName));
+ }
+ if (getSecretValueResult.getSecretString() != null) {
+ return getSecretValueResult.getSecretString();
+ } else {
+ throw new IllegalStateException(String.format("Secret Name [%s] string value not found", secretName));
+ }
+ }
+
+ private void cacheSupportedParameterNames() {
+ supportedParameterNames.clear();
+ final ListSecretsResult listSecretsResult = secretsManager.listSecrets(new ListSecretsRequest());
+ if (listSecretsResult != null) {
+ for (final SecretListEntry entry : listSecretsResult.getSecretList()) {
+ supportedParameterNames.add(entry.getName());
+ }
+ }
+ }
+
+ private Properties loadProperties(final String propertiesFilename) throws IOException {
+ final Properties properties = new Properties();
+
+ try (final InputStream in = new FileInputStream(Paths.get(propertiesFilename).toFile())) {
+ properties.load(in);
+ return properties;
+ }
+ }
+
+ AWSSecretsManager configureClient(final String awsCredentialsFilename) throws IOException {
+ if (awsCredentialsFilename == null) {
+ return getDefaultClient();
+ }
+ final Properties properties = loadProperties(awsCredentialsFilename);
+ final String accessKey = properties.getProperty(ACCESS_KEY_PROPS_NAME);
+ final String secretKey = properties.getProperty(SECRET_KEY_PROPS_NAME);
+ final String region = properties.getProperty(REGION_KEY_PROPS_NAME);
+
+ if (isNotBlank(accessKey) && isNotBlank(secretKey) && isNotBlank(region)) {
+ return AWSSecretsManagerClientBuilder.standard()
+ .withRegion(region)
+ .withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials(accessKey, secretKey)))
+ .build();
+ } else {
+ return getDefaultClient();
+ }
+ }
+
+ private AWSSecretsManager getDefaultClient() {
+ return AWSSecretsManagerClientBuilder.standard()
+ .withCredentials(DefaultAWSCredentialsProviderChain.getInstance())
+ .build();
+ }
+
+ private static String getSecretName(final String contextName, final String parameterName) {
+ return contextName == null ? parameterName : String.format(QUALIFIED_SECRET_FORMAT, contextName, parameterName);
Review comment:
Yes, exactly. Because 90% of the time a stateless flow will only use a single Parameter Context. So we don't want to require that they use the name of the parameter context each time. But, if they have a secret named "Password" or something like that, we do want to support disambiguating that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] gresockj commented on a change in pull request #5391: NIFI-9174: Adding AWS SecretsManager ParamValueProvider for Stateless
Posted by GitBox <gi...@apache.org>.
gresockj commented on a change in pull request #5391:
URL: https://github.com/apache/nifi/pull/5391#discussion_r712409195
##########
File path: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-parameter-value-providers/src/main/java/org/apache/nifi/stateless/parameter/aws/SecretsManagerParameterValueProvider.java
##########
@@ -0,0 +1,208 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.stateless.parameter.aws;
+
+import com.amazonaws.auth.AWSCredentials;
+import com.amazonaws.auth.AWSCredentialsProvider;
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
+import com.amazonaws.services.secretsmanager.AWSSecretsManager;
+import com.amazonaws.services.secretsmanager.AWSSecretsManagerClientBuilder;
+import com.amazonaws.services.secretsmanager.model.GetSecretValueRequest;
+import com.amazonaws.services.secretsmanager.model.GetSecretValueResult;
+import com.amazonaws.services.secretsmanager.model.ListSecretsRequest;
+import com.amazonaws.services.secretsmanager.model.ListSecretsResult;
+import com.amazonaws.services.secretsmanager.model.SecretListEntry;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.ObjectReader;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stateless.parameter.AbstractParameterValueProvider;
+import org.apache.nifi.stateless.parameter.ParameterValueProvider;
+import org.apache.nifi.stateless.parameter.ParameterValueProviderInitializationContext;
+
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Paths;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+
+/**
+ * Reads secrets from AWS Secrets Manager to provide parameter values. Secrets must be created similar to the following AWS cli command: <br/><br/>
+ * <code>aws secretsmanager create-secret --name "[ParamContextName]/[ParamName]" --secret-string '[ParamValue]'</code> <br/><br/>
+ *
+ * A standard configuration for this provider would be: <br/><br/>
+ *
+ * <code>
+ * nifi.stateless.parameter.provider.AWSSecretsManager.name=AWS Secrets Manager Value Provider
+ * nifi.stateless.parameter.provider.AWSSecretsManager.type=org.apache.nifi.stateless.parameter.aws.SecretsManagerParameterValueProvider
+ * nifi.stateless.parameter.provider.AWSSecretsManager.properties.aws-configuration-file=./conf/bootstrap-aws.conf
+ * </code>
+ */
+public class SecretsManagerParameterValueProvider extends AbstractParameterValueProvider implements ParameterValueProvider {
+ private static final String QUALIFIED_SECRET_FORMAT = "%s/%s";
+ private static final String ACCESS_KEY_PROPS_NAME = "aws.access.key.id";
+ private static final String SECRET_KEY_PROPS_NAME = "aws.secret.access.key";
+ private static final String REGION_KEY_PROPS_NAME = "aws.region";
+
+ public static final PropertyDescriptor AWS_CONFIG_FILE = new PropertyDescriptor.Builder()
+ .displayName("AWS Configuration File")
+ .name("aws-configuration-file")
+ .required(false)
+ .defaultValue("./conf/bootstrap-aws.conf")
+ .description("Location of the bootstrap-aws.conf file that configures the AWS credentials. If not provided, the default AWS credentials will be used.")
+ .addValidator(StandardValidators.FILE_EXISTS_VALIDATOR)
+ .build();
+
+ private final ObjectReader objectReader = new ObjectMapper().reader();
+
+ private final Set<String> supportedParameterNames = new HashSet<>();
+
+ private List<PropertyDescriptor> descriptors;
+
+ private AWSSecretsManager secretsManager;
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ protected void init(final ParameterValueProviderInitializationContext context) {
+ super.init(context);
+
+ this.descriptors = Collections.unmodifiableList(Arrays.asList(AWS_CONFIG_FILE));
+
+ final String awsBootstrapConfFilename = context.getProperty(AWS_CONFIG_FILE).getValue();
+ try {
+ this.secretsManager = this.configureClient(awsBootstrapConfFilename);
+ } catch (final IOException e) {
+ throw new IllegalStateException("Could not configure AWS Secrets Manager Client", e);
+ }
+
+ cacheSupportedParameterNames();
+ }
+
+ private void cacheSupportedParameterNames() {
+ supportedParameterNames.clear();
+ final ListSecretsResult listSecretsResult = secretsManager.listSecrets(new ListSecretsRequest());
+ if (listSecretsResult != null) {
+ for (final SecretListEntry entry : listSecretsResult.getSecretList()) {
+ supportedParameterNames.add(entry.getName());
+ }
+ }
+ }
+
+ @Override
+ public boolean isParameterDefined(final String contextName, final String parameterName) {
+ return supportedParameterNames.contains(getSecretName(contextName, parameterName));
+ }
+
+ @Override
+ public String getParameterValue(final String contextName, final String parameterName) {
+ final String secretName = getSecretName(contextName, parameterName);
+ final GetSecretValueRequest getSecretValueRequest = new GetSecretValueRequest()
+ .withSecretId(secretName);
+ final GetSecretValueResult getSecretValueResult = secretsManager.getSecretValue(getSecretValueRequest);
+
+ if (getSecretValueResult == null) {
+ throw new IllegalArgumentException(String.format("Secret [%s] not found", secretName));
+ }
+ if (getSecretValueResult.getSecretString() != null) {
+ return getSecretValueResult.getSecretString();
+ } else {
+ throw new IllegalStateException("Binary secrets are not supported");
+ }
+ }
+
+ private Properties loadProperties(final String propertiesFilename) throws IOException {
+ final Properties properties = new Properties();
+
+ try (final InputStream in = new FileInputStream(Paths.get(propertiesFilename).toFile())) {
+ properties.load(in);
+ return properties;
+ }
+ }
+
+ AWSSecretsManager configureClient(final String awsBootstrapConfFilename) throws IOException {
+ if (awsBootstrapConfFilename == null) {
+ return getDefaultClient();
+ }
+ final Properties properties = loadProperties(awsBootstrapConfFilename);
+ final String accessKey = properties.getProperty(ACCESS_KEY_PROPS_NAME);
+ final String secretKey = properties.getProperty(SECRET_KEY_PROPS_NAME);
+ final String region = properties.getProperty(REGION_KEY_PROPS_NAME);
+
+ if (isNotBlank(accessKey) && isNotBlank(secretKey) && isNotBlank(region)) {
+ return AWSSecretsManagerClientBuilder.standard()
+ .withRegion(region)
+ .withCredentials(new BasicAWSCredentialsProvider(accessKey, secretKey))
+ .build();
+ } else {
+ return getDefaultClient();
+ }
+ }
+
+ private AWSSecretsManager getDefaultClient() {
+ return AWSSecretsManagerClientBuilder.standard()
+ .withCredentials(DefaultAWSCredentialsProviderChain.getInstance())
+ .build();
+ }
+
+ private static String getSecretName(final String contextName, final String parameterName) {
+ return String.format(QUALIFIED_SECRET_FORMAT, contextName, parameterName);
+ }
+
+ private static boolean isNotBlank(final String value) {
+ return value != null && !value.trim().equals("");
+ }
+
+ private static class BasicAWSCredentialsProvider implements AWSCredentialsProvider {
Review comment:
Good catch with `BasicAWSCredentials`, I missed that class. Now, the `AwsClientBuilder#withCredentials` (of which `AWSSecretsManagerClientBuilder` is a subclass) requires an `AWSCredentialsProvider` rather than an `AWSCredentials`, but `BasicAWSCredentials` would definitely be preferable to the anonymous class currently returned by the `getCredentials` method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] markap14 commented on a change in pull request #5391: NIFI-9174: Adding AWS SecretsManager ParamValueProvider for Stateless
Posted by GitBox <gi...@apache.org>.
markap14 commented on a change in pull request #5391:
URL: https://github.com/apache/nifi/pull/5391#discussion_r719387912
##########
File path: nifi-stateless/nifi-stateless-assembly/README.md
##########
@@ -515,3 +515,31 @@ nifi.stateless.parameter.provider.Vault.name=HashiCorp Vault Provider
nifi.stateless.parameter.provider.Vault.type=org.apache.nifi.stateless.parameter.HashiCorpVaultParameterValueProvider
nifi.stateless.parameter.provider.Vault.properties.vault-configuration-file=./conf/bootstrap-hashicorp-vault.conf
```
+
+**AWS SecretsManagerParameterValueProvider**
+
+This provider reads parameter values from AWS SecretsManager. The AWS credentials can be configured
+via the `./conf/bootstrap-aws.conf` file, which comes with NiFi.
Review comment:
Might be worth mentioning here that whatever credentials are supplied must have permissions both to List Secrets and to retrieve the values of the secrets.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] gresockj commented on pull request #5391: NIFI-9174: Adding AWS SecretsManager ParamValueProvider for Stateless
Posted by GitBox <gi...@apache.org>.
gresockj commented on pull request #5391:
URL: https://github.com/apache/nifi/pull/5391#issuecomment-922316815
> Thanks for working on this feature @gresockj, this will be a very useful integration option. At a high level, is the a reason for requiring secrets to be stored as JSON objects? It seems like requiring the JSON object wrapping around `value` introduces potential complexity when populating secrets from other tools. Why not just use `secretString` directly as the parameter value?
>
> On closer inspection, I see that the HashiCorp Vault Parameter Value Provider does require the wrapping object, but perhaps that is something to revisit as well.
Great question -- regarding AWS SecretsManager, it looks like I missed in AWS's documentation the possibility of setting a plain value instead of the "key/value pair" format that translates to JSON. I agree that the simple value is the best approach, and will make that change.
Regarding HashiCorp Vault, I believe this is the only way Secrets are represented in the K/V Secrets Engine (see https://learn.hashicorp.com/tutorials/vault/getting-started-first-secret). So when you put a secret like this:
```
vault kv put secret/hello foo=world
```
then the secret is represented as { "foo": "world" } in the API. However, if I find a way to avoid using JSON with Vault, I'll address that as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a change in pull request #5391: NIFI-9174: Adding AWS SecretsManager ParamValueProvider for Stateless
Posted by GitBox <gi...@apache.org>.
exceptionfactory commented on a change in pull request #5391:
URL: https://github.com/apache/nifi/pull/5391#discussion_r712369520
##########
File path: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-parameter-value-providers/src/main/java/org/apache/nifi/stateless/parameter/aws/SecretsManagerParameterValueProvider.java
##########
@@ -0,0 +1,208 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.stateless.parameter.aws;
+
+import com.amazonaws.auth.AWSCredentials;
+import com.amazonaws.auth.AWSCredentialsProvider;
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
+import com.amazonaws.services.secretsmanager.AWSSecretsManager;
+import com.amazonaws.services.secretsmanager.AWSSecretsManagerClientBuilder;
+import com.amazonaws.services.secretsmanager.model.GetSecretValueRequest;
+import com.amazonaws.services.secretsmanager.model.GetSecretValueResult;
+import com.amazonaws.services.secretsmanager.model.ListSecretsRequest;
+import com.amazonaws.services.secretsmanager.model.ListSecretsResult;
+import com.amazonaws.services.secretsmanager.model.SecretListEntry;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.ObjectReader;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stateless.parameter.AbstractParameterValueProvider;
+import org.apache.nifi.stateless.parameter.ParameterValueProvider;
+import org.apache.nifi.stateless.parameter.ParameterValueProviderInitializationContext;
+
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Paths;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+
+/**
+ * Reads secrets from AWS Secrets Manager to provide parameter values. Secrets must be created similar to the following AWS cli command: <br/><br/>
+ * <code>aws secretsmanager create-secret --name "[ParamContextName]/[ParamName]" --secret-string '[ParamValue]'</code> <br/><br/>
+ *
+ * A standard configuration for this provider would be: <br/><br/>
+ *
+ * <code>
+ * nifi.stateless.parameter.provider.AWSSecretsManager.name=AWS Secrets Manager Value Provider
+ * nifi.stateless.parameter.provider.AWSSecretsManager.type=org.apache.nifi.stateless.parameter.aws.SecretsManagerParameterValueProvider
+ * nifi.stateless.parameter.provider.AWSSecretsManager.properties.aws-configuration-file=./conf/bootstrap-aws.conf
+ * </code>
+ */
+public class SecretsManagerParameterValueProvider extends AbstractParameterValueProvider implements ParameterValueProvider {
+ private static final String QUALIFIED_SECRET_FORMAT = "%s/%s";
+ private static final String ACCESS_KEY_PROPS_NAME = "aws.access.key.id";
+ private static final String SECRET_KEY_PROPS_NAME = "aws.secret.access.key";
+ private static final String REGION_KEY_PROPS_NAME = "aws.region";
+
+ public static final PropertyDescriptor AWS_CONFIG_FILE = new PropertyDescriptor.Builder()
+ .displayName("AWS Configuration File")
+ .name("aws-configuration-file")
+ .required(false)
+ .defaultValue("./conf/bootstrap-aws.conf")
+ .description("Location of the bootstrap-aws.conf file that configures the AWS credentials. If not provided, the default AWS credentials will be used.")
+ .addValidator(StandardValidators.FILE_EXISTS_VALIDATOR)
+ .build();
+
+ private final ObjectReader objectReader = new ObjectMapper().reader();
Review comment:
This `ObjectReader` instance does not appear to be used.
##########
File path: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-parameter-value-providers/src/main/java/org/apache/nifi/stateless/parameter/aws/SecretsManagerParameterValueProvider.java
##########
@@ -0,0 +1,208 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.stateless.parameter.aws;
+
+import com.amazonaws.auth.AWSCredentials;
+import com.amazonaws.auth.AWSCredentialsProvider;
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
+import com.amazonaws.services.secretsmanager.AWSSecretsManager;
+import com.amazonaws.services.secretsmanager.AWSSecretsManagerClientBuilder;
+import com.amazonaws.services.secretsmanager.model.GetSecretValueRequest;
+import com.amazonaws.services.secretsmanager.model.GetSecretValueResult;
+import com.amazonaws.services.secretsmanager.model.ListSecretsRequest;
+import com.amazonaws.services.secretsmanager.model.ListSecretsResult;
+import com.amazonaws.services.secretsmanager.model.SecretListEntry;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.ObjectReader;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stateless.parameter.AbstractParameterValueProvider;
+import org.apache.nifi.stateless.parameter.ParameterValueProvider;
+import org.apache.nifi.stateless.parameter.ParameterValueProviderInitializationContext;
+
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Paths;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+
+/**
+ * Reads secrets from AWS Secrets Manager to provide parameter values. Secrets must be created similar to the following AWS cli command: <br/><br/>
+ * <code>aws secretsmanager create-secret --name "[ParamContextName]/[ParamName]" --secret-string '[ParamValue]'</code> <br/><br/>
+ *
+ * A standard configuration for this provider would be: <br/><br/>
+ *
+ * <code>
+ * nifi.stateless.parameter.provider.AWSSecretsManager.name=AWS Secrets Manager Value Provider
+ * nifi.stateless.parameter.provider.AWSSecretsManager.type=org.apache.nifi.stateless.parameter.aws.SecretsManagerParameterValueProvider
+ * nifi.stateless.parameter.provider.AWSSecretsManager.properties.aws-configuration-file=./conf/bootstrap-aws.conf
+ * </code>
+ */
+public class SecretsManagerParameterValueProvider extends AbstractParameterValueProvider implements ParameterValueProvider {
+ private static final String QUALIFIED_SECRET_FORMAT = "%s/%s";
+ private static final String ACCESS_KEY_PROPS_NAME = "aws.access.key.id";
+ private static final String SECRET_KEY_PROPS_NAME = "aws.secret.access.key";
+ private static final String REGION_KEY_PROPS_NAME = "aws.region";
+
+ public static final PropertyDescriptor AWS_CONFIG_FILE = new PropertyDescriptor.Builder()
+ .displayName("AWS Configuration File")
+ .name("aws-configuration-file")
Review comment:
For clarity, what do you think about renaming this to AWS Credentials File?
```suggestion
public static final PropertyDescriptor AWS_CREDENTIALS_FILE = new PropertyDescriptor.Builder()
.displayName("AWS Credentials File")
.name("aws-credentials-file")
```
##########
File path: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-parameter-value-providers/src/main/java/org/apache/nifi/stateless/parameter/aws/SecretsManagerParameterValueProvider.java
##########
@@ -0,0 +1,208 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.stateless.parameter.aws;
+
+import com.amazonaws.auth.AWSCredentials;
+import com.amazonaws.auth.AWSCredentialsProvider;
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
+import com.amazonaws.services.secretsmanager.AWSSecretsManager;
+import com.amazonaws.services.secretsmanager.AWSSecretsManagerClientBuilder;
+import com.amazonaws.services.secretsmanager.model.GetSecretValueRequest;
+import com.amazonaws.services.secretsmanager.model.GetSecretValueResult;
+import com.amazonaws.services.secretsmanager.model.ListSecretsRequest;
+import com.amazonaws.services.secretsmanager.model.ListSecretsResult;
+import com.amazonaws.services.secretsmanager.model.SecretListEntry;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.ObjectReader;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stateless.parameter.AbstractParameterValueProvider;
+import org.apache.nifi.stateless.parameter.ParameterValueProvider;
+import org.apache.nifi.stateless.parameter.ParameterValueProviderInitializationContext;
+
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Paths;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+
+/**
+ * Reads secrets from AWS Secrets Manager to provide parameter values. Secrets must be created similar to the following AWS cli command: <br/><br/>
+ * <code>aws secretsmanager create-secret --name "[ParamContextName]/[ParamName]" --secret-string '[ParamValue]'</code> <br/><br/>
+ *
+ * A standard configuration for this provider would be: <br/><br/>
+ *
+ * <code>
+ * nifi.stateless.parameter.provider.AWSSecretsManager.name=AWS Secrets Manager Value Provider
+ * nifi.stateless.parameter.provider.AWSSecretsManager.type=org.apache.nifi.stateless.parameter.aws.SecretsManagerParameterValueProvider
+ * nifi.stateless.parameter.provider.AWSSecretsManager.properties.aws-configuration-file=./conf/bootstrap-aws.conf
+ * </code>
+ */
+public class SecretsManagerParameterValueProvider extends AbstractParameterValueProvider implements ParameterValueProvider {
+ private static final String QUALIFIED_SECRET_FORMAT = "%s/%s";
+ private static final String ACCESS_KEY_PROPS_NAME = "aws.access.key.id";
+ private static final String SECRET_KEY_PROPS_NAME = "aws.secret.access.key";
+ private static final String REGION_KEY_PROPS_NAME = "aws.region";
+
+ public static final PropertyDescriptor AWS_CONFIG_FILE = new PropertyDescriptor.Builder()
+ .displayName("AWS Configuration File")
+ .name("aws-configuration-file")
+ .required(false)
+ .defaultValue("./conf/bootstrap-aws.conf")
+ .description("Location of the bootstrap-aws.conf file that configures the AWS credentials. If not provided, the default AWS credentials will be used.")
+ .addValidator(StandardValidators.FILE_EXISTS_VALIDATOR)
+ .build();
+
+ private final ObjectReader objectReader = new ObjectMapper().reader();
+
+ private final Set<String> supportedParameterNames = new HashSet<>();
+
+ private List<PropertyDescriptor> descriptors;
+
+ private AWSSecretsManager secretsManager;
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ protected void init(final ParameterValueProviderInitializationContext context) {
+ super.init(context);
+
+ this.descriptors = Collections.unmodifiableList(Arrays.asList(AWS_CONFIG_FILE));
+
+ final String awsBootstrapConfFilename = context.getProperty(AWS_CONFIG_FILE).getValue();
+ try {
+ this.secretsManager = this.configureClient(awsBootstrapConfFilename);
+ } catch (final IOException e) {
+ throw new IllegalStateException("Could not configure AWS Secrets Manager Client", e);
+ }
+
+ cacheSupportedParameterNames();
+ }
+
+ private void cacheSupportedParameterNames() {
+ supportedParameterNames.clear();
+ final ListSecretsResult listSecretsResult = secretsManager.listSecrets(new ListSecretsRequest());
+ if (listSecretsResult != null) {
+ for (final SecretListEntry entry : listSecretsResult.getSecretList()) {
+ supportedParameterNames.add(entry.getName());
+ }
+ }
+ }
+
+ @Override
+ public boolean isParameterDefined(final String contextName, final String parameterName) {
+ return supportedParameterNames.contains(getSecretName(contextName, parameterName));
+ }
+
+ @Override
+ public String getParameterValue(final String contextName, final String parameterName) {
+ final String secretName = getSecretName(contextName, parameterName);
+ final GetSecretValueRequest getSecretValueRequest = new GetSecretValueRequest()
+ .withSecretId(secretName);
+ final GetSecretValueResult getSecretValueResult = secretsManager.getSecretValue(getSecretValueRequest);
+
+ if (getSecretValueResult == null) {
+ throw new IllegalArgumentException(String.format("Secret [%s] not found", secretName));
+ }
+ if (getSecretValueResult.getSecretString() != null) {
+ return getSecretValueResult.getSecretString();
+ } else {
+ throw new IllegalStateException("Binary secrets are not supported");
Review comment:
Reading the documentation for [getSecretString() ](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/secretsmanager/model/GetSecretValueResult.html#getSecretString--), it sounds like the secret might have been stored in some other format. Recommend adjusting the error message to include the secret name:
```suggestion
throw new IllegalStateException(String.format("Secret Name [%s] string value not found", secretName));
```
##########
File path: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-parameter-value-providers/src/main/java/org/apache/nifi/stateless/parameter/aws/SecretsManagerParameterValueProvider.java
##########
@@ -0,0 +1,208 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.stateless.parameter.aws;
+
+import com.amazonaws.auth.AWSCredentials;
+import com.amazonaws.auth.AWSCredentialsProvider;
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
+import com.amazonaws.services.secretsmanager.AWSSecretsManager;
+import com.amazonaws.services.secretsmanager.AWSSecretsManagerClientBuilder;
+import com.amazonaws.services.secretsmanager.model.GetSecretValueRequest;
+import com.amazonaws.services.secretsmanager.model.GetSecretValueResult;
+import com.amazonaws.services.secretsmanager.model.ListSecretsRequest;
+import com.amazonaws.services.secretsmanager.model.ListSecretsResult;
+import com.amazonaws.services.secretsmanager.model.SecretListEntry;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.ObjectReader;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stateless.parameter.AbstractParameterValueProvider;
+import org.apache.nifi.stateless.parameter.ParameterValueProvider;
+import org.apache.nifi.stateless.parameter.ParameterValueProviderInitializationContext;
+
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Paths;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+
+/**
+ * Reads secrets from AWS Secrets Manager to provide parameter values. Secrets must be created similar to the following AWS cli command: <br/><br/>
+ * <code>aws secretsmanager create-secret --name "[ParamContextName]/[ParamName]" --secret-string '[ParamValue]'</code> <br/><br/>
+ *
+ * A standard configuration for this provider would be: <br/><br/>
+ *
+ * <code>
+ * nifi.stateless.parameter.provider.AWSSecretsManager.name=AWS Secrets Manager Value Provider
+ * nifi.stateless.parameter.provider.AWSSecretsManager.type=org.apache.nifi.stateless.parameter.aws.SecretsManagerParameterValueProvider
+ * nifi.stateless.parameter.provider.AWSSecretsManager.properties.aws-configuration-file=./conf/bootstrap-aws.conf
+ * </code>
+ */
+public class SecretsManagerParameterValueProvider extends AbstractParameterValueProvider implements ParameterValueProvider {
+ private static final String QUALIFIED_SECRET_FORMAT = "%s/%s";
+ private static final String ACCESS_KEY_PROPS_NAME = "aws.access.key.id";
+ private static final String SECRET_KEY_PROPS_NAME = "aws.secret.access.key";
+ private static final String REGION_KEY_PROPS_NAME = "aws.region";
+
+ public static final PropertyDescriptor AWS_CONFIG_FILE = new PropertyDescriptor.Builder()
+ .displayName("AWS Configuration File")
+ .name("aws-configuration-file")
+ .required(false)
+ .defaultValue("./conf/bootstrap-aws.conf")
+ .description("Location of the bootstrap-aws.conf file that configures the AWS credentials. If not provided, the default AWS credentials will be used.")
+ .addValidator(StandardValidators.FILE_EXISTS_VALIDATOR)
+ .build();
+
+ private final ObjectReader objectReader = new ObjectMapper().reader();
+
+ private final Set<String> supportedParameterNames = new HashSet<>();
+
+ private List<PropertyDescriptor> descriptors;
+
+ private AWSSecretsManager secretsManager;
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ protected void init(final ParameterValueProviderInitializationContext context) {
+ super.init(context);
+
+ this.descriptors = Collections.unmodifiableList(Arrays.asList(AWS_CONFIG_FILE));
+
+ final String awsBootstrapConfFilename = context.getProperty(AWS_CONFIG_FILE).getValue();
+ try {
+ this.secretsManager = this.configureClient(awsBootstrapConfFilename);
+ } catch (final IOException e) {
+ throw new IllegalStateException("Could not configure AWS Secrets Manager Client", e);
+ }
+
+ cacheSupportedParameterNames();
+ }
+
+ private void cacheSupportedParameterNames() {
+ supportedParameterNames.clear();
+ final ListSecretsResult listSecretsResult = secretsManager.listSecrets(new ListSecretsRequest());
+ if (listSecretsResult != null) {
+ for (final SecretListEntry entry : listSecretsResult.getSecretList()) {
+ supportedParameterNames.add(entry.getName());
+ }
+ }
+ }
+
+ @Override
+ public boolean isParameterDefined(final String contextName, final String parameterName) {
+ return supportedParameterNames.contains(getSecretName(contextName, parameterName));
+ }
+
+ @Override
+ public String getParameterValue(final String contextName, final String parameterName) {
+ final String secretName = getSecretName(contextName, parameterName);
+ final GetSecretValueRequest getSecretValueRequest = new GetSecretValueRequest()
+ .withSecretId(secretName);
+ final GetSecretValueResult getSecretValueResult = secretsManager.getSecretValue(getSecretValueRequest);
+
+ if (getSecretValueResult == null) {
+ throw new IllegalArgumentException(String.format("Secret [%s] not found", secretName));
+ }
+ if (getSecretValueResult.getSecretString() != null) {
+ return getSecretValueResult.getSecretString();
+ } else {
+ throw new IllegalStateException("Binary secrets are not supported");
+ }
+ }
+
+ private Properties loadProperties(final String propertiesFilename) throws IOException {
+ final Properties properties = new Properties();
+
+ try (final InputStream in = new FileInputStream(Paths.get(propertiesFilename).toFile())) {
+ properties.load(in);
+ return properties;
+ }
+ }
+
+ AWSSecretsManager configureClient(final String awsBootstrapConfFilename) throws IOException {
+ if (awsBootstrapConfFilename == null) {
+ return getDefaultClient();
+ }
+ final Properties properties = loadProperties(awsBootstrapConfFilename);
+ final String accessKey = properties.getProperty(ACCESS_KEY_PROPS_NAME);
+ final String secretKey = properties.getProperty(SECRET_KEY_PROPS_NAME);
+ final String region = properties.getProperty(REGION_KEY_PROPS_NAME);
+
+ if (isNotBlank(accessKey) && isNotBlank(secretKey) && isNotBlank(region)) {
+ return AWSSecretsManagerClientBuilder.standard()
+ .withRegion(region)
+ .withCredentials(new BasicAWSCredentialsProvider(accessKey, secretKey))
+ .build();
+ } else {
+ return getDefaultClient();
+ }
+ }
+
+ private AWSSecretsManager getDefaultClient() {
+ return AWSSecretsManagerClientBuilder.standard()
+ .withCredentials(DefaultAWSCredentialsProviderChain.getInstance())
+ .build();
+ }
+
+ private static String getSecretName(final String contextName, final String parameterName) {
+ return String.format(QUALIFIED_SECRET_FORMAT, contextName, parameterName);
+ }
+
+ private static boolean isNotBlank(final String value) {
+ return value != null && !value.trim().equals("");
+ }
+
+ private static class BasicAWSCredentialsProvider implements AWSCredentialsProvider {
Review comment:
Is this class necessary? It seems like the standard [BasicAWSCredentials](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/BasicAWSCredentials.html) class should work.
##########
File path: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-parameter-value-providers/src/main/java/org/apache/nifi/stateless/parameter/aws/SecretsManagerParameterValueProvider.java
##########
@@ -0,0 +1,208 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.stateless.parameter.aws;
+
+import com.amazonaws.auth.AWSCredentials;
+import com.amazonaws.auth.AWSCredentialsProvider;
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
+import com.amazonaws.services.secretsmanager.AWSSecretsManager;
+import com.amazonaws.services.secretsmanager.AWSSecretsManagerClientBuilder;
+import com.amazonaws.services.secretsmanager.model.GetSecretValueRequest;
+import com.amazonaws.services.secretsmanager.model.GetSecretValueResult;
+import com.amazonaws.services.secretsmanager.model.ListSecretsRequest;
+import com.amazonaws.services.secretsmanager.model.ListSecretsResult;
+import com.amazonaws.services.secretsmanager.model.SecretListEntry;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.ObjectReader;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stateless.parameter.AbstractParameterValueProvider;
+import org.apache.nifi.stateless.parameter.ParameterValueProvider;
+import org.apache.nifi.stateless.parameter.ParameterValueProviderInitializationContext;
+
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Paths;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+
+/**
+ * Reads secrets from AWS Secrets Manager to provide parameter values. Secrets must be created similar to the following AWS cli command: <br/><br/>
+ * <code>aws secretsmanager create-secret --name "[ParamContextName]/[ParamName]" --secret-string '[ParamValue]'</code> <br/><br/>
+ *
+ * A standard configuration for this provider would be: <br/><br/>
+ *
+ * <code>
+ * nifi.stateless.parameter.provider.AWSSecretsManager.name=AWS Secrets Manager Value Provider
+ * nifi.stateless.parameter.provider.AWSSecretsManager.type=org.apache.nifi.stateless.parameter.aws.SecretsManagerParameterValueProvider
+ * nifi.stateless.parameter.provider.AWSSecretsManager.properties.aws-configuration-file=./conf/bootstrap-aws.conf
+ * </code>
+ */
+public class SecretsManagerParameterValueProvider extends AbstractParameterValueProvider implements ParameterValueProvider {
+ private static final String QUALIFIED_SECRET_FORMAT = "%s/%s";
+ private static final String ACCESS_KEY_PROPS_NAME = "aws.access.key.id";
+ private static final String SECRET_KEY_PROPS_NAME = "aws.secret.access.key";
+ private static final String REGION_KEY_PROPS_NAME = "aws.region";
+
+ public static final PropertyDescriptor AWS_CONFIG_FILE = new PropertyDescriptor.Builder()
+ .displayName("AWS Configuration File")
+ .name("aws-configuration-file")
+ .required(false)
+ .defaultValue("./conf/bootstrap-aws.conf")
+ .description("Location of the bootstrap-aws.conf file that configures the AWS credentials. If not provided, the default AWS credentials will be used.")
+ .addValidator(StandardValidators.FILE_EXISTS_VALIDATOR)
+ .build();
+
+ private final ObjectReader objectReader = new ObjectMapper().reader();
+
+ private final Set<String> supportedParameterNames = new HashSet<>();
+
+ private List<PropertyDescriptor> descriptors;
+
+ private AWSSecretsManager secretsManager;
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ protected void init(final ParameterValueProviderInitializationContext context) {
+ super.init(context);
+
+ this.descriptors = Collections.unmodifiableList(Arrays.asList(AWS_CONFIG_FILE));
+
+ final String awsBootstrapConfFilename = context.getProperty(AWS_CONFIG_FILE).getValue();
+ try {
+ this.secretsManager = this.configureClient(awsBootstrapConfFilename);
+ } catch (final IOException e) {
+ throw new IllegalStateException("Could not configure AWS Secrets Manager Client", e);
+ }
+
+ cacheSupportedParameterNames();
+ }
+
+ private void cacheSupportedParameterNames() {
+ supportedParameterNames.clear();
+ final ListSecretsResult listSecretsResult = secretsManager.listSecrets(new ListSecretsRequest());
+ if (listSecretsResult != null) {
+ for (final SecretListEntry entry : listSecretsResult.getSecretList()) {
+ supportedParameterNames.add(entry.getName());
+ }
+ }
+ }
+
+ @Override
+ public boolean isParameterDefined(final String contextName, final String parameterName) {
+ return supportedParameterNames.contains(getSecretName(contextName, parameterName));
+ }
+
+ @Override
+ public String getParameterValue(final String contextName, final String parameterName) {
Review comment:
Minor suggestion, what do you think about moving these `public` methods up in the class before other `protected` and `private` methods?
##########
File path: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-parameter-value-providers/src/main/java/org/apache/nifi/stateless/parameter/aws/SecretsManagerParameterValueProvider.java
##########
@@ -0,0 +1,208 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.stateless.parameter.aws;
+
+import com.amazonaws.auth.AWSCredentials;
+import com.amazonaws.auth.AWSCredentialsProvider;
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
+import com.amazonaws.services.secretsmanager.AWSSecretsManager;
+import com.amazonaws.services.secretsmanager.AWSSecretsManagerClientBuilder;
+import com.amazonaws.services.secretsmanager.model.GetSecretValueRequest;
+import com.amazonaws.services.secretsmanager.model.GetSecretValueResult;
+import com.amazonaws.services.secretsmanager.model.ListSecretsRequest;
+import com.amazonaws.services.secretsmanager.model.ListSecretsResult;
+import com.amazonaws.services.secretsmanager.model.SecretListEntry;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.ObjectReader;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stateless.parameter.AbstractParameterValueProvider;
+import org.apache.nifi.stateless.parameter.ParameterValueProvider;
+import org.apache.nifi.stateless.parameter.ParameterValueProviderInitializationContext;
+
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Paths;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+
+/**
+ * Reads secrets from AWS Secrets Manager to provide parameter values. Secrets must be created similar to the following AWS cli command: <br/><br/>
+ * <code>aws secretsmanager create-secret --name "[ParamContextName]/[ParamName]" --secret-string '[ParamValue]'</code> <br/><br/>
+ *
+ * A standard configuration for this provider would be: <br/><br/>
+ *
+ * <code>
+ * nifi.stateless.parameter.provider.AWSSecretsManager.name=AWS Secrets Manager Value Provider
+ * nifi.stateless.parameter.provider.AWSSecretsManager.type=org.apache.nifi.stateless.parameter.aws.SecretsManagerParameterValueProvider
+ * nifi.stateless.parameter.provider.AWSSecretsManager.properties.aws-configuration-file=./conf/bootstrap-aws.conf
+ * </code>
+ */
+public class SecretsManagerParameterValueProvider extends AbstractParameterValueProvider implements ParameterValueProvider {
+ private static final String QUALIFIED_SECRET_FORMAT = "%s/%s";
+ private static final String ACCESS_KEY_PROPS_NAME = "aws.access.key.id";
+ private static final String SECRET_KEY_PROPS_NAME = "aws.secret.access.key";
+ private static final String REGION_KEY_PROPS_NAME = "aws.region";
+
+ public static final PropertyDescriptor AWS_CONFIG_FILE = new PropertyDescriptor.Builder()
+ .displayName("AWS Configuration File")
+ .name("aws-configuration-file")
+ .required(false)
+ .defaultValue("./conf/bootstrap-aws.conf")
+ .description("Location of the bootstrap-aws.conf file that configures the AWS credentials. If not provided, the default AWS credentials will be used.")
+ .addValidator(StandardValidators.FILE_EXISTS_VALIDATOR)
+ .build();
+
+ private final ObjectReader objectReader = new ObjectMapper().reader();
+
+ private final Set<String> supportedParameterNames = new HashSet<>();
+
+ private List<PropertyDescriptor> descriptors;
+
+ private AWSSecretsManager secretsManager;
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ protected void init(final ParameterValueProviderInitializationContext context) {
+ super.init(context);
+
+ this.descriptors = Collections.unmodifiableList(Arrays.asList(AWS_CONFIG_FILE));
+
+ final String awsBootstrapConfFilename = context.getProperty(AWS_CONFIG_FILE).getValue();
Review comment:
Recommend renaming the variable for clarity:
```suggestion
final String awsCredentialsFilename = context.getProperty(AWS_CONFIG_FILE).getValue();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] markap14 commented on a change in pull request #5391: NIFI-9174: Adding AWS SecretsManager ParamValueProvider for Stateless
Posted by GitBox <gi...@apache.org>.
markap14 commented on a change in pull request #5391:
URL: https://github.com/apache/nifi/pull/5391#discussion_r719453065
##########
File path: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-parameter-value-providers/src/main/java/org/apache/nifi/stateless/parameter/aws/SecretsManagerParameterValueProvider.java
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.stateless.parameter.aws;
+
+import com.amazonaws.auth.AWSStaticCredentialsProvider;
+import com.amazonaws.auth.BasicAWSCredentials;
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
+import com.amazonaws.services.secretsmanager.AWSSecretsManager;
+import com.amazonaws.services.secretsmanager.AWSSecretsManagerClientBuilder;
+import com.amazonaws.services.secretsmanager.model.GetSecretValueRequest;
+import com.amazonaws.services.secretsmanager.model.GetSecretValueResult;
+import com.amazonaws.services.secretsmanager.model.ListSecretsRequest;
+import com.amazonaws.services.secretsmanager.model.ListSecretsResult;
+import com.amazonaws.services.secretsmanager.model.SecretListEntry;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stateless.parameter.AbstractParameterValueProvider;
+import org.apache.nifi.stateless.parameter.ParameterValueProvider;
+import org.apache.nifi.stateless.parameter.ParameterValueProviderInitializationContext;
+
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Paths;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+
+/**
+ * Reads secrets from AWS Secrets Manager to provide parameter values. Secrets must be created similar to the following AWS cli command: <br/><br/>
+ * <code>aws secretsmanager create-secret --name "[ParamContextName]/[ParamName]" --secret-string '[ParamValue]'</code> <br/><br/>
+ *
+ * A standard configuration for this provider would be: <br/><br/>
+ *
+ * <code>
+ * nifi.stateless.parameter.provider.AWSSecretsManager.name=AWS Secrets Manager Value Provider
+ * nifi.stateless.parameter.provider.AWSSecretsManager.type=org.apache.nifi.stateless.parameter.aws.SecretsManagerParameterValueProvider
+ * nifi.stateless.parameter.provider.AWSSecretsManager.properties.aws-credentials-file=./conf/bootstrap-aws.conf
+ * </code>
+ */
+public class SecretsManagerParameterValueProvider extends AbstractParameterValueProvider implements ParameterValueProvider {
+ private static final String QUALIFIED_SECRET_FORMAT = "%s/%s";
+ private static final String ACCESS_KEY_PROPS_NAME = "aws.access.key.id";
+ private static final String SECRET_KEY_PROPS_NAME = "aws.secret.access.key";
+ private static final String REGION_KEY_PROPS_NAME = "aws.region";
+
+ public static final PropertyDescriptor AWS_CREDENTIALS_FILE = new PropertyDescriptor.Builder()
+ .displayName("AWS Credentials File")
+ .name("aws-credentials-file")
+ .required(false)
+ .defaultValue("./conf/bootstrap-aws.conf")
+ .description("Location of the bootstrap-aws.conf file that configures the AWS credentials. If not provided, the default AWS credentials will be used.")
+ .addValidator(StandardValidators.FILE_EXISTS_VALIDATOR)
+ .build();
+
+ private final Set<String> supportedParameterNames = new HashSet<>();
+
+ private List<PropertyDescriptor> descriptors;
+
+ private AWSSecretsManager secretsManager;
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ protected void init(final ParameterValueProviderInitializationContext context) {
+ super.init(context);
+
+ this.descriptors = Collections.singletonList(AWS_CREDENTIALS_FILE);
+
+ final String awsCredentialsFilename = context.getProperty(AWS_CREDENTIALS_FILE).getValue();
+ try {
+ this.secretsManager = this.configureClient(awsCredentialsFilename);
+ } catch (final IOException e) {
+ throw new IllegalStateException("Could not configure AWS Secrets Manager Client", e);
+ }
+
+ cacheSupportedParameterNames();
+ }
+
+ @Override
+ public boolean isParameterDefined(final String contextName, final String parameterName) {
+ return supportedParameterNames.contains(getSecretName(contextName, parameterName));
+ }
+
+ @Override
+ public String getParameterValue(final String contextName, final String parameterName) {
+ final String secretName = getSecretName(contextName, parameterName);
+ final GetSecretValueRequest getSecretValueRequest = new GetSecretValueRequest()
+ .withSecretId(secretName);
+ final GetSecretValueResult getSecretValueResult = secretsManager.getSecretValue(getSecretValueRequest);
+
+ if (getSecretValueResult == null) {
+ throw new IllegalArgumentException(String.format("Secret [%s] not found", secretName));
+ }
+ if (getSecretValueResult.getSecretString() != null) {
+ return getSecretValueResult.getSecretString();
+ } else {
+ throw new IllegalStateException(String.format("Secret Name [%s] string value not found", secretName));
+ }
+ }
+
+ private void cacheSupportedParameterNames() {
+ supportedParameterNames.clear();
+ final ListSecretsResult listSecretsResult = secretsManager.listSecrets(new ListSecretsRequest());
+ if (listSecretsResult != null) {
+ for (final SecretListEntry entry : listSecretsResult.getSecretList()) {
+ supportedParameterNames.add(entry.getName());
+ }
+ }
+ }
+
+ private Properties loadProperties(final String propertiesFilename) throws IOException {
+ final Properties properties = new Properties();
+
+ try (final InputStream in = new FileInputStream(Paths.get(propertiesFilename).toFile())) {
+ properties.load(in);
+ return properties;
+ }
+ }
+
+ AWSSecretsManager configureClient(final String awsCredentialsFilename) throws IOException {
+ if (awsCredentialsFilename == null) {
+ return getDefaultClient();
+ }
+ final Properties properties = loadProperties(awsCredentialsFilename);
+ final String accessKey = properties.getProperty(ACCESS_KEY_PROPS_NAME);
+ final String secretKey = properties.getProperty(SECRET_KEY_PROPS_NAME);
+ final String region = properties.getProperty(REGION_KEY_PROPS_NAME);
+
+ if (isNotBlank(accessKey) && isNotBlank(secretKey) && isNotBlank(region)) {
+ return AWSSecretsManagerClientBuilder.standard()
+ .withRegion(region)
+ .withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials(accessKey, secretKey)))
+ .build();
+ } else {
+ return getDefaultClient();
+ }
+ }
+
+ private AWSSecretsManager getDefaultClient() {
+ return AWSSecretsManagerClientBuilder.standard()
+ .withCredentials(DefaultAWSCredentialsProviderChain.getInstance())
+ .build();
+ }
+
+ private static String getSecretName(final String contextName, final String parameterName) {
+ return contextName == null ? parameterName : String.format(QUALIFIED_SECRET_FORMAT, contextName, parameterName);
Review comment:
@gresockj sorry for the confusion. The context name will never be ignored. Rather, what I was suggesting is that it should be ignored if there's no secret name that uses it. For example, if the Context Name is MyContext and the parameter name is MyParam, then `isParameterDefined` should return true if there's a parameter named `MyContext/MyParam` OR if there's a parameter named `MyParam`. Similarly, when `getParameterValue` is called, if there's a secret named `MyContext/MyParam`, then its value should be returned. Otherwise, return the value for the `MyParam` secret. Does that make sense?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] gresockj commented on a change in pull request #5391: NIFI-9174: Adding AWS SecretsManager ParamValueProvider for Stateless
Posted by GitBox <gi...@apache.org>.
gresockj commented on a change in pull request #5391:
URL: https://github.com/apache/nifi/pull/5391#discussion_r718961645
##########
File path: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-parameter-value-providers/src/main/java/org/apache/nifi/stateless/parameter/aws/SecretsManagerParameterValueProvider.java
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.stateless.parameter.aws;
+
+import com.amazonaws.auth.AWSStaticCredentialsProvider;
+import com.amazonaws.auth.BasicAWSCredentials;
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
+import com.amazonaws.services.secretsmanager.AWSSecretsManager;
+import com.amazonaws.services.secretsmanager.AWSSecretsManagerClientBuilder;
+import com.amazonaws.services.secretsmanager.model.GetSecretValueRequest;
+import com.amazonaws.services.secretsmanager.model.GetSecretValueResult;
+import com.amazonaws.services.secretsmanager.model.ListSecretsRequest;
+import com.amazonaws.services.secretsmanager.model.ListSecretsResult;
+import com.amazonaws.services.secretsmanager.model.SecretListEntry;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stateless.parameter.AbstractParameterValueProvider;
+import org.apache.nifi.stateless.parameter.ParameterValueProvider;
+import org.apache.nifi.stateless.parameter.ParameterValueProviderInitializationContext;
+
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Paths;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+
+/**
+ * Reads secrets from AWS Secrets Manager to provide parameter values. Secrets must be created similar to the following AWS cli command: <br/><br/>
+ * <code>aws secretsmanager create-secret --name "[ParamContextName]/[ParamName]" --secret-string '[ParamValue]'</code> <br/><br/>
+ *
+ * A standard configuration for this provider would be: <br/><br/>
+ *
+ * <code>
+ * nifi.stateless.parameter.provider.AWSSecretsManager.name=AWS Secrets Manager Value Provider
+ * nifi.stateless.parameter.provider.AWSSecretsManager.type=org.apache.nifi.stateless.parameter.aws.SecretsManagerParameterValueProvider
+ * nifi.stateless.parameter.provider.AWSSecretsManager.properties.aws-credentials-file=./conf/bootstrap-aws.conf
+ * </code>
+ */
+public class SecretsManagerParameterValueProvider extends AbstractParameterValueProvider implements ParameterValueProvider {
+ private static final String QUALIFIED_SECRET_FORMAT = "%s/%s";
+ private static final String ACCESS_KEY_PROPS_NAME = "aws.access.key.id";
+ private static final String SECRET_KEY_PROPS_NAME = "aws.secret.access.key";
+ private static final String REGION_KEY_PROPS_NAME = "aws.region";
+
+ public static final PropertyDescriptor AWS_CREDENTIALS_FILE = new PropertyDescriptor.Builder()
+ .displayName("AWS Credentials File")
+ .name("aws-credentials-file")
+ .required(false)
+ .defaultValue("./conf/bootstrap-aws.conf")
+ .description("Location of the bootstrap-aws.conf file that configures the AWS credentials. If not provided, the default AWS credentials will be used.")
+ .addValidator(StandardValidators.FILE_EXISTS_VALIDATOR)
+ .build();
+
+ private final Set<String> supportedParameterNames = new HashSet<>();
+
+ private List<PropertyDescriptor> descriptors;
+
+ private AWSSecretsManager secretsManager;
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ protected void init(final ParameterValueProviderInitializationContext context) {
+ super.init(context);
+
+ this.descriptors = Collections.singletonList(AWS_CREDENTIALS_FILE);
+
+ final String awsCredentialsFilename = context.getProperty(AWS_CREDENTIALS_FILE).getValue();
+ try {
+ this.secretsManager = this.configureClient(awsCredentialsFilename);
+ } catch (final IOException e) {
+ throw new IllegalStateException("Could not configure AWS Secrets Manager Client", e);
+ }
+
+ cacheSupportedParameterNames();
+ }
+
+ @Override
+ public boolean isParameterDefined(final String contextName, final String parameterName) {
+ return supportedParameterNames.contains(getSecretName(contextName, parameterName));
+ }
+
+ @Override
+ public String getParameterValue(final String contextName, final String parameterName) {
+ final String secretName = getSecretName(contextName, parameterName);
+ final GetSecretValueRequest getSecretValueRequest = new GetSecretValueRequest()
+ .withSecretId(secretName);
+ final GetSecretValueResult getSecretValueResult = secretsManager.getSecretValue(getSecretValueRequest);
+
+ if (getSecretValueResult == null) {
+ throw new IllegalArgumentException(String.format("Secret [%s] not found", secretName));
+ }
+ if (getSecretValueResult.getSecretString() != null) {
+ return getSecretValueResult.getSecretString();
Review comment:
Are you adding the secret as a plain text secret? If you use the AWS key/value interface, it creates the JSON you posted instead of simply "Hello". Basically, if you use the command line example in the PR, it should resolve to "Hello". I will also take a look at this again, in case it is a bug in the code, though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] markap14 commented on a change in pull request #5391: NIFI-9174: Adding AWS SecretsManager ParamValueProvider for Stateless
Posted by GitBox <gi...@apache.org>.
markap14 commented on a change in pull request #5391:
URL: https://github.com/apache/nifi/pull/5391#discussion_r721546424
##########
File path: nifi-stateless/nifi-stateless-assembly/README.md
##########
@@ -515,3 +515,42 @@ nifi.stateless.parameter.provider.Vault.name=HashiCorp Vault Provider
nifi.stateless.parameter.provider.Vault.type=org.apache.nifi.stateless.parameter.HashiCorpVaultParameterValueProvider
nifi.stateless.parameter.provider.Vault.properties.vault-configuration-file=./conf/bootstrap-hashicorp-vault.conf
```
+
+**AWS SecretsManagerParameterValueProvider**
+
+This provider reads parameter values from AWS SecretsManager. Each AWS secret is mapped to a Parameter Context, with
+the Secret name representing the Parameter Context name and the key/value pairs in the Secret representing the
+Parameter names and values.
+
+The AWS credentials can be configured via the `./conf/bootstrap-aws.conf` file, which comes with NiFi.
+
+Note: The provided AWS credentials must have the `secretsmanager:GetSecretValue` permission in order to use this provider.
+An example of creating a single secret in the correct format is:
+
+```
+aws secretsmanager create-secret --name "Context" --secret-string '{ "Param": "secretValue", "Param2": "secretValue2" }'
+```
+
+In this example, `Context` is the name of a Parameter Context, `Param` is the name of the parameter whose value
+should be retrieved from the Vault server, and `secretValue` is the actual value of the parameter. Notice that
+there are multiple parameters stored in this secret: a second parameter named `Param2` has the value of `secretValue2`.
+
+Alternatively, if you use the AWS Console to create a secret, follow these steps:
+1. Select a secret type of "Other type of secrets (e.g. API key)"
+2. Enter one Secret key/value for each Parameter, where the key is the Parameter Name and the value is the Parameter value
+3. On the next page, enter the name of the Parameter Context as the Secret name. Save the Secret.
+
+This Parameter Provider requires the following properties:
+
+| Property Name | Description | Example Value |
+|---------------|-------------|---------------|
+| nifi.stateless.parameter.provider.\<key>.properties.aws-credentials-file | The filename of a configuration file optionally specifying the AWS credentials. If this property is not provided, or if the credentials are not provided in the file, the default AWS credentials chain will be followed. | `./conf/bootstrap-aws.conf` |
+| nifi.stateless.parameter.provider.\<key>.default-secret-name | The default AWS secret name to use. This secret represents a default Parameter Context if there is not a matching key within the mapped Parameter Context secret. | `Default` |
+
+An example of configuring this provider in the dataflow configuration file is:
+
+```
+nifi.stateless.parameter.provider.AWSSecretsManager.name=AWS SecretsManager Provider
+nifi.stateless.parameter.provider.AWSSecretsManager.type=org.apache.nifi.stateless.parameter.AwsSecretsManagerParameterValueProvider
+nifi.stateless.parameter.provider.AWSSecretsManager.properties.aws-credentials-file=./conf/bootstrap-aws.conf
Review comment:
Probably makes sense to update this example showing the new mapping properties and describe those, as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a change in pull request #5391: NIFI-9174: Adding AWS SecretsManager ParamValueProvider for Stateless
Posted by GitBox <gi...@apache.org>.
exceptionfactory commented on a change in pull request #5391:
URL: https://github.com/apache/nifi/pull/5391#discussion_r712410361
##########
File path: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-parameter-value-providers/src/main/java/org/apache/nifi/stateless/parameter/aws/SecretsManagerParameterValueProvider.java
##########
@@ -0,0 +1,208 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.stateless.parameter.aws;
+
+import com.amazonaws.auth.AWSCredentials;
+import com.amazonaws.auth.AWSCredentialsProvider;
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
+import com.amazonaws.services.secretsmanager.AWSSecretsManager;
+import com.amazonaws.services.secretsmanager.AWSSecretsManagerClientBuilder;
+import com.amazonaws.services.secretsmanager.model.GetSecretValueRequest;
+import com.amazonaws.services.secretsmanager.model.GetSecretValueResult;
+import com.amazonaws.services.secretsmanager.model.ListSecretsRequest;
+import com.amazonaws.services.secretsmanager.model.ListSecretsResult;
+import com.amazonaws.services.secretsmanager.model.SecretListEntry;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.ObjectReader;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stateless.parameter.AbstractParameterValueProvider;
+import org.apache.nifi.stateless.parameter.ParameterValueProvider;
+import org.apache.nifi.stateless.parameter.ParameterValueProviderInitializationContext;
+
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Paths;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+
+/**
+ * Reads secrets from AWS Secrets Manager to provide parameter values. Secrets must be created similar to the following AWS cli command: <br/><br/>
+ * <code>aws secretsmanager create-secret --name "[ParamContextName]/[ParamName]" --secret-string '[ParamValue]'</code> <br/><br/>
+ *
+ * A standard configuration for this provider would be: <br/><br/>
+ *
+ * <code>
+ * nifi.stateless.parameter.provider.AWSSecretsManager.name=AWS Secrets Manager Value Provider
+ * nifi.stateless.parameter.provider.AWSSecretsManager.type=org.apache.nifi.stateless.parameter.aws.SecretsManagerParameterValueProvider
+ * nifi.stateless.parameter.provider.AWSSecretsManager.properties.aws-configuration-file=./conf/bootstrap-aws.conf
+ * </code>
+ */
+public class SecretsManagerParameterValueProvider extends AbstractParameterValueProvider implements ParameterValueProvider {
+ private static final String QUALIFIED_SECRET_FORMAT = "%s/%s";
+ private static final String ACCESS_KEY_PROPS_NAME = "aws.access.key.id";
+ private static final String SECRET_KEY_PROPS_NAME = "aws.secret.access.key";
+ private static final String REGION_KEY_PROPS_NAME = "aws.region";
+
+ public static final PropertyDescriptor AWS_CONFIG_FILE = new PropertyDescriptor.Builder()
+ .displayName("AWS Configuration File")
+ .name("aws-configuration-file")
+ .required(false)
+ .defaultValue("./conf/bootstrap-aws.conf")
+ .description("Location of the bootstrap-aws.conf file that configures the AWS credentials. If not provided, the default AWS credentials will be used.")
+ .addValidator(StandardValidators.FILE_EXISTS_VALIDATOR)
+ .build();
+
+ private final ObjectReader objectReader = new ObjectMapper().reader();
+
+ private final Set<String> supportedParameterNames = new HashSet<>();
+
+ private List<PropertyDescriptor> descriptors;
+
+ private AWSSecretsManager secretsManager;
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ protected void init(final ParameterValueProviderInitializationContext context) {
+ super.init(context);
+
+ this.descriptors = Collections.unmodifiableList(Arrays.asList(AWS_CONFIG_FILE));
+
+ final String awsBootstrapConfFilename = context.getProperty(AWS_CONFIG_FILE).getValue();
+ try {
+ this.secretsManager = this.configureClient(awsBootstrapConfFilename);
+ } catch (final IOException e) {
+ throw new IllegalStateException("Could not configure AWS Secrets Manager Client", e);
+ }
+
+ cacheSupportedParameterNames();
+ }
+
+ private void cacheSupportedParameterNames() {
+ supportedParameterNames.clear();
+ final ListSecretsResult listSecretsResult = secretsManager.listSecrets(new ListSecretsRequest());
+ if (listSecretsResult != null) {
+ for (final SecretListEntry entry : listSecretsResult.getSecretList()) {
+ supportedParameterNames.add(entry.getName());
+ }
+ }
+ }
+
+ @Override
+ public boolean isParameterDefined(final String contextName, final String parameterName) {
+ return supportedParameterNames.contains(getSecretName(contextName, parameterName));
+ }
+
+ @Override
+ public String getParameterValue(final String contextName, final String parameterName) {
+ final String secretName = getSecretName(contextName, parameterName);
+ final GetSecretValueRequest getSecretValueRequest = new GetSecretValueRequest()
+ .withSecretId(secretName);
+ final GetSecretValueResult getSecretValueResult = secretsManager.getSecretValue(getSecretValueRequest);
+
+ if (getSecretValueResult == null) {
+ throw new IllegalArgumentException(String.format("Secret [%s] not found", secretName));
+ }
+ if (getSecretValueResult.getSecretString() != null) {
+ return getSecretValueResult.getSecretString();
+ } else {
+ throw new IllegalStateException("Binary secrets are not supported");
+ }
+ }
+
+ private Properties loadProperties(final String propertiesFilename) throws IOException {
+ final Properties properties = new Properties();
+
+ try (final InputStream in = new FileInputStream(Paths.get(propertiesFilename).toFile())) {
+ properties.load(in);
+ return properties;
+ }
+ }
+
+ AWSSecretsManager configureClient(final String awsBootstrapConfFilename) throws IOException {
+ if (awsBootstrapConfFilename == null) {
+ return getDefaultClient();
+ }
+ final Properties properties = loadProperties(awsBootstrapConfFilename);
+ final String accessKey = properties.getProperty(ACCESS_KEY_PROPS_NAME);
+ final String secretKey = properties.getProperty(SECRET_KEY_PROPS_NAME);
+ final String region = properties.getProperty(REGION_KEY_PROPS_NAME);
+
+ if (isNotBlank(accessKey) && isNotBlank(secretKey) && isNotBlank(region)) {
+ return AWSSecretsManagerClientBuilder.standard()
+ .withRegion(region)
+ .withCredentials(new BasicAWSCredentialsProvider(accessKey, secretKey))
+ .build();
+ } else {
+ return getDefaultClient();
+ }
+ }
+
+ private AWSSecretsManager getDefaultClient() {
+ return AWSSecretsManagerClientBuilder.standard()
+ .withCredentials(DefaultAWSCredentialsProviderChain.getInstance())
+ .build();
+ }
+
+ private static String getSecretName(final String contextName, final String parameterName) {
+ return String.format(QUALIFIED_SECRET_FORMAT, contextName, parameterName);
+ }
+
+ private static boolean isNotBlank(final String value) {
+ return value != null && !value.trim().equals("");
+ }
+
+ private static class BasicAWSCredentialsProvider implements AWSCredentialsProvider {
Review comment:
[AWSStaticCredentialsProvider](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/AWSStaticCredentialsProvider.html) should work the `AWSCredentialsProvider`.
##########
File path: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-parameter-value-providers/src/main/java/org/apache/nifi/stateless/parameter/aws/SecretsManagerParameterValueProvider.java
##########
@@ -0,0 +1,208 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.stateless.parameter.aws;
+
+import com.amazonaws.auth.AWSCredentials;
+import com.amazonaws.auth.AWSCredentialsProvider;
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
+import com.amazonaws.services.secretsmanager.AWSSecretsManager;
+import com.amazonaws.services.secretsmanager.AWSSecretsManagerClientBuilder;
+import com.amazonaws.services.secretsmanager.model.GetSecretValueRequest;
+import com.amazonaws.services.secretsmanager.model.GetSecretValueResult;
+import com.amazonaws.services.secretsmanager.model.ListSecretsRequest;
+import com.amazonaws.services.secretsmanager.model.ListSecretsResult;
+import com.amazonaws.services.secretsmanager.model.SecretListEntry;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.ObjectReader;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stateless.parameter.AbstractParameterValueProvider;
+import org.apache.nifi.stateless.parameter.ParameterValueProvider;
+import org.apache.nifi.stateless.parameter.ParameterValueProviderInitializationContext;
+
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Paths;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+
+/**
+ * Reads secrets from AWS Secrets Manager to provide parameter values. Secrets must be created similar to the following AWS cli command: <br/><br/>
+ * <code>aws secretsmanager create-secret --name "[ParamContextName]/[ParamName]" --secret-string '[ParamValue]'</code> <br/><br/>
+ *
+ * A standard configuration for this provider would be: <br/><br/>
+ *
+ * <code>
+ * nifi.stateless.parameter.provider.AWSSecretsManager.name=AWS Secrets Manager Value Provider
+ * nifi.stateless.parameter.provider.AWSSecretsManager.type=org.apache.nifi.stateless.parameter.aws.SecretsManagerParameterValueProvider
+ * nifi.stateless.parameter.provider.AWSSecretsManager.properties.aws-configuration-file=./conf/bootstrap-aws.conf
+ * </code>
+ */
+public class SecretsManagerParameterValueProvider extends AbstractParameterValueProvider implements ParameterValueProvider {
+ private static final String QUALIFIED_SECRET_FORMAT = "%s/%s";
+ private static final String ACCESS_KEY_PROPS_NAME = "aws.access.key.id";
+ private static final String SECRET_KEY_PROPS_NAME = "aws.secret.access.key";
+ private static final String REGION_KEY_PROPS_NAME = "aws.region";
+
+ public static final PropertyDescriptor AWS_CONFIG_FILE = new PropertyDescriptor.Builder()
+ .displayName("AWS Configuration File")
+ .name("aws-configuration-file")
+ .required(false)
+ .defaultValue("./conf/bootstrap-aws.conf")
+ .description("Location of the bootstrap-aws.conf file that configures the AWS credentials. If not provided, the default AWS credentials will be used.")
+ .addValidator(StandardValidators.FILE_EXISTS_VALIDATOR)
+ .build();
+
+ private final ObjectReader objectReader = new ObjectMapper().reader();
+
+ private final Set<String> supportedParameterNames = new HashSet<>();
+
+ private List<PropertyDescriptor> descriptors;
+
+ private AWSSecretsManager secretsManager;
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ protected void init(final ParameterValueProviderInitializationContext context) {
+ super.init(context);
+
+ this.descriptors = Collections.unmodifiableList(Arrays.asList(AWS_CONFIG_FILE));
+
+ final String awsBootstrapConfFilename = context.getProperty(AWS_CONFIG_FILE).getValue();
+ try {
+ this.secretsManager = this.configureClient(awsBootstrapConfFilename);
+ } catch (final IOException e) {
+ throw new IllegalStateException("Could not configure AWS Secrets Manager Client", e);
+ }
+
+ cacheSupportedParameterNames();
+ }
+
+ private void cacheSupportedParameterNames() {
+ supportedParameterNames.clear();
+ final ListSecretsResult listSecretsResult = secretsManager.listSecrets(new ListSecretsRequest());
+ if (listSecretsResult != null) {
+ for (final SecretListEntry entry : listSecretsResult.getSecretList()) {
+ supportedParameterNames.add(entry.getName());
+ }
+ }
+ }
+
+ @Override
+ public boolean isParameterDefined(final String contextName, final String parameterName) {
+ return supportedParameterNames.contains(getSecretName(contextName, parameterName));
+ }
+
+ @Override
+ public String getParameterValue(final String contextName, final String parameterName) {
+ final String secretName = getSecretName(contextName, parameterName);
+ final GetSecretValueRequest getSecretValueRequest = new GetSecretValueRequest()
+ .withSecretId(secretName);
+ final GetSecretValueResult getSecretValueResult = secretsManager.getSecretValue(getSecretValueRequest);
+
+ if (getSecretValueResult == null) {
+ throw new IllegalArgumentException(String.format("Secret [%s] not found", secretName));
+ }
+ if (getSecretValueResult.getSecretString() != null) {
+ return getSecretValueResult.getSecretString();
+ } else {
+ throw new IllegalStateException("Binary secrets are not supported");
+ }
+ }
+
+ private Properties loadProperties(final String propertiesFilename) throws IOException {
+ final Properties properties = new Properties();
+
+ try (final InputStream in = new FileInputStream(Paths.get(propertiesFilename).toFile())) {
+ properties.load(in);
+ return properties;
+ }
+ }
+
+ AWSSecretsManager configureClient(final String awsBootstrapConfFilename) throws IOException {
+ if (awsBootstrapConfFilename == null) {
+ return getDefaultClient();
+ }
+ final Properties properties = loadProperties(awsBootstrapConfFilename);
+ final String accessKey = properties.getProperty(ACCESS_KEY_PROPS_NAME);
+ final String secretKey = properties.getProperty(SECRET_KEY_PROPS_NAME);
+ final String region = properties.getProperty(REGION_KEY_PROPS_NAME);
+
+ if (isNotBlank(accessKey) && isNotBlank(secretKey) && isNotBlank(region)) {
+ return AWSSecretsManagerClientBuilder.standard()
+ .withRegion(region)
+ .withCredentials(new BasicAWSCredentialsProvider(accessKey, secretKey))
+ .build();
+ } else {
+ return getDefaultClient();
+ }
+ }
+
+ private AWSSecretsManager getDefaultClient() {
+ return AWSSecretsManagerClientBuilder.standard()
+ .withCredentials(DefaultAWSCredentialsProviderChain.getInstance())
+ .build();
+ }
+
+ private static String getSecretName(final String contextName, final String parameterName) {
+ return String.format(QUALIFIED_SECRET_FORMAT, contextName, parameterName);
+ }
+
+ private static boolean isNotBlank(final String value) {
+ return value != null && !value.trim().equals("");
+ }
+
+ private static class BasicAWSCredentialsProvider implements AWSCredentialsProvider {
Review comment:
[AWSStaticCredentialsProvider](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/AWSStaticCredentialsProvider.html) should work for the `AWSCredentialsProvider`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] markap14 merged pull request #5391: NIFI-9174: Adding AWS SecretsManager ParamValueProvider for Stateless
Posted by GitBox <gi...@apache.org>.
markap14 merged pull request #5391:
URL: https://github.com/apache/nifi/pull/5391
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] markap14 commented on pull request #5391: NIFI-9174: Adding AWS SecretsManager ParamValueProvider for Stateless
Posted by GitBox <gi...@apache.org>.
markap14 commented on pull request #5391:
URL: https://github.com/apache/nifi/pull/5391#issuecomment-931741211
@gresockj @exceptionfactory thanks for the feedback. Yes, I agree there could be some benefit to also allowing for a 'plaintext' approach, but I also think we should choose an approach to start and say this is how it works. If a need arises to allow for plaintext, we can look into it then. I do not think we should attempt to parse JSON and if it fails, fallback to 'plaintext', though. Instead, it makes sense to me to either have two separate Provider implementations or to have a configuration option in the Provider that says how to handle the data. A really good thing to consider is for Google Cloud credentials, you generally will be given a JSON document that contains a Base64 encoded certificate. So we have a secret JSON document, which would cause a lot of confusion if we attempted to separate those into separate key/value pairs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] gresockj commented on a change in pull request #5391: NIFI-9174: Adding AWS SecretsManager ParamValueProvider for Stateless
Posted by GitBox <gi...@apache.org>.
gresockj commented on a change in pull request #5391:
URL: https://github.com/apache/nifi/pull/5391#discussion_r719455044
##########
File path: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-parameter-value-providers/src/main/java/org/apache/nifi/stateless/parameter/aws/SecretsManagerParameterValueProvider.java
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.stateless.parameter.aws;
+
+import com.amazonaws.auth.AWSStaticCredentialsProvider;
+import com.amazonaws.auth.BasicAWSCredentials;
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
+import com.amazonaws.services.secretsmanager.AWSSecretsManager;
+import com.amazonaws.services.secretsmanager.AWSSecretsManagerClientBuilder;
+import com.amazonaws.services.secretsmanager.model.GetSecretValueRequest;
+import com.amazonaws.services.secretsmanager.model.GetSecretValueResult;
+import com.amazonaws.services.secretsmanager.model.ListSecretsRequest;
+import com.amazonaws.services.secretsmanager.model.ListSecretsResult;
+import com.amazonaws.services.secretsmanager.model.SecretListEntry;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stateless.parameter.AbstractParameterValueProvider;
+import org.apache.nifi.stateless.parameter.ParameterValueProvider;
+import org.apache.nifi.stateless.parameter.ParameterValueProviderInitializationContext;
+
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Paths;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+
+/**
+ * Reads secrets from AWS Secrets Manager to provide parameter values. Secrets must be created similar to the following AWS cli command: <br/><br/>
+ * <code>aws secretsmanager create-secret --name "[ParamContextName]/[ParamName]" --secret-string '[ParamValue]'</code> <br/><br/>
+ *
+ * A standard configuration for this provider would be: <br/><br/>
+ *
+ * <code>
+ * nifi.stateless.parameter.provider.AWSSecretsManager.name=AWS Secrets Manager Value Provider
+ * nifi.stateless.parameter.provider.AWSSecretsManager.type=org.apache.nifi.stateless.parameter.aws.SecretsManagerParameterValueProvider
+ * nifi.stateless.parameter.provider.AWSSecretsManager.properties.aws-credentials-file=./conf/bootstrap-aws.conf
+ * </code>
+ */
+public class SecretsManagerParameterValueProvider extends AbstractParameterValueProvider implements ParameterValueProvider {
+ private static final String QUALIFIED_SECRET_FORMAT = "%s/%s";
+ private static final String ACCESS_KEY_PROPS_NAME = "aws.access.key.id";
+ private static final String SECRET_KEY_PROPS_NAME = "aws.secret.access.key";
+ private static final String REGION_KEY_PROPS_NAME = "aws.region";
+
+ public static final PropertyDescriptor AWS_CREDENTIALS_FILE = new PropertyDescriptor.Builder()
+ .displayName("AWS Credentials File")
+ .name("aws-credentials-file")
+ .required(false)
+ .defaultValue("./conf/bootstrap-aws.conf")
+ .description("Location of the bootstrap-aws.conf file that configures the AWS credentials. If not provided, the default AWS credentials will be used.")
+ .addValidator(StandardValidators.FILE_EXISTS_VALIDATOR)
+ .build();
+
+ private final Set<String> supportedParameterNames = new HashSet<>();
+
+ private List<PropertyDescriptor> descriptors;
+
+ private AWSSecretsManager secretsManager;
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ protected void init(final ParameterValueProviderInitializationContext context) {
+ super.init(context);
+
+ this.descriptors = Collections.singletonList(AWS_CREDENTIALS_FILE);
+
+ final String awsCredentialsFilename = context.getProperty(AWS_CREDENTIALS_FILE).getValue();
+ try {
+ this.secretsManager = this.configureClient(awsCredentialsFilename);
+ } catch (final IOException e) {
+ throw new IllegalStateException("Could not configure AWS Secrets Manager Client", e);
+ }
+
+ cacheSupportedParameterNames();
+ }
+
+ @Override
+ public boolean isParameterDefined(final String contextName, final String parameterName) {
+ return supportedParameterNames.contains(getSecretName(contextName, parameterName));
+ }
+
+ @Override
+ public String getParameterValue(final String contextName, final String parameterName) {
+ final String secretName = getSecretName(contextName, parameterName);
+ final GetSecretValueRequest getSecretValueRequest = new GetSecretValueRequest()
+ .withSecretId(secretName);
+ final GetSecretValueResult getSecretValueResult = secretsManager.getSecretValue(getSecretValueRequest);
+
+ if (getSecretValueResult == null) {
+ throw new IllegalArgumentException(String.format("Secret [%s] not found", secretName));
+ }
+ if (getSecretValueResult.getSecretString() != null) {
+ return getSecretValueResult.getSecretString();
+ } else {
+ throw new IllegalStateException(String.format("Secret Name [%s] string value not found", secretName));
+ }
+ }
+
+ private void cacheSupportedParameterNames() {
+ supportedParameterNames.clear();
+ final ListSecretsResult listSecretsResult = secretsManager.listSecrets(new ListSecretsRequest());
+ if (listSecretsResult != null) {
+ for (final SecretListEntry entry : listSecretsResult.getSecretList()) {
+ supportedParameterNames.add(entry.getName());
+ }
+ }
+ }
+
+ private Properties loadProperties(final String propertiesFilename) throws IOException {
+ final Properties properties = new Properties();
+
+ try (final InputStream in = new FileInputStream(Paths.get(propertiesFilename).toFile())) {
+ properties.load(in);
+ return properties;
+ }
+ }
+
+ AWSSecretsManager configureClient(final String awsCredentialsFilename) throws IOException {
+ if (awsCredentialsFilename == null) {
+ return getDefaultClient();
+ }
+ final Properties properties = loadProperties(awsCredentialsFilename);
+ final String accessKey = properties.getProperty(ACCESS_KEY_PROPS_NAME);
+ final String secretKey = properties.getProperty(SECRET_KEY_PROPS_NAME);
+ final String region = properties.getProperty(REGION_KEY_PROPS_NAME);
+
+ if (isNotBlank(accessKey) && isNotBlank(secretKey) && isNotBlank(region)) {
+ return AWSSecretsManagerClientBuilder.standard()
+ .withRegion(region)
+ .withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials(accessKey, secretKey)))
+ .build();
+ } else {
+ return getDefaultClient();
+ }
+ }
+
+ private AWSSecretsManager getDefaultClient() {
+ return AWSSecretsManagerClientBuilder.standard()
+ .withCredentials(DefaultAWSCredentialsProviderChain.getInstance())
+ .build();
+ }
+
+ private static String getSecretName(final String contextName, final String parameterName) {
+ return contextName == null ? parameterName : String.format(QUALIFIED_SECRET_FORMAT, contextName, parameterName);
Review comment:
Ah, I see -- it's fallback logic, rather than simply handling a null Context.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] gresockj commented on pull request #5391: NIFI-9174: Adding AWS SecretsManager ParamValueProvider for Stateless
Posted by GitBox <gi...@apache.org>.
gresockj commented on pull request #5391:
URL: https://github.com/apache/nifi/pull/5391#issuecomment-931597824
> > Thoughts on this approach?
>
> @markap14 Thanks for describing some options for implementation, particularly in relation to the AWS Secrets Manager web user interface. PR #5410 for NIFI-9221 provides similar capabilities for NiFi Sensitive Properties. Using a plain string is easier to handle in code since it avoids the need for JSON parsing, but it also looks like the AWS Secrets Manager UI encourages using a JSON object as the default representation for generic secret values.
>
> Going with the JSON object approach provides the ability to store multiple keys and values in a single Secret as you described, which could be useful. On the other hand, requiring a JSON obejct representation would break use cases where the Secret is a simple string. Without getting too complicated, a potential hybrid approach might be to attempt JSON parsing, and otherwise return the plain string, at least in the case of the NiFi Sensitive Property Provider for NIFI-9221.
>
> Either way, it would be helpful to have a consistent approach, even though these are different use cases.
An additional benefit of the JSON approach is that it would store fewer secrets (less cost). In the case of the AWS Secrets Manager Sensitive Property Provider, in order to stay consistent we could map the `ProtectedPropertyContext.contextName` to the Secret name, and `ProtectedPropertyContext.propertyName` to the key within the secret.
As for allowing Parameter Contexts to be arbitrarily mapped to different Secret names, @markap14, I'm going to suggest we go for simplicity here and simply enforce that a Parameter Context represents exactly one Secret, and so its name would become the Secret name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] markap14 commented on a change in pull request #5391: NIFI-9174: Adding AWS SecretsManager ParamValueProvider for Stateless
Posted by GitBox <gi...@apache.org>.
markap14 commented on a change in pull request #5391:
URL: https://github.com/apache/nifi/pull/5391#discussion_r719404692
##########
File path: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-parameter-value-providers/src/main/java/org/apache/nifi/stateless/parameter/aws/SecretsManagerParameterValueProvider.java
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.stateless.parameter.aws;
+
+import com.amazonaws.auth.AWSStaticCredentialsProvider;
+import com.amazonaws.auth.BasicAWSCredentials;
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
+import com.amazonaws.services.secretsmanager.AWSSecretsManager;
+import com.amazonaws.services.secretsmanager.AWSSecretsManagerClientBuilder;
+import com.amazonaws.services.secretsmanager.model.GetSecretValueRequest;
+import com.amazonaws.services.secretsmanager.model.GetSecretValueResult;
+import com.amazonaws.services.secretsmanager.model.ListSecretsRequest;
+import com.amazonaws.services.secretsmanager.model.ListSecretsResult;
+import com.amazonaws.services.secretsmanager.model.SecretListEntry;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stateless.parameter.AbstractParameterValueProvider;
+import org.apache.nifi.stateless.parameter.ParameterValueProvider;
+import org.apache.nifi.stateless.parameter.ParameterValueProviderInitializationContext;
+
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Paths;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+
+/**
+ * Reads secrets from AWS Secrets Manager to provide parameter values. Secrets must be created similar to the following AWS cli command: <br/><br/>
+ * <code>aws secretsmanager create-secret --name "[ParamContextName]/[ParamName]" --secret-string '[ParamValue]'</code> <br/><br/>
+ *
+ * A standard configuration for this provider would be: <br/><br/>
+ *
+ * <code>
+ * nifi.stateless.parameter.provider.AWSSecretsManager.name=AWS Secrets Manager Value Provider
+ * nifi.stateless.parameter.provider.AWSSecretsManager.type=org.apache.nifi.stateless.parameter.aws.SecretsManagerParameterValueProvider
+ * nifi.stateless.parameter.provider.AWSSecretsManager.properties.aws-credentials-file=./conf/bootstrap-aws.conf
+ * </code>
+ */
+public class SecretsManagerParameterValueProvider extends AbstractParameterValueProvider implements ParameterValueProvider {
+ private static final String QUALIFIED_SECRET_FORMAT = "%s/%s";
+ private static final String ACCESS_KEY_PROPS_NAME = "aws.access.key.id";
+ private static final String SECRET_KEY_PROPS_NAME = "aws.secret.access.key";
+ private static final String REGION_KEY_PROPS_NAME = "aws.region";
+
+ public static final PropertyDescriptor AWS_CREDENTIALS_FILE = new PropertyDescriptor.Builder()
+ .displayName("AWS Credentials File")
+ .name("aws-credentials-file")
+ .required(false)
+ .defaultValue("./conf/bootstrap-aws.conf")
+ .description("Location of the bootstrap-aws.conf file that configures the AWS credentials. If not provided, the default AWS credentials will be used.")
+ .addValidator(StandardValidators.FILE_EXISTS_VALIDATOR)
+ .build();
+
+ private final Set<String> supportedParameterNames = new HashSet<>();
+
+ private List<PropertyDescriptor> descriptors;
+
+ private AWSSecretsManager secretsManager;
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ protected void init(final ParameterValueProviderInitializationContext context) {
+ super.init(context);
+
+ this.descriptors = Collections.singletonList(AWS_CREDENTIALS_FILE);
+
+ final String awsCredentialsFilename = context.getProperty(AWS_CREDENTIALS_FILE).getValue();
+ try {
+ this.secretsManager = this.configureClient(awsCredentialsFilename);
+ } catch (final IOException e) {
+ throw new IllegalStateException("Could not configure AWS Secrets Manager Client", e);
+ }
+
+ cacheSupportedParameterNames();
+ }
+
+ @Override
+ public boolean isParameterDefined(final String contextName, final String parameterName) {
+ return supportedParameterNames.contains(getSecretName(contextName, parameterName));
+ }
+
+ @Override
+ public String getParameterValue(final String contextName, final String parameterName) {
+ final String secretName = getSecretName(contextName, parameterName);
+ final GetSecretValueRequest getSecretValueRequest = new GetSecretValueRequest()
+ .withSecretId(secretName);
+ final GetSecretValueResult getSecretValueResult = secretsManager.getSecretValue(getSecretValueRequest);
+
+ if (getSecretValueResult == null) {
+ throw new IllegalArgumentException(String.format("Secret [%s] not found", secretName));
+ }
+ if (getSecretValueResult.getSecretString() != null) {
+ return getSecretValueResult.getSecretString();
Review comment:
Ahhh yes. I used the key/value interface. The README gives an example of using the CLI, but we should probably indicate in the README also that if you're going to use the AWS Console (i.e., the UI) you need to create the secret and use a type of "Other type of secrets (e.g. API key)" and then go to the plaintext tab, remove the JSON completely, and then just type the secret value that you'd like to use.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on pull request #5391: NIFI-9174: Adding AWS SecretsManager ParamValueProvider for Stateless
Posted by GitBox <gi...@apache.org>.
exceptionfactory commented on pull request #5391:
URL: https://github.com/apache/nifi/pull/5391#issuecomment-931591215
> Thoughts on this approach?
@markap14 Thanks for describing some options for implementation, particularly in relation to the AWS Secrets Manager web user interface. PR #5410 for NIFI-9221 provides similar capabilities for NiFi Sensitive Properties. Using a plain string is easier to handle in code since it avoids the need for JSON parsing, but it also looks like the AWS Secrets Manager UI encourages using a JSON object as the default representation for generic secret values.
Going with the JSON object approach provides the ability to store multiple keys and values in a single Secret as you described, which could be useful. On the other hand, requiring a JSON obejct representation would break use cases where the Secret is a simple string. Without getting too complicated, a potential hybrid approach might be to attempt JSON parsing, and otherwise return the plain string, at least in the case of the NiFi Sensitive Property Provider for NIFI-9221.
Either way, it would be helpful to have a consistent approach, even though these are different use cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] markap14 commented on pull request #5391: NIFI-9174: Adding AWS SecretsManager ParamValueProvider for Stateless
Posted by GitBox <gi...@apache.org>.
markap14 commented on pull request #5391:
URL: https://github.com/apache/nifi/pull/5391#issuecomment-931391414
Actually @gresockj after playing with the AWS Secrets Manager a bit more, I'm wondering if we should actually go a slightly different route here. Rather than having the value of a secret be a 'plaintext' value, perhaps it makes more sense to use the key/value pairs that the UI is tailored to?
So then, instead of treating each secret as a separate parameter, we would treat each AWS Secret like a Parameter Context. And each parameter would then map to one of those key/value pairs. Then users can just go into AWS Secrets Manager, create a new Secret, and enter all of their parameters that they care about. Then the provider would be configured with a mapping of ParameterContext Name to Secret Name, with a default Secret Name to be used if there is no mapping.
For example, if my flow has ContextA and ContextB, I could configure the Provider so that ContextA maps to secret SecretA and ContextB maps to SecretB. Or I could configure it so that ContextA maps to SecretA and ContextB maps to SecretA. Or configure it so that ContextA maps to SecretA and not provide a mapping for ContextB, just specifying SecretA as the default secret name.
Thoughts on this approach?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] markap14 commented on a change in pull request #5391: NIFI-9174: Adding AWS SecretsManager ParamValueProvider for Stateless
Posted by GitBox <gi...@apache.org>.
markap14 commented on a change in pull request #5391:
URL: https://github.com/apache/nifi/pull/5391#discussion_r722327357
##########
File path: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-parameter-value-providers/src/main/java/org/apache/nifi/stateless/parameter/AwsSecretsManagerParameterValueProvider.java
##########
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.stateless.parameter;
+
+import com.amazonaws.auth.AWSStaticCredentialsProvider;
+import com.amazonaws.auth.BasicAWSCredentials;
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
+import com.amazonaws.services.secretsmanager.AWSSecretsManager;
+import com.amazonaws.services.secretsmanager.AWSSecretsManagerClientBuilder;
+import com.amazonaws.services.secretsmanager.model.GetSecretValueRequest;
+import com.amazonaws.services.secretsmanager.model.GetSecretValueResult;
+import com.amazonaws.services.secretsmanager.model.ResourceNotFoundException;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Paths;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+
+/**
+ * Reads secrets from AWS Secrets Manager to provide parameter values. Secrets must be created similar to the following AWS cli command: <br/><br/>
+ * <code>aws secretsmanager create-secret --name "[Context]" --secret-string '{ "[Param]": "[secretValue]", "[Param2]": "[secretValue2]" }'</code> <br/><br/>
+ *
+ * A standard configuration for this provider would be: <br/><br/>
+ *
+ * <code>
+ * nifi.stateless.parameter.provider.AWSSecretsManager.name=AWS Secrets Manager Value Provider
+ * nifi.stateless.parameter.provider.AWSSecretsManager.type=org.apache.nifi.stateless.parameter.AwsSecretsManagerParameterValueProvider
+ * nifi.stateless.parameter.provider.AWSSecretsManager.properties.aws-credentials-file=./conf/bootstrap-aws.conf
+ * </code>
+ */
+public class AwsSecretsManagerParameterValueProvider extends AbstractSecretBasedParameterValueProvider implements ParameterValueProvider {
+ private static final Logger logger = LoggerFactory.getLogger(AwsSecretsManagerParameterValueProvider.class);
+
+ private static final String ACCESS_KEY_PROPS_NAME = "aws.access.key.id";
+ private static final String SECRET_KEY_PROPS_NAME = "aws.secret.access.key";
+ private static final String REGION_KEY_PROPS_NAME = "aws.region";
+
+ public static final PropertyDescriptor AWS_CREDENTIALS_FILE = new PropertyDescriptor.Builder()
+ .displayName("AWS Credentials File")
+ .name("aws-credentials-file")
+ .required(false)
+ .defaultValue("./conf/bootstrap-aws.conf")
+ .description("Location of the bootstrap-aws.conf file that configures the AWS credentials. If not provided, the default AWS credentials will be used.")
+ .addValidator(StandardValidators.FILE_EXISTS_VALIDATOR)
+ .build();
+
+ private final ObjectMapper objectMapper = new ObjectMapper();
+
+ private AWSSecretsManager secretsManager;
+
+ @Override
+ protected List<PropertyDescriptor> getAdditionalSupportedPropertyDescriptors() {
+ return Collections.singletonList(AWS_CREDENTIALS_FILE);
+ }
+
+ @Override
+ protected void additionalInit(final ParameterValueProviderInitializationContext context) {
+ final String awsCredentialsFilename = context.getProperty(AWS_CREDENTIALS_FILE).getValue();
+ try {
+ this.secretsManager = this.configureClient(awsCredentialsFilename);
+ } catch (final IOException e) {
+ throw new IllegalStateException("Could not configure AWS Secrets Manager Client", e);
+ }
+ }
+
+ @Override
+ protected String getSecretValue(final String secretName, final String keyName) {
+ final GetSecretValueRequest getSecretValueRequest = new GetSecretValueRequest()
+ .withSecretId(secretName);
+ try {
+ final GetSecretValueResult getSecretValueResult = secretsManager.getSecretValue(getSecretValueRequest);
Review comment:
If the given secret name is not a valid secret name, this throws an AWSSecretsManagerException. In that case, I think we need to catch the Exception and delegate to the default Secret or return null, but we should throw an Exception here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] markap14 commented on pull request #5391: NIFI-9174: Adding AWS SecretsManager ParamValueProvider for Stateless
Posted by GitBox <gi...@apache.org>.
markap14 commented on pull request #5391:
URL: https://github.com/apache/nifi/pull/5391#issuecomment-939106801
Thanks for all the work here @gresockj ! And thanks for helping with the reviews @exceptionfactory. I've done a good bit of testing and everything is looking good to me. +1 will merge to main if you're good also @exceptionfactory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] gresockj commented on a change in pull request #5391: NIFI-9174: Adding AWS SecretsManager ParamValueProvider for Stateless
Posted by GitBox <gi...@apache.org>.
gresockj commented on a change in pull request #5391:
URL: https://github.com/apache/nifi/pull/5391#discussion_r718961733
##########
File path: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-parameter-value-providers/src/main/java/org/apache/nifi/stateless/parameter/aws/SecretsManagerParameterValueProvider.java
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.stateless.parameter.aws;
+
+import com.amazonaws.auth.AWSStaticCredentialsProvider;
+import com.amazonaws.auth.BasicAWSCredentials;
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
+import com.amazonaws.services.secretsmanager.AWSSecretsManager;
+import com.amazonaws.services.secretsmanager.AWSSecretsManagerClientBuilder;
+import com.amazonaws.services.secretsmanager.model.GetSecretValueRequest;
+import com.amazonaws.services.secretsmanager.model.GetSecretValueResult;
+import com.amazonaws.services.secretsmanager.model.ListSecretsRequest;
+import com.amazonaws.services.secretsmanager.model.ListSecretsResult;
+import com.amazonaws.services.secretsmanager.model.SecretListEntry;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stateless.parameter.AbstractParameterValueProvider;
+import org.apache.nifi.stateless.parameter.ParameterValueProvider;
+import org.apache.nifi.stateless.parameter.ParameterValueProviderInitializationContext;
+
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Paths;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+
+/**
+ * Reads secrets from AWS Secrets Manager to provide parameter values. Secrets must be created similar to the following AWS cli command: <br/><br/>
+ * <code>aws secretsmanager create-secret --name "[ParamContextName]/[ParamName]" --secret-string '[ParamValue]'</code> <br/><br/>
+ *
+ * A standard configuration for this provider would be: <br/><br/>
+ *
+ * <code>
+ * nifi.stateless.parameter.provider.AWSSecretsManager.name=AWS Secrets Manager Value Provider
+ * nifi.stateless.parameter.provider.AWSSecretsManager.type=org.apache.nifi.stateless.parameter.aws.SecretsManagerParameterValueProvider
+ * nifi.stateless.parameter.provider.AWSSecretsManager.properties.aws-credentials-file=./conf/bootstrap-aws.conf
+ * </code>
+ */
+public class SecretsManagerParameterValueProvider extends AbstractParameterValueProvider implements ParameterValueProvider {
+ private static final String QUALIFIED_SECRET_FORMAT = "%s/%s";
+ private static final String ACCESS_KEY_PROPS_NAME = "aws.access.key.id";
+ private static final String SECRET_KEY_PROPS_NAME = "aws.secret.access.key";
+ private static final String REGION_KEY_PROPS_NAME = "aws.region";
+
+ public static final PropertyDescriptor AWS_CREDENTIALS_FILE = new PropertyDescriptor.Builder()
+ .displayName("AWS Credentials File")
+ .name("aws-credentials-file")
+ .required(false)
+ .defaultValue("./conf/bootstrap-aws.conf")
+ .description("Location of the bootstrap-aws.conf file that configures the AWS credentials. If not provided, the default AWS credentials will be used.")
+ .addValidator(StandardValidators.FILE_EXISTS_VALIDATOR)
+ .build();
+
+ private final Set<String> supportedParameterNames = new HashSet<>();
+
+ private List<PropertyDescriptor> descriptors;
+
+ private AWSSecretsManager secretsManager;
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ protected void init(final ParameterValueProviderInitializationContext context) {
+ super.init(context);
+
+ this.descriptors = Collections.singletonList(AWS_CREDENTIALS_FILE);
+
+ final String awsCredentialsFilename = context.getProperty(AWS_CREDENTIALS_FILE).getValue();
+ try {
+ this.secretsManager = this.configureClient(awsCredentialsFilename);
+ } catch (final IOException e) {
+ throw new IllegalStateException("Could not configure AWS Secrets Manager Client", e);
+ }
+
+ cacheSupportedParameterNames();
+ }
+
+ @Override
+ public boolean isParameterDefined(final String contextName, final String parameterName) {
+ return supportedParameterNames.contains(getSecretName(contextName, parameterName));
Review comment:
Thanks for the additional context! I'll make that update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] gresockj commented on a change in pull request #5391: NIFI-9174: Adding AWS SecretsManager ParamValueProvider for Stateless
Posted by GitBox <gi...@apache.org>.
gresockj commented on a change in pull request #5391:
URL: https://github.com/apache/nifi/pull/5391#discussion_r712415979
##########
File path: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-parameter-value-providers/src/main/java/org/apache/nifi/stateless/parameter/aws/SecretsManagerParameterValueProvider.java
##########
@@ -0,0 +1,208 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.stateless.parameter.aws;
+
+import com.amazonaws.auth.AWSCredentials;
+import com.amazonaws.auth.AWSCredentialsProvider;
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
+import com.amazonaws.services.secretsmanager.AWSSecretsManager;
+import com.amazonaws.services.secretsmanager.AWSSecretsManagerClientBuilder;
+import com.amazonaws.services.secretsmanager.model.GetSecretValueRequest;
+import com.amazonaws.services.secretsmanager.model.GetSecretValueResult;
+import com.amazonaws.services.secretsmanager.model.ListSecretsRequest;
+import com.amazonaws.services.secretsmanager.model.ListSecretsResult;
+import com.amazonaws.services.secretsmanager.model.SecretListEntry;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.ObjectReader;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stateless.parameter.AbstractParameterValueProvider;
+import org.apache.nifi.stateless.parameter.ParameterValueProvider;
+import org.apache.nifi.stateless.parameter.ParameterValueProviderInitializationContext;
+
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Paths;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+
+/**
+ * Reads secrets from AWS Secrets Manager to provide parameter values. Secrets must be created similar to the following AWS cli command: <br/><br/>
+ * <code>aws secretsmanager create-secret --name "[ParamContextName]/[ParamName]" --secret-string '[ParamValue]'</code> <br/><br/>
+ *
+ * A standard configuration for this provider would be: <br/><br/>
+ *
+ * <code>
+ * nifi.stateless.parameter.provider.AWSSecretsManager.name=AWS Secrets Manager Value Provider
+ * nifi.stateless.parameter.provider.AWSSecretsManager.type=org.apache.nifi.stateless.parameter.aws.SecretsManagerParameterValueProvider
+ * nifi.stateless.parameter.provider.AWSSecretsManager.properties.aws-configuration-file=./conf/bootstrap-aws.conf
+ * </code>
+ */
+public class SecretsManagerParameterValueProvider extends AbstractParameterValueProvider implements ParameterValueProvider {
+ private static final String QUALIFIED_SECRET_FORMAT = "%s/%s";
+ private static final String ACCESS_KEY_PROPS_NAME = "aws.access.key.id";
+ private static final String SECRET_KEY_PROPS_NAME = "aws.secret.access.key";
+ private static final String REGION_KEY_PROPS_NAME = "aws.region";
+
+ public static final PropertyDescriptor AWS_CONFIG_FILE = new PropertyDescriptor.Builder()
+ .displayName("AWS Configuration File")
+ .name("aws-configuration-file")
+ .required(false)
+ .defaultValue("./conf/bootstrap-aws.conf")
+ .description("Location of the bootstrap-aws.conf file that configures the AWS credentials. If not provided, the default AWS credentials will be used.")
+ .addValidator(StandardValidators.FILE_EXISTS_VALIDATOR)
+ .build();
+
+ private final ObjectReader objectReader = new ObjectMapper().reader();
+
+ private final Set<String> supportedParameterNames = new HashSet<>();
+
+ private List<PropertyDescriptor> descriptors;
+
+ private AWSSecretsManager secretsManager;
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ protected void init(final ParameterValueProviderInitializationContext context) {
+ super.init(context);
+
+ this.descriptors = Collections.unmodifiableList(Arrays.asList(AWS_CONFIG_FILE));
+
+ final String awsBootstrapConfFilename = context.getProperty(AWS_CONFIG_FILE).getValue();
+ try {
+ this.secretsManager = this.configureClient(awsBootstrapConfFilename);
+ } catch (final IOException e) {
+ throw new IllegalStateException("Could not configure AWS Secrets Manager Client", e);
+ }
+
+ cacheSupportedParameterNames();
+ }
+
+ private void cacheSupportedParameterNames() {
+ supportedParameterNames.clear();
+ final ListSecretsResult listSecretsResult = secretsManager.listSecrets(new ListSecretsRequest());
+ if (listSecretsResult != null) {
+ for (final SecretListEntry entry : listSecretsResult.getSecretList()) {
+ supportedParameterNames.add(entry.getName());
+ }
+ }
+ }
+
+ @Override
+ public boolean isParameterDefined(final String contextName, final String parameterName) {
+ return supportedParameterNames.contains(getSecretName(contextName, parameterName));
+ }
+
+ @Override
+ public String getParameterValue(final String contextName, final String parameterName) {
+ final String secretName = getSecretName(contextName, parameterName);
+ final GetSecretValueRequest getSecretValueRequest = new GetSecretValueRequest()
+ .withSecretId(secretName);
+ final GetSecretValueResult getSecretValueResult = secretsManager.getSecretValue(getSecretValueRequest);
+
+ if (getSecretValueResult == null) {
+ throw new IllegalArgumentException(String.format("Secret [%s] not found", secretName));
+ }
+ if (getSecretValueResult.getSecretString() != null) {
+ return getSecretValueResult.getSecretString();
+ } else {
+ throw new IllegalStateException("Binary secrets are not supported");
+ }
+ }
+
+ private Properties loadProperties(final String propertiesFilename) throws IOException {
+ final Properties properties = new Properties();
+
+ try (final InputStream in = new FileInputStream(Paths.get(propertiesFilename).toFile())) {
+ properties.load(in);
+ return properties;
+ }
+ }
+
+ AWSSecretsManager configureClient(final String awsBootstrapConfFilename) throws IOException {
+ if (awsBootstrapConfFilename == null) {
+ return getDefaultClient();
+ }
+ final Properties properties = loadProperties(awsBootstrapConfFilename);
+ final String accessKey = properties.getProperty(ACCESS_KEY_PROPS_NAME);
+ final String secretKey = properties.getProperty(SECRET_KEY_PROPS_NAME);
+ final String region = properties.getProperty(REGION_KEY_PROPS_NAME);
+
+ if (isNotBlank(accessKey) && isNotBlank(secretKey) && isNotBlank(region)) {
+ return AWSSecretsManagerClientBuilder.standard()
+ .withRegion(region)
+ .withCredentials(new BasicAWSCredentialsProvider(accessKey, secretKey))
+ .build();
+ } else {
+ return getDefaultClient();
+ }
+ }
+
+ private AWSSecretsManager getDefaultClient() {
+ return AWSSecretsManagerClientBuilder.standard()
+ .withCredentials(DefaultAWSCredentialsProviderChain.getInstance())
+ .build();
+ }
+
+ private static String getSecretName(final String contextName, final String parameterName) {
+ return String.format(QUALIFIED_SECRET_FORMAT, contextName, parameterName);
+ }
+
+ private static boolean isNotBlank(final String value) {
+ return value != null && !value.trim().equals("");
+ }
+
+ private static class BasicAWSCredentialsProvider implements AWSCredentialsProvider {
Review comment:
Ah nice, that's exactly what I want. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org