You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by GitBox <gi...@apache.org> on 2020/06/29 15:15:19 UTC
[GitHub] [nifi] tpalfy opened a new pull request #4369: NIFI-7581 Add CS-based credential settings support for ADLS processors
tpalfy opened a new pull request #4369:
URL: https://github.com/apache/nifi/pull/4369
https://issues.apache.org/jira/browse/NIFI-7581
Added ControllerService-based credential settings support for ADLS processors
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with **NIFI-XXXX** where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [ ] Has your PR been rebased against the latest commit within the target branch (typically `master`)?
- [ ] Is your initial contribution a single, squashed commit? _Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not `squash` or use `--force` when pushing to allow for clean monitoring of changes._
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via `mvn -Pcontrib-check clean install` at the root `nifi` folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] Have you verified that the full build is successful on JDK 8?
- [ ] Have you verified that the full build is successful on JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main `LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main `NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to .name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for build issues and submit an update to your PR as soon as possible.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] asfgit closed pull request #4369: NIFI-7581 Add CS-based credential settings support for ADLS processors
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #4369:
URL: https://github.com/apache/nifi/pull/4369
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] MuazmaZ commented on a change in pull request #4369: NIFI-7581 Add CS-based credential settings support for ADLS processors
Posted by GitBox <gi...@apache.org>.
MuazmaZ commented on a change in pull request #4369:
URL: https://github.com/apache/nifi/pull/4369#discussion_r450387709
##########
File path: nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/AbstractAzureDataLakeStorageProcessor.java
##########
@@ -158,41 +173,51 @@
public static Collection<ValidationResult> validateCredentialProperties(final ValidationContext validationContext) {
final List<ValidationResult> results = new ArrayList<>();
- final boolean useManagedIdentity = validationContext.getProperty(USE_MANAGED_IDENTITY).asBoolean();
- final boolean accountKeyIsSet = validationContext.getProperty(ACCOUNT_KEY).isSet();
- final boolean sasTokenIsSet = validationContext.getProperty(SAS_TOKEN).isSet();
-
- int credential_config_found = 0;
- if(useManagedIdentity) credential_config_found++;
- if(accountKeyIsSet) credential_config_found++;
- if(sasTokenIsSet) credential_config_found++;
-
- if(credential_config_found == 0){
- final String msg = String.format(
- "At least one of ['%s', '%s', '%s'] should be set",
- ACCOUNT_KEY.getDisplayName(),
- SAS_TOKEN.getDisplayName(),
- USE_MANAGED_IDENTITY.getDisplayName()
- );
- results.add(new ValidationResult.Builder().subject("Credentials config").valid(false).explanation(msg).build());
- } else if(credential_config_found > 1) {
- final String msg = String.format(
- "Only one of ['%s', '%s', '%s'] should be set",
- ACCOUNT_KEY.getDisplayName(),
- SAS_TOKEN.getDisplayName(),
- USE_MANAGED_IDENTITY.getDisplayName()
- );
- results.add(new ValidationResult.Builder().subject("Credentials config").valid(false).explanation(msg).build());
+ if (!validationContext.getProperty(AzureStorageUtils.STORAGE_CREDENTIALS_SERVICE).isSet()) {
Review comment:
@tpalfy makes sense to me especially if that is the pattern for the newer processor.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] tpalfy commented on a change in pull request #4369: NIFI-7581 Add CS-based credential settings support for ADLS processors
Posted by GitBox <gi...@apache.org>.
tpalfy commented on a change in pull request #4369:
URL: https://github.com/apache/nifi/pull/4369#discussion_r450917508
##########
File path: nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/AbstractAzureDataLakeStorageProcessor.java
##########
@@ -158,41 +173,51 @@
public static Collection<ValidationResult> validateCredentialProperties(final ValidationContext validationContext) {
final List<ValidationResult> results = new ArrayList<>();
- final boolean useManagedIdentity = validationContext.getProperty(USE_MANAGED_IDENTITY).asBoolean();
- final boolean accountKeyIsSet = validationContext.getProperty(ACCOUNT_KEY).isSet();
- final boolean sasTokenIsSet = validationContext.getProperty(SAS_TOKEN).isSet();
-
- int credential_config_found = 0;
- if(useManagedIdentity) credential_config_found++;
- if(accountKeyIsSet) credential_config_found++;
- if(sasTokenIsSet) credential_config_found++;
-
- if(credential_config_found == 0){
- final String msg = String.format(
- "At least one of ['%s', '%s', '%s'] should be set",
- ACCOUNT_KEY.getDisplayName(),
- SAS_TOKEN.getDisplayName(),
- USE_MANAGED_IDENTITY.getDisplayName()
- );
- results.add(new ValidationResult.Builder().subject("Credentials config").valid(false).explanation(msg).build());
- } else if(credential_config_found > 1) {
- final String msg = String.format(
- "Only one of ['%s', '%s', '%s'] should be set",
- ACCOUNT_KEY.getDisplayName(),
- SAS_TOKEN.getDisplayName(),
- USE_MANAGED_IDENTITY.getDisplayName()
- );
- results.add(new ValidationResult.Builder().subject("Credentials config").valid(false).explanation(msg).build());
+ if (!validationContext.getProperty(AzureStorageUtils.STORAGE_CREDENTIALS_SERVICE).isSet()) {
Review comment:
Great, thanks, I did the change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] turcsanyip commented on a change in pull request #4369: NIFI-7581 Add CS-based credential settings support for ADLS processors
Posted by GitBox <gi...@apache.org>.
turcsanyip commented on a change in pull request #4369:
URL: https://github.com/apache/nifi/pull/4369#discussion_r452307527
##########
File path: nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/services/azure/storage/ADLSCredentialsControllerService.java
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.azure.storage;
+
+import com.google.common.base.Function;
Review comment:
Is Guava's `Function` used intentionally?
##########
File path: nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/AbstractAzureDataLakeStorageProcessor.java
##########
@@ -33,66 +33,26 @@
import org.apache.commons.lang3.StringUtils;
import org.apache.nifi.components.PropertyDescriptor;
-import org.apache.nifi.components.ValidationContext;
-import org.apache.nifi.components.ValidationResult;
import org.apache.nifi.components.Validator;
import org.apache.nifi.context.PropertyContext;
import org.apache.nifi.expression.ExpressionLanguageScope;
import org.apache.nifi.flowfile.FlowFile;
import org.apache.nifi.processor.AbstractProcessor;
import org.apache.nifi.processor.Relationship;
import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.storage.ADLSCredentialsDetails;
+import org.apache.nifi.services.azure.storage.ADLSCredentialsService;
+import reactor.core.publisher.Mono;
public abstract class AbstractAzureDataLakeStorageProcessor extends AbstractProcessor {
- public static final PropertyDescriptor ACCOUNT_NAME = new PropertyDescriptor.Builder()
- .name("storage-account-name").displayName("Storage Account Name")
- .description("The storage account name. There are certain risks in allowing the account name to be stored as a flowfile " +
- "attribute. While it does provide for a more flexible flow by allowing the account name to " +
- "be fetched dynamically from a flowfile attribute, care must be taken to restrict access to " +
- "the event provenance data (e.g. by strictly controlling the policies governing provenance for this Processor). " +
- "In addition, the provenance repositories may be put on encrypted disk partitions." +
- " Instead of defining the Storage Account Name, Storage Account Key and SAS Token properties directly on the processor, " +
- "the preferred way is to configure them through a controller service")
- .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
- .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
- .required(true)
- .sensitive(true).build();
-
- public static final PropertyDescriptor ACCOUNT_KEY = new PropertyDescriptor.Builder()
- .name("storage-account-key").displayName("Storage Account Key")
- .description("The storage account key. This is an admin-like password providing access to every container in this account. It is recommended " +
- "one uses Shared Access Signature (SAS) token instead for fine-grained control with policies. " +
- "There are certain risks in allowing the account key to be stored as a flowfile " +
- "attribute. While it does provide for a more flexible flow by allowing the account key to " +
- "be fetched dynamically from a flow file attribute, care must be taken to restrict access to " +
- "the event provenance data (e.g. by strictly controlling the policies governing provenance for this Processor). " +
- "In addition, the provenance repositories may be put on encrypted disk partitions.")
- .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
- .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
- .required(false)
- .sensitive(true).build();
-
- public static final PropertyDescriptor SAS_TOKEN = new PropertyDescriptor.Builder()
- .name("storage-sas-token").displayName("SAS Token")
- .description("Shared Access Signature token, including the leading '?'. Specify either SAS Token (recommended) or Account Key. " +
- "There are certain risks in allowing the SAS token to be stored as a flowfile " +
- "attribute. While it does provide for a more flexible flow by allowing the account name to " +
- "be fetched dynamically from a flowfile attribute, care must be taken to restrict access to " +
- "the event provenance data (e.g. by strictly controlling the policies governing provenance for this Processor). " +
- "In addition, the provenance repositories may be put on encrypted disk partitions.")
- .required(false)
- .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
- .sensitive(true)
- .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
- .build();
-
- public static final PropertyDescriptor USE_MANAGED_IDENTITY = new PropertyDescriptor.Builder()
- .name("use-managed-identity")
- .displayName("Use Azure Managed Identity")
- .description("Choose whether or not to use the managed identity of Azure VM/VMSS ")
- .required(false).defaultValue("false").allowableValues("true", "false")
- .addValidator(StandardValidators.BOOLEAN_VALIDATOR).build();
+ public static final PropertyDescriptor CREDENTIALS_SERVICE = new PropertyDescriptor.Builder()
+ .name("adls-credentials-service")
+ .displayName("Azure Credentials")
Review comment:
I would rather call it "ADLS Credentials" on the UI which is more specific.
##########
File path: nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/services/azure/storage/TestADLSCredentialsControllerService.java
##########
@@ -0,0 +1,242 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.azure.storage;
+
+import org.apache.nifi.processors.azure.storage.utils.AzureStorageUtils;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.util.NoOpProcessor;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.HashMap;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertTrue;
+
+public class TestADLSCredentialsControllerService {
+
+ public static final String CREDENTIALS_SERVICE_IDENTIIFIER = "credentials-service";
+
+ private static final String ACCOUNT_NAME_VALUE = "AccountName";
+ private static final String ACCOUNT_KEY_VALUE = "AccountKey";
+ private static final String SAS_TOKEN_VALUE = "SasToken";
+ public static final String END_POINT_SUFFIX_VALUE = "end.point.suffix";
+
+ private TestRunner runner;
+ private ADLSCredentialsControllerService credentialsService;
+
+ @Before
+ public void setUp() throws InitializationException {
+ runner = TestRunners.newTestRunner(NoOpProcessor.class);
+ credentialsService = new ADLSCredentialsControllerService();
+ runner.addControllerService(CREDENTIALS_SERVICE_IDENTIIFIER, credentialsService);
+ }
+
+ @Test
+ public void testNotValidBecauseAccountNameMissing() {
+ configureAccountKey();
+
+ runner.assertNotValid(credentialsService);
+ }
+
+ @Test
+ public void testNotValidBecauseNoCredentialsIsSet() {
+ configureAccountName();
+
+ runner.assertNotValid(credentialsService);
+ }
+
+ @Test
+ public void testNotValidBecauseBothAccountKeyAndSasTokenSpecified() {
+ configureAccountName();
+
+ configureAccountKey();
+ configureSasToken();
+
+ runner.assertNotValid(credentialsService);
+ }
+
+ @Test
+ public void testNotValidBecauseBothAccountKeyAndUseManagedIdentitySpecified() {
+ configureAccountName();
+
+ configureSasToken();
Review comment:
Typo: configureAccountKey()
##########
File path: nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/TestAbstractAzureDataLakeStorage.java
##########
@@ -34,57 +36,20 @@
private TestRunner runner;
@Before
- public void setUp() {
+ public void setUp() throws Exception {
// test the property validation in the abstract class via the put processor
runner = TestRunners.newTestRunner(PutAzureDataLakeStorage.class);
- runner.setProperty(ACCOUNT_NAME, "accountName");
- runner.setProperty(ACCOUNT_KEY, "accountKey");
+ ControllerService credentialsService = mock(ControllerService.class, withSettings().extraInterfaces(ADLSCredentialsService.class));
Review comment:
What is the difference between this and `mock(ADLSCredentialsService.class)` ?
##########
File path: nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/services/azure/storage/TestADLSCredentialsControllerService.java
##########
@@ -0,0 +1,242 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.azure.storage;
+
+import org.apache.nifi.processors.azure.storage.utils.AzureStorageUtils;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.util.NoOpProcessor;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.HashMap;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertTrue;
+
+public class TestADLSCredentialsControllerService {
+
+ public static final String CREDENTIALS_SERVICE_IDENTIIFIER = "credentials-service";
Review comment:
Typo: IDENTIFIER
All the constants can be private.
##########
File path: nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/services/azure/storage/TestADLSCredentialsControllerService.java
##########
@@ -0,0 +1,242 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.azure.storage;
+
+import org.apache.nifi.processors.azure.storage.utils.AzureStorageUtils;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.util.NoOpProcessor;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.HashMap;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertTrue;
+
+public class TestADLSCredentialsControllerService {
+
+ public static final String CREDENTIALS_SERVICE_IDENTIIFIER = "credentials-service";
+
+ private static final String ACCOUNT_NAME_VALUE = "AccountName";
+ private static final String ACCOUNT_KEY_VALUE = "AccountKey";
+ private static final String SAS_TOKEN_VALUE = "SasToken";
+ public static final String END_POINT_SUFFIX_VALUE = "end.point.suffix";
+
+ private TestRunner runner;
+ private ADLSCredentialsControllerService credentialsService;
+
+ @Before
+ public void setUp() throws InitializationException {
+ runner = TestRunners.newTestRunner(NoOpProcessor.class);
+ credentialsService = new ADLSCredentialsControllerService();
+ runner.addControllerService(CREDENTIALS_SERVICE_IDENTIIFIER, credentialsService);
+ }
+
+ @Test
+ public void testNotValidBecauseAccountNameMissing() {
+ configureAccountKey();
+
+ runner.assertNotValid(credentialsService);
+ }
+
+ @Test
+ public void testNotValidBecauseNoCredentialsIsSet() {
+ configureAccountName();
+
+ runner.assertNotValid(credentialsService);
+ }
+
+ @Test
+ public void testNotValidBecauseBothAccountKeyAndSasTokenSpecified() {
+ configureAccountName();
+
+ configureAccountKey();
+ configureSasToken();
+
+ runner.assertNotValid(credentialsService);
+ }
+
+ @Test
+ public void testNotValidBecauseBothAccountKeyAndUseManagedIdentitySpecified() {
+ configureAccountName();
+
+ configureSasToken();
+ configureUseManagedIdentity();
+
+ runner.assertNotValid(credentialsService);
+ }
+
+ @Test
+ public void testNotValidBecauseBothSasTokenAndUseManagedIdentitySpecified() {
+ configureAccountName();
+
+ configureAccountKey();
Review comment:
Typo: configureSasToken()
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] tpalfy commented on a change in pull request #4369: NIFI-7581 Add CS-based credential settings support for ADLS processors
Posted by GitBox <gi...@apache.org>.
tpalfy commented on a change in pull request #4369:
URL: https://github.com/apache/nifi/pull/4369#discussion_r450368696
##########
File path: nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/AbstractAzureDataLakeStorageProcessor.java
##########
@@ -158,41 +173,51 @@
public static Collection<ValidationResult> validateCredentialProperties(final ValidationContext validationContext) {
final List<ValidationResult> results = new ArrayList<>();
- final boolean useManagedIdentity = validationContext.getProperty(USE_MANAGED_IDENTITY).asBoolean();
- final boolean accountKeyIsSet = validationContext.getProperty(ACCOUNT_KEY).isSet();
- final boolean sasTokenIsSet = validationContext.getProperty(SAS_TOKEN).isSet();
-
- int credential_config_found = 0;
- if(useManagedIdentity) credential_config_found++;
- if(accountKeyIsSet) credential_config_found++;
- if(sasTokenIsSet) credential_config_found++;
-
- if(credential_config_found == 0){
- final String msg = String.format(
- "At least one of ['%s', '%s', '%s'] should be set",
- ACCOUNT_KEY.getDisplayName(),
- SAS_TOKEN.getDisplayName(),
- USE_MANAGED_IDENTITY.getDisplayName()
- );
- results.add(new ValidationResult.Builder().subject("Credentials config").valid(false).explanation(msg).build());
- } else if(credential_config_found > 1) {
- final String msg = String.format(
- "Only one of ['%s', '%s', '%s'] should be set",
- ACCOUNT_KEY.getDisplayName(),
- SAS_TOKEN.getDisplayName(),
- USE_MANAGED_IDENTITY.getDisplayName()
- );
- results.add(new ValidationResult.Builder().subject("Credentials config").valid(false).explanation(msg).build());
+ if (!validationContext.getProperty(AzureStorageUtils.STORAGE_CREDENTIALS_SERVICE).isSet()) {
Review comment:
I think it would make sense to have the credentials configurable only via the controller service.
I see the newer processors follow this pattern as well.
@MuazmaZ, would you be okay with removing all the Azure credentials-related properties from the processors and move them to the controller service?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] MuazmaZ commented on a change in pull request #4369: NIFI-7581 Add CS-based credential settings support for ADLS processors
Posted by GitBox <gi...@apache.org>.
MuazmaZ commented on a change in pull request #4369:
URL: https://github.com/apache/nifi/pull/4369#discussion_r452971444
##########
File path: nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/AbstractAzureDataLakeStorageProcessor.java
##########
@@ -33,66 +33,26 @@
import org.apache.commons.lang3.StringUtils;
import org.apache.nifi.components.PropertyDescriptor;
-import org.apache.nifi.components.ValidationContext;
-import org.apache.nifi.components.ValidationResult;
import org.apache.nifi.components.Validator;
import org.apache.nifi.context.PropertyContext;
import org.apache.nifi.expression.ExpressionLanguageScope;
import org.apache.nifi.flowfile.FlowFile;
import org.apache.nifi.processor.AbstractProcessor;
import org.apache.nifi.processor.Relationship;
import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.storage.ADLSCredentialsDetails;
+import org.apache.nifi.services.azure.storage.ADLSCredentialsService;
+import reactor.core.publisher.Mono;
public abstract class AbstractAzureDataLakeStorageProcessor extends AbstractProcessor {
- public static final PropertyDescriptor ACCOUNT_NAME = new PropertyDescriptor.Builder()
- .name("storage-account-name").displayName("Storage Account Name")
- .description("The storage account name. There are certain risks in allowing the account name to be stored as a flowfile " +
- "attribute. While it does provide for a more flexible flow by allowing the account name to " +
- "be fetched dynamically from a flowfile attribute, care must be taken to restrict access to " +
- "the event provenance data (e.g. by strictly controlling the policies governing provenance for this Processor). " +
- "In addition, the provenance repositories may be put on encrypted disk partitions." +
- " Instead of defining the Storage Account Name, Storage Account Key and SAS Token properties directly on the processor, " +
- "the preferred way is to configure them through a controller service")
- .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
- .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
- .required(true)
- .sensitive(true).build();
-
- public static final PropertyDescriptor ACCOUNT_KEY = new PropertyDescriptor.Builder()
- .name("storage-account-key").displayName("Storage Account Key")
- .description("The storage account key. This is an admin-like password providing access to every container in this account. It is recommended " +
- "one uses Shared Access Signature (SAS) token instead for fine-grained control with policies. " +
- "There are certain risks in allowing the account key to be stored as a flowfile " +
- "attribute. While it does provide for a more flexible flow by allowing the account key to " +
- "be fetched dynamically from a flow file attribute, care must be taken to restrict access to " +
- "the event provenance data (e.g. by strictly controlling the policies governing provenance for this Processor). " +
- "In addition, the provenance repositories may be put on encrypted disk partitions.")
- .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
- .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
- .required(false)
- .sensitive(true).build();
-
- public static final PropertyDescriptor SAS_TOKEN = new PropertyDescriptor.Builder()
- .name("storage-sas-token").displayName("SAS Token")
- .description("Shared Access Signature token, including the leading '?'. Specify either SAS Token (recommended) or Account Key. " +
- "There are certain risks in allowing the SAS token to be stored as a flowfile " +
- "attribute. While it does provide for a more flexible flow by allowing the account name to " +
- "be fetched dynamically from a flowfile attribute, care must be taken to restrict access to " +
- "the event provenance data (e.g. by strictly controlling the policies governing provenance for this Processor). " +
- "In addition, the provenance repositories may be put on encrypted disk partitions.")
- .required(false)
- .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
- .sensitive(true)
- .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
- .build();
-
- public static final PropertyDescriptor USE_MANAGED_IDENTITY = new PropertyDescriptor.Builder()
- .name("use-managed-identity")
- .displayName("Use Azure Managed Identity")
- .description("Choose whether or not to use the managed identity of Azure VM/VMSS ")
- .required(false).defaultValue("false").allowableValues("true", "false")
- .addValidator(StandardValidators.BOOLEAN_VALIDATOR).build();
+ public static final PropertyDescriptor CREDENTIALS_SERVICE = new PropertyDescriptor.Builder()
Review comment:
Blob property is named as STORAGE_CREDENTIALS_SERVICE, would need to clean up that as separate PR to avoid confusion.
being specific like ADLS_CREDENTIALS_SERVICE would be clear.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] turcsanyip commented on a change in pull request #4369: NIFI-7581 Add CS-based credential settings support for ADLS processors
Posted by GitBox <gi...@apache.org>.
turcsanyip commented on a change in pull request #4369:
URL: https://github.com/apache/nifi/pull/4369#discussion_r450057671
##########
File path: nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/AbstractAzureDataLakeStorageProcessor.java
##########
@@ -158,41 +173,51 @@
public static Collection<ValidationResult> validateCredentialProperties(final ValidationContext validationContext) {
final List<ValidationResult> results = new ArrayList<>();
- final boolean useManagedIdentity = validationContext.getProperty(USE_MANAGED_IDENTITY).asBoolean();
- final boolean accountKeyIsSet = validationContext.getProperty(ACCOUNT_KEY).isSet();
- final boolean sasTokenIsSet = validationContext.getProperty(SAS_TOKEN).isSet();
-
- int credential_config_found = 0;
- if(useManagedIdentity) credential_config_found++;
- if(accountKeyIsSet) credential_config_found++;
- if(sasTokenIsSet) credential_config_found++;
-
- if(credential_config_found == 0){
- final String msg = String.format(
- "At least one of ['%s', '%s', '%s'] should be set",
- ACCOUNT_KEY.getDisplayName(),
- SAS_TOKEN.getDisplayName(),
- USE_MANAGED_IDENTITY.getDisplayName()
- );
- results.add(new ValidationResult.Builder().subject("Credentials config").valid(false).explanation(msg).build());
- } else if(credential_config_found > 1) {
- final String msg = String.format(
- "Only one of ['%s', '%s', '%s'] should be set",
- ACCOUNT_KEY.getDisplayName(),
- SAS_TOKEN.getDisplayName(),
- USE_MANAGED_IDENTITY.getDisplayName()
- );
- results.add(new ValidationResult.Builder().subject("Credentials config").valid(false).explanation(msg).build());
+ if (!validationContext.getProperty(AzureStorageUtils.STORAGE_CREDENTIALS_SERVICE).isSet()) {
Review comment:
When the CS defined, the processor level credential properties should not be defined. It should be validated.
Another option would be to get rid of the processor level credential properties. That would make the validation easier and the whole code simpler.
##########
File path: nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/AbstractAzureDataLakeStorageProcessor.java
##########
@@ -158,41 +173,51 @@
public static Collection<ValidationResult> validateCredentialProperties(final ValidationContext validationContext) {
final List<ValidationResult> results = new ArrayList<>();
- final boolean useManagedIdentity = validationContext.getProperty(USE_MANAGED_IDENTITY).asBoolean();
- final boolean accountKeyIsSet = validationContext.getProperty(ACCOUNT_KEY).isSet();
- final boolean sasTokenIsSet = validationContext.getProperty(SAS_TOKEN).isSet();
-
- int credential_config_found = 0;
- if(useManagedIdentity) credential_config_found++;
- if(accountKeyIsSet) credential_config_found++;
- if(sasTokenIsSet) credential_config_found++;
-
- if(credential_config_found == 0){
- final String msg = String.format(
- "At least one of ['%s', '%s', '%s'] should be set",
- ACCOUNT_KEY.getDisplayName(),
- SAS_TOKEN.getDisplayName(),
- USE_MANAGED_IDENTITY.getDisplayName()
- );
- results.add(new ValidationResult.Builder().subject("Credentials config").valid(false).explanation(msg).build());
- } else if(credential_config_found > 1) {
- final String msg = String.format(
- "Only one of ['%s', '%s', '%s'] should be set",
- ACCOUNT_KEY.getDisplayName(),
- SAS_TOKEN.getDisplayName(),
- USE_MANAGED_IDENTITY.getDisplayName()
- );
- results.add(new ValidationResult.Builder().subject("Credentials config").valid(false).explanation(msg).build());
+ if (!validationContext.getProperty(AzureStorageUtils.STORAGE_CREDENTIALS_SERVICE).isSet()) {
+ final boolean useManagedIdentity = validationContext.getProperty(USE_MANAGED_IDENTITY).asBoolean();
+ final boolean accountKeyIsSet = validationContext.getProperty(ACCOUNT_KEY).isSet();
+ final boolean sasTokenIsSet = validationContext.getProperty(SAS_TOKEN).isSet();
+
+ int credential_config_found = 0;
+ if (useManagedIdentity) credential_config_found++;
+ if (accountKeyIsSet) credential_config_found++;
+ if (sasTokenIsSet) credential_config_found++;
+
+ if (credential_config_found == 0) {
+ final String msg = String.format(
+ "At least one of ['%s', '%s', '%s'] should be set",
+ ACCOUNT_KEY.getDisplayName(),
+ SAS_TOKEN.getDisplayName(),
+ USE_MANAGED_IDENTITY.getDisplayName()
+ );
+ results.add(new ValidationResult.Builder().subject("Credentials config").valid(false).explanation(msg).build());
+ } else if (credential_config_found > 1) {
+ final String msg = String.format(
+ "Only one of ['%s', '%s', '%s'] should be set",
+ ACCOUNT_KEY.getDisplayName(),
+ SAS_TOKEN.getDisplayName(),
+ USE_MANAGED_IDENTITY.getDisplayName()
+ );
+ results.add(new ValidationResult.Builder().subject("Credentials config").valid(false).explanation(msg).build());
+ }
}
+
return results;
}
public static DataLakeServiceClient getStorageClient(PropertyContext context, FlowFile flowFile) {
- final Map<String, String> attributes = flowFile != null ? flowFile.getAttributes() : Collections.emptyMap();
- final String accountName = context.getProperty(ACCOUNT_NAME).evaluateAttributeExpressions(attributes).getValue();
- final String accountKey = context.getProperty(ACCOUNT_KEY).evaluateAttributeExpressions(attributes).getValue();
- final String sasToken = context.getProperty(SAS_TOKEN).evaluateAttributeExpressions(attributes).getValue();
- final String endpointSuffix = context.getProperty(ENDPOINT_SUFFIX).evaluateAttributeExpressions(attributes).getValue();
+ AzureStorageCredentialsDetails storageCredentialsDetails = AzureStorageUtils.getStorageCredentialsDetails(context, flowFile);
+
+ final String accountName = storageCredentialsDetails.getStorageAccountName();
+ final String accountKey = storageCredentialsDetails.getAccountKey();
+ final String sasToken = storageCredentialsDetails.getSasToken();
+ final AccessToken accessToken = storageCredentialsDetails.getAccessToken();
+
+ // ControllerService - if set - may or may not have endpoint suffix defined
+ final String endpointSuffix = Optional
+ .ofNullable(storageCredentialsDetails.getStorageSuffix())
+ .orElse(context.getProperty(ENDPOINT_SUFFIX).getValue());
Review comment:
The processor level property is not a fallback option. When the CS defined, the processor level properties should be null (see my comment on `validateCredentialProperties()`).
##########
File path: nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/AbstractAzureDataLakeStorageProcessor.java
##########
@@ -60,7 +71,8 @@
.sensitive(true).build();
public static final PropertyDescriptor ACCOUNT_KEY = new PropertyDescriptor.Builder()
- .name("storage-account-key").displayName("Storage Account Key")
+ .name(STORAGE_ACCOUNT_KEY_PROPERTY_DESCRIPTOR_NAME)
Review comment:
The property descriptor definition seems to be the same as in `AzureStorageUtils`.
The same descriptors could be used here for the Account Key and SAS Token (maybe for Endpoint Override too).
##########
File path: nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/AbstractAzureDataLakeStorageProcessor.java
##########
@@ -204,6 +229,11 @@ public static DataLakeServiceClient getStorageClient(PropertyContext context, Fl
} else if (StringUtils.isNotBlank(sasToken)) {
storageClient = new DataLakeServiceClientBuilder().endpoint(endpoint).sasToken(sasToken)
.buildClient();
+ } else if (accessToken != null) {
+ TokenCredential credential = tokenRequestContext -> Mono.just(accessToken);
+
+ storageClient = new DataLakeServiceClientBuilder().endpoint(endpoint).credential(credential)
+ .buildClient();
} else if(useManagedIdentity){
Review comment:
Use Managed Identity option should be available on the CS too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] turcsanyip commented on a change in pull request #4369: NIFI-7581 Add CS-based credential settings support for ADLS processors
Posted by GitBox <gi...@apache.org>.
turcsanyip commented on a change in pull request #4369:
URL: https://github.com/apache/nifi/pull/4369#discussion_r452068857
##########
File path: nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/services/azure/storage/ADLSCredentialsControllerService.java
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.azure.storage;
+
+import com.google.common.base.Function;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.azure.storage.utils.AzureStorageUtils;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.StringJoiner;
+import java.util.function.BiConsumer;
+
+/**
+ * Provides credentials details for ADLS
+ *
+ * @see AbstractControllerService
+ */
+@Tags({"azure", "microsoft", "cloud", "storage", "adls", "credentials"})
+@CapabilityDescription("Defines credentials for ADLS processors.")
+public class ADLSCredentialsControllerService extends AbstractControllerService implements ADLSCredentialsService {
+
+ public static final PropertyDescriptor ACCOUNT_NAME = new PropertyDescriptor.Builder()
+ .fromPropertyDescriptor(AzureStorageUtils.ACCOUNT_NAME)
+ .description(AzureStorageUtils.ACCOUNT_NAME_BASE_DESCRIPTION)
+ .required(true)
+ .build();
+
+ public static final PropertyDescriptor ENDPOINT_SUFFIX = new PropertyDescriptor.Builder()
+ .fromPropertyDescriptor(AzureStorageUtils.ENDPOINT_SUFFIX)
+ .description(
+ "Storage accounts in public Azure always use a common FQDN suffix. " +
+ "Override this endpoint suffix with a different suffix in certain circumsances (like Azure Stack or non-public Azure regions).")
+ .required(true)
+ .defaultValue("dfs.core.windows.net")
+ .build();
+
+ public static final PropertyDescriptor USE_MANAGED_IDENTITY = new PropertyDescriptor.Builder()
+ .name("storage-use-managed-identity")
+ .displayName("Use Azure Managed Identity")
+ .description("Choose whether or not to use the managed identity of Azure VM/VMSS ")
+ .required(false)
+ .defaultValue("false")
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .build();
+
+ private static final List<PropertyDescriptor> PROPERTIES = Collections.unmodifiableList(Arrays.asList(
+ ACCOUNT_NAME,
+ AzureStorageUtils.ACCOUNT_KEY,
+ AzureStorageUtils.PROP_SAS_TOKEN,
+ ENDPOINT_SUFFIX,
Review comment:
I would move it up after the Account Name. It is not really a credential attribute, but rather belongs to the storage "location" and Account Name is also related to the location / storage url.
##########
File path: nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/AbstractAzureDataLakeStorageProcessor.java
##########
@@ -136,11 +88,7 @@
.build();
private static final List<PropertyDescriptor> PROPERTIES = Collections.unmodifiableList(
- Arrays.asList(AbstractAzureDataLakeStorageProcessor.ACCOUNT_NAME,
- AbstractAzureDataLakeStorageProcessor.ACCOUNT_KEY,
- AbstractAzureDataLakeStorageProcessor.SAS_TOKEN,
- AbstractAzureDataLakeStorageProcessor.USE_MANAGED_IDENTITY,
- AbstractAzureDataLakeStorageProcessor.ENDPOINT_SUFFIX,
+ Arrays.asList(AzureStorageUtils.STORAGE_CREDENTIALS_SERVICE,
Review comment:
This is the Blob credential service.
##########
File path: nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/services/azure/storage/ADLSCredentialsControllerService.java
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.azure.storage;
+
+import com.google.common.base.Function;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.azure.storage.utils.AzureStorageUtils;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.StringJoiner;
+import java.util.function.BiConsumer;
+
+/**
+ * Provides credentials details for ADLS
+ *
+ * @see AbstractControllerService
+ */
+@Tags({"azure", "microsoft", "cloud", "storage", "adls", "credentials"})
+@CapabilityDescription("Defines credentials for ADLS processors.")
+public class ADLSCredentialsControllerService extends AbstractControllerService implements ADLSCredentialsService {
+
+ public static final PropertyDescriptor ACCOUNT_NAME = new PropertyDescriptor.Builder()
+ .fromPropertyDescriptor(AzureStorageUtils.ACCOUNT_NAME)
+ .description(AzureStorageUtils.ACCOUNT_NAME_BASE_DESCRIPTION)
+ .required(true)
+ .build();
+
+ public static final PropertyDescriptor ENDPOINT_SUFFIX = new PropertyDescriptor.Builder()
+ .fromPropertyDescriptor(AzureStorageUtils.ENDPOINT_SUFFIX)
+ .description(
+ "Storage accounts in public Azure always use a common FQDN suffix. " +
+ "Override this endpoint suffix with a different suffix in certain circumsances (like Azure Stack or non-public Azure regions).")
Review comment:
Typo: circumstances
(old issue, pls. also fix it in AzureStorageUtils as well)
##########
File path: nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/services/azure/storage/ADLSCredentialsControllerService.java
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.azure.storage;
+
+import com.google.common.base.Function;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.azure.storage.utils.AzureStorageUtils;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.StringJoiner;
+import java.util.function.BiConsumer;
+
+/**
+ * Provides credentials details for ADLS
+ *
+ * @see AbstractControllerService
+ */
+@Tags({"azure", "microsoft", "cloud", "storage", "adls", "credentials"})
+@CapabilityDescription("Defines credentials for ADLS processors.")
+public class ADLSCredentialsControllerService extends AbstractControllerService implements ADLSCredentialsService {
+
+ public static final PropertyDescriptor ACCOUNT_NAME = new PropertyDescriptor.Builder()
+ .fromPropertyDescriptor(AzureStorageUtils.ACCOUNT_NAME)
+ .description(AzureStorageUtils.ACCOUNT_NAME_BASE_DESCRIPTION)
+ .required(true)
+ .build();
+
+ public static final PropertyDescriptor ENDPOINT_SUFFIX = new PropertyDescriptor.Builder()
+ .fromPropertyDescriptor(AzureStorageUtils.ENDPOINT_SUFFIX)
+ .description(
Review comment:
I think we should customize the display name as well.
Something like "Endpoint Suffix" (as it was in the processor level property) or "Storage Endpoint Suffix" would be fine.
I don't really understand why it has the long "Common Storage Account Endpoint Suffix" name in the Blob processors.
##########
File path: nifi-nar-bundles/nifi-azure-bundle/nifi-azure-services-api/src/main/java/org/apache/nifi/services/azure/storage/ADLSCredentialsDetails.java
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.azure.storage;
+
+import com.azure.core.credential.AccessToken;
+
+public class ADLSCredentialsDetails {
+ private final String accountName;
+
+ private final String accountKey;
+ private final String sasToken;
+ private final String endpointSuffix;
+
+ private final AccessToken accessToken;
+
+ private final boolean useManagedIdentity;
+
+ public ADLSCredentialsDetails(
+ String accountName,
+ String accountKey,
+ String sasToken,
+ String endpointSuffix,
+ AccessToken accessToken,
+ boolean useManagedIdentity
+ ) {
+ this.accountName = accountName;
+ this.accountKey = accountKey;
+ this.sasToken = sasToken;
+ this.accessToken = accessToken;
+ this.endpointSuffix = endpointSuffix;
+ this.useManagedIdentity = useManagedIdentity;
Review comment:
Could you please use the same order of the fields above (also adjust constructor parameters and assignments) as in the builder / getters (those looks fine)?
##########
File path: nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/AbstractAzureDataLakeStorageProcessor.java
##########
@@ -136,11 +88,7 @@
.build();
private static final List<PropertyDescriptor> PROPERTIES = Collections.unmodifiableList(
- Arrays.asList(AbstractAzureDataLakeStorageProcessor.ACCOUNT_NAME,
- AbstractAzureDataLakeStorageProcessor.ACCOUNT_KEY,
- AbstractAzureDataLakeStorageProcessor.SAS_TOKEN,
- AbstractAzureDataLakeStorageProcessor.USE_MANAGED_IDENTITY,
- AbstractAzureDataLakeStorageProcessor.ENDPOINT_SUFFIX,
+ Arrays.asList(AzureStorageUtils.STORAGE_CREDENTIALS_SERVICE,
AbstractAzureDataLakeStorageProcessor.FILESYSTEM,
AbstractAzureDataLakeStorageProcessor.DIRECTORY,
AbstractAzureDataLakeStorageProcessor.FILE));
Review comment:
Unnecessary class name in static reference. It is an old issue, but please fix it.
Another old issue below: the local REL_SUCCESS / REL_FAILURE constants should be used.
Pls. fix it too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org