You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "pvillard31 (via GitHub)" <gi...@apache.org> on 2025/11/10 11:47:37 UTC
[PR] NIFI-15200 - Introduce Authentication Strategy for GCP Credentials Controller Service [nifi]
pvillard31 opened a new pull request, #10510:
URL: https://github.com/apache/nifi/pull/10510
# Summary
NIFI-15200 - Introduce Authentication Strategy for GCP Credentials Controller Service
# Tracking
Please complete the following tracking steps prior to pull request creation.
### Issue Tracking
- [ ] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created
### Pull Request Tracking
- [ ] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-00000`
- [ ] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-00000`
### Pull Request Formatting
- [ ] Pull Request based on current revision of the `main` branch
- [ ] Pull Request refers to a feature branch with one commit containing changes
# Verification
Please indicate the verification steps performed prior to pull request creation.
### Build
- [ ] Build completed using `./mvnw clean install -P contrib-check`
- [ ] JDK 21
- [ ] JDK 25
### Licensing
- [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html)
- [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files
### Documentation
- [ ] Documentation formatting appears as expected in rendered files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-15200 - Introduce Authentication Strategy for GCP Credentials Controller Service [nifi]
Posted by "turcsanyip (via GitHub)" <gi...@apache.org>.
turcsanyip commented on code in PR #10510:
URL: https://github.com/apache/nifi/pull/10510#discussion_r2514169191
##########
nifi-extension-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/credentials/service/GCPCredentialsControllerService.java:
##########
@@ -140,6 +142,35 @@ public void migrateProperties(PropertyConfiguration config) {
config.renameProperty("compute-engine-credentials", USE_COMPUTE_ENGINE_CREDENTIALS.getName());
config.renameProperty("service-account-json-file", SERVICE_ACCOUNT_JSON_FILE.getName());
config.renameProperty("service-account-json", SERVICE_ACCOUNT_JSON.getName());
+
+ if (config.getRawPropertyValue(AUTHENTICATION_STRATEGY).isEmpty()) {
+ final AuthenticationStrategy authenticationStrategy = determineAuthenticationStrategy(config);
+ if (authenticationStrategy != null) {
+ config.setProperty(AUTHENTICATION_STRATEGY, authenticationStrategy.getValue());
+ }
+ }
+
+ config.removeProperty(USE_APPLICATION_DEFAULT_CREDENTIALS.getName());
+ config.removeProperty(USE_COMPUTE_ENGINE_CREDENTIALS.getName());
+ }
+
+ private AuthenticationStrategy determineAuthenticationStrategy(final PropertyConfiguration config) {
+ if (isTrue(config, USE_APPLICATION_DEFAULT_CREDENTIALS)) {
+ return AuthenticationStrategy.APPLICATION_DEFAULT;
+ } else if (config.isPropertySet(SERVICE_ACCOUNT_JSON_FILE)) {
+ return AuthenticationStrategy.SERVICE_ACCOUNT_JSON_FILE;
+ } else if (config.isPropertySet(SERVICE_ACCOUNT_JSON)) {
+ return AuthenticationStrategy.SERVICE_ACCOUNT_JSON;
+ } else if (isTrue(config, USE_COMPUTE_ENGINE_CREDENTIALS)) {
+ return AuthenticationStrategy.COMPUTE_ENGINE;
+ }
+ return null;
Review Comment:
I think `AuthenticationStrategy.APPLICATION_DEFAULT` should be the default case because the old version fell back to Application Default Credentials when none of the properties was configured (empty or false).
##########
nifi-extension-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/credentials/factory/CredentialPropertyDescriptors.java:
##########
@@ -83,6 +92,7 @@ private CredentialPropertyDescriptors() { }
.expressionLanguageSupported(ExpressionLanguageScope.ENVIRONMENT)
.required(false)
.addValidator(JsonValidator.INSTANCE)
+ .dependsOn(AUTHENTICATION_STRATEGY, AuthenticationStrategy.SERVICE_ACCOUNT_JSON.getValue())
Review Comment:
`Service Account JSON` property should be `required(true)` for validation.
##########
nifi-extension-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/credentials/factory/CredentialPropertyDescriptors.java:
##########
@@ -75,6 +83,7 @@ private CredentialPropertyDescriptors() { }
.expressionLanguageSupported(ExpressionLanguageScope.NONE)
.required(false)
.identifiesExternalResource(ResourceCardinality.SINGLE, ResourceType.FILE)
+ .dependsOn(AUTHENTICATION_STRATEGY, AuthenticationStrategy.SERVICE_ACCOUNT_JSON_FILE.getValue())
Review Comment:
`Service Account JSON File` property should be `required(true)` for validation.
##########
nifi-extension-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/credentials/factory/CredentialPropertyDescriptors.java:
##########
Review Comment:
`USE_APPLICATION_DEFAULT_CREDENTIALS` and `USE_COMPUTE_ENGINE_CREDENTIALS` properties are not used in the regular way anymore, only in migration. So it might make sense to delete the property definition and keep the name only:
```
public static final PropertyDescriptor USE_APPLICATION_DEFAULT_CREDENTIALS = new PropertyDescriptor.Builder()
.name("Use Application Default Credentials")
.build();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-15200 - Introduce Authentication Strategy for GCP Credentials Controller Service [nifi]
Posted by "asf-gitbox-commits (via GitHub)" <gi...@apache.org>.
asf-gitbox-commits closed pull request #10510: NIFI-15200 - Introduce Authentication Strategy for GCP Credentials Controller Service
URL: https://github.com/apache/nifi/pull/10510
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-15200 - Introduce Authentication Strategy for GCP Credentials Controller Service [nifi]
Posted by "pvillard31 (via GitHub)" <gi...@apache.org>.
pvillard31 commented on PR #10510:
URL: https://github.com/apache/nifi/pull/10510#issuecomment-3517051994
Pushed another commit @turcsanyip, thanks for the review. I did test the change this way: in NiFi without the changes of that PR, I have 4 instances of the GCP credentials controller service configured with each combination, I then rebuild NiFi with the changes and restart with the same flow.json.gz and I can confirm that all 4 controller services are properly configured with the right allowable value set.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-15200 - Introduce Authentication Strategy for GCP Credentials Controller Service [nifi]
Posted by "pvillard31 (via GitHub)" <gi...@apache.org>.
pvillard31 commented on PR #10510:
URL: https://github.com/apache/nifi/pull/10510#issuecomment-3513774823
Thanks for the thorough review @turcsanyip - I pushed a commit to address your comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-15200 - Introduce Authentication Strategy for GCP Credentials Controller Service [nifi]
Posted by "pvillard31 (via GitHub)" <gi...@apache.org>.
pvillard31 commented on code in PR #10510:
URL: https://github.com/apache/nifi/pull/10510#discussion_r2515005947
##########
nifi-extension-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/credentials/service/GCPCredentialsControllerService.java:
##########
@@ -136,10 +139,52 @@ public void onConfigured(final ConfigurationContext context) throws Initializati
@Override
public void migrateProperties(PropertyConfiguration config) {
- config.renameProperty("application-default-credentials", USE_APPLICATION_DEFAULT_CREDENTIALS.getName());
- config.renameProperty("compute-engine-credentials", USE_COMPUTE_ENGINE_CREDENTIALS.getName());
+ config.renameProperty("application-default-credentials", LEGACY_USE_APPLICATION_DEFAULT_CREDENTIALS.getName());
+ config.renameProperty("compute-engine-credentials", LEGACY_USE_COMPUTE_ENGINE_CREDENTIALS.getName());
config.renameProperty("service-account-json-file", SERVICE_ACCOUNT_JSON_FILE.getName());
config.renameProperty("service-account-json", SERVICE_ACCOUNT_JSON.getName());
+
+ final boolean legacyPropertiesPresent = config.hasProperty(LEGACY_USE_APPLICATION_DEFAULT_CREDENTIALS)
+ || config.hasProperty(LEGACY_USE_COMPUTE_ENGINE_CREDENTIALS)
+ || config.hasProperty(SERVICE_ACCOUNT_JSON_FILE)
+ || config.hasProperty(SERVICE_ACCOUNT_JSON);
Review Comment:
:( thanks for finding this edge case Peter! I pushed another commit and manually verified all of the scenarios I could think of...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-15200 - Introduce Authentication Strategy for GCP Credentials Controller Service [nifi]
Posted by "turcsanyip (via GitHub)" <gi...@apache.org>.
turcsanyip commented on code in PR #10510:
URL: https://github.com/apache/nifi/pull/10510#discussion_r2511398891
##########
nifi-extension-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/credentials/service/GCPCredentialsControllerService.java:
##########
@@ -140,6 +141,35 @@ public void migrateProperties(PropertyConfiguration config) {
config.renameProperty("compute-engine-credentials", USE_COMPUTE_ENGINE_CREDENTIALS.getName());
config.renameProperty("service-account-json-file", SERVICE_ACCOUNT_JSON_FILE.getName());
config.renameProperty("service-account-json", SERVICE_ACCOUNT_JSON.getName());
+
+ if (!config.hasProperty(AUTHENTICATION_STRATEGY.getName())) {
Review Comment:
I tested the property migration in different ways (in-place flow.json.gz and flow json import) and it always results in `Authentication Strategy = Application Default Credentials` which is not the expected behavior.
It looks the framework calls `migrateProperties()` twice:
1. the component contains the old properties only => everything is fine
2. the component contains the union of the old and the new properties (with default values if defined) => problem if the code checks the existence of a new property (as it is already there for some reason)
Unfortunately, the 2nd execution overrides the result of the 1st.
Not sure, if the duplicated call is a bug or a feature in the migration framework. Based on [this comment](https://github.com/apache/nifi/blob/28c526d9bdb963a6966b469bd9707723ce0aa1b0/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/flow/synchronization/StandardVersionedComponentSynchronizer.java#L751-L753), it is by design.
A possible client-side workaround could be to check for the existence of a removed property instead of the non-existence of an added property, like:
```
if (config.hasProperty(USE_APPLICATION_DEFAULT_CREDENTIALS)) { ... }
```
##########
nifi-extension-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/credentials/service/GCPCredentialsControllerService.java:
##########
@@ -140,6 +141,35 @@ public void migrateProperties(PropertyConfiguration config) {
config.renameProperty("compute-engine-credentials", USE_COMPUTE_ENGINE_CREDENTIALS.getName());
config.renameProperty("service-account-json-file", SERVICE_ACCOUNT_JSON_FILE.getName());
config.renameProperty("service-account-json", SERVICE_ACCOUNT_JSON.getName());
+
+ if (!config.hasProperty(AUTHENTICATION_STRATEGY.getName())) {
Review Comment:
Minor: `hasProperty()` / `isPropertySet()` / `get[Raw]PropertyValue()` methods have signatures with `PropertyDescriptor` parameter so `getName()` could be omitted which is less verbose.
##########
nifi-extension-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/credentials/factory/CredentialsFactory.java:
##########
@@ -46,19 +48,35 @@
public class CredentialsFactory {
private final List<CredentialsStrategy> strategies = new ArrayList<>();
+ private final Map<AuthenticationStrategy, CredentialsStrategy> strategiesByAuthentication = new EnumMap<>(AuthenticationStrategy.class);
public CredentialsFactory() {
// Primary Credential Strategies
- strategies.add(new ExplicitApplicationDefaultCredentialsStrategy());
- strategies.add(new JsonFileServiceAccountCredentialsStrategy());
- strategies.add(new JsonStringServiceAccountCredentialsStrategy());
- strategies.add(new ComputeEngineCredentialsStrategy());
+ final CredentialsStrategy explicitApplicationDefaultCredentialsStrategy = new ExplicitApplicationDefaultCredentialsStrategy();
+ final CredentialsStrategy jsonFileServiceAccountCredentialsStrategy = new JsonFileServiceAccountCredentialsStrategy();
+ final CredentialsStrategy jsonStringServiceAccountCredentialsStrategy = new JsonStringServiceAccountCredentialsStrategy();
+ final CredentialsStrategy computeEngineCredentialsStrategy = new ComputeEngineCredentialsStrategy();
+
+ strategies.add(explicitApplicationDefaultCredentialsStrategy);
+ strategies.add(jsonFileServiceAccountCredentialsStrategy);
+ strategies.add(jsonStringServiceAccountCredentialsStrategy);
+ strategies.add(computeEngineCredentialsStrategy);
// Implicit Default is the catch-all primary strategy
strategies.add(new ImplicitApplicationDefaultCredentialsStrategy());
Review Comment:
I think `ImplicitApplicationDefaultCredentialsStrategy` should be removed as it is not used anymore. Also, `ExplicitApplicationDefaultCredentialsStrategy` could be renamed to simply `ApplicationDefaultCredentialsStrategy`.
##########
nifi-extension-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/credentials/factory/CredentialsFactory.java:
##########
@@ -46,19 +48,35 @@
public class CredentialsFactory {
private final List<CredentialsStrategy> strategies = new ArrayList<>();
+ private final Map<AuthenticationStrategy, CredentialsStrategy> strategiesByAuthentication = new EnumMap<>(AuthenticationStrategy.class);
public CredentialsFactory() {
// Primary Credential Strategies
- strategies.add(new ExplicitApplicationDefaultCredentialsStrategy());
- strategies.add(new JsonFileServiceAccountCredentialsStrategy());
- strategies.add(new JsonStringServiceAccountCredentialsStrategy());
- strategies.add(new ComputeEngineCredentialsStrategy());
+ final CredentialsStrategy explicitApplicationDefaultCredentialsStrategy = new ExplicitApplicationDefaultCredentialsStrategy();
+ final CredentialsStrategy jsonFileServiceAccountCredentialsStrategy = new JsonFileServiceAccountCredentialsStrategy();
+ final CredentialsStrategy jsonStringServiceAccountCredentialsStrategy = new JsonStringServiceAccountCredentialsStrategy();
+ final CredentialsStrategy computeEngineCredentialsStrategy = new ComputeEngineCredentialsStrategy();
+
+ strategies.add(explicitApplicationDefaultCredentialsStrategy);
+ strategies.add(jsonFileServiceAccountCredentialsStrategy);
+ strategies.add(jsonStringServiceAccountCredentialsStrategy);
+ strategies.add(computeEngineCredentialsStrategy);
// Implicit Default is the catch-all primary strategy
strategies.add(new ImplicitApplicationDefaultCredentialsStrategy());
+
+ strategiesByAuthentication.put(AuthenticationStrategy.APPLICATION_DEFAULT, explicitApplicationDefaultCredentialsStrategy);
+ strategiesByAuthentication.put(AuthenticationStrategy.SERVICE_ACCOUNT_JSON_FILE, jsonFileServiceAccountCredentialsStrategy);
+ strategiesByAuthentication.put(AuthenticationStrategy.SERVICE_ACCOUNT_JSON, jsonStringServiceAccountCredentialsStrategy);
+ strategiesByAuthentication.put(AuthenticationStrategy.COMPUTE_ENGINE, computeEngineCredentialsStrategy);
}
public CredentialsStrategy selectPrimaryStrategy(final Map<PropertyDescriptor, String> properties) {
+ final CredentialsStrategy selectedStrategy = selectStrategyFromAuthenticationProperty(properties);
+ if (selectedStrategy != null) {
+ return selectedStrategy;
+ }
+
for (CredentialsStrategy strategy : strategies) {
if (strategy.canCreatePrimaryCredential(properties)) {
return strategy;
Review Comment:
I think we can get rid of the old `canCreatePrimaryCredential()` based strategy routing. The new `Authentication Strategy` property (with proper property migration) entirely replaces the old way. The method can even be removed from the `CredentialsStrategy` interface.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-15200 - Introduce Authentication Strategy for GCP Credentials Controller Service [nifi]
Posted by "turcsanyip (via GitHub)" <gi...@apache.org>.
turcsanyip commented on code in PR #10510:
URL: https://github.com/apache/nifi/pull/10510#discussion_r2514838289
##########
nifi-extension-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/credentials/service/GCPCredentialsControllerService.java:
##########
@@ -136,10 +139,52 @@ public void onConfigured(final ConfigurationContext context) throws Initializati
@Override
public void migrateProperties(PropertyConfiguration config) {
- config.renameProperty("application-default-credentials", USE_APPLICATION_DEFAULT_CREDENTIALS.getName());
- config.renameProperty("compute-engine-credentials", USE_COMPUTE_ENGINE_CREDENTIALS.getName());
+ config.renameProperty("application-default-credentials", LEGACY_USE_APPLICATION_DEFAULT_CREDENTIALS.getName());
+ config.renameProperty("compute-engine-credentials", LEGACY_USE_COMPUTE_ENGINE_CREDENTIALS.getName());
config.renameProperty("service-account-json-file", SERVICE_ACCOUNT_JSON_FILE.getName());
config.renameProperty("service-account-json", SERVICE_ACCOUNT_JSON.getName());
+
+ final boolean legacyPropertiesPresent = config.hasProperty(LEGACY_USE_APPLICATION_DEFAULT_CREDENTIALS)
+ || config.hasProperty(LEGACY_USE_COMPUTE_ENGINE_CREDENTIALS)
+ || config.hasProperty(SERVICE_ACCOUNT_JSON_FILE)
+ || config.hasProperty(SERVICE_ACCOUNT_JSON);
Review Comment:
`SERVICE_ACCOUNT_JSON_FILE` and `SERVICE_ACCOUNT_JSON` are not legacy-only properties but they are present in both the old and the new services. As a result, `legacyPropertiesPresent` flag will always be `true` and it can lead to the following incorrect scenario (non-real migration):
1. Create a new controller service (using the updated version)
2. First choose `Service Account Credentials (Json File)` and specify a path (even invalid path)
3. Then choose `Application Default Credentials` => the controller service works with this credential, the service account json file is not used
4. Restart NiFi => the `AuthenticationStrategy` migration is triggered for the component (because the `if` condition in line 160 is evaluated to `true`) and modifies the strategy to `Service Account Credentials (Json File)` (it should not)
I haven't tried it but it seems to me that if `legacyPropertiesPresent` counts only `LEGACY_USE_APPLICATION_DEFAULT_CREDENTIALS` and `LEGACY_USE_COMPUTE_ENGINE_CREDENTIALS`, then it should work. In this case `authenticationStrategyIsDefault` check may not be needed either.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org