You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "markap14 (via GitHub)" <gi...@apache.org> on 2026/08/20 19:52:06 UTC
[PR] NIFI-16201: Reload Parameter-driven extra classpath for Connector Methods [nifi]
markap14 opened a new pull request, #11574:
URL: https://github.com/apache/nifi/pull/11574
## Summary
- When a Parameter or asset changes a Processor or Controller Service extra classpath, stopped Working components never restart, so their `InstanceClassLoader` can omit the new JAR. Connector Method discovery and invocation then fail with `NoClassDefFoundError` / `LinkageError`.
- This change reloads additional classpath resources on Parameter update, always invokes Connector Methods on the live component, wraps `LinkageError` as `InvocationFailedException`, and treats `Throwable` as a failed async request so verification cannot hang.
- Adds `ClasspathVerifyIT` covering Processor verify, Connector Method `loadClass`, instance state across calls, and method-signature resolution when the classpath comes from a Parameter, Processor property, or Controller Service property.
https://issues.apache.org/jira/browse/NIFI-16201
## Test plan
- [x] `ClasspathVerifyIT` (9 tests)
- [x] Unit tests for `nifi-framework-core-api`, `nifi-framework-components`, `nifi-web-api`
- [x] Connector system tests
- [x] Checkstyle and PMD on modified modules
- [ ] CI (`./mvnw clean install -P contrib-check`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-16201: Reload Parameter-driven extra classpath for Connector Methods [nifi]
Posted by "bbende (via GitHub)" <gi...@apache.org>.
bbende merged PR #11574:
URL: https://github.com/apache/nifi/pull/11574
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-16201: Reload Parameter-driven extra classpath for Connector Methods [nifi]
Posted by "markap14 (via GitHub)" <gi...@apache.org>.
markap14 commented on PR #11574:
URL: https://github.com/apache/nifi/pull/11574#issuecomment-5372916734
Two CI failures on `f6be5026` look unrelated to this PR. Reran only the failed shards.
**integration-tests โ [Ubuntu Java 21](https://github.com/apache/nifi/actions/runs/32497455285/job/96819149028)** in [run 32497455285](https://github.com/apache/nifi/actions/runs/32497455285)
`CreateConnectorIT.testStopConnectorWithTimeoutStopsRunningConnector` timed out after 2 minutes waiting for a mock Connector to reach STOPPED. That test starts `GenerateAndLog` and immediately stops it; it does not exercise Parameter classpath reloads or Connector Method discovery. The other integration/compliance shards on this head passed.
```
[ERROR] org.apache.nifi.mock.connectors.tests.CreateConnectorIT.testStopConnectorWithTimeoutStopsRunningConnector -- Time elapsed: 120.9 s <<< FAILURE!
org.opentest4j.AssertionFailedError: Unexpected exception thrown: java.util.concurrent.TimeoutException: Timed out waiting for the Connector to stop after PT2M
at org.apache.nifi.mock.connectors.tests.CreateConnectorIT.testStopConnectorWithTimeoutStopsRunningConnector(CreateConnectorIT.java:85)
Caused by: java.util.concurrent.TimeoutException: Timed out waiting for the Connector to stop after PT2M
at org.apache.nifi.mock.connector.server.StandardConnectorMockServer.stopConnector(StandardConnectorMockServer.java:387)
```
**system-tests โ [macos-15 Java 21](https://github.com/apache/nifi/actions/runs/32497455268/job/96819148793)** in [run 32497455268](https://github.com/apache/nifi/actions/runs/32497455268)
`ClusteredConnectorBacklogIT.testClusteredReadQueueAttributesLoadBalancedConnection` failed in teardown: delete of `BacklogReportingTestConnector` got HTTP 409 because 2 FlowFiles were still queued after a ROUND_ROBIN load-balanced purge. `ClasspathVerifyIT` passed on this shard, and ubuntu-24.04 Java 21/25 plus macos-15 Java 25 all passed.
```
[ERROR] ClusteredConnectorBacklogIT.testClusteredReadQueueAttributesLoadBalancedConnection -- Time elapsed: 11.44 s <<< ERROR!
org.apache.nifi.toolkit.client.NiFiClientException: Error deleting Connector: Node localhost:5671 is unable to fulfill this request due to: Cannot delete StandardConnectorNode[..., name=BacklogReportingTestConnector, state=STOPPED] because its Process Group has 2 FlowFiles queued; all FlowFiles must be removed before it can be deleted.
Caused by: jakarta.ws.rs.ClientErrorException: HTTP 409 Conflict
```
Node logs show the purge reported success, then the cluster DELETE was rejected for leftover queue data โ in-flight load-balance leftovers, not a ClassLoader reload.
Reran: `gh run rerun 32497455285 --failed` and `gh run rerun 32497455268 --failed`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-16201: Reload Parameter-driven extra classpath for Connector Methods [nifi]
Posted by "markap14 (via GitHub)" <gi...@apache.org>.
markap14 commented on PR #11574:
URL: https://github.com/apache/nifi/pull/11574#issuecomment-5361146991
[cursor-grok-4.6] Code-compliance Validate failed on two Checkstyle `LineLength` violations in `ClasspathVerifyIT` (lines 129 and 141 were over 200 characters). Wrapped those `verifyConnectorStepConfigWithReferences` calls in ae4dfc5073f.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-16201: Reload Parameter-driven extra classpath for Connector Methods [nifi]
Posted by "markap14 (via GitHub)" <gi...@apache.org>.
markap14 commented on code in PR #11574:
URL: https://github.com/apache/nifi/pull/11574#discussion_r3831536048
##########
nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/StandardProcessorNode.java:
##########
@@ -2082,6 +2082,11 @@ public String invokeConnectorMethod(final String methodName, final Map<String, S
} catch (final Exception e) {
throw new InvocationFailedException(e);
}
+ } catch (final LinkageError e) {
+ // Argument types, return types, and the invoked method body can all reference classes supplied by additional classpath
+ // resources. Those failures are LinkageError, not Exception, so they would otherwise escape as an uncaught Error.
+ throw new InvocationFailedException("Failed to invoke Connector Method '" + methodName + "' on " + this
+ + " because a class required by the component could not be loaded from the component's ClassLoader", e);
Review Comment:
@sfc-gh-bbende Agreed. Addressed in f6be5026c9f: the outer catch is now `TypeNotPresentException | LinkageError`, and the comment describes the remaining paths (`MethodArgument.type()`, implementation Method parameter types, vs discovery wrapping `LinkageError` and method-body Errors arriving as `InvocationTargetException`). The same change is on `StandardControllerServiceNode`.
##########
nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/facades/standalone/StandaloneParameterContextFacade.java:
##########
@@ -95,8 +95,14 @@ public void updateParameters(final Collection<ParameterValue> updatedValues) {
final Map<String, Parameter> updatedParameters = createParameterMap(updatedValues);
managedProcessGroup.getParameterContext().setParameters(updatedParameters);
- allReferencingProcessors.forEach(ProcessorNode::resetValidationState);
+ // A component whose additional classpath resources are supplied by a Parameter must be reloaded so that its ClassLoader reflects the
+ // updated Parameter values. Components that are not running are not restarted below, so reloading here is the only point at which their
+ // ClassLoader is rebuilt.
+ allReferencingServices.forEach(ControllerServiceNode::reloadAdditionalResourcesIfNecessary);
+ allReferencingProcessors.forEach(ProcessorNode::reloadAdditionalResourcesIfNecessary);
Review Comment:
@sfc-gh-bbende Also addressed in f6be5026c9f. After reloading each service in `allReferencingServices`, we now call `findRecursiveReferences(ComponentNode.class)` and reload those components, matching `StandardControllerServiceNode.setProperties()`. Covered by `StandaloneParameterContextFacadeTest`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-16201: Reload Parameter-driven extra classpath for Connector Methods [nifi]
Posted by "sfc-gh-bbende (via GitHub)" <gi...@apache.org>.
sfc-gh-bbende commented on code in PR #11574:
URL: https://github.com/apache/nifi/pull/11574#discussion_r3831249597
##########
nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/facades/standalone/StandaloneParameterContextFacade.java:
##########
@@ -95,8 +95,14 @@ public void updateParameters(final Collection<ParameterValue> updatedValues) {
final Map<String, Parameter> updatedParameters = createParameterMap(updatedValues);
managedProcessGroup.getParameterContext().setParameters(updatedParameters);
- allReferencingProcessors.forEach(ProcessorNode::resetValidationState);
+ // A component whose additional classpath resources are supplied by a Parameter must be reloaded so that its ClassLoader reflects the
+ // updated Parameter values. Components that are not running are not restarted below, so reloading here is the only point at which their
+ // ClassLoader is rebuilt.
+ allReferencingServices.forEach(ControllerServiceNode::reloadAdditionalResourcesIfNecessary);
+ allReferencingProcessors.forEach(ProcessorNode::reloadAdditionalResourcesIfNecessary);
Review Comment:
This reloads the directly referencing components, which is narrower than the equivalent property-change path. `StandardControllerServiceNode.setProperties` reloads recursively, because changing a Controller Service property can alter the classloader isolation key of components that reference it:
```java
// It's possible that changing the properties of this Controller Service could alter the Classloader Isolation Key of a referencing
// component so reload any referencing component as necessary.
getReferences().findRecursiveReferences(ComponentNode.class).forEach(ComponentNode::reloadAdditionalResourcesIfNecessary);
```
A Parameter update can change that same isolation-key-relevant service property, so a stopped Processor that references one of these services will still keep a stale `InstanceClassLoader` after this call. Active referencing components are covered, since `activeSet` includes service references and gets restarted below, but stopped ones are not. Worth applying the same recursive reload for `allReferencingServices`.
##########
nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/StandardProcessorNode.java:
##########
@@ -2082,6 +2082,11 @@ public String invokeConnectorMethod(final String methodName, final Map<String, S
} catch (final Exception e) {
throw new InvocationFailedException(e);
}
+ } catch (final LinkageError e) {
+ // Argument types, return types, and the invoked method body can all reference classes supplied by additional classpath
+ // resources. Those failures are LinkageError, not Exception, so they would otherwise escape as an uncaught Error.
+ throw new InvocationFailedException("Failed to invoke Connector Method '" + methodName + "' on " + this
+ + " because a class required by the component could not be loaded from the component's ClassLoader", e);
Review Comment:
This `catch (LinkageError)` is mostly unreachable, and the comment describes coverage that it doesn't actually provide:
- The `getDeclaredMethods()` path that this JIRA is about now throws from `discoverConnectorMethod`, which this PR changed to wrap `LinkageError` into `InvocationFailedException` itself. That error is converted one layer down and never reaches here.
- "the invoked method body" is already handled today: a `NoClassDefFoundError` thrown by the target method is wrapped by `Method.invoke` into an `InvocationTargetException`, which is an `Exception` and is caught by the existing inner `catch (final Exception e)` just above.
- The gap that does remain is not a `LinkageError`. Resolving `methodArgument.type()` when that class is missing from the component's ClassLoader throws `TypeNotPresentException`, a `RuntimeException` produced by annotation `Class`-member resolution, so it still escapes unwrapped. The `AsyncRequestManager` change keeps that from hanging the request, but the caller gets a generic failure rather than an `InvocationFailedException`.
Suggest either dropping this block as redundant, or broadening it to also cover `TypeNotPresentException` and tightening the comment to describe what it really catches. The same applies to the identical block added to `StandardControllerServiceNode`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-16201: Reload Parameter-driven extra classpath for Connector Methods [nifi]
Posted by "markap14 (via GitHub)" <gi...@apache.org>.
markap14 commented on PR #11574:
URL: https://github.com/apache/nifi/pull/11574#issuecomment-5361223741
[cursor-grok-4.6] Validate Pull Request failed because `ae4dfc50` was not a verified signed commit. Replaced it with signed `62fc913219e`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-16201: Reload Parameter-driven extra classpath for Connector Methods [nifi]
Posted by "bbende (via GitHub)" <gi...@apache.org>.
bbende commented on PR #11574:
URL: https://github.com/apache/nifi/pull/11574#issuecomment-5371978537
+1 latest updates look good, will merge after CI passes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-16201: Reload Parameter-driven extra classpath for Connector Methods [nifi]
Posted by "markap14 (via GitHub)" <gi...@apache.org>.
markap14 commented on PR #11574:
URL: https://github.com/apache/nifi/pull/11574#issuecomment-5362436383
[cursor-grok-4.6] Pre-existing flake on **ubuntu-24.04 Java 25** (system-tests): [job 96569764221](https://github.com/apache/nifi/actions/runs/32413629346/job/96569764221) in [run 32413629346](https://github.com/apache/nifi/actions/runs/32413629346).
`OffloadIT.testOffload` timed out while waiting for a disconnected node, then `OffloadContentClaimTruncationIT` failed to start node 2 because the port was still in use. This PR does not change clustering or offload; the same tests passed on ubuntu-24.04 Java 21 and both macOS shards.
```
[ERROR] OffloadIT.testOffload ยป Timeout testOffload() timed out after 10 minutes
[ERROR] OffloadContentClaimTruncationIT>NiFiSystemIT.setup:134->...waitForAllNodesConnected:333
Waited up to 60 seconds for all 2 nodes to connect but only 1 nodes connected.
Node [localhost:5671] Status=CONNECTED
```
```
org.apache.nifi.toolkit.client.NiFiClientException: Error retrieving node status: Cannot replicate request to Node localhost:5672 because the node is not connected
at org.apache.nifi.toolkit.client.impl.AbstractJerseyClient.executeAction(AbstractJerseyClient.java:107)
...
at org.apache.nifi.tests.system.clustering.OffloadIT.disconnectNode(OffloadIT.java:156)
at org.apache.nifi.tests.system.clustering.OffloadIT.testIteration(OffloadIT.java:77)
Caused by: jakarta.ws.rs.ClientErrorException: HTTP 409 Conflict
```
```
Caused by: java.net.BindException: Address already in use
```
Reran the failed shard: `gh run rerun 32413629346 --failed`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org