You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@nifi.apache.org by ma...@apache.org on 2026/03/10 19:35:13 UTC
(nifi) 01/01: NIFI-15258: Initial implementation of Connectors
This is an automated email from the ASF dual-hosted git repository.
markap14 pushed a commit to branch NIFI-15258
in repository https://gitbox.apache.org/repos/asf/nifi.git
commit 6071ce67df48c94388066da02e86b83486754d55
Author: Mark Payne <ma...@hotmail.com>
AuthorDate: Tue Sep 2 18:22:14 2025 -0400
NIFI-15258: Initial implementation of Connectors
NIFI-15258: Fixed vulnerable lz4 dependency
NIFI-15258: Fixed checkstyle violations/pmd
NIFI-15258: Addressed PMD findings
NIFI-15259: Added REST API for Connectors
Introduced comprehensive REST endpoints for managing Connectors through
the NiFi web interface and API clients.
Key capabilities:
- Full lifecycle management (create, read, update, delete)
- Start/stop control with run status endpoints
- Multi-step configuration with property groups supporting string
literals, asset references, and secret references
- Asynchronous configuration verification with polling support
- Access to connector's managed flow and process group status
- Search functionality within connector's encapsulated flow
Cluster support:
- Request replication across cluster nodes
- Response merging for dynamic property allowable values that may
differ between nodes
- Consistent configuration step and property descriptor handling
Infrastructure:
- Audit logging for all connector configuration changes
- Authorization integration with NiFi's policy framework
- UI extension support for connector-specific interfaces
- Frontend updates for connector access policy management
NIFI-15323: Adding operate permissions to the ConnectorEntity. (#10625)
This closes #10625
NIFI-15322: Require all property descriptors within a PropertyGroup / ConfigurationStep are unique (#10626)
* NIFI-15322: Huge refactoring of Connectors to require that <Step Name, Property Name> tuple is unique. Requiring that is simple, but now that we enforce that, we can greatly simplify much
of the code.
* NIFI-15322: Added StepConfiguration to simplify API rather than using Map<String, ConnectorValueReference> in some places
* NIFI-15322: Now that we enforce unique property names per step, introduced StepConfigurationContext to further reduce API complexity
Signed-off-by: Kevin Doran <kd...@apache.org>
NIFI-15326: Adding support of configuration step documentation. (#10631)
NIFI-15322: Adapting to new connector configuration step model.
NIFI-15330: Allow GhostConnector to be updated (#10632)
NIFI-15312: Implementation of Parameter Provider based Secrets Manager; updated mock framework to support secrets; validation cleanup (#10638)
NIFI-15336: Created AuthorizableSecret and implemented ParameterProviderSecretProvider to provide the Parameter Provider as the associated Authorizable (#10644)
NIFI-15258: Enable Custom UI to retrieve connector with flow contexts (#10652)
NIFI-15352: Added ConnectorClient to toolkit-cli so that it can be us… (#10654)
* NIFI-15352: Added ConnectorClient to toolkit-cli so that it can be used in system tests; created system tests for Connectors; bug fixes found running tests; updated Secret/SecretReference to support fully qualified names
* NIFI-15352: Addressed review feedback
NIFI-15343: Adding an endpoint to return available secrets to the con… (#10645)
* NIFI-15343: Adding an endpoint to return available secrets to the connector configuration wizard.
* NIFI-15343: Adding providerId to SecretDTO.
* NIFI-15343: Skipping ParameterProviders that are invalid or validating when fetching secrets.
* NIFI-15343: Adding fully qualified name to SecretDTO.
- Fixing rebase issue.
This closes #10645
NIFI-15315 Add support for assets in connectors (#10647)
- Support multiple asset references
- Clean up of assets in StandardConnectorDAO after apply and discard
- Add endpoint for listing assets for a given connector
- Add endpoint for discarding the working config of a connector
- Add endpoint for retrieving asset content
- Add system test for connectors that uploads an asset
Signed-off-by: Kevin Doran <kd...@apache.org>
* Fix issues after rebasing, enable ClusteredConnectorAssetsIT
NIFI-15361: Allowing configuration step documentation to be returned … (#10667)
* NIFI-15361: Allowing configuration step documentation to be returned independent of any connector instances.
NIFI-15369: Allow ConfigurationStep to depend on another (ConfigurationStep,Property) tuple (#10669)
NIFI-15367: Ensure that Connectors' implicit parameter contexts are n… (#10665)
* NIFI-15367: Ensure that Connectors' implicit parameter contexts are not registered with ParameterContextManager; added system test to verify; fixed bug in ConnectorAssetsIT where it did not wait for connector to return to its 'STOPPED' state before attempting to delete
* NIFI-15367: Bug fixes; review feedback
NIFI-15370: Add Connector ID to process groups and do not register create process groups if they have a connector ID. (#10673)
NIFI-15356: Adding authorization to the StandardNiFiConnectorWebContext. (#10660)
NIFI-15353: Adding support for rendering Connector Documentation. (#10683)
* NIFI-15353: Adding support for rendering Connector Documentation.
* NIFI-15353: Addressing review feedback.
* NIFI-15353: Using mergeMap instead of switchMap to allow for concurrent doc loading.
This closes #10683
NIFI-15429: Adding an optional query parameter for specifying which p… (#10732)
* NIFI-15429: Adding an optional query parameter for specifying which process group should be returned within the Connectors flow.
* NIFI-15429: Including managed process group id in the ConnectorDTO.
* NIFI-15429: Addressing review feedback.
* NIFI-15429: Updating path for fetching a Connector flow to require the desired Process Group ID.
* NIFI-15429: Fixing failing unit test.
NIFI-15430: Ensure that if we fail to initialize a Connector, we crea… (#10733)
* NIFI-15430: Ensure that if we fail to initialize a Connector, we create a GhostConnector instead and ensure that we also proivde the extensionMissing flag on ConnectorNode
* NIFI-15430: Added extensionMissing flag to Connector DTO
* NIFI-15430: If unable to load initial flow of a Connector, make ghosted connector instead
NIFI-15427: Added abiliy to drop flowfiles / drain flowfiles from a C… (#10730)
* NIFI-15427: Added abiliy to drop flowfiles / drain flowfiles from a Connector; added some system tests to verify existing behavior; fixed existing issue with IT
* NIFI-15427: Fixed Exception message in case unable to purge FlowFiles due to Connector state
* NIFI-15427: Ensured thread safety of state transitions when draining/purging FlowFiles
* NIFI-15427: Addressed review feedback
NIFI-15433: If connector validation throws an Exception keep trying u… (#10736)
* NIFI-15433: If connector validation throws an Exception keep trying until it completes (with a timeout between retries); some bug fixes
* NIFI-15433: Addressed review feedback
* NIFI-15433: Fixed failing unit tests
NIFI-15434: Ensure that we start or stop connectors on flow sync (#10737)
NIFI-15445: Removed 'disabled' state from Connectors; also fixed bug … (#10749)
* NIFI-15445: Removed 'disabled' state from Connectors; also fixed bug to ensure that if we fail to apply an update to a Connector that we stop it.
* NIFI-15445: Addressed review feedback
* NIFI-15445: Fixed checkstyle
NIFI-15440: Implementation of ConnectorActions (#10748)
* NIFI-15440: Implementation of ConnectorActions
* NIFI-15440: Addressed review feedback
* NIFI-15440: Addressed issues that occurred after rebase
* NIFI-15440: Addressed review feedback
NIFI-15439: Ensure that Process Groups are accessible from the approp… (#10742)
* NIFI-15439: Ensure that Process Groups are accessible from the appropriate endpoint but not accessible elsewhere
* NIFI-15439: FlowAnalyzingRegistryClient should use an explicit Connector ID of null when retrieving Process Group from FlowManager
NIFI-15468: Ensure that we properly initialize ConnectorNode even if a Ghost Connector is created (#10775)
This closes #10775
NIFI-15446: When invoking ConnectorMethod, make sure to serialize/des… (#10750)
* NIFI-15446: When invoking ConnectorMethod, make sure to serialize/deserialize the arguments and return value
* NIFI-15446: Addressed review feedback
NIFI-15376: Adding dependencies to the configuration step dto. (#10674)
NIFI-15453: Adding an endpoint to return all controller services for … (#10770)
* NIFI-15453: Adding an endpoint to return all controller services for a process group in a connector.
* NIFI-15453: Addressing review feedback.
This closes #10770
NIFI-15472: Added addAsset methods to ConnectorTestRunner (#10776)
NIFI-15482: MockConnectorAssetManager requires SPI file to start (#10783)
NIFI-15451: Added ability for Connectors to retrieve bundles availabl… (#10756)
* NIFI-15451: Added ability for Connectors to retrieve bundles available for component types and updated VersionedFlowUtils to make use of it to easily update Versioned components
* NIFI-15451: Added additional unit tests to VersionedFlowUtils; updates to how we compare component versions
* NIFI-15451: Fixed PMD violation
NIFI-15481: Updated ConnectorActions to reflect that starting is not … (#10782)
* NIFI-15481: Updated ConnectorActions to reflect that starting is not allowed when invalid. Also updated ValidationState to ensure that we don't return any ValidationResult with a valid flag of true as a 'validation error'.
* NIFI-15481: Addressed review feedback
NIFI-15258: Fixes necessary after rebasing against main
NIFI-15258: Addressed checkstyle violations
NIFI-15461: Added ability to initiate drainage of Connector's FlowFiles and calce; added tests to verify; some bug fixes (#10767)
NIFI-15489: Addressed the 4 bugs called out in Jira: Wrong delimiter in StandardConnectorPropertyValue.asList, ConnectorValidationContextBridge.getProperty() does not resolve property value properly; StandaloneProcessGroupLifecycle.stop() does not await controller service disabling; switched hashmap in StandaloneProcessGroupLifecycle to ConcurrentHashMap (#10798)
- Addressed another similar bug in which an invalid connector that is expected to be running would prevent NiFi from starting
NIFI-15490: If any processors or controller services are missing, make Connector invalid (#10799)
NIFI-15485: Include provider name when providing fqn for secrets (#10801)
NIFI-15497: Fixed unit tests (#10805)
NIFI-15258: Fixes for system tests that needed to be updated from 2.7.0-SNAPSHOT to 2.8.0-SNAPSHOT (#10807)
NIFI-15495 Restart Connectors that reference assets that were synchro… (#10806)
* NIFI-15495 Restart Connectors that reference assets that were synchronized
- Ensure Connectors re-resolve property values before starting
- Ensure asset clean up happens only after applyUpdate fully finishes
- Add connector asset properties to default nifi.properties
* Fix system test
* Encapsulate restart logic in new method on ConnectorNode
* Fix JavaDoc
NIFI-15465 Allow client-specified connector component ids (#10768)
Add verifyCreate to ConnectorDAO and test cases
NIFI-15467: Ensure that Connectors' versions are automatically resolv… (#10811)
* NIFI-15467: Ensure that Connectors' versions are automatically resolved on startup
- Allow connectors to specify Bundle Compatability when updating flow
- Default to BundleCompatability.RESOLVE_BUNDLE when connectors are updating flows
- Bug fixes; updated MockConnectorInitializationContext to extend from StandardConnectorInitializationContext to cut down on code duplication
* NIFI-15467: Addressed review feedback
NIFI-15529: Fix to StatelessGroupLifecycle to ensure that we set the desired state of Stateless Group Node before attempting to schedule; updated StandardProcessScheduler.startStatelessGroup to make it more clear if this is left out again at some point (#10832)
NIFI-15525: Ensure that we call onStepConfigured when we restore flow; ensure that we synchronized Parmaeter Providers before Connectors (#10835)
NIFI-15509: Enable Component reload additional classpath resources if needed (#10812)
NIFI-15536: Change Connector API to use DescribedValue vs AllowableValue (#10841)
NIFI-15511: Added endpoints for purging all FlowFiles for a given Con… (#10833)
* NIFI-15511: Added endpoints for purging all FlowFiles for a given Connector; added method to ConnectorClient to call these endpoints; updated system tests to use these endpoints when tearing down flows; some bug fixes
* NIFI-15511: Added WRITE permission requirement for retrieving and deleting a Connector Purge request
NIFI-15514: Ensure that Parameter Contexts are assigned to all Proces… (#10815)
* NIFI-15514: Ensure that Parameter Contexts are assigned to all Process Groups in a Connector and not just the top-level group. Ensure that all components are started when Connector starts instead of just Processors and Controller Services
- When Working Context is recreated, ensure that we appropriately apply Parameter Context to newly created Process Group(s)
- Ensure that when we cleanup unused assets for Connectors that we consider any assets that are referenced in either the Working or Active context instead of just the Active context
- Ensure that when we stop Process Group we call all tasks in background threads instead of calling .thenRun which could potentially run in the foreground thread
* NIFI-15514: Update parameter context assignment to occur during sync.
* NIFI-15514: Fixes around ensuring that processors/controller services are properly configured and notified of any configuration changes when parameters change
- Removed the updateParameterContexts from ProcessGroup.updateFlow, which was added in a previous commit as we went a different direction for the fix
---------
Co-authored-by: Bob Paulin <bo...@bobpaulin.com>
NIFI-15538: When starting/stopping components allow specifying whethe… (#10843)
* NIFI-15538: When starting/stopping components allow specifying whether or not the action should be recursive; code cleanup and simplification to use a Virtual Thread to execute code sequentially instead of chaining CompletableFutures.
* NIFI-15538: Addressed review feedback
---------
Co-authored-by: Mark Payne <ma...@snowflake.com>
NIFI-15480 Added ability to drop FlowFiles selectively using a Predicate (#10848)
* NIFI-15480: Added ability to drop FlowFiles selectively using a Predicate
Added system tests to verify selective dropping of flowfiles from a Connector. Also made significant updates so that diagnostic bundles are captured when a system test fails and diagnostic bundle also contains details of each Connection's FlowFileQueue. This was necessary in aiding the debugging of the new feature but is also something that has been missing for some time.
* NIFI-15480: Ensure that when we perform a selective drop on a FlowFileQueue that we pass appropriate SWAP_FILE_DELETED / SWAP_FILE_RENAMED events to the FlowFile Repository
NIFI-15549: Adding endpoints for viewing and clearing state for compo… (#10855)
* NIFI-15549: Adding endpoints for viewing and clearing state for components that are managed by a Connector.
* NIFI-15549: Addressing review feedback.
NIFI-15557: Allow Connectors to enable a Controller Service using overridden property values (#10862)
NIFI-15488: Added significant number of debug log messages as well as some info/error for connector-related events (#10803)
NIFI-15544: If a Connector requires processors or controller services that are unavailable, make Connector invalid but not ghosted (#10851)
NIFI-15558: Sensitive Parameter References should resolve to value. (#10863)
NIFI-15539: Updating the actions in FlowFileQueueResource to support … (#10846)
* NIFI-15539: Updating the actions in FlowFileQueueResource to support interacting with queues in flows managed by a Connector.
* NIFI-15539: Updating resource endpoint to include data policies for connectors.
NIFI-15565: Code cleanup; improvements to .cursorrules to provide bet… (#10869)
* NIFI-15565: Code cleanup; improvements to .cursorrules to provide better instructions regarding automated tests
* NIFI-15565: Broke apart cursorrules file into the new format of .cursor/rules/*
NIFI-15355 Connector Configuration Repository (#10876)
* NIFI-15255 Introduce ConnectorConfigurationRepository Extension Interface
* Add new property key to default nifi.properties file
* Address peer review comments
NIFI-15559: Enforce data permissions when purging data from a Connector. (#10867)
This closes #10867
NIFI-15258: Fixes as a result of rebasing against main
NIFI-15577: Including connector id in provenance event dto. (#10883)
* NIFI-15577: Including connector id in provenance event dto.
- Fixing go to routing logic in the provenance event table.
* NIFI-15577: Fixing back navigation context when navigating to provenance from the flow designer.
This closes #10883
NIFI-15581: Add support for ControllerService Mocks for Connectors. (#10885)
NIFI-15579: When synchronizing PG when Versioned flow for stateless group node, do not synchronize parameter contexts if group is within a Connector (#10884)
NIFI-15605 - Rebase Connectors branch with main for 2.9.0-SNAPSHOT
NIFI-15606: Connector Mock War implementation. (#10907)
* NIFI-15606: Connector Mock War implementation.
* Load war files from Connector Nar
* Mock Connector Web Context
* Allow test runner to set http port
* NIFI-15606: Connector Mock War implementation.
* Code Review Feedback
NIFI-15611: Fetchable Values should pull from current FlowContext (#10906)
NIFI-15618: Bug fixes around handling of FlowFileActivity with child groups and stateless groups (#10912)
NIFI-15598: When enabling referenced controller services in a connector, skip references from properties with unsatisfied dependencies (#10897)
NIFI-15615: Ensure that we include necessary values in the ConnectorStatusDTO (#10913)
NIFI-15610 Add asset management to ConnectorConfigurationProvider (#10909)
NIFI-15621: Allow Configuration to Validate with ValidationContext (#10916)
* Currently this is only implemented for processors in schedule periods
* Allows connector to pass Validation Context to be used instead of
current parameter context.
NIFI-15640: NPE in the applyFlow should NOT crash the runtime. (#10931)
* NIFI-15640: NPE in the applyFlow should NOT crash the runtime.
* Leave the flow in a state of UPDATE_FAILED
* With validationState of INVALID.
* NIFI-15640: NPE in the applyFlow should NOT crash the runtime.
* Code Review Feedback
NIFI-15629 Add ConnectorRequestContext (#10924)
Signed-off-by: David Handermann <ex...@apache.org>
NIFI-15663: Update to use new ComponentHierarchyScope instead of on facade/lifecycle classes (#10958)
NIFI-15680: Not including global parameter contexts and parameters in connector search results. (#10980)
This closes #10980
NIFI-15604: Expose VersionedExternalFlow that represents the Active/Working flow from ConnectorTestRunner (#10901)
Signed-off-by: Kevin Doran <kd...@apache.org>
NIFI-15648 - Resolve SECRET property values before fetchAllowableValues and verify in working flow context (#10937)
---
.cursor/rules/building.mdc | 18 +
.cursor/rules/code-style.mdc | 74 +
.cursor/rules/ending-conditions.mdc | 30 +
.cursor/rules/extension-development.mdc | 107 +
.cursor/rules/extension-testing.mdc | 80 +
.cursor/rules/framework-testing.mdc | 46 +
.cursor/rules/persona.mdc | 16 +
.cursor/rules/testing-standards.mdc | 81 +
.gitignore | 2 +-
.../protocol/component/api/ComponentManifest.java | 10 +
.../protocol/component/api/ConfigurationStep.java | 81 +
.../component/api/ConfigurationStepDependency.java | 61 +
.../component/api/ConnectorDefinition.java | 37 +-
.../component/api/ConnectorPropertyDependency.java | 51 +
.../component/api/ConnectorPropertyDescriptor.java | 111 +
.../component/api/ConnectorPropertyGroup.java | 61 +
.../component/api/ConnectorPropertyType.java | 25 +-
checkstyle.xml | 7 -
.../c2/command/AgentPropertyValidationContext.java | 5 +
.../c2/command/DefaultFlowStateStrategyTest.java | 9 +-
nifi-assembly/pom.xml | 6 +
nifi-commons/nifi-connector-utils/pom.xml | 42 +
.../connector/util/VersionedFlowUtils.java | 592 +++++
.../connector/util/TestVersionedFlowUtils.java | 285 +++
.../java/org/apache/nifi/util/NiFiProperties.java | 64 +-
.../java/org/apache/nifi/wali/HashMapSnapshot.java | 35 +-
.../apache/nifi/wali/LengthDelimitedJournal.java | 43 +-
.../src/main/java/org/wali/SerDe.java | 13 +
.../src/main/java/org/wali/SerDeFactory.java | 13 +
.../src/main/java/org/wali/UpdateType.java | 10 +-
nifi-commons/pom.xml | 1 +
.../nifi-connector-mock-api}/pom.xml | 30 +-
.../server/ConnectorConfigVerificationResult.java | 19 +-
.../mock/connector/server/ConnectorMockServer.java | 22 +-
.../mock/connector/server/ConnectorTestRunner.java | 103 +
.../nifi-connector-mock-server-nar}/pom.xml | 42 +-
.../nifi-connector-mock-server}/pom.xml | 47 +-
.../mock/connector/server/MockAuditService.java | 63 +
.../server/MockConnectorAssetManager.java | 179 ++
.../server/MockConnectorInitializationContext.java | 78 +
.../connector/server/MockConnectorRepository.java | 28 +-
.../server/MockExtensionDiscoveringManager.java | 74 +
.../mock/connector/server/MockExtensionMapper.java | 61 +
.../server/MockNiFiConnectorWebContext.java | 52 +
.../server/MockRuleViolationsManager.java | 79 +
.../server/MockServerConfigVerificationResult.java | 52 +
.../mock/connector/server/MockStateManager.java | 89 +
.../connector/server/MockStateManagerProvider.java | 59 +
.../nifi/mock/connector/server/MockStateMap.java | 56 +
.../connector/server/NopPropertyEncryptor.java | 26 +-
.../mock/connector/server/PermitAllAuthorizer.java | 46 +
.../server/StandardConnectorMockServer.java | 629 +++++
.../secrets/ConnectorTestRunnerAuthorizable.java | 28 +-
.../secrets/ConnectorTestRunnerSecretProvider.java | 94 +
.../secrets/ConnectorTestRunnerSecretsManager.java | 99 +
.../META-INF/services/org.apache.nifi.NiFiServer | 2 +-
.../services/org.apache.nifi.asset.AssetManager | 2 +-
...e.nifi.components.connector.ConnectorRepository | 2 +-
...ifi.components.connector.secrets.SecretsManager | 2 +-
.../server/MockConnectorAssetManagerTest.java | 245 ++
.../server/MockNiFiConnectorWebContextTest.java | 80 +
.../server/MockStateManagerProviderTest.java | 267 +++
.../StandardConnectorMockServerJettyTest.java | 186 ++
.../nifi-connector-mock-integration-tests/pom.xml | 130 ++
.../mock/connectors/tests/AllowableValuesIT.java | 87 +
.../mock/connectors/tests/CreateConnectorIT.java | 92 +
.../nifi/mock/connectors/tests/CronScheduleIT.java | 68 +
.../connectors/tests/MockControllerServiceIT.java | 87 +
.../mock/connectors/tests/MockProcessorIT.java | 91 +
.../pom.xml | 32 +-
.../nifi-connector-mock-test-connectors}/pom.xml | 35 +-
.../mock/connectors/AllowableValuesConnector.java | 106 +
.../mock/connectors/CronScheduleConnector.java | 106 +
.../nifi/mock/connectors/GenerateAndLog.java | 55 +
.../mock/connectors/MissingBundleConnector.java | 108 +
.../org.apache.nifi.components.connector.Connector | 5 +-
.../resources/flows/Cron_Schedule_Connector.json | 410 ++++
.../main/resources/flows/Generate_and_Update.json | 393 ++++
.../nifi-connector-mock-test-bundle}/pom.xml | 42 +-
.../nifi-connector-mock/pom.xml | 147 ++
.../connector/StandardConnectorTestRunner.java | 282 +++
.../src/main/resources/nifi.properties | 165 ++
.../pom.xml | 41 +-
.../nifi-kafka-to-s3-connector}/pom.xml | 31 +-
.../connectors/kafkas3/KafkaConnectionStep.java | 135 ++
.../apache/nifi/connectors/kafkas3/KafkaToS3.java | 297 +++
.../connectors/kafkas3/KafkaToS3FlowBuilder.java | 213 ++
.../nifi/connectors/kafkas3/KafkaTopicsStep.java | 85 +
.../org/apache/nifi/connectors/kafkas3/S3Step.java | 156 ++
.../org.apache.nifi.components.connector.Connector | 3 +-
.../steps/Kafka_Connection.md | 27 +
.../steps/Kafka_Topics.md | 34 +
.../steps/S3_Configuration.md | 65 +
.../src/main/resources/flows/Kafka_to_S3.json | 2105 +++++++++++++++++
.../nifi-kafka-to-s3-integration-tests/pom.xml | 238 ++
.../nifi/connectors/kafkas3/KafkaToS3IT.java | 758 ++++++
.../nifi-kafka-to-s3-nar/pom.xml | 36 +
nifi-connectors/nifi-kafka-to-s3-bundle/pom.xml | 39 +
nifi-connectors/pom.xml | 42 +
.../src/main/asciidoc/administration-guide.adoc | 5 +
.../processors/aws/s3/AbstractS3Processor.java | 11 +
.../nifi/processors/aws/region/RegionUtil.java | 2 +-
.../apache/nifi/kafka/processors/ConsumeKafka.java | 118 +-
.../nifi/kafka/processors/ConsumeKafkaTest.java | 22 +-
.../kafka/service/api/consumer/PollingContext.java | 4 +-
.../kafka/service/Kafka3ConnectionService.java | 43 +-
.../service/consumer/Kafka3AssignmentService.java | 100 +
.../kafka/service/consumer/RecordIterable.java | 79 +
.../nifi/script/impl/ValidationContextAdapter.java | 5 +
.../processors/standard/DuplicateFlowFile.java | 2 +-
.../java/org/apache/nifi/asset/AssetManager.java | 30 +-
.../connector/ConnectorConfigurationProvider.java | 161 ++
.../ConnectorConfigurationProviderException.java | 37 +
...ConfigurationProviderInitializationContext.java | 50 +
.../connector/ConnectorRequestContext.java | 88 +
.../connector/ConnectorRequestContextHolder.java | 64 +
.../connector/ConnectorWorkingConfiguration.java | 55 +
.../connector/StandardConnectorRequestContext.java | 84 +
.../connector/secrets/AuthorizableSecret.java} | 12 +-
.../nifi/controller/queue/DropFlowFileStatus.java | 7 +
.../nifi/controller/queue/FlowFileQueue.java | 15 +
.../queue/LocalQueuePartitionDiagnostics.java | 10 +
.../controller/repository/FlowFileSwapManager.java | 10 +
.../controller/repository/RepositoryRecord.java | 8 +
.../repository/RepositoryRecordType.java | 2 +-
.../java/org/apache/nifi/web/UiExtensionType.java | 1 +
.../ConnectorRequestContextHolderTest.java | 69 +
.../TestVolatileProvenanceRepository.java | 6 +-
.../web/api/dto/ConfigVerificationResultDTO.java | 10 +
.../api/dto/ConfigurationStepConfigurationDTO.java | 96 +
.../api/dto/ConfigurationStepDependencyDTO.java | 70 +
.../nifi/web/api/dto/ConnectorActionDTO.java | 70 +
.../web/api/dto/ConnectorConfigurationDTO.java | 43 +
.../org/apache/nifi/web/api/dto/ConnectorDTO.java | 172 ++
.../api/dto/ConnectorPropertyDependencyDTO.java | 57 +
.../api/dto/ConnectorPropertyDescriptorDTO.java | 137 ++
.../web/api/dto/ConnectorValueReferenceDTO.java | 120 +
.../web/api/dto/PropertyGroupConfigurationDTO.java | 82 +
.../org/apache/nifi/web/api/dto/SecretDTO.java | 89 +
.../dto/VerifyConnectorConfigStepRequestDTO.java | 69 +
.../web/api/dto/provenance/ProvenanceEventDTO.java | 14 +
.../web/api/dto/status/ConnectorStatusDTO.java | 186 ++
.../api/dto/status/ConnectorStatusSnapshotDTO.java | 322 +++
.../dto/status/NodeConnectorStatusSnapshotDTO.java | 79 +
.../web/api/entity/ConfigurationStepEntity.java | 77 +
.../api/entity/ConfigurationStepNamesEntity.java | 44 +
.../nifi/web/api/entity/ConnectorEntity.java | 65 +
.../ConnectorPropertyAllowableValuesEntity.java | 71 +
.../web/api/entity/ConnectorRunStatusEntity.java | 29 +-
.../nifi/web/api/entity/ConnectorTypesEntity.java} | 34 +-
.../nifi/web/api/entity/ConnectorsEntity.java | 55 +
.../nifi/web/api/entity/CurrentUserEntity.java | 13 +
.../nifi/web/api/entity/NarDetailsEntity.java | 10 +
.../apache/nifi/web/api/entity/SecretsEntity.java} | 39 +-
.../web/api/entity/StepDocumentationEntity.java | 25 +-
.../VerifyConnectorConfigStepRequestEntity.java | 30 +-
.../ReconstitutedSerializedRepositoryRecord.java | 13 +
.../repository/RepositoryRecordSerde.java | 13 +-
.../repository/SchemaRepositoryRecordSerde.java | 34 +-
.../repository/SerializedRepositoryRecord.java | 8 +
.../StandardRepositoryRecordSerdeFactory.java | 13 +-
.../schema/RepositoryRecordFieldMap.java | 17 +-
.../repository/schema/RepositoryRecordSchema.java | 65 +-
.../repository/schema/RepositoryRecordUpdate.java | 15 +-
.../authorization/resource/ResourceFactory.java | 27 +
.../nifi/authorization/resource/ResourceType.java | 3 +-
.../http/StandardHttpResponseMapper.java | 14 +
.../http/endpoints/AssetsEndpointMerger.java | 2 +-
.../endpoints/ComponentStateEndpointMerger.java | 6 +-
.../http/endpoints/ConnectorEndpointMerger.java | 60 +
.../endpoints/ConnectorFlowEndpointMerger.java | 56 +
.../ConnectorPropertyGroupEndpointMerger.java | 51 +
.../ConnectorPropertyGroupNamesEndpointMerger.java | 63 +
.../endpoints/ConnectorStatusEndpointMerger.java | 86 +
.../http/endpoints/ConnectorsEndpointMerger.java | 66 +
.../http/endpoints/CurrentUserEndpointMerger.java | 1 +
.../coordination/http/endpoints/FlowMerger.java | 11 +
.../VerifyConnectorConfigStepEndpointMerger.java | 72 +
.../ClusteredConnectorRequestReplicator.java | 131 ++
.../replication/ThreadPoolRequestReplicator.java | 2 +
.../manager/ConfigurationStepEntityMerger.java | 160 ++
.../ConfigurationStepNamesEntityMerger.java | 62 +
.../cluster/manager/ConnectorEntityMerger.java | 212 ++
.../ConnectorPropertyDescriptorDtoMerger.java | 64 +
.../cluster/manager/ConnectorsEntityMerger.java} | 38 +-
.../apache/nifi/cluster/manager/StatusMerger.java | 95 +
.../ConnectorRequestReplicatorConfiguration.java | 50 +
.../FrameworkClusterConfiguration.java | 9 +-
.../endpoints/CurrentUserEndpointMergerTest.java | 4 +
.../TestThreadPoolRequestReplicator.java | 6 +-
.../manager/ConfigurationStepEntityMergerTest.java | 184 ++
.../cluster/manager/ConnectorEntityMergerTest.java | 404 ++++
.../ConnectorPropertyDescriptorDtoMergerTest.java | 171 ++
.../java/org/apache/nifi/asset/StandardAsset.java | 13 +-
.../apache/nifi/asset/StandardAssetManager.java | 103 +-
.../nifi/asset/StandardConnectorAssetManager.java | 225 ++
.../StandardConnectorValidationTrigger.java | 92 +
.../secrets/ParameterProviderSecretProvider.java | 89 +
.../secrets/ParameterProviderSecretsManager.java | 180 ++
.../connector/secrets/StandardSecret.java | 186 ++
...tandardSecretsManagerInitializationContext.java | 25 +-
.../nifi/controller/StandardProcessorNode.java | 246 +-
.../nifi/controller/flow/AbstractFlowManager.java | 55 +-
.../controller/flowanalysis/FlowAnalysisUtil.java | 6 +-
.../parameter/StandardParameterProviderNode.java | 24 +-
.../repository/StandardProcessSession.java | 32 +-
...StandardControllerServiceInvocationHandler.java | 40 +-
.../service/StandardControllerServiceNode.java | 141 +-
.../service/StandardControllerServiceProvider.java | 5 +-
.../service/StandardControllerServiceResolver.java | 6 +-
.../nifi/events/VolatileBulletinRepository.java | 1 +
.../StandardVersionedComponentSynchronizer.java | 50 +-
.../apache/nifi/groups/StandardProcessGroup.java | 96 +-
...rLogObserver.java => ConnectorLogObserver.java} | 21 +-
.../apache/nifi/logging/ProcessorLogObserver.java | 2 +-
.../nifi/parameter/StandardParameterContext.java | 28 +-
.../StandardParameterReferenceManager.java | 17 +-
.../nifi/processor/StandardValidationContext.java | 38 +-
.../StandardValidationContextFactory.java | 6 +-
.../flow/FlowAnalyzingRegistryClientNode.java | 16 +-
.../flow/StandardFlowRegistryClientNode.java | 3 +-
...pper.java => VersionedComponentFlowMapper.java} | 95 +-
.../nifi/remote/StandardRemoteProcessGroup.java | 11 +-
.../services/org.apache.nifi.asset.AssetManager | 3 +-
...fi.components.connector.secrets.SecretsManager} | 2 +-
.../asset/StandardConnectorAssetManagerTest.java | 178 ++
.../TestParameterProviderSecretsManager.java | 374 +++
.../repository/StandardProcessSessionTest.java | 7 +
.../StandardControllerServiceResolverTest.java | 6 +-
.../nifi/groups/StandardProcessGroupTest.java | 3 +-
.../flow/FlowAnalyzingRegistryClientNodeTest.java | 6 +-
....java => TestVersionedComponentFlowMapper.java} | 10 +-
.../connector/ConfigurationUpdateResult.java | 11 +-
.../nifi/components/connector/ConnectorAction.java | 40 +-
.../connector/ConnectorConfiguration.java | 65 +
.../nifi/components/connector/ConnectorNode.java | 281 +++
.../components/connector/ConnectorRepository.java | 181 ++
.../ConnectorRepositoryInitializationContext.java | 50 +
.../connector/ConnectorRequestReplicator.java | 18 +-
.../nifi/components/connector/ConnectorState.java | 18 +-
.../connector/ConnectorStateTransition.java | 82 +
.../connector/ConnectorUpdateContext.java | 13 +-
.../connector/ConnectorValidationTrigger.java | 44 +
.../components/connector/FlowContextFactory.java | 21 +-
.../FrameworkConnectorInitializationContext.java | 19 +-
...eworkConnectorInitializationContextBuilder.java | 39 +
.../components/connector/FrameworkFlowContext.java | 24 +-
.../MutableConnectorConfigurationContext.java | 64 +
.../connector/NamedStepConfiguration.java | 8 +-
.../connector/ParameterContextFacadeFactory.java | 21 +-
.../connector/ProcessGroupFacadeFactory.java | 22 +-
.../components/connector/ProcessGroupFactory.java | 11 +-
.../connector/secrets/SecretProvider.java | 20 +-
.../connector/secrets/SecretsManager.java | 26 +-
.../SecretsManagerInitializationContext.java | 13 +-
.../DisabledServiceValidationResult.java | 14 +
.../components/validation/ValidationState.java | 16 +-
.../org/apache/nifi/connectable/Connectable.java | 2 +
.../connectable/ConnectableFlowFileActivity.java | 70 +
.../apache/nifi/connectable/FlowFileActivity.java | 20 +-
.../nifi/connectable/FlowFileTransferCounts.java | 88 +
.../connectable/ProcessGroupFlowFileActivity.java | 170 ++
.../nifi/controller/AbstractComponentNode.java | 69 +-
.../org/apache/nifi/controller/AbstractPort.java | 10 +
.../org/apache/nifi/controller/ComponentNode.java | 3 +
.../nifi/controller/ParameterProviderNode.java | 13 +
.../apache/nifi/controller/ProcessScheduler.java | 22 +-
.../org/apache/nifi/controller/ProcessorNode.java | 17 +-
.../org/apache/nifi/controller/StandardFunnel.java | 9 +
.../nifi/controller/ValidationContextFactory.java | 4 +-
.../apache/nifi/controller/flow/FlowManager.java | 60 +-
.../nifi/controller/flow/VersionedDataflow.java | 10 +
.../nifi/controller/queue/DropFlowFileRequest.java | 15 +
.../controller/service/ControllerServiceNode.java | 24 +-
.../service/ControllerServiceProvider.java | 4 +-
.../org/apache/nifi/events/BulletinFactory.java | 6 +
.../java/org/apache/nifi/groups/ProcessGroup.java | 31 +-
.../org/apache/nifi/groups/RemoteProcessGroup.java | 4 +-
.../nifi/manifest/RuntimeManifestService.java | 16 +-
.../ProcessGroupFlowFileActivityTest.java | 162 ++
.../nifi/controller/TestAbstractComponentNode.java | 2 +-
.../nifi-framework/nifi-framework-core/pom.xml | 10 +-
.../org/apache/nifi/asset/AssetsRestApiClient.java | 55 +-
.../nifi/asset/StandardAssetComponentManager.java | 4 +-
.../nifi/asset/StandardAssetSynchronizer.java | 4 +-
...ava => StandardConnectorAssetSynchronizer.java} | 184 +-
.../components/connector/ConnectorDetails.java} | 34 +-
.../connector/ConnectorParameterLookup.java | 193 ++
.../ConnectorValidationContextBridge.java | 146 ++
.../connector/DescribedValueProvider.java | 18 +-
.../nifi/components/connector/GhostConnector.java | 138 ++
.../StandaloneConnectorRequestReplicator.java | 29 +-
.../connector/StandardComponentBundleLookup.java | 172 ++
.../connector/StandardConnectorAction.java | 81 +
.../StandardConnectorConfigurationContext.java | 323 +++
...ConfigurationProviderInitializationContext.java | 44 +
.../StandardConnectorInitializationContext.java | 205 ++
.../connector/StandardConnectorNode.java | 1592 +++++++++++++
.../connector/StandardConnectorPropertyValue.java | 101 +
...StandardConnectorRepoInitializationContext.java | 85 +
.../connector/StandardConnectorRepository.java | 594 +++++
.../StandardConnectorStateTransition.java | 113 +
.../StandardConnectorValidationContext.java | 107 +
.../components/connector/StandardFlowContext.java | 130 ++
.../StandardStepConfigurationContext.java | 59 +
.../standalone/ComponentContextProvider.java | 41 +
...egalExecutionEngineStatelessGroupLifecycle.java | 46 +
.../standalone/StandaloneConnectionFacade.java | 82 +
.../StandaloneControllerServiceFacade.java | 243 ++
.../StandaloneControllerServiceLifecycle.java | 77 +
.../StandaloneParameterContextFacade.java | 229 ++
.../standalone/StandaloneProcessGroupFacade.java | 424 ++++
.../StandaloneProcessGroupLifecycle.java | 463 ++++
.../standalone/StandaloneProcessorFacade.java | 253 ++
.../standalone/StandaloneProcessorLifecycle.java | 93 +
.../StandaloneStatelessGroupLifecycle.java | 54 +
.../apache/nifi/controller/ExtensionBuilder.java | 409 ++--
.../nifi/controller/FileSystemSwapManager.java | 9 +
.../org/apache/nifi/controller/FlowController.java | 226 +-
.../nifi/controller/StandardFlowService.java | 30 +-
.../flow/FlowControllerFlowContextFactory.java | 140 ++
.../FlowControllerProcessGroupFacadeFactory.java | 70 +
.../flow/StandardComponentContextProvider.java | 105 +
.../nifi/controller/flow/StandardFlowManager.java | 147 +-
.../flow/StandardStatelessGroupNodeFactory.java | 23 +-
.../flowanalysis/StandardFlowAnalysisContext.java | 4 +-
.../flowanalysis/StandardFlowAnalysisRuleNode.java | 3 +-
.../controller/queue/AbstractFlowFileQueue.java | 34 +-
.../nifi/controller/queue/SelectiveDropResult.java | 75 +
.../controller/queue/StandardFlowFileQueue.java | 57 +
.../StandardLocalQueuePartitionDiagnostics.java | 17 +-
.../controller/queue/SwappablePriorityQueue.java | 254 +-
.../clustered/SocketLoadBalancedFlowFileQueue.java | 66 +
.../queue/clustered/partition/QueuePartition.java | 13 +
.../clustered/partition/RemoteQueuePartition.java | 9 +
.../partition/StandardRebalancingPartition.java | 9 +
.../SwappablePriorityQueueLocalPartition.java | 8 +
.../reporting/StandardReportingTaskNode.java | 3 +-
.../repository/LiveSerializedRepositoryRecord.java | 5 +
.../repository/WriteAheadFlowFileRepository.java | 33 +-
.../scheduling/StandardProcessScheduler.java | 91 +-
.../serialization/VersionedDataflowMapper.java | 20 +-
.../serialization/VersionedFlowSynchronizer.java | 170 +-
.../VersionedReportingTaskSnapshotMapper.java | 6 +-
.../nifi/controller/tasks/StatelessFlowTask.java | 69 +
.../bootstrap/BootstrapDiagnosticsFactory.java | 2 +
.../bootstrap/tasks/ConnectionDiagnosticTask.java | 265 +++
.../nifi/flowanalysis/StandardFlowAnalyzer.java | 10 +-
.../configuration/FlowControllerConfiguration.java | 52 +-
.../nifi/groups/StandardStatelessGroupNode.java | 10 +
.../manifest/StandardRuntimeManifestService.java | 45 +
.../java/org/apache/nifi/nar/NarInstallTask.java | 4 +-
....nifi.components.connector.ConnectorRepository} | 2 +-
.../components/connector/BlockingConnector.java | 132 ++
.../connector/DynamicAllowableValuesConnector.java | 142 ++
.../components/connector/DynamicFlowConnector.java | 336 +++
.../connector/MissingBundleConnector.java | 111 +
.../connector/OnPropertyModifiedConnector.java | 88 +
.../components/connector/ParameterConnector.java | 131 ++
.../components/connector/SleepingConnector.java | 126 +
.../connector/StandardConnectorNodeIT.java | 675 ++++++
.../TestStandardComponentBundleLookup.java | 246 ++
.../TestStandardConnectorConfigurationContext.java | 256 +++
...TestStandardConnectorInitializationContext.java | 387 ++++
.../connector/TestStandardConnectorNode.java | 848 +++++++
.../connector/TestStandardConnectorRepository.java | 877 +++++++
.../TestStandaloneProcessGroupLifecycle.java | 292 +++
.../connector/processors/CreateDummyFlowFile.java | 64 +-
.../connector/processors/DuplicateFlowFile.java | 128 ++
.../connector/processors/ExposeFileValues.java | 72 +
.../connector/processors/LogFlowFileContents.java | 29 +-
.../processors/OnPropertyModifiedTracker.java | 116 +
.../connector/processors/OverwriteFlowFile.java | 60 +-
.../components/connector/processors/Sleep.java | 151 ++
.../connector/processors}/TerminateFlowFile.java | 12 +-
.../connector/services/CounterService.java | 13 +-
.../services/impl/StandardCounterService.java | 27 +-
.../nifi/controller/MockStateManagerProvider.java | 72 +
.../apache/nifi/controller/MockSwapManager.java | 5 +
.../nifi/controller/TestStandardFlowFileQueue.java | 118 +-
.../apache/nifi/controller/flow/NopConnector.java | 167 ++
.../controller/flow/TestStandardFlowManager.java | 174 ++
.../TestSocketLoadBalancedFlowFileQueue.java | 117 +-
.../clustered/TestSwappablePriorityQueue.java | 138 +-
.../SchemaRepositoryRecordSerdeTest.java | 10 +-
.../repository/StandardProcessSessionIT.java | 5 +-
.../TestWriteAheadFlowFileRepository.java | 137 +-
.../scheduling/TestStandardProcessScheduler.java | 81 +
.../VersionedFlowSynchronizerTest.java | 6 +
.../TestStandardControllerServiceProvider.java | 42 +-
.../controller/service/mock/MockProcessGroup.java | 26 +-
.../controller/tasks/TestStatelessFlowTask.java | 140 +-
.../StandardRuntimeManifestServiceTest.java | 48 +
....java => VersionedComponentFlowMapperTest.java} | 4 +-
...org.apache.nifi.components.connector.Connector} | 4 +-
.../org.apache.nifi.controller.ControllerService | 5 +-
.../services/org.apache.nifi.processor.Processor | 11 +-
.../org.example.TestConnector/Another_Test_Step.md | 4 +
.../steps/org.example.TestConnector/Test_Step.md | 8 +
.../src/test/resources/colors.txt | 3 +
.../src/test/resources/flows/choose-color.json | 76 +
.../flows/generate-and-log-with-parameter.json | 253 ++
.../flows/generate-duplicate-log-flow.json | 690 ++++++
.../flows/on-property-modified-tracker.json | 98 +
.../nar/StandardExtensionDiscoveringManager.java | 20 +-
.../java/org/apache/nifi/nar/DummyConnector.java | 120 +
...rdExtensionDiscoveringManagerConnectorTest.java | 49 +
...org.apache.nifi.components.connector.Connector} | 4 +-
.../apache/nifi/headless/HeadlessNiFiServer.java | 17 +-
.../main/java/org/apache/nifi/nar/NarUnpacker.java | 4 +-
.../repository/StandardRepositoryRecord.java | 24 +-
.../nifi-framework/nifi-resources/pom.xml | 2 +
.../src/main/resources/conf/nifi.properties | 5 +
.../org/apache/nifi/web/server/JettyServer.java | 117 +-
.../web/server/StandardServerProviderTest.java | 8 +-
.../org/apache/nifi/ui/extension/UiExtension.java | 16 +
.../org/apache/nifi/audit/ConnectorAuditor.java | 416 ++++
.../nifi/authorization/AuthorizableLookup.java | 31 +
.../authorization/StandardAuthorizableLookup.java | 46 +-
.../org/apache/nifi/web/NiFiServiceFacade.java | 187 ++
.../apache/nifi/web/NiFiWebApiResourceConfig.java | 1 +
.../apache/nifi/web/StandardNiFiServiceFacade.java | 589 ++++-
.../org/apache/nifi/web/api/ConnectorResource.java | 2419 ++++++++++++++++++++
.../apache/nifi/web/api/FlowFileQueueResource.java | 12 +-
.../java/org/apache/nifi/web/api/FlowResource.java | 210 +-
.../nifi/web/api/ParameterContextResource.java | 4 +-
.../apache/nifi/web/api/ProcessGroupResource.java | 2 +-
.../org/apache/nifi/web/api/dto/DtoFactory.java | 384 +++-
.../org/apache/nifi/web/api/dto/EntityFactory.java | 89 +
.../configuration/WebApplicationConfiguration.java | 29 +
.../connector/StandardNiFiConnectorWebContext.java | 99 +
.../authorization/AuthorizingConnectionFacade.java | 89 +
.../AuthorizingConnectorInvocationHandler.java | 130 ++
.../AuthorizingControllerServiceFacade.java | 122 +
.../AuthorizingControllerServiceLifecycle.java | 63 +
.../authorization/AuthorizingFlowContext.java | 70 +
.../AuthorizingParameterContextFacade.java | 95 +
.../AuthorizingProcessGroupFacade.java | 178 ++
.../AuthorizingProcessGroupLifecycle.java | 129 ++
.../authorization/AuthorizingProcessorFacade.java | 122 +
.../AuthorizingProcessorLifecycle.java | 80 +
.../AuthorizingStatelessGroupLifecycle.java | 55 +
.../ConnectorAuthorizationContext.java | 74 +
.../nifi/web/controller/ControllerFacade.java | 115 +-
.../org/apache/nifi/web/dao/ConnectionDAO.java | 9 +
.../java/org/apache/nifi/web/dao/ConnectorDAO.java | 84 +
.../org/apache/nifi/web/dao/ProcessGroupDAO.java | 9 +
.../org/apache/nifi/web/dao/impl/ComponentDAO.java | 31 +-
.../nifi/web/dao/impl/StandardConnectionDAO.java | 52 +-
.../nifi/web/dao/impl/StandardConnectorDAO.java | 284 +++
.../web/dao/impl/StandardControllerServiceDAO.java | 2 +-
.../web/dao/impl/StandardParameterContextDAO.java | 2 +-
.../nifi/web/dao/impl/StandardProcessGroupDAO.java | 19 +-
.../nifi/web/dao/impl/StandardProcessorDAO.java | 3 +-
.../web/filter/ConnectorRequestContextFilter.java | 89 +
.../nifi-web-api/src/main/webapp/WEB-INF/web.xml | 8 +
.../org/apache/nifi/audit/TestLabelAuditor.java | 2 +-
.../apache/nifi/audit/TestProcessGroupAuditor.java | 6 +-
.../apache/nifi/audit/TestProcessorAuditor.java | 2 +-
.../StandardAuthorizableLookupTest.java | 109 +
.../nifi/web/StandardNiFiServiceFacadeTest.java | 372 ++-
.../apache/nifi/web/api/TestConnectorResource.java | 748 ++++++
.../nifi/web/api/TestDataTransferResource.java | 18 +-
.../org/apache/nifi/web/api/TestFlowResource.java | 46 +-
.../StandardNiFiConnectorWebContextTest.java | 219 ++
.../AuthorizingConnectorInvocationHandlerTest.java | 239 ++
.../authorization/AuthorizingFlowContextTest.java | 167 ++
.../AuthorizingParameterContextFacadeTest.java | 162 ++
.../nifi/web/controller/ControllerFacadeTest.java | 306 +++
.../web/dao/impl/StandardConnectionDAOTest.java | 196 ++
.../web/dao/impl/StandardConnectorDAOTest.java | 311 +++
.../web/dao/impl/StandardProcessGroupDAOTest.java | 130 ++
.../web/dao/impl/StandardProcessorDAOTest.java | 2 +-
.../dao/impl/TestStandardParameterContextDAO.java | 2 +-
.../web/dao/impl/TestStandardProcessGroupDAO.java | 15 +-
.../filter/ConnectorRequestContextFilterTest.java | 178 ++
.../StandardClientRegistrationProviderTest.java | 2 +-
.../StandardRegistrationBuilderProviderTest.java | 4 +-
...dardRelyingPartyRegistrationRepositoryTest.java | 4 +-
.../x509/X509AuthenticationProviderTest.java | 2 +-
.../component-access-policies.component.ts | 6 +-
.../global-access-policies.component.ts | 1 +
.../feature/documentation-routing.module.ts | 5 +
.../feature/documentation.component.html | 13 +
.../feature/documentation.component.ts | 8 +
.../documentation/feature/documentation.module.ts | 2 +
.../documentation/service/documentation.service.ts | 17 +
.../connector-definition.actions.ts | 56 +
.../connector-definition.effects.ts | 79 +
.../connector-definition.reducer.ts | 92 +
.../connector-definition.selectors.ts | 31 +-
.../state/connector-definition/index.ts | 73 +
.../src/app/pages/documentation/state/index.ts | 4 +
.../connector-property-definition.component.html | 95 +
.../connector-property-definition.component.scss | 11 +-
...connector-property-definition.component.spec.ts | 176 ++
.../connector-property-definition.component.ts | 88 +
.../connector-definition.component.html | 209 ++
.../connector-definition.component.scss | 11 +-
.../connector-definition.component.spec.ts | 415 ++++
.../connector-definition.component.ts | 204 ++
.../flow-designer/feature/flow-designer.module.ts | 4 +-
.../service/canvas-context-menu.service.ts | 3 +-
.../pages/flow-designer/state/flow/flow.actions.ts | 3 +-
.../flow-designer/state/flow/flow.effects.spec.ts | 104 +-
.../pages/flow-designer/state/flow/flow.effects.ts | 10 +-
.../state/provenance-event-listing/index.ts | 1 +
.../provenance-event-listing.effects.ts | 18 +-
.../provenance-event-table.component.ts | 3 +-
.../flow-analysis-rule-table.component.spec.ts | 4 +
...anagement-controller-services.component.spec.ts | 4 +
.../parameter-providers-table.component.spec.ts | 4 +
.../reporting-task-table.component.spec.ts | 4 +
.../user-table/user-table.component.spec.ts | 4 +
.../src/app/service/extension-types.service.ts | 4 +
.../app/state/current-user/current-user.reducer.ts | 1 +
.../apps/nifi/src/app/state/current-user/index.ts | 1 +
.../extension-types/extension-types.effects.ts | 9 +-
.../extension-types/extension-types.reducer.ts | 3 +
.../extension-types/extension-types.selectors.ts | 8 +-
.../nifi/src/app/state/extension-types/index.ts | 2 +
.../shared/src/services/nifi-common.service.ts | 5 +
.../main/frontend/libs/shared/src/types/index.ts | 1 +
.../nifi/extension/manifest/ConfigurationStep.java | 80 +
.../manifest/ConfigurationStepDependency.java | 67 +
.../nifi/extension/manifest/ConnectorProperty.java | 120 +
.../manifest/ConnectorPropertyDependency.java | 57 +
.../extension/manifest/ConnectorPropertyGroup.java | 67 +
...tensionType.java => ConnectorPropertyType.java} | 23 +-
.../apache/nifi/extension/manifest/Extension.java | 14 +
.../nifi/extension/manifest/ExtensionType.java | 4 +-
.../runtime/manifest/ComponentManifestBuilder.java | 7 +
.../impl/StandardComponentManifestBuilder.java | 12 +
.../impl/StandardRuntimeManifestBuilder.java | 126 +
...tandardRuntimeManifestBuilderConnectorTest.java | 349 +++
.../apache/nifi/util/MockValidationContext.java | 8 +
.../java/org/apache/nifi/nar/ExtensionMapping.java | 25 +-
.../nifi/stateless/flow/StatelessDataflow.java | 9 +
.../stateless/bootstrap/AllowListClassLoader.java | 5 +
.../stateless/bootstrap/StatelessBootstrap.java | 2 +-
.../reporting/StatelessReportingTaskNode.java | 3 +-
.../scheduling/StatelessProcessScheduler.java | 25 +-
.../stateless/engine/StatelessFlowManager.java | 21 +-
.../nifi/stateless/flow/StandardStatelessFlow.java | 5 +
.../stateless/queue/StatelessFlowFileQueue.java | 8 +
.../nifi-system-test-extensions/pom.xml | 5 +
.../connectors/tests/system/AssetConnector.java | 93 +
.../tests/system/BundleResolutionConnector.java | 163 ++
.../tests/system/CalculateConnector.java | 226 ++
.../tests/system/ComponentLifecycleConnector.java | 177 ++
.../tests/system/DataQueuingConnector.java | 79 +
.../tests/system/GatedDataQueuingConnector.java | 122 +
.../tests/system/NestedProcessGroupConnector.java | 75 +
.../nifi/connectors/tests/system/NopConnector.java | 133 ++
.../tests/system/ParameterContextConnector.java | 286 +++
.../tests/system/SelectiveDropConnector.java | 150 ++
.../nifi/processors/tests/system/Calculate.java | 71 +
.../processors/tests/system/TerminateFlowFile.java | 28 +
.../processors/tests/system/UpdateContent.java | 41 +-
.../org.apache.nifi.components.connector.Connector | 11 +-
.../services/org.apache.nifi.processor.Processor | 1 +
.../apache/nifi/tests/system/NiFiClientUtil.java | 315 +++
.../org/apache/nifi/tests/system/NiFiSystemIT.java | 26 +
.../SpawnedStandaloneNiFiInstanceFactory.java | 47 +
.../tests/system/TroubleshootingTestWatcher.java | 105 +-
.../system/clustering/FlowSynchronizationIT.java | 3 +
.../connectors/ClusteredConnectorAssetsIT.java | 158 ++
.../connectors/ClusteredConnectorDrainIT.java | 228 ++
.../system/connectors/ClusteredConnectorIT.java | 220 ++
.../tests/system/connectors/ConnectorAssetsIT.java | 207 ++
.../tests/system/connectors/ConnectorCrudIT.java | 280 +++
.../tests/system/connectors/ConnectorDrainIT.java | 178 ++
.../system/connectors/ConnectorLifecycleIT.java | 235 ++
.../connectors/ConnectorMethodMarshallingIT.java | 71 +
.../connectors/ConnectorParameterContextIT.java | 119 +
.../connectors/ConnectorVersionResolutionIT.java | 97 +
.../connectors/SelectiveDropConnectorIT.java | 188 ++
.../flows/connector-version-mismatch/flow.json | 52 +
.../controller-service-enabling-failure/flow.json | 2 +-
.../resources/flows/migrate-properties/flow.json | 56 +-
.../toolkit/cli/impl/client/NiFiClientFactory.java | 11 +
.../nifi/toolkit/client/ConnectorClient.java | 419 ++++
.../org/apache/nifi/toolkit/client/FlowClient.java | 8 +
.../org/apache/nifi/toolkit/client/NiFiClient.java | 8 +-
.../toolkit/client/impl/JerseyConnectorClient.java | 673 ++++++
.../nifi/toolkit/client/impl/JerseyFlowClient.java | 9 +
.../nifi/toolkit/client/impl/JerseyNiFiClient.java | 13 +-
pom.xml | 2 +
588 files changed, 54629 insertions(+), 1829 deletions(-)
diff --git a/.cursor/rules/building.mdc b/.cursor/rules/building.mdc
new file mode 100644
index 00000000000..af4829a4262
--- /dev/null
+++ b/.cursor/rules/building.mdc
@@ -0,0 +1,18 @@
+---
+description: Maven build instructions for the NiFi codebase
+alwaysApply: true
+---
+
+# Building
+
+NiFi is a complex Maven codebase. Never build code (testing or otherwise) using javac.
+Always use `mvn` instead, or preferably the `.mvnw` wrapper script.
+
+Additionally, building a maven module using the also-make flag (`-am`) is often very
+expensive and slow. Instead, only build the specific module you are modifying. Assume that
+the user has already built the entire codebase and that only the specific module you are
+modifying needs to be built again. If this fails, you can prompt the user to build the entire
+codebase, but only after you have attempted to build the relevant modules yourself first.
+It is important not to run `mvn clean` at the root level or at the `nifi-assembly` level without
+the user's express permission, as this may delete a running instance of NiFi, causing permanent
+loss of flows and configuration.
diff --git a/.cursor/rules/code-style.mdc b/.cursor/rules/code-style.mdc
new file mode 100644
index 00000000000..ca7e9cfed5a
--- /dev/null
+++ b/.cursor/rules/code-style.mdc
@@ -0,0 +1,74 @@
+---
+description: Java code style conventions for the NiFi codebase
+globs: "**/*.java"
+alwaysApply: false
+---
+
+# Code Style
+
+NiFi adheres to a few code styles that are not necessarily common. Please ensure that you
+observe these code styles.
+
+1. Any variable that can be marked `final` must be marked `final`. This includes
+ declarations of Exceptions, method arguments, local variables, member variables, etc.
+2. Short-hand is highly discouraged in names of variables, classes, methods, etc., as well
+ as in documentation. Exceptions to this include in the framework, you may see references to
+ `procNode` for `ProcessorNode` or other such short-hand that is very difficult to confuse with
+ other terms, and it is used only when clearly defined such as `final ProcessorNode procNode = ...`.
+ Even though, however, we would not abbreviate `ControllerService` as `cs` because `cs` is too vague
+ and easily misunderstood. Instead, a value of `serviceNode` might be used.
+3. Private / helper methods should not be placed before the first public/protected method
+ that calls it.
+4. Unless the method is to be heavily reused, avoid creating trivial 1-2 line methods and
+ instead just place the code inline.
+5. Code is allowed to be up to 200 characters wide. Avoid breaking lines into many short lines.
+6. Avoid creating private methods that are called only once unless they are at least 10
+ lines long or are complex.
+7. It is never acceptable to use star imports. Import each individual class that is to be used.
+8. Never use underscores in class names, variables, or filenames.
+9. Never use System.out.println but instead use SLF4J Loggers.
+10. Avoid excessive whitespace in method invocations. For example, instead of writing:
+
+```java
+myObject.doSomething(
+ arg1,
+ arg2,
+ arg3,
+ arg4,
+ arg5
+);
+```
+
+Write this instead:
+
+```java
+myObject.doSomething(arg1, arg2, arg3, arg4, arg5);
+```
+
+It is okay to use many newlines in a builder pattern, such as:
+```java
+final MyObject myObject = MyObject.builder()
+ .arg1(arg1)
+ .arg2(arg2)
+ .arg3(arg3)
+ .build();
+```
+
+It is also acceptable when chaining methods in a functional style such as:
+```java
+final List<String> result = myList.stream()
+ .filter(s -> s.startsWith("A"))
+ .map(String::toUpperCase)
+ .toList();
+```
+
+11. When possible, prefer importing a class, rather than using fully qualified classname
+ inline in the code.
+12. Avoid statically importing methods, except in methods that are frequently used in testing
+ frameworks, such as the `Assertions` and `Mockito` classes.
+13. Avoid trailing whitespace at the end of lines, especially in blank lines.
+14. The `var` keyword is never allowed in the codebase. Always explicitly declare the type of variables.
+15. Prefer procedural code over functional code. For example, prefer using a for loop instead of a stream
+ when the logic is not simple and straightforward. The stream API is powerful but can be difficult to
+ read when overused or used in complex scenarios. Functional style is best used when the logic is simple
+ and chains together no more than 3-4 operations.
diff --git a/.cursor/rules/ending-conditions.mdc b/.cursor/rules/ending-conditions.mdc
new file mode 100644
index 00000000000..aa2b607801a
--- /dev/null
+++ b/.cursor/rules/ending-conditions.mdc
@@ -0,0 +1,30 @@
+---
+description: Task completion checklist that must be verified before considering any task done
+alwaysApply: true
+---
+
+# Ending Conditions
+
+When you have completed a task, ensure that you have verified the following:
+
+1. All code compiles and builds successfully using `mvn`.
+2. All relevant unit tests pass successfully using `mvn`.
+3. All code adheres to the Code Style rules.
+4. Checkstyle and PMD pass successfully using
+ `mvn checkstyle:check pmd:check -T 1C` from the appropriate directory.
+5. Unit tests have been added to verify the functionality of any sufficiently complex method.
+6. A system test or an integration test has been added if the change makes significant
+ changes to the framework and the interaction between a significant number of classes.
+7. You have performed a full review of the code to ensure that there are no logical errors
+ and that the code is not duplicative or difficult to understand. If you find any code that
+ is in need of refactoring due to clarity or duplication, you should report this to the user
+ and offer to make those changes as well.
+8. If creating a new Processor or Controller Service, ensure that all relevant annotations
+ have been added, including `@Tags`, `@CapabilityDescription`, `@UseCase`, and
+ `@MultiProcessorUseCase` as appropriate.
+
+
+Do not consider the task complete until all of the above conditions have been met. When you
+do consider the task complete, provide a summary of what you changed and which tests were
+added or modified and what the behavior is that they verify. Additionally, provide any feedback
+about your work that may need further review or that is not entirely complete.
diff --git a/.cursor/rules/extension-development.mdc b/.cursor/rules/extension-development.mdc
new file mode 100644
index 00000000000..74e53f34d5f
--- /dev/null
+++ b/.cursor/rules/extension-development.mdc
@@ -0,0 +1,107 @@
+---
+description: Development patterns for NiFi extensions (Processors, Controller Services, Connectors). Covers Property Descriptors, Relationships, and common patterns.
+alwaysApply: false
+---
+
+# Extension Development
+
+This rule applies when developing NiFi extensions: Processors, Controller Services, and Connectors.
+
+## Property Descriptors
+
+Property Descriptors are defined as `static final` fields on the component class using
+`PropertyDescriptor.Builder`.
+
+- **Naming:** Use clear, descriptive names. The `displayName` field should never be used. Make the
+ name itself clear and concise. Use Title Case for property names.
+- **Required vs. optional:** Mark properties as `.required(true)` when the component cannot
+ function without them. Prefer sensible defaults via `.defaultValue(...)` when possible.
+ When a default value is provided, the property will always have a value. The `required` flag in this
+ case is more of a documentation aid to indicate the importance of the property.
+- **Validators:** Always attach an appropriate `Validator` (e.g., `StandardValidators.NON_EMPTY_VALIDATOR`,
+ `StandardValidators.POSITIVE_INTEGER_VALIDATOR`). The Validator can be left off only when Allowable Values
+ are provided. In this case, do not include a Validator because it is redundant and confusing.
+- **Expression Language:** If a property should support Expression Language, add
+ `.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)` or the
+ appropriate scope. Always document when Expression Language is supported in the property
+ description. Some developers tend to go overboard here and feel like Expression Language should be supported
+ everywhere, but this is a mistake! The default assumption should be that Expression Language is not supported
+ unless the value is expected to be different for every FlowFile that is processed.
+- **Dependencies:** Use `.dependsOn(...)` to conditionally show properties based on the
+ values of other properties. This keeps the configuration UI clean and avoids exposing
+ irrelevant properties. If there is a dependency, it is important to understand that `.required(true)` means that
+ this property is required IF AND ONLY IF the dependency condition is met.
+
+## Processors
+
+- The `onTrigger` method should be focused on processing FlowFiles. Keep setup and teardown
+ logic in lifecycle methods when possible.
+- Prefer `session.read()` and `session.write()` with callbacks over directly working with
+ streams to ensure proper resource management.
+- Prefer `session.commitAsync()` over `session.commit()`. The `commit` method was the original implementation,
+ but it has now been deprecated in favor of `commitAsync`. The `commitAsync` call provide a clearer, cleaner
+ interface for handling post-commit actions including success and failure callbacks. In addition, the async
+ method allows Processors to be used much more efficiently in a Stateless NiFi flow.
+
+### Processor Lifecycle Annotations
+
+- Use `@OnScheduled` for setup that should happen once before the processor starts
+ running (e.g., creating clients, compiling patterns).
+- Use `@OnStopped` for cleanup (e.g., closing clients, releasing resources).
+- `@OnUnscheduled` is rarely used but can be used to interrupt long-running processes when the Processor is stopped.
+ Generally, though, it is preferable to write the Processor in such a way that long-running processes check `isScheduled()`
+ and stop gracefully if the return value is `false`.
+
+### Relationships
+- **Declaration**: Relationships are defined as `static final` fields using `new Relationship.Builder()`.
+ Relationship names should generally be lowercase.
+- **Success and Failure:** Most processors define at least a `success` and `failure`
+ relationship. Use `REL_SUCCESS` and `REL_FAILURE` as constant names.
+- **Original relationship:** Processors that enrich or fork FlowFiles often include an
+ `original` relationship for the unmodified input FlowFile.
+
+### Use Case Documentation
+The `@UseCase` and `@MultiProcessorUseCase` annotations help document common usage patterns for Processors.
+This is helpful for users to understand when and how to use the component effectively. It is equally important
+for Agents that can determine which components should be used for a given task.
+
+- Use `@UseCase` to document common use cases for the Processor. This helps users understand
+ when and how to use the component effectively. This is unnecessary for Processors that serve a single use case
+ that is clearly described by the component name and description. For example, a Processor that consumes messages
+ from a specific service likely does not need a `@UseCase` annotation because its purpose is clear.
+- Use `@MultiProcessorUseCase` to document well-known patterns that involve multiple Processors working
+ together to achieve a common goal. Examples include List/Fetch patterns, Fork/Join patterns, etc.
+ The `@MultiProcessorUseCase` annotation should not be added to each individual Processor involved in the pattern.
+ Rather, the convention is to add the annotation to the last Processor in the flow that completes the pattern.
+ Some Processors will have one or more `@UseCase` annotations and no `@MultiProcessorUseCase` annotations,
+ while some will have one or more `@MultiProcessorUseCase` annotations and no `@UseCase` annotations.
+
+
+## Controller Services
+
+Controller Services are objects that can be shared across multiple components. This is typically done for
+clients that connect to external systems in order to avoid creating many connections, or in order to share
+configuration across multiple components without the user having to duplicate configuration. Controller Services
+can also be helpful for abstracting away some piece of functionality into a separate extension point so that the
+implementation can be swapped out by the user. For example, Record Readers and Writers are implemented as Controller
+Services so that the user can simply choose which format they want to read and write in a flexible and reusable way.
+
+That said, Controller Services can be more onerous to configure and maintain for the user, so they should
+be used sparingly and only when there is a clear benefit to doing so.
+
+### Controller Service Lifecycle Annotations
+
+- Use `@OnScheduled` for setup that should happen once before the service is enabled (e.g., creating clients, compiling patterns).
+- Use `@OnDisabled` for cleanup (e.g., closing clients, releasing resources).
+
+
+## General Patterns
+
+- Use `ComponentLog` (obtained via `getLogger()`) for all logging, not SLF4J directly.
+ This ensures log messages are associated with the component instance and that they generate Bulletins.
+- Use `@CapabilityDescription` to provide a clear and concise description of what the component does. This should not
+ be used for configuration details.
+- Use `@Tags` to provide relevant keywords that help users find the component.
+- Use `@SeeAlso` to reference related components.
+- Use `@WritesAttributes` and `@ReadsAttributes` to document which FlowFile attributes are read and written by the component.
+- Use `@DynamicProperty` to document any dynamic properties supported by the component.
diff --git a/.cursor/rules/extension-testing.mdc b/.cursor/rules/extension-testing.mdc
new file mode 100644
index 00000000000..97692657289
--- /dev/null
+++ b/.cursor/rules/extension-testing.mdc
@@ -0,0 +1,80 @@
+---
+description: Testing guidance for NiFi extensions (Processors, Controller Services, Connectors). Covers nifi-mock and TestRunner usage.
+alwaysApply: false
+---
+
+# Extension Testing
+
+This rule applies when writing tests for NiFi extensions: Processors, Controller Services, and Connectors.
+
+## Unit Tests
+
+Unit tests should be used to test individual classes and methods in isolation. This often
+will result in mocking dependency classes. However, if there already exists a Mock
+implementation of an interface or dependency class, it is preferred to use the existing
+Mock implementation. Similarly, for simple classes, it is preferable to make use of the
+real implementation of a class rather than creating a Mock implementation. We are infinitely
+more interested in having tests that are fast, reliable, correct, and easy to maintain than
+we are in having tests that adhere to strict and arbitrary definitions of what constitutes
+a "unit test."
+
+## Use nifi-mock
+
+Tests for extensions should always make use of the `nifi-mock` mocking framework. This is
+done through the `TestRunner` interface and its standard implementation, obtained via
+`TestRunners.newTestRunner(processor)`.
+
+The `TestRunner` provides methods for:
+- Setting property values (`setProperty`)
+- Enqueueing FlowFiles (`enqueue`)
+- Running the processor (`run`)
+- Asserting transfer to relationships (`assertTransferCount`, `assertAllFlowFilesTransferred`)
+- Validating processor configuration (`assertValid`, `assertNotValid`)
+- Asserting content and attributes of FlowFiles (`assertContentEquals`, `assertAttributeEquals`, etc.)
+
+## No System Tests for Extensions
+
+System tests are not expected for extensions. Extensions are tested at the unit level using
+`nifi-mock`. The `nifi-mock` framework provides sufficient isolation and simulation of the
+NiFi runtime environment.
+
+## What to Test
+
+- **Property validation:** If the extension has a custom Validator, it
+- **customValidate:** If the extension overrides the `customValidate` method, test that it correctly
+ validates the configuration and produces appropriate validation results.
+- **Relationship routing:** Verify that FlowFiles are routed to the correct relationship
+ based on input and configuration.
+- **Content transformation:** For processors that modify FlowFile content, verify that
+ output content matches expectations.
+- **Attribute handling:** Verify that expected attributes are set on output FlowFiles.
+- **Error handling:** Verify that error conditions (bad input, misconfiguration, simulated
+ failures) are handled correctly, typically by routing to a failure relationship.
+
+## What NOT to Test
+
+- **NiFi framework behavior:** Do not attempt to test the behavior of the NiFi framework itself.
+ For example, do not test that `session.commitAsync()` actually commits a transaction. Instead,
+ focus on testing that your extension behaves correctly when `commitAsync` is called, and trust
+ that the NiFi framework will handle the commit correctly.
+- **Validator behavior:** If a custom validator is used by an extension, that custom validator should
+ be tested separate as a unit test for the validator itself. However, if the extension point provides
+ a `customValidate` method, that should absolutely be tested as part of the extension's unit tests.
+- **The PropertyDescriptors that are returned:** Do not test that the `getSupportedPropertyDescriptors`
+ method returns the expected PropertyDescriptors. This is an anti-pattern because it does not properly
+ test that the extension abides by the contract of the API. For example, if a new PropertyDescriptor is
+ added whose default is to behave the same way as the old behavior, the test should absolutely pass.
+ However, if the test is written to expect a specific set of PropertyDescriptors, then the test will fail,
+ leading to confusion and unnecessary maintenance.
+
+## Controller Service Testing
+
+When a processor depends on a Controller Service, use `TestRunner.addControllerService`
+and `TestRunner.enableControllerService` to wire up either a real or mock implementation
+of the service for testing.
+
+## TestContainers
+
+For Processors that interact with external systems, it can be helpful to use TestContainers to spin up
+a temporary instance of the external system for testing. This allows for more realistic integration tests
+without requiring the user to have the external system installed and running on their machine.
diff --git a/.cursor/rules/framework-testing.mdc b/.cursor/rules/framework-testing.mdc
new file mode 100644
index 00000000000..d25680b5021
--- /dev/null
+++ b/.cursor/rules/framework-testing.mdc
@@ -0,0 +1,46 @@
+---
+description: Testing guidance for NiFi framework code (not extensions). Covers when to use unit, integration, and system tests for framework classes.
+alwaysApply: false
+---
+
+# Framework Testing
+
+This rule applies when working on NiFi framework code (not Processors, Controller
+Services, or Connectors).
+
+## Unit Tests
+
+Unit tests should be used to test individual classes and methods in isolation. This often
+will result in mocking dependency classes. However, if there already exists a Mock
+implementation of an interface or dependency class, it is preferred to use the existing
+Mock implementation. Similarly, for simple classes, it is preferable to make use of the
+real implementation of a class rather than creating a Mock implementation. We are infinitely
+more interested in having tests that are fast, reliable, correct, and easy to maintain than
+we are in having tests that adhere to strict and arbitrary definitions of what constitutes
+a "unit test."
+
+## Integration Tests
+
+When working in the framework, unit tests are still important, but integration tests and
+system tests are often more important. Integration tests are still allowed to use mocks but
+typically we prefer to use real implementations of classes in order to ensure a more
+realistic and holistic test.
+
+## System Tests
+
+System tests live in the `nifi-system-tests` module and should be used for any changes
+that make significant changes to the framework and the interaction between a significant
+number of classes. They should also be used for any changes that may be fairly isolated but
+which are in a critical path of the framework, especially those that affect how data is
+persisted, processed, or accessed; or those that affect how components are created,
+configured, scheduled, or executed.
+
+Good candidates for system tests include changes to `ProcessScheduler`, `ProcessorNode`,
+`ControllerServiceNode`, `FlowController`, `FlowManager`, how Parameters are handled, flow
+synchronization, the repositories, etc.
+
+## Escalation
+
+Any unit test that ends up requiring a large number of mocks is a good candidate for an
+integration test, and any integration test that ends up requiring a large number of mocks
+is a good candidate for a system test.
diff --git a/.cursor/rules/persona.mdc b/.cursor/rules/persona.mdc
new file mode 100644
index 00000000000..3e4a66ed645
--- /dev/null
+++ b/.cursor/rules/persona.mdc
@@ -0,0 +1,16 @@
+---
+description: AI persona and general approach for working on the Apache NiFi codebase
+alwaysApply: true
+---
+
+# AI Persona
+
+Act as an experienced Java software engineer. When considering how to implement a task,
+first consider the big picture of what is being asked. Then determine which classes will
+need to be updated.
+
+Quite often, a single request will require manipulating many different classes. Generally
+speaking, it is best to avoid changing established interfaces, especially those in nifi-api.
+It is acceptable when necessary, but any change in nifi-api needs to be backward compatible.
+For example, you might introduce a new method with a default implementation, or add a new method
+and deprecate an old one without removing it.
diff --git a/.cursor/rules/testing-standards.mdc b/.cursor/rules/testing-standards.mdc
new file mode 100644
index 00000000000..6cb98f4853c
--- /dev/null
+++ b/.cursor/rules/testing-standards.mdc
@@ -0,0 +1,81 @@
+---
+description: Shared test coding conventions for all NiFi automated tests (framework and extensions)
+globs: "**/src/test/**/*.java"
+alwaysApply: false
+---
+
+# Testing Standards
+
+In addition to the general rules defined in the Code Style rule, follow these rules when
+creating or manipulating automated tests.
+
+## Test Code Conventions
+
+1. NEVER add comments such as `// Given`, `// When`, `// Then`. These comments are
+ considered an anti-pattern and should be removed or replaced whenever they are
+ encountered. Instead, leave them out all together (preferred) or use comments that
+ clearly articulate what is happening, such as `// Setup`, `// Invoke method`,
+ `// Assert expected results`. One of the reasons that this is considered an anti-pattern
+ (in addition to the fact that the given/when/then nomenclature itself provides no
+ meaning) is that it assumes a very specific pattern in unit tests, that we will create a
+ bunch of objects, invoke the method we care about, make assertions, and then end. This
+ often results in many tests that are extremely repetitive. Instead, whenever it makes
+ sense to do so, create the prerequisite objects, invoke the method we care about with
+ appropriate arguments, make assertions, and then invoke again with a different set of
+ arguments, make assertions, etc. There is no need to have many repetitive methods that
+ each create many repetitive objects.
+
+2. Unit tests are Java. They are not English. As such, they should be written like Java.
+ Frameworks such as assertj that strive to make the unit tests look more "English-like"
+ should be avoided. Use of these frameworks sometimes works well but often quickly
+ devolves into automated tests that read like neither English nor Java.
+
+3. Like any other code, unit tests should be created using reusable methods where
+ appropriate. Do not create 15 methods that are all very similar and repetitive. Instead,
+ create reusable methods that can be called from each of the methods.
+
+4. Never use the `assert` keyword. Use JUnit assertions instead.
+
+5. Never create a test file that is centered around testing a method or capability. Unit
+ tests must always be named after the class they are testing. It is okay if a given unit
+ test class is very, very long.
+
+6. This is a Java project using the Maven structure. Java test files must always fall under
+ src/test/java of the appropriate sub-module.
+
+7. Never use pointless assertions such as assertDoesNotThrow. This adds nothing but
+ complexity. Just call the method, and if it throws an Exception, the test will fail. It
+ is assumed by default that each line does not throw an Exception.
+
+8. Avoid providing messages in assert statements when they do not offer benefits over the default message.
+ For example, rather than:
+```
+ assertEquals(3, processors.size(), "Expected 3 processors in the initial flow");
+```
+ Just use:
+```
+ assertEquals(3, processors.size());
+```
+Because the expectation is already clear from assertion, and this message is actually harmful because it hides the size
+of the `processors` Collection.
+
+## General Testing Philosophy
+
+- Unit tests should be used to verify any sufficiently complex method in a class. We should
+ *NOT* have unit tests for trivial methods such as getters and setters, or methods that are
+ only a few lines long and are not complex. A good general rule of thumb is that if a
+ person can understand a method and verify that it is correct in a few seconds, then it is
+ not necessary to have a unit test for that method.
+
+- Avoid adding multiple tests that are redundant. For example, if you have a method that
+ takes a single `String` argument it may make sense to test with `null`, a 0-length
+ string, and a long String. But do not test with 4 different Strings each with a different
+ value but which test the same lines of code. Avoid over-testing by adding multiple tests
+ whose differences are conditions that do not truly affect the behavior of the code.
+
+- Unit tests should always focus on the "contract" of the method. That is, given a certain
+ input, the test should assert that it receives the correct output. The test should NOT be
+ focused on the inner details of how the method works. Focusing on the inner details of
+ how the method works is an anti-pattern because it results in tests that are more brittle,
+ more difficult to maintain, and tests that fail when the method is improved or refactored
+ but still adheres to the same contract.
diff --git a/.gitignore b/.gitignore
index ff8a118e9a5..e299e15e43c 100644
--- a/.gitignore
+++ b/.gitignore
@@ -21,7 +21,7 @@ nb-configuration.xml
.java-version
/nifi-nar-bundles/nifi-py4j-bundle/nifi-python-extension-api/src/main/python/dist/
__pycache__
-.cursor/
+.cursor/debug.log
# Develocity
.mvn/.develocity/
diff --git a/c2/c2-protocol/c2-protocol-component-api/src/main/java/org/apache/nifi/c2/protocol/component/api/ComponentManifest.java b/c2/c2-protocol/c2-protocol-component-api/src/main/java/org/apache/nifi/c2/protocol/component/api/ComponentManifest.java
index 1bb5fb07d98..38bd55573e5 100644
--- a/c2/c2-protocol/c2-protocol-component-api/src/main/java/org/apache/nifi/c2/protocol/component/api/ComponentManifest.java
+++ b/c2/c2-protocol/c2-protocol-component-api/src/main/java/org/apache/nifi/c2/protocol/component/api/ComponentManifest.java
@@ -33,6 +33,7 @@ public class ComponentManifest implements Serializable {
private List<ParameterProviderDefinition> parameterProviders;
private List<FlowRegistryClientDefinition> flowRegistryClients;
private List<FlowAnalysisRuleDefinition> flowAnalysisRules;
+ private List<ConnectorDefinition> connectors;
@Schema(description = "Public interfaces defined in this bundle")
public List<DefinedType> getApis() {
@@ -97,4 +98,13 @@ public class ComponentManifest implements Serializable {
this.flowRegistryClients = flowRegistryClients;
}
+ @Schema(description = "Connectors provided in this bundle")
+ public List<ConnectorDefinition> getConnectors() {
+ return (connectors != null ? Collections.unmodifiableList(connectors) : null);
+ }
+
+ public void setConnectors(List<ConnectorDefinition> connectors) {
+ this.connectors = connectors;
+ }
+
}
diff --git a/c2/c2-protocol/c2-protocol-component-api/src/main/java/org/apache/nifi/c2/protocol/component/api/ConfigurationStep.java b/c2/c2-protocol/c2-protocol-component-api/src/main/java/org/apache/nifi/c2/protocol/component/api/ConfigurationStep.java
new file mode 100644
index 00000000000..88d76b69af7
--- /dev/null
+++ b/c2/c2-protocol/c2-protocol-component-api/src/main/java/org/apache/nifi/c2/protocol/component/api/ConfigurationStep.java
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.c2.protocol.component.api;
+
+import io.swagger.v3.oas.annotations.media.Schema;
+
+import java.io.Serializable;
+import java.util.List;
+
+/**
+ * Represents a configuration step for a Connector.
+ */
+public class ConfigurationStep implements Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private String name;
+ private String description;
+ private boolean documented;
+ private List<ConfigurationStepDependency> stepDependencies;
+ private List<ConnectorPropertyGroup> propertyGroups;
+
+ @Schema(description = "The name of the configuration step")
+ public String getName() {
+ return name;
+ }
+
+ public void setName(String name) {
+ this.name = name;
+ }
+
+ @Schema(description = "The description of the configuration step")
+ public String getDescription() {
+ return description;
+ }
+
+ public void setDescription(String description) {
+ this.description = description;
+ }
+
+ @Schema(description = "Whether this configuration step has additional documentation")
+ public boolean isDocumented() {
+ return documented;
+ }
+
+ public void setDocumented(boolean documented) {
+ this.documented = documented;
+ }
+
+ @Schema(description = "The dependencies that this step has on other steps")
+ public List<ConfigurationStepDependency> getStepDependencies() {
+ return stepDependencies;
+ }
+
+ public void setStepDependencies(List<ConfigurationStepDependency> stepDependencies) {
+ this.stepDependencies = stepDependencies;
+ }
+
+ @Schema(description = "The property groups in this configuration step")
+ public List<ConnectorPropertyGroup> getPropertyGroups() {
+ return propertyGroups;
+ }
+
+ public void setPropertyGroups(List<ConnectorPropertyGroup> propertyGroups) {
+ this.propertyGroups = propertyGroups;
+ }
+}
+
diff --git a/c2/c2-protocol/c2-protocol-component-api/src/main/java/org/apache/nifi/c2/protocol/component/api/ConfigurationStepDependency.java b/c2/c2-protocol/c2-protocol-component-api/src/main/java/org/apache/nifi/c2/protocol/component/api/ConfigurationStepDependency.java
new file mode 100644
index 00000000000..1f1bc2a24d9
--- /dev/null
+++ b/c2/c2-protocol/c2-protocol-component-api/src/main/java/org/apache/nifi/c2/protocol/component/api/ConfigurationStepDependency.java
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.c2.protocol.component.api;
+
+import io.swagger.v3.oas.annotations.media.Schema;
+
+import java.io.Serializable;
+import java.util.List;
+
+/**
+ * Represents a dependency that a configuration step has on another step's property.
+ */
+public class ConfigurationStepDependency implements Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private String stepName;
+ private String propertyName;
+ private List<String> dependentValues;
+
+ @Schema(description = "The name of the step that this step depends on")
+ public String getStepName() {
+ return stepName;
+ }
+
+ public void setStepName(String stepName) {
+ this.stepName = stepName;
+ }
+
+ @Schema(description = "The name of the property within the step that this step depends on")
+ public String getPropertyName() {
+ return propertyName;
+ }
+
+ public void setPropertyName(String propertyName) {
+ this.propertyName = propertyName;
+ }
+
+ @Schema(description = "The values of the dependent property that enable this step")
+ public List<String> getDependentValues() {
+ return dependentValues;
+ }
+
+ public void setDependentValues(List<String> dependentValues) {
+ this.dependentValues = dependentValues;
+ }
+}
+
diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-ui-extension/src/main/java/org/apache/nifi/ui/extension/UiExtension.java b/c2/c2-protocol/c2-protocol-component-api/src/main/java/org/apache/nifi/c2/protocol/component/api/ConnectorDefinition.java
similarity index 52%
copy from nifi-framework-bundle/nifi-framework/nifi-web/nifi-ui-extension/src/main/java/org/apache/nifi/ui/extension/UiExtension.java
copy to c2/c2-protocol/c2-protocol-component-api/src/main/java/org/apache/nifi/c2/protocol/component/api/ConnectorDefinition.java
index da921b4535d..9ac1c85eea3 100644
--- a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-ui-extension/src/main/java/org/apache/nifi/ui/extension/UiExtension.java
+++ b/c2/c2-protocol/c2-protocol-component-api/src/main/java/org/apache/nifi/c2/protocol/component/api/ConnectorDefinition.java
@@ -14,35 +14,28 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.nifi.ui.extension;
+package org.apache.nifi.c2.protocol.component.api;
-import org.apache.nifi.web.UiExtensionType;
+import io.swagger.v3.oas.annotations.media.Schema;
+
+import java.util.Collections;
+import java.util.List;
/**
- * Information about a UI extension required to be invoked.
+ * Definition of a Connector component.
*/
-public class UiExtension {
-
- private final UiExtensionType extensionType;
- private final String contextPath;
+public class ConnectorDefinition extends ExtensionComponent {
+ private static final long serialVersionUID = 1L;
- public UiExtension(final UiExtensionType extensionType, final String contextPath) {
- this.extensionType = extensionType;
- this.contextPath = contextPath;
- }
+ private List<ConfigurationStep> configurationSteps;
- /**
- * @return type of this UI extension
- */
- public UiExtensionType getExtensionType() {
- return extensionType;
+ @Schema(description = "The configuration steps for this connector")
+ public List<ConfigurationStep> getConfigurationSteps() {
+ return (configurationSteps != null ? Collections.unmodifiableList(configurationSteps) : null);
}
- /**
- * @return The context path of this UI extension
- */
- public String getContextPath() {
- return contextPath;
+ public void setConfigurationSteps(List<ConfigurationStep> configurationSteps) {
+ this.configurationSteps = configurationSteps;
}
-
}
+
diff --git a/c2/c2-protocol/c2-protocol-component-api/src/main/java/org/apache/nifi/c2/protocol/component/api/ConnectorPropertyDependency.java b/c2/c2-protocol/c2-protocol-component-api/src/main/java/org/apache/nifi/c2/protocol/component/api/ConnectorPropertyDependency.java
new file mode 100644
index 00000000000..b01b4143e0f
--- /dev/null
+++ b/c2/c2-protocol/c2-protocol-component-api/src/main/java/org/apache/nifi/c2/protocol/component/api/ConnectorPropertyDependency.java
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.c2.protocol.component.api;
+
+import io.swagger.v3.oas.annotations.media.Schema;
+
+import java.io.Serializable;
+import java.util.List;
+
+/**
+ * Represents a dependency that a connector property has on another property.
+ */
+public class ConnectorPropertyDependency implements Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private String propertyName;
+ private List<String> dependentValues;
+
+ @Schema(description = "The name of the property that this property depends on")
+ public String getPropertyName() {
+ return propertyName;
+ }
+
+ public void setPropertyName(String propertyName) {
+ this.propertyName = propertyName;
+ }
+
+ @Schema(description = "The values of the dependent property that enable this property")
+ public List<String> getDependentValues() {
+ return dependentValues;
+ }
+
+ public void setDependentValues(List<String> dependentValues) {
+ this.dependentValues = dependentValues;
+ }
+}
+
diff --git a/c2/c2-protocol/c2-protocol-component-api/src/main/java/org/apache/nifi/c2/protocol/component/api/ConnectorPropertyDescriptor.java b/c2/c2-protocol/c2-protocol-component-api/src/main/java/org/apache/nifi/c2/protocol/component/api/ConnectorPropertyDescriptor.java
new file mode 100644
index 00000000000..fc9b4519e4e
--- /dev/null
+++ b/c2/c2-protocol/c2-protocol-component-api/src/main/java/org/apache/nifi/c2/protocol/component/api/ConnectorPropertyDescriptor.java
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.c2.protocol.component.api;
+
+import io.swagger.v3.oas.annotations.media.Schema;
+
+import java.io.Serializable;
+import java.util.List;
+
+/**
+ * Represents a property descriptor for a Connector.
+ */
+public class ConnectorPropertyDescriptor implements Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private String name;
+ private String description;
+ private String defaultValue;
+ private boolean required;
+ private ConnectorPropertyType propertyType;
+ private boolean allowableValuesFetchable;
+ private List<PropertyAllowableValue> allowableValues;
+ private List<ConnectorPropertyDependency> dependencies;
+
+ @Schema(description = "The name of the property")
+ public String getName() {
+ return name;
+ }
+
+ public void setName(String name) {
+ this.name = name;
+ }
+
+ @Schema(description = "The description of the property")
+ public String getDescription() {
+ return description;
+ }
+
+ public void setDescription(String description) {
+ this.description = description;
+ }
+
+ @Schema(description = "The default value of the property")
+ public String getDefaultValue() {
+ return defaultValue;
+ }
+
+ public void setDefaultValue(String defaultValue) {
+ this.defaultValue = defaultValue;
+ }
+
+ @Schema(description = "Whether or not the property is required")
+ public boolean isRequired() {
+ return required;
+ }
+
+ public void setRequired(boolean required) {
+ this.required = required;
+ }
+
+ @Schema(description = "The type of the property")
+ public ConnectorPropertyType getPropertyType() {
+ return propertyType;
+ }
+
+ public void setPropertyType(ConnectorPropertyType propertyType) {
+ this.propertyType = propertyType;
+ }
+
+ @Schema(description = "Whether or not the allowable values can be fetched dynamically")
+ public boolean isAllowableValuesFetchable() {
+ return allowableValuesFetchable;
+ }
+
+ public void setAllowableValuesFetchable(boolean allowableValuesFetchable) {
+ this.allowableValuesFetchable = allowableValuesFetchable;
+ }
+
+ @Schema(description = "The allowable values for this property")
+ public List<PropertyAllowableValue> getAllowableValues() {
+ return allowableValues;
+ }
+
+ public void setAllowableValues(List<PropertyAllowableValue> allowableValues) {
+ this.allowableValues = allowableValues;
+ }
+
+ @Schema(description = "The properties that this property depends on")
+ public List<ConnectorPropertyDependency> getDependencies() {
+ return dependencies;
+ }
+
+ public void setDependencies(List<ConnectorPropertyDependency> dependencies) {
+ this.dependencies = dependencies;
+ }
+}
+
diff --git a/c2/c2-protocol/c2-protocol-component-api/src/main/java/org/apache/nifi/c2/protocol/component/api/ConnectorPropertyGroup.java b/c2/c2-protocol/c2-protocol-component-api/src/main/java/org/apache/nifi/c2/protocol/component/api/ConnectorPropertyGroup.java
new file mode 100644
index 00000000000..8eaab4ec974
--- /dev/null
+++ b/c2/c2-protocol/c2-protocol-component-api/src/main/java/org/apache/nifi/c2/protocol/component/api/ConnectorPropertyGroup.java
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.c2.protocol.component.api;
+
+import io.swagger.v3.oas.annotations.media.Schema;
+
+import java.io.Serializable;
+import java.util.List;
+
+/**
+ * Represents a group of properties within a configuration step.
+ */
+public class ConnectorPropertyGroup implements Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private String name;
+ private String description;
+ private List<ConnectorPropertyDescriptor> properties;
+
+ @Schema(description = "The name of the property group")
+ public String getName() {
+ return name;
+ }
+
+ public void setName(String name) {
+ this.name = name;
+ }
+
+ @Schema(description = "The description of the property group")
+ public String getDescription() {
+ return description;
+ }
+
+ public void setDescription(String description) {
+ this.description = description;
+ }
+
+ @Schema(description = "The properties in this group")
+ public List<ConnectorPropertyDescriptor> getProperties() {
+ return properties;
+ }
+
+ public void setProperties(List<ConnectorPropertyDescriptor> properties) {
+ this.properties = properties;
+ }
+}
+
diff --git a/nifi-manifest/nifi-extension-manifest-model/src/main/java/org/apache/nifi/extension/manifest/ExtensionType.java b/c2/c2-protocol/c2-protocol-component-api/src/main/java/org/apache/nifi/c2/protocol/component/api/ConnectorPropertyType.java
similarity index 76%
copy from nifi-manifest/nifi-extension-manifest-model/src/main/java/org/apache/nifi/extension/manifest/ExtensionType.java
copy to c2/c2-protocol/c2-protocol-component-api/src/main/java/org/apache/nifi/c2/protocol/component/api/ConnectorPropertyType.java
index 0413e260ed1..906673ec9d1 100644
--- a/nifi-manifest/nifi-extension-manifest-model/src/main/java/org/apache/nifi/extension/manifest/ExtensionType.java
+++ b/c2/c2-protocol/c2-protocol-component-api/src/main/java/org/apache/nifi/c2/protocol/component/api/ConnectorPropertyType.java
@@ -14,23 +14,30 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.nifi.extension.manifest;
+package org.apache.nifi.c2.protocol.component.api;
/**
- * Possible types of extensions.
+ * Possible types for connector properties.
*/
-public enum ExtensionType {
+public enum ConnectorPropertyType {
- PROCESSOR,
+ STRING,
- CONTROLLER_SERVICE,
+ INTEGER,
- REPORTING_TASK,
+ BOOLEAN,
- FLOW_ANALYSIS_RULE,
+ FLOAT,
- PARAMETER_PROVIDER,
+ DOUBLE,
- FLOW_REGISTRY_CLIENT;
+ STRING_LIST,
+
+ SECRET,
+
+ ASSET,
+
+ ASSET_LIST;
}
+
diff --git a/checkstyle.xml b/checkstyle.xml
index dc5e8a81d6a..cd784213ed3 100644
--- a/checkstyle.xml
+++ b/checkstyle.xml
@@ -25,10 +25,6 @@
<property name="eachLine" value="true"/>
</module>
<module name="NewlineAtEndOfFile"/>
- <module name="RegexpMultiline">
- <property name="format" value="\n{3,}"/>
- <property name="message" value="Multiple consecutive blank lines are not allowed."/>
- </module>
<module name="LineLength">
<!-- needs extra, because Eclipse formatter
ignores the ending left brace -->
@@ -154,8 +150,5 @@
<module name="TypecastParenPad"/>
<module name="UnusedLocalVariable"/>
<module name="InnerAssignment"/>
- <module name="AvoidNestedBlocks">
- <property name="allowInSwitchCase" value="true"/>
- </module>
</module>
</module>
diff --git a/minifi/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-framework-core/src/main/java/org/apache/nifi/minifi/c2/command/AgentPropertyValidationContext.java b/minifi/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-framework-core/src/main/java/org/apache/nifi/minifi/c2/command/AgentPropertyValidationContext.java
index ed0434949cd..7998a92065a 100644
--- a/minifi/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-framework-core/src/main/java/org/apache/nifi/minifi/c2/command/AgentPropertyValidationContext.java
+++ b/minifi/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-framework-core/src/main/java/org/apache/nifi/minifi/c2/command/AgentPropertyValidationContext.java
@@ -109,4 +109,9 @@ public class AgentPropertyValidationContext implements ValidationContext {
public PropertyValue getProperty(PropertyDescriptor descriptor) {
throw new UnsupportedOperationException();
}
+
+ @Override
+ public String evaluateParameters(final String value) {
+ throw new UnsupportedOperationException();
+ }
}
diff --git a/minifi/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-framework-core/src/test/java/org/apache/nifi/minifi/c2/command/DefaultFlowStateStrategyTest.java b/minifi/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-framework-core/src/test/java/org/apache/nifi/minifi/c2/command/DefaultFlowStateStrategyTest.java
index ad23f6204af..18252878cff 100644
--- a/minifi/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-framework-core/src/test/java/org/apache/nifi/minifi/c2/command/DefaultFlowStateStrategyTest.java
+++ b/minifi/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-framework-core/src/test/java/org/apache/nifi/minifi/c2/command/DefaultFlowStateStrategyTest.java
@@ -31,7 +31,6 @@ import org.mockito.junit.jupiter.MockitoExtension;
import java.util.Set;
import java.util.concurrent.CompletableFuture;
-import java.util.concurrent.Future;
import static org.apache.nifi.c2.protocol.api.C2OperationState.OperationState.FULLY_APPLIED;
import static org.apache.nifi.c2.protocol.api.C2OperationState.OperationState.NOT_APPLIED;
@@ -111,13 +110,13 @@ public class DefaultFlowStateStrategyTest {
ProcessGroup rootProcessGroup = mock(ProcessGroup.class);
RemoteProcessGroup remoteProcessGroup = mock(RemoteProcessGroup.class);
Set<RemoteProcessGroup> remoteProcessGroups = Set.of(remoteProcessGroup);
- CompletableFuture<Void> rootProcessGroupStopFuture = mock(CompletableFuture.class);
- Future remoteProcessGroupStopFuture = mock(Future.class);
+ CompletableFuture<Void> rootProcessGroupStopFuture = CompletableFuture.completedFuture(null);
+ CompletableFuture<Void> remoteProcessGroupStopFuture = CompletableFuture.completedFuture(null);
ProcessGroup nestedProcessGroup = mock(ProcessGroup.class);
RemoteProcessGroup nestedRemoteProcessGroup = mock(RemoteProcessGroup.class);
Set<RemoteProcessGroup> nestedRemoteProcessGroups = Set.of(nestedRemoteProcessGroup);
- CompletableFuture<Void> nestedProcessGroupStopFuture = mock(CompletableFuture.class);
- Future nestedRemoteProcessGroupStopFuture = mock(Future.class);
+ CompletableFuture<Void> nestedProcessGroupStopFuture = CompletableFuture.completedFuture(null);
+ CompletableFuture<Void> nestedRemoteProcessGroupStopFuture = CompletableFuture.completedFuture(null);
when(flowManager.getRootGroup()).thenReturn(rootProcessGroup);
when(rootProcessGroup.getRemoteProcessGroups()).thenReturn(remoteProcessGroups);
diff --git a/nifi-assembly/pom.xml b/nifi-assembly/pom.xml
index 8308a723bdf..def628fa54e 100644
--- a/nifi-assembly/pom.xml
+++ b/nifi-assembly/pom.xml
@@ -915,6 +915,12 @@ language governing permissions and limitations under the License. -->
<version>2.9.0-SNAPSHOT</version>
<type>nar</type>
</dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-kafka-to-s3-nar</artifactId>
+ <version>2.9.0-SNAPSHOT</version>
+ <type>nar</type>
+ </dependency>
<!-- AspectJ library needed by the Java Agent used for native library loading (see bootstrap.conf) -->
<dependency>
<groupId>org.aspectj</groupId>
diff --git a/nifi-commons/nifi-connector-utils/pom.xml b/nifi-commons/nifi-connector-utils/pom.xml
new file mode 100644
index 00000000000..c44893483cd
--- /dev/null
+++ b/nifi-commons/nifi-connector-utils/pom.xml
@@ -0,0 +1,42 @@
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-commons</artifactId>
+ <version>2.9.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-connector-utils</artifactId>
+ <name>nifi-connector-utils</name>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-databind</artifactId>
+ </dependency>
+ </dependencies>
+
+</project>
diff --git a/nifi-commons/nifi-connector-utils/src/main/java/org/apache/nifi/components/connector/util/VersionedFlowUtils.java b/nifi-commons/nifi-connector-utils/src/main/java/org/apache/nifi/components/connector/util/VersionedFlowUtils.java
new file mode 100644
index 00000000000..739280d16fa
--- /dev/null
+++ b/nifi-commons/nifi-connector-utils/src/main/java/org/apache/nifi/components/connector/util/VersionedFlowUtils.java
@@ -0,0 +1,592 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.components.connector.util;
+
+import com.fasterxml.jackson.databind.DeserializationFeature;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.components.connector.ComponentBundleLookup;
+import org.apache.nifi.flow.Bundle;
+import org.apache.nifi.flow.ComponentType;
+import org.apache.nifi.flow.ConnectableComponent;
+import org.apache.nifi.flow.ConnectableComponentType;
+import org.apache.nifi.flow.PortType;
+import org.apache.nifi.flow.Position;
+import org.apache.nifi.flow.ScheduledState;
+import org.apache.nifi.flow.VersionedConnection;
+import org.apache.nifi.flow.VersionedControllerService;
+import org.apache.nifi.flow.VersionedExternalFlow;
+import org.apache.nifi.flow.VersionedParameter;
+import org.apache.nifi.flow.VersionedParameterContext;
+import org.apache.nifi.flow.VersionedPort;
+import org.apache.nifi.flow.VersionedProcessGroup;
+import org.apache.nifi.flow.VersionedProcessor;
+
+import java.io.InputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+import java.util.UUID;
+import java.util.function.Predicate;
+
+public class VersionedFlowUtils {
+ private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper()
+ .configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
+
+ public static VersionedExternalFlow loadFlowFromResource(final String resourceName) {
+ try (final InputStream in = Thread.currentThread().getContextClassLoader().getResourceAsStream(resourceName)) {
+ if (in == null) {
+ throw new IllegalArgumentException("Resource not found: " + resourceName);
+ }
+
+ return OBJECT_MAPPER.readValue(in, VersionedExternalFlow.class);
+ } catch (final Exception e) {
+ throw new IllegalStateException("Unable to load resource: " + resourceName, e);
+ }
+ }
+
+ public static Optional<VersionedProcessor> findProcessor(final VersionedProcessGroup group, final Predicate<VersionedProcessor> predicate) {
+ final List<VersionedProcessor> processors = findProcessors(group, predicate);
+ if (processors.size() == 1) {
+ return Optional.of(processors.getFirst());
+ }
+ return Optional.empty();
+ }
+
+ public static List<VersionedProcessor> findProcessors(final VersionedProcessGroup group, final Predicate<VersionedProcessor> predicate) {
+ final List<VersionedProcessor> processors = new ArrayList<>();
+ findProcessors(group, predicate, processors);
+ return processors;
+ }
+
+ private static void findProcessors(final VersionedProcessGroup group, final Predicate<VersionedProcessor> predicate, final List<VersionedProcessor> processors) {
+ for (final VersionedProcessor processor : group.getProcessors()) {
+ if (predicate.test(processor)) {
+ processors.add(processor);
+ }
+ }
+
+ for (final VersionedProcessGroup childGroup : group.getProcessGroups()) {
+ findProcessors(childGroup, predicate, processors);
+ }
+ }
+
+ public static ConnectableComponent createConnectableComponent(final VersionedProcessor processor) {
+ final ConnectableComponent component = new ConnectableComponent();
+ component.setId(processor.getIdentifier());
+ component.setName(processor.getName());
+ component.setType(ConnectableComponentType.PROCESSOR);
+ component.setGroupId(processor.getGroupIdentifier());
+ return component;
+ }
+
+ public static ConnectableComponent createConnectableComponent(final VersionedPort port) {
+ final ConnectableComponent component = new ConnectableComponent();
+ component.setId(port.getIdentifier());
+ component.setName(port.getName());
+ component.setType(port.getComponentType() == ComponentType.INPUT_PORT ? ConnectableComponentType.INPUT_PORT : ConnectableComponentType.OUTPUT_PORT);
+ component.setGroupId(port.getGroupIdentifier());
+ return component;
+ }
+
+ public static VersionedConnection addConnection(final VersionedProcessGroup group, final ConnectableComponent source, final ConnectableComponent destination,
+ final Set<String> relationships) {
+ final VersionedConnection connection = new VersionedConnection();
+ connection.setSource(source);
+ connection.setDestination(destination);
+ connection.setSelectedRelationships(relationships);
+ connection.setBends(List.of());
+ connection.setLabelIndex(0);
+ connection.setzIndex(0L);
+ connection.setGroupIdentifier(group.getIdentifier());
+ connection.setLoadBalanceStrategy("DO_NOT_LOAD_BALANCE");
+ connection.setBackPressureDataSizeThreshold("1 GB");
+ connection.setBackPressureObjectThreshold(10000L);
+ connection.setFlowFileExpiration("0 sec");
+ connection.setPrioritizers(new ArrayList<>());
+ connection.setComponentType(ComponentType.CONNECTION);
+
+ Set<VersionedConnection> connections = group.getConnections();
+ if (connections == null) {
+ connections = new HashSet<>();
+ group.setConnections(connections);
+ }
+ connections.add(connection);
+
+ final String uuid = generateDeterministicUuid(group, ComponentType.CONNECTION);
+ connection.setIdentifier(uuid);
+ return connection;
+ }
+
+ public static List<VersionedConnection> findOutboundConnections(final VersionedProcessGroup group, final VersionedProcessor processor) {
+ final VersionedProcessGroup processorGroup = findGroupForProcessor(group, processor);
+ if (processorGroup == null) {
+ return List.of();
+ }
+
+ final List<VersionedConnection> outboundConnections = new ArrayList<>();
+ final Set<VersionedConnection> connections = processorGroup.getConnections();
+ if (connections == null) {
+ return outboundConnections;
+ }
+
+ for (final VersionedConnection connection : connections) {
+ final ConnectableComponent source = connection.getSource();
+ if (Objects.equals(source.getId(), processor.getIdentifier()) && source.getType() == ConnectableComponentType.PROCESSOR) {
+ outboundConnections.add(connection);
+ }
+ }
+
+ return outboundConnections;
+ }
+
+ public static VersionedProcessGroup findGroupForProcessor(final VersionedProcessGroup rootGroup, final VersionedProcessor processor) {
+ if (rootGroup.getProcessors().contains(processor)) {
+ return rootGroup;
+ }
+
+ for (final VersionedProcessGroup childGroup : rootGroup.getProcessGroups()) {
+ final VersionedProcessGroup foundGroup = findGroupForProcessor(childGroup, processor);
+ if (foundGroup != null) {
+ return foundGroup;
+ }
+ }
+
+ return null;
+ }
+
+ public static String generateDeterministicUuid(final VersionedProcessGroup group, final ComponentType componentType) {
+ final int componentCount = getComponentCount(group, componentType);
+ final String uuidSeed = "%s-%s-%d".formatted(group.getIdentifier(), componentType.name(), componentCount);
+ return UUID.nameUUIDFromBytes(uuidSeed.getBytes(StandardCharsets.UTF_8)).toString();
+ }
+
+ private static int getComponentCount(final VersionedProcessGroup group, final ComponentType componentType) {
+ final Collection<?> components = switch (componentType) {
+ case PROCESSOR -> group.getProcessors();
+ case INPUT_PORT -> group.getInputPorts();
+ case OUTPUT_PORT -> group.getOutputPorts();
+ case CONNECTION -> group.getConnections();
+ case FUNNEL -> group.getFunnels();
+ case LABEL -> group.getLabels();
+ case PROCESS_GROUP -> group.getProcessGroups();
+ case CONTROLLER_SERVICE -> group.getControllerServices();
+ default -> List.of();
+ };
+
+ return components == null ? 0 : components.size();
+ }
+
+ public static VersionedProcessor addProcessor(final VersionedProcessGroup group, final String processorType, final Bundle bundle, final String name, final Position position) {
+ final VersionedProcessor processor = new VersionedProcessor();
+
+ // Generate deterministic UUID based on group and component type
+ processor.setIdentifier(generateDeterministicUuid(group, ComponentType.PROCESSOR));
+
+ processor.setName(name);
+ processor.setType(processorType);
+ processor.setPosition(position);
+ processor.setBundle(bundle);
+
+ // Set default processor configuration
+ processor.setProperties(new HashMap<>());
+ processor.setPropertyDescriptors(new HashMap<>());
+ processor.setStyle(new HashMap<>());
+ processor.setSchedulingPeriod("0 sec");
+ processor.setSchedulingStrategy("TIMER_DRIVEN");
+ processor.setExecutionNode("ALL");
+ processor.setPenaltyDuration("30 sec");
+ processor.setYieldDuration("1 sec");
+ processor.setBulletinLevel("WARN");
+ processor.setRunDurationMillis(25L);
+ processor.setConcurrentlySchedulableTaskCount(1);
+ processor.setAutoTerminatedRelationships(new HashSet<>());
+ processor.setScheduledState(ScheduledState.ENABLED);
+ processor.setRetryCount(10);
+ processor.setRetriedRelationships(new HashSet<>());
+ processor.setBackoffMechanism("PENALIZE_FLOWFILE");
+ processor.setMaxBackoffPeriod("10 mins");
+ processor.setComponentType(ComponentType.PROCESSOR);
+ processor.setGroupIdentifier(group.getIdentifier());
+
+ group.getProcessors().add(processor);
+ return processor;
+ }
+
+ public static VersionedControllerService addControllerService(final VersionedProcessGroup group, final String serviceType, final Bundle bundle, final String name) {
+ final VersionedControllerService controllerService = new VersionedControllerService();
+
+ // Generate deterministic UUID based on group and component type
+ controllerService.setIdentifier(generateDeterministicUuid(group, ComponentType.CONTROLLER_SERVICE));
+
+ controllerService.setName(name);
+ controllerService.setType(serviceType);
+ controllerService.setBundle(bundle);
+ controllerService.setComponentType(ComponentType.CONTROLLER_SERVICE);
+
+ // Set default controller service configuration
+ controllerService.setProperties(new HashMap<>());
+ controllerService.setPropertyDescriptors(new HashMap<>());
+ controllerService.setControllerServiceApis(new ArrayList<>());
+ controllerService.setAnnotationData(null);
+ controllerService.setScheduledState(ScheduledState.DISABLED);
+ controllerService.setBulletinLevel("WARN");
+ controllerService.setComments(null);
+ controllerService.setGroupIdentifier(group.getIdentifier());
+
+ // Initialize controller services collection if it doesn't exist
+ Set<VersionedControllerService> controllerServices = group.getControllerServices();
+ if (controllerServices == null) {
+ controllerServices = new HashSet<>();
+ group.setControllerServices(controllerServices);
+ }
+ controllerServices.add(controllerService);
+
+ return controllerService;
+ }
+
+ public static VersionedProcessGroup createProcessGroup(final String identifier, final String name) {
+ final VersionedProcessGroup group = new VersionedProcessGroup();
+ group.setIdentifier(identifier);
+ group.setName(name);
+ group.setProcessors(new HashSet<>());
+ group.setProcessGroups(new HashSet<>());
+ group.setConnections(new HashSet<>());
+ group.setControllerServices(new HashSet<>());
+ group.setInputPorts(new HashSet<>());
+ group.setOutputPorts(new HashSet<>());
+ group.setFunnels(new HashSet<>());
+ group.setLabels(new HashSet<>());
+ group.setComponentType(ComponentType.PROCESS_GROUP);
+ return group;
+ }
+
+ public static VersionedPort addInputPort(final VersionedProcessGroup group, final String name, final Position position) {
+ return addPort(group, name, position, PortType.INPUT_PORT);
+ }
+
+ public static VersionedPort addOutputPort(final VersionedProcessGroup group, final String name, final Position position) {
+ return addPort(group, name, position, PortType.OUTPUT_PORT);
+ }
+
+ private static VersionedPort addPort(final VersionedProcessGroup group, final String name, final Position position, final PortType portType) {
+ final boolean isInput = portType == PortType.INPUT_PORT;
+ final ComponentType componentType = isInput ? ComponentType.INPUT_PORT : ComponentType.OUTPUT_PORT;
+
+ final VersionedPort port = new VersionedPort();
+ port.setIdentifier(generateDeterministicUuid(group, componentType));
+ port.setName(name);
+ port.setPosition(position);
+ port.setType(portType);
+ port.setComponentType(componentType);
+ port.setScheduledState(ScheduledState.ENABLED);
+ port.setConcurrentlySchedulableTaskCount(1);
+ port.setAllowRemoteAccess(false);
+ port.setGroupIdentifier(group.getIdentifier());
+
+ if (isInput) {
+ Set<VersionedPort> inputPorts = group.getInputPorts();
+ if (inputPorts == null) {
+ inputPorts = new HashSet<>();
+ group.setInputPorts(inputPorts);
+ }
+ inputPorts.add(port);
+ } else {
+ Set<VersionedPort> outputPorts = group.getOutputPorts();
+ if (outputPorts == null) {
+ outputPorts = new HashSet<>();
+ group.setOutputPorts(outputPorts);
+ }
+ outputPorts.add(port);
+ }
+
+ return port;
+ }
+
+ public static Set<VersionedControllerService> getReferencedControllerServices(final VersionedProcessGroup group) {
+ final Set<VersionedControllerService> referencedServices = new HashSet<>();
+ collectReferencedControllerServices(group, referencedServices);
+ return referencedServices;
+ }
+
+ private static void collectReferencedControllerServices(final VersionedProcessGroup group, final Set<VersionedControllerService> referencedServices) {
+ final Map<String, VersionedControllerService> serviceMap = new HashMap<>();
+ for (final VersionedControllerService service : group.getControllerServices()) {
+ serviceMap.put(service.getIdentifier(), service);
+ }
+
+ for (final VersionedProcessor processor : group.getProcessors()) {
+ for (final String propertyValue : processor.getProperties().values()) {
+ final VersionedControllerService referencedService = serviceMap.get(propertyValue);
+ if (referencedService != null) {
+ referencedServices.add(referencedService);
+ }
+ }
+ }
+
+ while (true) {
+ final Set<VersionedControllerService> newlyAddedServices = new HashSet<>();
+
+ for (final VersionedControllerService service : referencedServices) {
+ for (final String propertyValue : service.getProperties().values()) {
+ final VersionedControllerService referencedService = serviceMap.get(propertyValue);
+ if (referencedService != null && !referencedServices.contains(referencedService)) {
+ newlyAddedServices.add(referencedService);
+ }
+ }
+ }
+
+ referencedServices.addAll(newlyAddedServices);
+ if (newlyAddedServices.isEmpty()) {
+ break;
+ }
+ }
+
+ for (final VersionedProcessGroup childGroup : group.getProcessGroups()) {
+ collectReferencedControllerServices(childGroup, referencedServices);
+ }
+ }
+
+ /**
+ * Returns the set of controller services that are transitively referenced by the given processor.
+ * This includes any services directly referenced by the processor's properties, as well as any services
+ * that those services reference, and so on. Only services that are accessible to the processor are considered,
+ * meaning services in the processor's own group and its ancestor groups.
+ *
+ * @param rootGroup the root process group to search for controller services
+ * @param processor the processor whose referenced services should be found
+ * @return the set of transitively referenced controller services
+ */
+ public static Set<VersionedControllerService> getReferencedControllerServices(final VersionedProcessGroup rootGroup, final VersionedProcessor processor) {
+ return findTransitivelyReferencedServices(rootGroup, processor.getGroupIdentifier(), processor.getProperties());
+ }
+
+ /**
+ * Returns the set of controller services that are transitively referenced by the given controller service.
+ * This includes any services directly referenced by the service's properties, as well as any services
+ * that those services reference, and so on. Only services that are accessible to the given service are considered,
+ * meaning services in the service's own group and its ancestor groups.
+ *
+ * @param rootGroup the root process group to search for controller services
+ * @param controllerService the controller service whose referenced services should be found
+ * @return the set of transitively referenced controller services
+ */
+ public static Set<VersionedControllerService> getReferencedControllerServices(final VersionedProcessGroup rootGroup, final VersionedControllerService controllerService) {
+ return findTransitivelyReferencedServices(rootGroup, controllerService.getGroupIdentifier(), controllerService.getProperties());
+ }
+
+ private static Set<VersionedControllerService> findTransitivelyReferencedServices(final VersionedProcessGroup rootGroup, final String componentGroupId,
+ final Map<String, String> properties) {
+ final Map<String, VersionedControllerService> serviceMap = new HashMap<>();
+ collectAccessibleControllerServices(rootGroup, componentGroupId, serviceMap);
+
+ final Set<VersionedControllerService> referencedServices = new HashSet<>();
+ for (final String propertyValue : properties.values()) {
+ final VersionedControllerService referencedService = serviceMap.get(propertyValue);
+ if (referencedService != null) {
+ referencedServices.add(referencedService);
+ }
+ }
+
+ resolveTransitiveServiceReferences(referencedServices, serviceMap);
+ return referencedServices;
+ }
+
+ /**
+ * Collects controller services that are accessible from the given target group. In NiFi, a component can reference
+ * controller services in its own group or any ancestor group. This method traverses from the root group down to the
+ * target group, collecting services from each group along the path.
+ *
+ * @param group the current group being examined
+ * @param targetGroupId the identifier of the group whose accessible services should be collected
+ * @param serviceMap the map to populate with accessible service identifiers and their corresponding services
+ * @return true if the target group was found at or beneath this group, false otherwise
+ */
+ private static boolean collectAccessibleControllerServices(final VersionedProcessGroup group, final String targetGroupId,
+ final Map<String, VersionedControllerService> serviceMap) {
+ final boolean isTarget = group.getIdentifier().equals(targetGroupId);
+
+ boolean foundInChild = false;
+ if (!isTarget) {
+ for (final VersionedProcessGroup childGroup : group.getProcessGroups()) {
+ if (collectAccessibleControllerServices(childGroup, targetGroupId, serviceMap)) {
+ foundInChild = true;
+ break;
+ }
+ }
+ }
+
+ if (isTarget || foundInChild) {
+ for (final VersionedControllerService service : group.getControllerServices()) {
+ serviceMap.put(service.getIdentifier(), service);
+ }
+ return true;
+ }
+
+ return false;
+ }
+
+ private static void resolveTransitiveServiceReferences(final Set<VersionedControllerService> referencedServices, final Map<String, VersionedControllerService> serviceMap) {
+ while (true) {
+ final Set<VersionedControllerService> newlyAddedServices = new HashSet<>();
+
+ for (final VersionedControllerService service : referencedServices) {
+ for (final String propertyValue : service.getProperties().values()) {
+ final VersionedControllerService referencedService = serviceMap.get(propertyValue);
+ if (referencedService != null && !referencedServices.contains(referencedService)) {
+ newlyAddedServices.add(referencedService);
+ }
+ }
+ }
+
+ referencedServices.addAll(newlyAddedServices);
+ if (newlyAddedServices.isEmpty()) {
+ break;
+ }
+ }
+ }
+
+ public static void removeControllerServiceReferences(final VersionedProcessGroup processGroup, final String serviceIdentifier) {
+ for (final VersionedProcessor processor : processGroup.getProcessors()) {
+ removeValuesFromMap(processor.getProperties(), serviceIdentifier);
+ }
+
+ for (final VersionedControllerService service : processGroup.getControllerServices()) {
+ removeValuesFromMap(service.getProperties(), serviceIdentifier);
+ }
+
+ for (final VersionedProcessGroup childGroup : processGroup.getProcessGroups()) {
+ removeControllerServiceReferences(childGroup, serviceIdentifier);
+ }
+ }
+
+ private static void removeValuesFromMap(final Map<String, String> properties, final String valueToRemove) {
+ final List<String> keysToRemove = new ArrayList<>();
+ for (final Map.Entry<String, String> entry : properties.entrySet()) {
+ if (Objects.equals(entry.getValue(), valueToRemove)) {
+ keysToRemove.add(entry.getKey());
+ }
+ }
+
+ keysToRemove.forEach(properties::remove);
+ }
+
+ public static void setParameterValue(final VersionedExternalFlow externalFlow, final String parameterName, final String parameterValue) {
+ for (final VersionedParameterContext context : externalFlow.getParameterContexts().values()) {
+ setParameterValue(context, parameterName, parameterValue);
+ }
+ }
+
+ public static void setParameterValue(final VersionedParameterContext parameterContext, final String parameterName, final String parameterValue) {
+ final Set<VersionedParameter> parameters = parameterContext.getParameters();
+ for (final VersionedParameter parameter : parameters) {
+ if (parameter.getName().equals(parameterName)) {
+ parameter.setValue(parameterValue);
+ }
+ }
+ }
+
+ public static void setParameterValues(final VersionedExternalFlow externalFlow, final Map<String, String> parameterValues) {
+ for (final Map.Entry<String, String> entry : parameterValues.entrySet()) {
+ setParameterValue(externalFlow, entry.getKey(), entry.getValue());
+ }
+ }
+
+ public static void setParameterValues(final VersionedParameterContext parameterContext, final Map<String, String> parameterValues) {
+ for (final Map.Entry<String, String> entry : parameterValues.entrySet()) {
+ setParameterValue(parameterContext, entry.getKey(), entry.getValue());
+ }
+ }
+
+ public static void removeUnreferencedControllerServices(final VersionedProcessGroup processGroup) {
+ final Set<VersionedControllerService> referencedServices = getReferencedControllerServices(processGroup);
+ final Set<String> referencedServiceIds = new HashSet<>();
+ for (final VersionedControllerService service : referencedServices) {
+ referencedServiceIds.add(service.getIdentifier());
+ }
+
+ removeUnreferencedControllerServices(processGroup, referencedServiceIds);
+ }
+
+ private static void removeUnreferencedControllerServices(final VersionedProcessGroup processGroup, final Set<String> referencedServiceIds) {
+ processGroup.getControllerServices().removeIf(service -> !referencedServiceIds.contains(service.getIdentifier()));
+
+ for (final VersionedProcessGroup childGroup : processGroup.getProcessGroups()) {
+ removeUnreferencedControllerServices(childGroup, referencedServiceIds);
+ }
+ }
+
+ /**
+ * Updates all processors and controller services in the given process group (and its child groups)
+ * to use the latest available bundle version. See {@link ComponentBundleLookup#getLatestBundle(String)} for details on how
+ * version comparison is performed.
+ *
+ * @param processGroup the process group containing components to update
+ * @param componentBundleLookup the lookup used to find available bundles for each component type
+ */
+ public static void updateToLatestBundles(final VersionedProcessGroup processGroup, final ComponentBundleLookup componentBundleLookup) {
+ for (final VersionedProcessor processor : processGroup.getProcessors()) {
+ updateToLatestBundle(processor, componentBundleLookup);
+ }
+
+ for (final VersionedControllerService service : processGroup.getControllerServices()) {
+ updateToLatestBundle(service, componentBundleLookup);
+ }
+
+ for (final VersionedProcessGroup childGroup : processGroup.getProcessGroups()) {
+ updateToLatestBundles(childGroup, componentBundleLookup);
+ }
+ }
+
+ /**
+ * Updates the given processor to use the latest available bundle version.
+ * If no bundle is available, the processor's bundle is left unchanged.
+ * See {@link ComponentBundleLookup#getLatestBundle(String)} for details on how version comparison is performed.
+ *
+ * @param processor the processor to update
+ * @param componentBundleLookup the lookup used to find available bundles for the processor type
+ * @return true if the bundle was updated, false if no bundle was available
+ */
+ public static boolean updateToLatestBundle(final VersionedProcessor processor, final ComponentBundleLookup componentBundleLookup) {
+ final Optional<Bundle> latestBundle = componentBundleLookup.getLatestBundle(processor.getType());
+ latestBundle.ifPresent(processor::setBundle);
+ return latestBundle.isPresent();
+ }
+
+ /**
+ * Updates the given controller service to use the latest available bundle version.
+ * If no bundle is available, the service's bundle is left unchanged.
+ * See {@link ComponentBundleLookup#getLatestBundle(String)} for details on how version comparison is performed.
+ *
+ * @param service the controller service to update
+ * @param componentBundleLookup the lookup used to find available bundles for the service type
+ * @return true if the bundle was updated, false if no bundle was available
+ */
+ public static boolean updateToLatestBundle(final VersionedControllerService service, final ComponentBundleLookup componentBundleLookup) {
+ final Optional<Bundle> latestBundle = componentBundleLookup.getLatestBundle(service.getType());
+ latestBundle.ifPresent(service::setBundle);
+ return latestBundle.isPresent();
+ }
+
+}
diff --git a/nifi-commons/nifi-connector-utils/src/test/java/org/apache/nifi/components/connector/util/TestVersionedFlowUtils.java b/nifi-commons/nifi-connector-utils/src/test/java/org/apache/nifi/components/connector/util/TestVersionedFlowUtils.java
new file mode 100644
index 00000000000..a7534d9088f
--- /dev/null
+++ b/nifi-commons/nifi-connector-utils/src/test/java/org/apache/nifi/components/connector/util/TestVersionedFlowUtils.java
@@ -0,0 +1,285 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.components.connector.util;
+
+import org.apache.nifi.components.connector.ComponentBundleLookup;
+import org.apache.nifi.flow.Bundle;
+import org.apache.nifi.flow.Position;
+import org.apache.nifi.flow.VersionedControllerService;
+import org.apache.nifi.flow.VersionedProcessGroup;
+import org.apache.nifi.flow.VersionedProcessor;
+import org.junit.jupiter.api.Nested;
+import org.junit.jupiter.api.Test;
+
+import java.util.Optional;
+import java.util.Set;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestVersionedFlowUtils {
+
+ @Nested
+ class UpdateToLatestBundles {
+ private static final String PROCESSOR_TYPE = "org.apache.nifi.processors.TestProcessor";
+ private static final String SERVICE_TYPE = "org.apache.nifi.services.TestService";
+
+ @Test
+ void testLookupReturnsNewerBundle() {
+ final VersionedProcessGroup group = createProcessGroup();
+ final VersionedProcessor processor = VersionedFlowUtils.addProcessor(group, PROCESSOR_TYPE, new Bundle("group", "artifact", "2.0.0"), "Test Processor", new Position(0, 0));
+
+ final ComponentBundleLookup lookup = mock(ComponentBundleLookup.class);
+ when(lookup.getLatestBundle(PROCESSOR_TYPE)).thenReturn(Optional.of(new Bundle("group", "artifact", "3.0.0")));
+
+ VersionedFlowUtils.updateToLatestBundles(group, lookup);
+
+ assertEquals("3.0.0", processor.getBundle().getVersion());
+ }
+
+ @Test
+ void testLookupReturnsLatestBundle() {
+ final VersionedProcessGroup group = createProcessGroup();
+ final VersionedProcessor processor = VersionedFlowUtils.addProcessor(group, PROCESSOR_TYPE, new Bundle("group", "artifact", "2.0.0"), "Test Processor", new Position(0, 0));
+
+ final ComponentBundleLookup lookup = mock(ComponentBundleLookup.class);
+ when(lookup.getLatestBundle(PROCESSOR_TYPE)).thenReturn(Optional.of(new Bundle("group", "artifact", "4.0.0")));
+
+ VersionedFlowUtils.updateToLatestBundles(group, lookup);
+
+ assertEquals("4.0.0", processor.getBundle().getVersion());
+ }
+
+ @Test
+ void testControllerServiceUpdatedToNewerBundle() {
+ final VersionedProcessGroup group = createProcessGroup();
+ final VersionedControllerService service = VersionedFlowUtils.addControllerService(group, SERVICE_TYPE, new Bundle("group", "artifact", "1.5.0"), "Test Service");
+
+ final ComponentBundleLookup lookup = mock(ComponentBundleLookup.class);
+ when(lookup.getLatestBundle(SERVICE_TYPE)).thenReturn(Optional.of(new Bundle("group", "artifact", "2.5.0")));
+
+ VersionedFlowUtils.updateToLatestBundles(group, lookup);
+
+ assertEquals("2.5.0", service.getBundle().getVersion());
+ }
+
+ @Test
+ void testNestedProcessorUpdatedToNewerBundle() {
+ final VersionedProcessGroup rootGroup = createProcessGroup();
+ final VersionedProcessGroup childGroup = createChildProcessGroup(rootGroup, "child-group-id");
+ final VersionedProcessor nestedProcessor = VersionedFlowUtils.addProcessor(childGroup, PROCESSOR_TYPE, new Bundle("group", "artifact", "1.0.0"), "Nested Processor", new Position(0, 0));
+
+ final ComponentBundleLookup lookup = mock(ComponentBundleLookup.class);
+ when(lookup.getLatestBundle(PROCESSOR_TYPE)).thenReturn(Optional.of(new Bundle("group", "artifact", "5.0.0")));
+
+ VersionedFlowUtils.updateToLatestBundles(rootGroup, lookup);
+
+ assertEquals("5.0.0", nestedProcessor.getBundle().getVersion());
+ }
+
+ @Test
+ void testEmptyOptionalDoesNotChangeBundle() {
+ final VersionedProcessGroup group = createProcessGroup();
+ final VersionedProcessor processor = VersionedFlowUtils.addProcessor(group, PROCESSOR_TYPE, new Bundle("group", "artifact", "2.0.0"), "Test Processor", new Position(0, 0));
+
+ final ComponentBundleLookup lookup = mock(ComponentBundleLookup.class);
+ when(lookup.getLatestBundle(PROCESSOR_TYPE)).thenReturn(Optional.empty());
+
+ VersionedFlowUtils.updateToLatestBundles(group, lookup);
+
+ assertEquals("2.0.0", processor.getBundle().getVersion());
+ }
+
+ @Test
+ void testUpdateToLatestBundleReturnsTrueWhenUpdated() {
+ final VersionedProcessGroup group = createProcessGroup();
+ final VersionedProcessor processor = VersionedFlowUtils.addProcessor(group, PROCESSOR_TYPE, new Bundle("group", "artifact", "2.0.0"), "Test Processor", new Position(0, 0));
+
+ final ComponentBundleLookup lookup = mock(ComponentBundleLookup.class);
+ when(lookup.getLatestBundle(PROCESSOR_TYPE)).thenReturn(Optional.of(new Bundle("group", "artifact", "3.0.0")));
+
+ final boolean updated = VersionedFlowUtils.updateToLatestBundle(processor, lookup);
+
+ assertTrue(updated);
+ assertEquals("3.0.0", processor.getBundle().getVersion());
+ }
+
+ @Test
+ void testUpdateToLatestBundleReturnsFalseWhenNotUpdated() {
+ final VersionedProcessGroup group = createProcessGroup();
+ final VersionedProcessor processor = VersionedFlowUtils.addProcessor(group, PROCESSOR_TYPE, new Bundle("group", "artifact", "2.0.0"), "Test Processor", new Position(0, 0));
+
+ final ComponentBundleLookup lookup = mock(ComponentBundleLookup.class);
+ when(lookup.getLatestBundle(PROCESSOR_TYPE)).thenReturn(Optional.empty());
+
+ final boolean updated = VersionedFlowUtils.updateToLatestBundle(processor, lookup);
+
+ assertFalse(updated);
+ assertEquals("2.0.0", processor.getBundle().getVersion());
+ }
+
+ }
+
+ @Nested
+ class GetReferencedControllerServices {
+ private static final String PROCESSOR_TYPE = "org.apache.nifi.processors.TestProcessor";
+ private static final String SERVICE_TYPE = "org.apache.nifi.services.TestService";
+ private static final Bundle TEST_BUNDLE = new Bundle("group", "artifact", "1.0.0");
+
+ @Test
+ void testProcessorWithNoReferences() {
+ final VersionedProcessGroup group = createProcessGroup();
+ VersionedFlowUtils.addControllerService(group, SERVICE_TYPE, TEST_BUNDLE, "Unreferenced Service");
+ final VersionedProcessor processor = VersionedFlowUtils.addProcessor(group, PROCESSOR_TYPE, TEST_BUNDLE, "Processor", new Position(0, 0));
+ processor.getProperties().put("some-property", "not-a-service-id");
+
+ final Set<VersionedControllerService> result = VersionedFlowUtils.getReferencedControllerServices(group, processor);
+
+ assertTrue(result.isEmpty());
+ }
+
+ @Test
+ void testProcessorWithDirectReferences() {
+ final VersionedProcessGroup group = createProcessGroup();
+ final VersionedControllerService service1 = VersionedFlowUtils.addControllerService(group, SERVICE_TYPE, TEST_BUNDLE, "Service 1");
+ final VersionedControllerService service2 = VersionedFlowUtils.addControllerService(group, SERVICE_TYPE, TEST_BUNDLE, "Service 2");
+ final VersionedControllerService unreferencedService = VersionedFlowUtils.addControllerService(group, SERVICE_TYPE, TEST_BUNDLE, "Unreferenced Service");
+
+ final VersionedProcessor processor = VersionedFlowUtils.addProcessor(group, PROCESSOR_TYPE, TEST_BUNDLE, "Processor", new Position(0, 0));
+ processor.getProperties().put("service-prop-1", service1.getIdentifier());
+ processor.getProperties().put("service-prop-2", service2.getIdentifier());
+ processor.getProperties().put("non-service-prop", "some-value");
+
+ final Set<VersionedControllerService> result = VersionedFlowUtils.getReferencedControllerServices(group, processor);
+
+ assertEquals(2, result.size());
+ assertTrue(result.contains(service1));
+ assertTrue(result.contains(service2));
+ assertFalse(result.contains(unreferencedService));
+ }
+
+ @Test
+ void testProcessorWithTransitiveReferences() {
+ final VersionedProcessGroup group = createProcessGroup();
+ final VersionedControllerService serviceC = VersionedFlowUtils.addControllerService(group, SERVICE_TYPE, TEST_BUNDLE, "Service C");
+ final VersionedControllerService serviceB = VersionedFlowUtils.addControllerService(group, SERVICE_TYPE, TEST_BUNDLE, "Service B");
+ serviceB.getProperties().put("nested-service", serviceC.getIdentifier());
+ final VersionedControllerService serviceA = VersionedFlowUtils.addControllerService(group, SERVICE_TYPE, TEST_BUNDLE, "Service A");
+ serviceA.getProperties().put("nested-service", serviceB.getIdentifier());
+
+ final VersionedProcessor processor = VersionedFlowUtils.addProcessor(group, PROCESSOR_TYPE, TEST_BUNDLE, "Processor", new Position(0, 0));
+ processor.getProperties().put("service-prop", serviceA.getIdentifier());
+
+ final Set<VersionedControllerService> result = VersionedFlowUtils.getReferencedControllerServices(group, processor);
+
+ assertEquals(3, result.size());
+ assertTrue(result.contains(serviceA));
+ assertTrue(result.contains(serviceB));
+ assertTrue(result.contains(serviceC));
+ }
+
+ @Test
+ void testProcessorReferencingServiceInAncestorGroup() {
+ final VersionedProcessGroup rootGroup = createProcessGroup();
+ final VersionedControllerService parentService = VersionedFlowUtils.addControllerService(rootGroup, SERVICE_TYPE, TEST_BUNDLE, "Parent Service");
+ final VersionedProcessGroup childGroup = createChildProcessGroup(rootGroup, "child-group-id");
+ final VersionedProcessor processor = VersionedFlowUtils.addProcessor(childGroup, PROCESSOR_TYPE, TEST_BUNDLE, "Processor", new Position(0, 0));
+ processor.getProperties().put("service-prop", parentService.getIdentifier());
+
+ final Set<VersionedControllerService> result = VersionedFlowUtils.getReferencedControllerServices(rootGroup, processor);
+
+ assertEquals(1, result.size());
+ assertTrue(result.contains(parentService));
+ }
+
+ @Test
+ void testProcessorDoesNotFindServiceInDescendantGroup() {
+ final VersionedProcessGroup rootGroup = createProcessGroup();
+ final VersionedProcessGroup childGroup = createChildProcessGroup(rootGroup, "child-group-id");
+ final VersionedControllerService childService = VersionedFlowUtils.addControllerService(childGroup, SERVICE_TYPE, TEST_BUNDLE, "Child Service");
+
+ final VersionedProcessor processor = VersionedFlowUtils.addProcessor(rootGroup, PROCESSOR_TYPE, TEST_BUNDLE, "Processor", new Position(0, 0));
+ processor.getProperties().put("service-prop", childService.getIdentifier());
+
+ final Set<VersionedControllerService> result = VersionedFlowUtils.getReferencedControllerServices(rootGroup, processor);
+
+ assertTrue(result.isEmpty());
+ }
+
+ @Test
+ void testProcessorWithCircularServiceReferences() {
+ final VersionedProcessGroup group = createProcessGroup();
+ final VersionedControllerService serviceA = VersionedFlowUtils.addControllerService(group, SERVICE_TYPE, TEST_BUNDLE, "Service A");
+ final VersionedControllerService serviceB = VersionedFlowUtils.addControllerService(group, SERVICE_TYPE, TEST_BUNDLE, "Service B");
+ serviceA.getProperties().put("nested-service", serviceB.getIdentifier());
+ serviceB.getProperties().put("nested-service", serviceA.getIdentifier());
+
+ final VersionedProcessor processor = VersionedFlowUtils.addProcessor(group, PROCESSOR_TYPE, TEST_BUNDLE, "Processor", new Position(0, 0));
+ processor.getProperties().put("service-prop", serviceA.getIdentifier());
+
+ final Set<VersionedControllerService> result = VersionedFlowUtils.getReferencedControllerServices(group, processor);
+
+ assertEquals(2, result.size());
+ assertTrue(result.contains(serviceA));
+ assertTrue(result.contains(serviceB));
+ }
+
+ @Test
+ void testControllerServiceWithNoReferences() {
+ final VersionedProcessGroup group = createProcessGroup();
+ final VersionedControllerService service = VersionedFlowUtils.addControllerService(group, SERVICE_TYPE, TEST_BUNDLE, "Service");
+ service.getProperties().put("some-property", "not-a-service-id");
+
+ final Set<VersionedControllerService> result = VersionedFlowUtils.getReferencedControllerServices(group, service);
+
+ assertTrue(result.isEmpty());
+ }
+
+ @Test
+ void testControllerServiceWithDirectAndTransitiveReferences() {
+ final VersionedProcessGroup group = createProcessGroup();
+ final VersionedControllerService serviceC = VersionedFlowUtils.addControllerService(group, SERVICE_TYPE, TEST_BUNDLE, "Service C");
+ final VersionedControllerService serviceB = VersionedFlowUtils.addControllerService(group, SERVICE_TYPE, TEST_BUNDLE, "Service B");
+ serviceB.getProperties().put("nested-service", serviceC.getIdentifier());
+ final VersionedControllerService serviceA = VersionedFlowUtils.addControllerService(group, SERVICE_TYPE, TEST_BUNDLE, "Service A");
+ serviceA.getProperties().put("nested-service", serviceB.getIdentifier());
+
+ final Set<VersionedControllerService> result = VersionedFlowUtils.getReferencedControllerServices(group, serviceA);
+
+ assertEquals(2, result.size());
+ assertTrue(result.contains(serviceB));
+ assertTrue(result.contains(serviceC));
+ assertFalse(result.contains(serviceA));
+ }
+ }
+
+ private static VersionedProcessGroup createProcessGroup() {
+ return VersionedFlowUtils.createProcessGroup("test-group-id", "Test Process Group");
+ }
+
+ private static VersionedProcessGroup createChildProcessGroup(final VersionedProcessGroup parent, final String identifier) {
+ final VersionedProcessGroup childGroup = VersionedFlowUtils.createProcessGroup(identifier, "Child Process Group");
+ childGroup.setGroupIdentifier(parent.getIdentifier());
+ parent.getProcessGroups().add(childGroup);
+ return childGroup;
+ }
+
+}
diff --git a/nifi-commons/nifi-properties/src/main/java/org/apache/nifi/util/NiFiProperties.java b/nifi-commons/nifi-properties/src/main/java/org/apache/nifi/util/NiFiProperties.java
index 0d6a8412e65..eef06942f01 100644
--- a/nifi-commons/nifi-properties/src/main/java/org/apache/nifi/util/NiFiProperties.java
+++ b/nifi-commons/nifi-properties/src/main/java/org/apache/nifi/util/NiFiProperties.java
@@ -124,6 +124,9 @@ public class NiFiProperties extends ApplicationProperties {
public static final String ASSET_MANAGER_IMPLEMENTATION = "nifi.asset.manager.implementation";
public static final String ASSET_MANAGER_PREFIX = "nifi.asset.manager.properties.";
+ public static final String CONNECTOR_ASSET_MANAGER_IMPLEMENTATION = "nifi.connector.asset.manager.implementation";
+ public static final String CONNECTOR_ASSET_MANAGER_PREFIX = "nifi.connector.asset.manager.properties.";
+
// status repository properties
public static final String COMPONENT_STATUS_REPOSITORY_IMPLEMENTATION = "nifi.components.status.repository.implementation";
public static final String COMPONENT_STATUS_SNAPSHOT_FREQUENCY = "nifi.components.status.snapshot.frequency";
@@ -136,6 +139,16 @@ public class NiFiProperties extends ApplicationProperties {
public static final String STATUS_REPOSITORY_QUESTDB_PERSIST_BATCH_SIZE = "nifi.status.repository.questdb.persist.batchsize";
public static final String STATUS_REPOSITORY_QUESTDB_PERSIST_FREQUENCY = "nifi.status.repository.questdb.persist.frequency";
+ // Connector Repository properties
+ public static final String CONNECTOR_REPOSITORY_IMPLEMENTATION = "nifi.components.connectors.repository.implementation";
+
+ // Connector Configuration Provider properties
+ public static final String CONNECTOR_CONFIGURATION_PROVIDER_IMPLEMENTATION = "nifi.components.connectors.configuration.provider.implementation";
+ public static final String CONNECTOR_CONFIGURATION_PROVIDER_PROPERTIES_PREFIX = "nifi.components.connectors.configuration.provider.";
+
+ // Secrets Manager properties
+ public static final String SECRETS_MANAGER_IMPLEMENTATION = "nifi.secrets.manager.implementation";
+
// security properties
public static final String SECURITY_KEYSTORE = "nifi.security.keystore";
public static final String SECURITY_KEYSTORE_TYPE = "nifi.security.keystoreType";
@@ -1807,8 +1820,28 @@ public class NiFiProperties extends ApplicationProperties {
// The Properties(Properties) constructor does NOT inherit the provided values, just uses them as default values
if (additionalProperties != null) {
- additionalProperties.forEach(properties::put);
+ properties.putAll(additionalProperties);
}
+
+ return createNiFiProperties(properties);
+ }
+
+ public static NiFiProperties createBasicNiFiProperties(final InputStream inputStream) {
+ return createBasicNiFiProperties(inputStream, null);
+ }
+
+ public static NiFiProperties createBasicNiFiProperties(final InputStream inputStream, final Properties additionalProperties) {
+ final Properties properties = new Properties();
+ readFromInputStream(inputStream, properties);
+
+ if (additionalProperties != null) {
+ properties.putAll(additionalProperties);
+ }
+
+ return createNiFiProperties(properties);
+ }
+
+ private static NiFiProperties createNiFiProperties(final Properties properties) {
return new NiFiProperties() {
@Override
public String getProperty(String key) {
@@ -1828,9 +1861,7 @@ public class NiFiProperties extends ApplicationProperties {
}
private static void readFromPropertiesFile(String propertiesFilePath, Properties properties) {
- final String nfPropertiesFilePath = (propertiesFilePath == null)
- ? System.getProperty(PROPERTIES_FILE_PATH)
- : propertiesFilePath;
+ final String nfPropertiesFilePath = (propertiesFilePath == null) ? System.getProperty(PROPERTIES_FILE_PATH) : propertiesFilePath;
if (nfPropertiesFilePath != null) {
final File propertiesFile = new File(nfPropertiesFilePath.trim());
if (!propertiesFile.exists()) {
@@ -1841,24 +1872,25 @@ public class NiFiProperties extends ApplicationProperties {
throw new RuntimeException("Properties file exists but cannot be read '"
+ propertiesFile.getAbsolutePath() + "'");
}
- InputStream inStream = null;
- try {
- inStream = new BufferedInputStream(new FileInputStream(propertiesFile));
+
+ try (final InputStream fileIn = new FileInputStream(propertiesFile);
+ final InputStream inStream = new BufferedInputStream(fileIn)) {
+
properties.load(inStream);
} catch (final Exception ex) {
- throw new RuntimeException("Cannot load properties file due to "
- + ex.getLocalizedMessage(), ex);
- } finally {
- if (null != inStream) {
- try {
- inStream.close();
- } catch (final Exception ignored) {
- }
- }
+ throw new RuntimeException("Cannot load properties file due to " + ex.getLocalizedMessage(), ex);
}
}
}
+ private static void readFromInputStream(final InputStream inputStream, final Properties properties) {
+ try {
+ properties.load(inputStream);
+ } catch (final Exception ex) {
+ throw new RuntimeException("Cannot load properties file due to " + ex.getLocalizedMessage(), ex);
+ }
+ }
+
/**
* This method is used to validate the NiFi properties when the file is loaded
* for the first time. The objective is to stop NiFi startup in case a property
diff --git a/nifi-commons/nifi-write-ahead-log/src/main/java/org/apache/nifi/wali/HashMapSnapshot.java b/nifi-commons/nifi-write-ahead-log/src/main/java/org/apache/nifi/wali/HashMapSnapshot.java
index 4c768ed76c8..b6d7682ae71 100644
--- a/nifi-commons/nifi-write-ahead-log/src/main/java/org/apache/nifi/wali/HashMapSnapshot.java
+++ b/nifi-commons/nifi-write-ahead-log/src/main/java/org/apache/nifi/wali/HashMapSnapshot.java
@@ -175,14 +175,18 @@ public class HashMapSnapshot<T> implements WriteAheadSnapshot<T>, RecordLookup<T
switch (updateType) {
case DELETE:
- recordMap.remove(recordId);
+ if (recordId != null) {
+ recordMap.remove(recordId);
+ }
break;
case SWAP_OUT:
final String location = serdeFactory.getLocation(record);
if (location == null) {
logger.error(logMessage, recordId, UpdateType.SWAP_OUT, "Swapped Out to", "lost");
} else {
- recordMap.remove(recordId);
+ if (recordId != null) {
+ recordMap.remove(recordId);
+ }
this.swapLocations.add(location);
}
break;
@@ -193,10 +197,30 @@ public class HashMapSnapshot<T> implements WriteAheadSnapshot<T>, RecordLookup<T
} else {
swapLocations.remove(swapLocation);
}
- recordMap.put(recordId, record);
+ if (recordId != null) {
+ recordMap.put(recordId, record);
+ }
+ break;
+ case SWAP_FILE_DELETED:
+ final String deletedSwapLocation = serdeFactory.getLocation(record);
+ if (deletedSwapLocation != null) {
+ swapLocations.remove(deletedSwapLocation);
+ }
+ break;
+ case SWAP_FILE_RENAMED:
+ final String originalLocation = serdeFactory.getOriginalLocation(record);
+ final String newLocation = serdeFactory.getLocation(record);
+ if (originalLocation != null) {
+ swapLocations.remove(originalLocation);
+ }
+ if (newLocation != null) {
+ swapLocations.add(newLocation);
+ }
break;
default:
- recordMap.put(recordId, record);
+ if (recordId != null) {
+ recordMap.put(recordId, record);
+ }
break;
}
}
@@ -209,6 +233,9 @@ public class HashMapSnapshot<T> implements WriteAheadSnapshot<T>, RecordLookup<T
@Override
public T lookup(final Object recordId) {
+ if (recordId == null) {
+ return null;
+ }
return recordMap.get(recordId);
}
diff --git a/nifi-commons/nifi-write-ahead-log/src/main/java/org/apache/nifi/wali/LengthDelimitedJournal.java b/nifi-commons/nifi-write-ahead-log/src/main/java/org/apache/nifi/wali/LengthDelimitedJournal.java
index 36350d0c8e1..9b9428539e4 100644
--- a/nifi-commons/nifi-write-ahead-log/src/main/java/org/apache/nifi/wali/LengthDelimitedJournal.java
+++ b/nifi-commons/nifi-write-ahead-log/src/main/java/org/apache/nifi/wali/LengthDelimitedJournal.java
@@ -475,8 +475,10 @@ public class LengthDelimitedJournal<T> implements WriteAheadJournal<T> {
switch (updateType) {
case DELETE: {
- idsRemoved.add(recordId);
- transactionRecordMap.remove(recordId);
+ if (recordId != null) {
+ idsRemoved.add(recordId);
+ transactionRecordMap.remove(recordId);
+ }
break;
}
case SWAP_IN: {
@@ -486,7 +488,9 @@ public class LengthDelimitedJournal<T> implements WriteAheadJournal<T> {
} else {
swapLocationsRemoved.add(location);
swapLocationsAdded.remove(location);
- transactionRecordMap.put(recordId, record);
+ if (recordId != null) {
+ transactionRecordMap.put(recordId, record);
+ }
}
break;
}
@@ -497,15 +501,40 @@ public class LengthDelimitedJournal<T> implements WriteAheadJournal<T> {
} else {
swapLocationsRemoved.remove(location);
swapLocationsAdded.add(location);
- idsRemoved.add(recordId);
- transactionRecordMap.remove(recordId);
+ if (recordId != null) {
+ idsRemoved.add(recordId);
+ transactionRecordMap.remove(recordId);
+ }
}
break;
}
+ case SWAP_FILE_DELETED: {
+ final String location = serde.getLocation(record);
+ if (location != null) {
+ swapLocationsRemoved.add(location);
+ swapLocationsAdded.remove(location);
+ }
+ break;
+ }
+ case SWAP_FILE_RENAMED: {
+ final String originalLocation = serde.getOriginalLocation(record);
+ final String newLocation = serde.getLocation(record);
+ if (originalLocation != null) {
+ swapLocationsRemoved.add(originalLocation);
+ swapLocationsAdded.remove(originalLocation);
+ }
+ if (newLocation != null) {
+ swapLocationsAdded.add(newLocation);
+ swapLocationsRemoved.remove(newLocation);
+ }
+ break;
+ }
default: {
- transactionRecordMap.put(recordId, record);
- idsRemoved.remove(recordId);
+ if (recordId != null) {
+ transactionRecordMap.put(recordId, record);
+ idsRemoved.remove(recordId);
+ }
break;
}
}
diff --git a/nifi-commons/nifi-write-ahead-log/src/main/java/org/wali/SerDe.java b/nifi-commons/nifi-write-ahead-log/src/main/java/org/wali/SerDe.java
index 356cf845733..990484bdf96 100644
--- a/nifi-commons/nifi-write-ahead-log/src/main/java/org/wali/SerDe.java
+++ b/nifi-commons/nifi-write-ahead-log/src/main/java/org/wali/SerDe.java
@@ -136,6 +136,19 @@ public interface SerDe<T> {
*/
String getLocation(T record);
+ /**
+ * Returns the original external location of the given record; this is used when a
+ * swap file is renamed. For {@link UpdateType#SWAP_FILE_RENAMED} records, this
+ * returns the original location before the rename, while {@link #getLocation(Object)}
+ * returns the new location after the rename.
+ *
+ * @param record to get original location of
+ * @return original location, or null if not applicable
+ */
+ default String getOriginalLocation(T record) {
+ return null;
+ }
+
/**
* Returns the version that this SerDe will use when writing. This used used
* when serializing/deserializing the edit logs so that if the version
diff --git a/nifi-commons/nifi-write-ahead-log/src/main/java/org/wali/SerDeFactory.java b/nifi-commons/nifi-write-ahead-log/src/main/java/org/wali/SerDeFactory.java
index 09e6f7b50c1..26a177ece88 100644
--- a/nifi-commons/nifi-write-ahead-log/src/main/java/org/wali/SerDeFactory.java
+++ b/nifi-commons/nifi-write-ahead-log/src/main/java/org/wali/SerDeFactory.java
@@ -57,4 +57,17 @@ public interface SerDeFactory<T> {
* @return location
*/
String getLocation(T record);
+
+ /**
+ * Returns the original external location of the given record; this is used when a
+ * swap file is renamed. For {@link UpdateType#SWAP_FILE_RENAMED} records, this
+ * returns the original location before the rename, while {@link #getLocation(Object)}
+ * returns the new location after the rename.
+ *
+ * @param record to get original location of
+ * @return original location, or null if not applicable
+ */
+ default String getOriginalLocation(T record) {
+ return null;
+ }
}
diff --git a/nifi-commons/nifi-write-ahead-log/src/main/java/org/wali/UpdateType.java b/nifi-commons/nifi-write-ahead-log/src/main/java/org/wali/UpdateType.java
index 1b039f82fcb..0d383cbe377 100644
--- a/nifi-commons/nifi-write-ahead-log/src/main/java/org/wali/UpdateType.java
+++ b/nifi-commons/nifi-write-ahead-log/src/main/java/org/wali/UpdateType.java
@@ -45,5 +45,13 @@ public enum UpdateType {
* Used to indicate that a Record that was previously Swapped Out is now
* being Swapped In
*/
- SWAP_IN;
+ SWAP_IN,
+ /**
+ * Used to indicate that a Swap File has been deleted
+ */
+ SWAP_FILE_DELETED,
+ /**
+ * Used to indicate that a Swap File has been renamed
+ */
+ SWAP_FILE_RENAMED;
}
diff --git a/nifi-commons/pom.xml b/nifi-commons/pom.xml
index eba472df95d..488b7ef209a 100644
--- a/nifi-commons/pom.xml
+++ b/nifi-commons/pom.xml
@@ -68,6 +68,7 @@
<module>nifi-write-ahead-log</module>
<module>nifi-xml-processing</module>
<module>nifi-calcite-utils</module>
+ <module>nifi-connector-utils</module>
</modules>
<properties>
<guava.version>33.5.0-jre</guava.version>
diff --git a/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/pom.xml b/nifi-connector-mock-bundle/nifi-connector-mock-api/pom.xml
similarity index 58%
copy from nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/pom.xml
copy to nifi-connector-mock-bundle/nifi-connector-mock-api/pom.xml
index f877db14d04..d43dffe3192 100644
--- a/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/pom.xml
+++ b/nifi-connector-mock-bundle/nifi-connector-mock-api/pom.xml
@@ -14,40 +14,22 @@
limitations under the License.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
<parent>
- <artifactId>nifi-system-test-extensions-bundle</artifactId>
<groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-connector-mock-bundle</artifactId>
<version>2.9.0-SNAPSHOT</version>
</parent>
- <modelVersion>4.0.0</modelVersion>
- <artifactId>nifi-system-test-extensions</artifactId>
+ <artifactId>nifi-connector-mock-api</artifactId>
+ <packaging>jar</packaging>
<dependencies>
<dependency>
<groupId>org.apache.nifi</groupId>
- <artifactId>nifi-stateless-api</artifactId>
+ <artifactId>nifi-server-api</artifactId>
<version>2.9.0-SNAPSHOT</version>
</dependency>
- <dependency>
- <groupId>org.apache.nifi</groupId>
- <artifactId>nifi-utils</artifactId>
- <version>2.9.0-SNAPSHOT</version>
- </dependency>
- <dependency>
- <groupId>org.apache.nifi</groupId>
- <artifactId>nifi-bin-manager</artifactId>
- <version>2.9.0-SNAPSHOT</version>
- </dependency>
- <dependency>
- <groupId>com.fasterxml.jackson.core</groupId>
- <artifactId>jackson-databind</artifactId>
- </dependency>
- <dependency>
- <groupId>org.apache.nifi</groupId>
- <artifactId>nifi-system-test-extensions-services-api</artifactId>
- <version>2.9.0-SNAPSHOT</version>
- <scope>provided</scope>
- </dependency>
</dependencies>
+
</project>
\ No newline at end of file
diff --git a/nifi-manifest/nifi-extension-manifest-model/src/main/java/org/apache/nifi/extension/manifest/ExtensionType.java b/nifi-connector-mock-bundle/nifi-connector-mock-api/src/main/java/org/apache/nifi/mock/connector/server/ConnectorConfigVerificationResult.java
similarity index 71%
copy from nifi-manifest/nifi-extension-manifest-model/src/main/java/org/apache/nifi/extension/manifest/ExtensionType.java
copy to nifi-connector-mock-bundle/nifi-connector-mock-api/src/main/java/org/apache/nifi/mock/connector/server/ConnectorConfigVerificationResult.java
index 0413e260ed1..9ec3ac2878c 100644
--- a/nifi-manifest/nifi-extension-manifest-model/src/main/java/org/apache/nifi/extension/manifest/ExtensionType.java
+++ b/nifi-connector-mock-bundle/nifi-connector-mock-api/src/main/java/org/apache/nifi/mock/connector/server/ConnectorConfigVerificationResult.java
@@ -14,23 +14,18 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.nifi.extension.manifest;
-/**
- * Possible types of extensions.
- */
-public enum ExtensionType {
-
- PROCESSOR,
+package org.apache.nifi.mock.connector.server;
- CONTROLLER_SERVICE,
+import org.apache.nifi.components.ConfigVerificationResult;
- REPORTING_TASK,
+import java.util.List;
- FLOW_ANALYSIS_RULE,
+public interface ConnectorConfigVerificationResult {
- PARAMETER_PROVIDER,
+ List<ConfigVerificationResult> getAllResults();
- FLOW_REGISTRY_CLIENT;
+ List<ConfigVerificationResult> getFailedResults();
+ void assertNoFailures();
}
diff --git a/nifi-manifest/nifi-extension-manifest-model/src/main/java/org/apache/nifi/extension/manifest/ExtensionType.java b/nifi-connector-mock-bundle/nifi-connector-mock-api/src/main/java/org/apache/nifi/mock/connector/server/ConnectorMockServer.java
similarity index 56%
copy from nifi-manifest/nifi-extension-manifest-model/src/main/java/org/apache/nifi/extension/manifest/ExtensionType.java
copy to nifi-connector-mock-bundle/nifi-connector-mock-api/src/main/java/org/apache/nifi/mock/connector/server/ConnectorMockServer.java
index 0413e260ed1..ade061a9d47 100644
--- a/nifi-manifest/nifi-extension-manifest-model/src/main/java/org/apache/nifi/extension/manifest/ExtensionType.java
+++ b/nifi-connector-mock-bundle/nifi-connector-mock-api/src/main/java/org/apache/nifi/mock/connector/server/ConnectorMockServer.java
@@ -14,23 +14,23 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.nifi.extension.manifest;
-/**
- * Possible types of extensions.
- */
-public enum ExtensionType {
+package org.apache.nifi.mock.connector.server;
+
+import org.apache.nifi.NiFiServer;
+import org.apache.nifi.controller.ControllerService;
+import org.apache.nifi.processor.Processor;
- PROCESSOR,
+import java.io.File;
- CONTROLLER_SERVICE,
+public interface ConnectorMockServer extends NiFiServer, ConnectorTestRunner {
- REPORTING_TASK,
+ void instantiateConnector(String connectorClassName);
- FLOW_ANALYSIS_RULE,
+ void registerMockBundle(ClassLoader classLoader, File workingDirectory);
- PARAMETER_PROVIDER,
+ void mockProcessor(String processorType, Class<? extends Processor> mockProcessorClass);
- FLOW_REGISTRY_CLIENT;
+ void mockControllerService(String controllerServiceType, Class<? extends ControllerService> mockControllerServiceClass);
}
diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-api/src/main/java/org/apache/nifi/mock/connector/server/ConnectorTestRunner.java b/nifi-connector-mock-bundle/nifi-connector-mock-api/src/main/java/org/apache/nifi/mock/connector/server/ConnectorTestRunner.java
new file mode 100644
index 00000000000..669f7be22d2
--- /dev/null
+++ b/nifi-connector-mock-bundle/nifi-connector-mock-api/src/main/java/org/apache/nifi/mock/connector/server/ConnectorTestRunner.java
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.mock.connector.server;
+
+import org.apache.nifi.components.DescribedValue;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.connector.AssetReference;
+import org.apache.nifi.components.connector.ConnectorValueReference;
+import org.apache.nifi.components.connector.FlowUpdateException;
+import org.apache.nifi.components.connector.SecretReference;
+import org.apache.nifi.components.connector.StepConfiguration;
+import org.apache.nifi.flow.VersionedExternalFlow;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.InputStream;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+
+public interface ConnectorTestRunner extends Closeable {
+ String SECRET_PROVIDER_ID = "TestRunnerSecretsManager";
+ String SECRET_PROVIDER_NAME = "TestRunnerSecretsManager";
+
+ void applyUpdate() throws FlowUpdateException;
+
+ void configure(String stepName, StepConfiguration configuration) throws FlowUpdateException;
+
+ void configure(String stepName, Map<String, String> propertyValues) throws FlowUpdateException;
+
+ void configure(String stepName, Map<String, String> propertyValues, Map<String, ConnectorValueReference> propertyReferences) throws FlowUpdateException;
+
+ SecretReference createSecretReference(String secretName);
+
+ ConnectorConfigVerificationResult verifyConfiguration(String stepName, Map<String, String> propertyValueOverrides);
+
+ ConnectorConfigVerificationResult verifyConfiguration(String stepName, Map<String, String> propertyValueOverrides, Map<String, ConnectorValueReference> referenceOverrides);
+
+ ConnectorConfigVerificationResult verifyConfiguration(String stepName, StepConfiguration configurationOverrides);
+
+ void addSecret(String name, String value);
+
+ AssetReference addAsset(File file);
+
+ AssetReference addAsset(String assetName, InputStream contents);
+
+ void startConnector();
+
+ void stopConnector();
+
+ void waitForDataIngested(Duration maxWaitTime);
+
+ void waitForIdle(Duration maxWaitTime);
+
+ void waitForIdle(Duration minimumIdleTime, Duration maxWaitTime);
+
+ List<ValidationResult> validate();
+
+ /**
+ * Returns the HTTP port on which the embedded Jetty server is listening, or -1 if no server is running.
+ *
+ * @return the HTTP port, or -1 if not applicable
+ */
+ default int getHttpPort() {
+ return -1;
+ }
+
+ List<DescribedValue> fetchAllowableValues(String stepName, String propertyName);
+
+ /*
+ * Returns a {@link VersionedExternalFlow} representing the current state of the Active Flow Context.
+ * The Active Flow Context is the flow that is currently running (or most recently ran) in the Connector.
+ * This is useful for making assertions about how the flow is configured after updates have been applied.
+ *
+ * @return the VersionedExternalFlow for the Active Flow Context
+ */
+ VersionedExternalFlow getActiveFlowSnapshot();
+
+ /**
+ * Returns a {@link VersionedExternalFlow} representing the current state of the Working Flow Context.
+ * The Working Flow Context is the flow that reflects configuration changes that have been made
+ * but not yet applied. This is useful for making assertions about how the flow will be configured
+ * once the update is applied.
+ *
+ * @return the VersionedExternalFlow for the Working Flow Context
+ */
+ VersionedExternalFlow getWorkingFlowSnapshot();
+}
diff --git a/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/pom.xml b/nifi-connector-mock-bundle/nifi-connector-mock-server-nar/pom.xml
similarity index 63%
copy from nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/pom.xml
copy to nifi-connector-mock-bundle/nifi-connector-mock-server-nar/pom.xml
index f877db14d04..4b970dc4867 100644
--- a/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/pom.xml
+++ b/nifi-connector-mock-bundle/nifi-connector-mock-server-nar/pom.xml
@@ -14,40 +14,42 @@
limitations under the License.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
<parent>
- <artifactId>nifi-system-test-extensions-bundle</artifactId>
<groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-connector-mock-bundle</artifactId>
<version>2.9.0-SNAPSHOT</version>
</parent>
- <modelVersion>4.0.0</modelVersion>
- <artifactId>nifi-system-test-extensions</artifactId>
+ <artifactId>nifi-connector-mock-server-nar</artifactId>
+ <packaging>nar</packaging>
<dependencies>
<dependency>
<groupId>org.apache.nifi</groupId>
- <artifactId>nifi-stateless-api</artifactId>
+ <artifactId>nifi-connector-mock-server</artifactId>
<version>2.9.0-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
- <artifactId>nifi-utils</artifactId>
+ <artifactId>nifi-framework-nar</artifactId>
<version>2.9.0-SNAPSHOT</version>
- </dependency>
- <dependency>
- <groupId>org.apache.nifi</groupId>
- <artifactId>nifi-bin-manager</artifactId>
- <version>2.9.0-SNAPSHOT</version>
- </dependency>
- <dependency>
- <groupId>com.fasterxml.jackson.core</groupId>
- <artifactId>jackson-databind</artifactId>
- </dependency>
- <dependency>
- <groupId>org.apache.nifi</groupId>
- <artifactId>nifi-system-test-extensions-services-api</artifactId>
- <version>2.9.0-SNAPSHOT</version>
- <scope>provided</scope>
+ <type>nar</type>
</dependency>
</dependencies>
+
+ <repositories>
+ <!-- Shibboleth Repository required for OpenSAML -->
+ <!-- This is required because nifi-framework-nar is the parent/NAR Dependency, andi t requires shibboleth. -->
+ <repository>
+ <id>shibboleth</id>
+ <url>https://build.shibboleth.net/nexus/content/repositories/releases/</url>
+ <releases>
+ <enabled>true</enabled>
+ </releases>
+ <snapshots>
+ <enabled>false</enabled>
+ </snapshots>
+ </repository>
+ </repositories>
</project>
\ No newline at end of file
diff --git a/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/pom.xml b/nifi-connector-mock-bundle/nifi-connector-mock-server/pom.xml
similarity index 57%
copy from nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/pom.xml
copy to nifi-connector-mock-bundle/nifi-connector-mock-server/pom.xml
index f877db14d04..bcfb88f3e5f 100644
--- a/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/pom.xml
+++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/pom.xml
@@ -14,40 +14,67 @@
limitations under the License.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
<parent>
- <artifactId>nifi-system-test-extensions-bundle</artifactId>
<groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-connector-mock-bundle</artifactId>
<version>2.9.0-SNAPSHOT</version>
</parent>
- <modelVersion>4.0.0</modelVersion>
- <artifactId>nifi-system-test-extensions</artifactId>
+ <artifactId>nifi-connector-mock-server</artifactId>
+ <packaging>jar</packaging>
<dependencies>
+ <dependency>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-api</artifactId>
+ </dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
- <artifactId>nifi-stateless-api</artifactId>
+ <artifactId>nifi-framework-core-api</artifactId>
<version>2.9.0-SNAPSHOT</version>
+ <scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
- <artifactId>nifi-utils</artifactId>
+ <artifactId>nifi-framework-core</artifactId>
<version>2.9.0-SNAPSHOT</version>
+ <scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
- <artifactId>nifi-bin-manager</artifactId>
+ <artifactId>nifi-property-encryptor</artifactId>
<version>2.9.0-SNAPSHOT</version>
</dependency>
<dependency>
- <groupId>com.fasterxml.jackson.core</groupId>
- <artifactId>jackson-databind</artifactId>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-connector-mock-api</artifactId>
+ <version>2.9.0-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
- <artifactId>nifi-system-test-extensions-services-api</artifactId>
+ <artifactId>nifi-server-api</artifactId>
<version>2.9.0-SNAPSHOT</version>
- <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.eclipse.jetty</groupId>
+ <artifactId>jetty-server</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.eclipse.jetty.ee11</groupId>
+ <artifactId>jetty-ee11-webapp</artifactId>
+ </dependency>
+
+ <!-- Test Dependencies -->
+ <dependency>
+ <groupId>org.junit.jupiter</groupId>
+ <artifactId>junit-jupiter</artifactId>
+ <scope>test</scope>
</dependency>
</dependencies>
+
</project>
\ No newline at end of file
diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockAuditService.java b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockAuditService.java
new file mode 100644
index 00000000000..3aa7bbc543e
--- /dev/null
+++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockAuditService.java
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.mock.connector.server;
+
+import org.apache.nifi.action.Action;
+import org.apache.nifi.admin.service.AuditService;
+import org.apache.nifi.history.History;
+import org.apache.nifi.history.HistoryQuery;
+import org.apache.nifi.history.PreviousValue;
+
+import java.util.Collection;
+import java.util.Date;
+import java.util.List;
+import java.util.Map;
+
+public class MockAuditService implements AuditService {
+ @Override
+ public void addActions(final Collection<Action> actions) {
+ }
+
+ @Override
+ public Map<String, List<PreviousValue>> getPreviousValues(final String componentId) {
+ return Map.of();
+ }
+
+ @Override
+ public void deletePreviousValues(final String propertyName, final String componentId) {
+ }
+
+ @Override
+ public History getActions(final HistoryQuery actionQuery) {
+ return null;
+ }
+
+ @Override
+ public History getActions(final int firstActionId, final int maxActions) {
+ return null;
+ }
+
+ @Override
+ public Action getAction(final Integer actionId) {
+ return null;
+ }
+
+ @Override
+ public void purgeActions(final Date end, final Action purgeAction) {
+ }
+}
diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockConnectorAssetManager.java b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockConnectorAssetManager.java
new file mode 100644
index 00000000000..92892f6275a
--- /dev/null
+++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockConnectorAssetManager.java
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.mock.connector.server;
+
+import org.apache.nifi.asset.Asset;
+import org.apache.nifi.asset.AssetManager;
+import org.apache.nifi.asset.AssetManagerInitializationContext;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.StandardCopyOption;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * A Mock implementation of AssetManager for use in ConnectorTestRunner tests.
+ * This implementation stores assets in a temporary directory structure.
+ */
+public class MockConnectorAssetManager implements AssetManager {
+
+ private static final String ASSET_STORAGE_LOCATION_PROPERTY = "directory";
+ private static final String DEFAULT_ASSET_STORAGE_LOCATION = "target/mock_connector_assets";
+
+ private final Map<String, Asset> assets = new ConcurrentHashMap<>();
+ private volatile File assetStorageLocation;
+
+ @Override
+ public void initialize(final AssetManagerInitializationContext context) {
+ final String storageLocation = context.getProperties().getOrDefault(ASSET_STORAGE_LOCATION_PROPERTY, DEFAULT_ASSET_STORAGE_LOCATION);
+ assetStorageLocation = new File(storageLocation);
+
+ if (!assetStorageLocation.exists()) {
+ try {
+ Files.createDirectories(assetStorageLocation.toPath());
+ } catch (final IOException e) {
+ throw new RuntimeException("Failed to create asset storage directory: " + storageLocation, e);
+ }
+ }
+ }
+
+ @Override
+ public Asset createAsset(final String ownerId, final String assetName, final InputStream contents) throws IOException {
+ final String assetId = UUID.randomUUID().toString();
+ return saveAsset(ownerId, assetId, assetName, contents);
+ }
+
+ @Override
+ public Asset saveAsset(final String ownerId, final String assetId, final String assetName, final InputStream contents) throws IOException {
+ final File assetFile = getFile(ownerId, assetId, assetName);
+ final File parentDir = assetFile.getParentFile();
+
+ if (!parentDir.exists()) {
+ Files.createDirectories(parentDir.toPath());
+ }
+
+ Files.copy(contents, assetFile.toPath(), StandardCopyOption.REPLACE_EXISTING);
+
+ final Asset asset = new MockAsset(assetId, ownerId, assetName, assetFile, null);
+ assets.put(assetId, asset);
+ return asset;
+ }
+
+ @Override
+ public Optional<Asset> getAsset(final String id) {
+ return Optional.ofNullable(assets.get(id));
+ }
+
+ @Override
+ public List<Asset> getAssets(final String ownerId) {
+ final List<Asset> ownerAssets = new ArrayList<>();
+ for (final Asset asset : assets.values()) {
+ if (asset.getOwnerIdentifier().equals(ownerId)) {
+ ownerAssets.add(asset);
+ }
+ }
+ return ownerAssets;
+ }
+
+ @Override
+ public Asset createMissingAsset(final String ownerId, final String assetName) {
+ final String assetId = UUID.randomUUID().toString();
+ final File file = getFile(ownerId, assetId, assetName);
+ final Asset asset = new MockAsset(assetId, ownerId, assetName, file, null);
+ assets.put(assetId, asset);
+ return asset;
+ }
+
+ @Override
+ public Optional<Asset> deleteAsset(final String id) {
+ final Asset removed = assets.remove(id);
+ if (removed != null && removed.getFile().exists()) {
+ try {
+ Files.delete(removed.getFile().toPath());
+ } catch (final IOException e) {
+ throw new RuntimeException("Failed to delete asset " + id + " from storage file " + removed.getFile().getAbsolutePath(), e);
+ }
+ }
+
+ return Optional.ofNullable(removed);
+ }
+
+ private File getFile(final String ownerId, final String assetId, final String assetName) {
+ final Path parentPath = assetStorageLocation.toPath().normalize();
+ final Path assetPath = Path.of(ownerId, assetId, assetName).normalize();
+ return parentPath.resolve(assetPath).toFile();
+ }
+
+ /**
+ * A simple Asset implementation for the mock.
+ */
+ private static class MockAsset implements Asset {
+ private final String identifier;
+ private final String ownerIdentifier;
+ private final String name;
+ private final File file;
+ private final String digest;
+
+ MockAsset(final String identifier, final String ownerIdentifier, final String name, final File file, final String digest) {
+ this.identifier = identifier;
+ this.ownerIdentifier = ownerIdentifier;
+ this.name = name;
+ this.file = file;
+ this.digest = digest;
+ }
+
+ @Override
+ public String getIdentifier() {
+ return identifier;
+ }
+
+ @Override
+ @Deprecated
+ public String getParameterContextIdentifier() {
+ return ownerIdentifier;
+ }
+
+ @Override
+ public String getOwnerIdentifier() {
+ return ownerIdentifier;
+ }
+
+ @Override
+ public String getName() {
+ return name;
+ }
+
+ @Override
+ public File getFile() {
+ return file;
+ }
+
+ @Override
+ public Optional<String> getDigest() {
+ return Optional.ofNullable(digest);
+ }
+ }
+}
diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockConnectorInitializationContext.java b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockConnectorInitializationContext.java
new file mode 100644
index 00000000000..02d276bc28d
--- /dev/null
+++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockConnectorInitializationContext.java
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.mock.connector.server;
+
+import org.apache.nifi.components.connector.BundleCompatibility;
+import org.apache.nifi.components.connector.FlowUpdateException;
+import org.apache.nifi.components.connector.StandardConnectorInitializationContext;
+import org.apache.nifi.components.connector.components.FlowContext;
+import org.apache.nifi.flow.VersionedControllerService;
+import org.apache.nifi.flow.VersionedExternalFlow;
+import org.apache.nifi.flow.VersionedProcessGroup;
+import org.apache.nifi.flow.VersionedProcessor;
+
+public class MockConnectorInitializationContext extends StandardConnectorInitializationContext {
+
+ private final MockExtensionMapper mockExtensionMapper;
+
+ protected MockConnectorInitializationContext(final Builder builder) {
+ super(builder);
+ this.mockExtensionMapper = builder.mockExtensionMapper;
+ }
+
+ @Override
+ public void updateFlow(final FlowContext flowContext, final VersionedExternalFlow versionedExternalFlow,
+ final BundleCompatibility bundleCompatability) throws FlowUpdateException {
+ replaceMocks(versionedExternalFlow.getFlowContents());
+ super.updateFlow(flowContext, versionedExternalFlow, bundleCompatability);
+ }
+
+ private void replaceMocks(final VersionedProcessGroup group) {
+ if (group.getProcessors() != null) {
+ for (final VersionedProcessor processor : group.getProcessors()) {
+ mockExtensionMapper.mapProcessor(processor);
+ }
+ }
+
+ if (group.getControllerServices() != null) {
+ for (final VersionedControllerService controllerService : group.getControllerServices()) {
+ mockExtensionMapper.mapControllerService(controllerService);
+ }
+ }
+
+ if (group.getProcessGroups() != null) {
+ for (final VersionedProcessGroup childGroup : group.getProcessGroups()) {
+ replaceMocks(childGroup);
+ }
+ }
+ }
+
+
+ public static class Builder extends StandardConnectorInitializationContext.Builder {
+ private final MockExtensionMapper mockExtensionMapper;
+
+ public Builder(final MockExtensionMapper mockExtensionMapper) {
+ this.mockExtensionMapper = mockExtensionMapper;
+ }
+
+ @Override
+ public MockConnectorInitializationContext build() {
+ return new MockConnectorInitializationContext(this);
+ }
+ }
+}
diff --git a/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/java/org/apache/nifi/processors/tests/system/TerminateFlowFile.java b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockConnectorRepository.java
similarity index 55%
copy from nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/java/org/apache/nifi/processors/tests/system/TerminateFlowFile.java
copy to nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockConnectorRepository.java
index 734766fa37e..9f951116747 100644
--- a/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/java/org/apache/nifi/processors/tests/system/TerminateFlowFile.java
+++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockConnectorRepository.java
@@ -14,22 +14,22 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.nifi.processors.tests.system;
-import org.apache.nifi.flowfile.FlowFile;
-import org.apache.nifi.processor.AbstractProcessor;
-import org.apache.nifi.processor.ProcessContext;
-import org.apache.nifi.processor.ProcessSession;
-import org.apache.nifi.processor.exception.ProcessException;
+package org.apache.nifi.mock.connector.server;
-public class TerminateFlowFile extends AbstractProcessor {
- @Override
- public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
- FlowFile flowFile = session.get();
- if (flowFile == null) {
- return;
- }
+import org.apache.nifi.components.connector.FrameworkConnectorInitializationContextBuilder;
+import org.apache.nifi.components.connector.StandardConnectorRepository;
+
+public class MockConnectorRepository extends StandardConnectorRepository {
+
+ private volatile MockExtensionMapper mockExtensionMapper;
- session.remove(flowFile);
+ public void setMockExtensionMapper(final MockExtensionMapper mapper) {
+ mockExtensionMapper = mapper;
+ }
+
+ @Override
+ public FrameworkConnectorInitializationContextBuilder createInitializationContextBuilder() {
+ return new MockConnectorInitializationContext.Builder(mockExtensionMapper);
}
}
diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockExtensionDiscoveringManager.java b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockExtensionDiscoveringManager.java
new file mode 100644
index 00000000000..22b2ca043ce
--- /dev/null
+++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockExtensionDiscoveringManager.java
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.mock.connector.server;
+
+import org.apache.nifi.bundle.Bundle;
+import org.apache.nifi.bundle.BundleCoordinate;
+import org.apache.nifi.bundle.BundleDetails;
+import org.apache.nifi.controller.ControllerService;
+import org.apache.nifi.nar.InstanceClassLoader;
+import org.apache.nifi.nar.StandardExtensionDiscoveringManager;
+import org.apache.nifi.processor.Processor;
+
+import java.io.File;
+import java.net.URL;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+
+public class MockExtensionDiscoveringManager extends StandardExtensionDiscoveringManager {
+ private final ConcurrentMap<String, ClassLoader> mockComponentClassLoaders = new ConcurrentHashMap<>();
+
+ public synchronized void addProcessor(final Class<? extends Processor> mockProcessorClass) {
+ final BundleDetails bundleDetails = new BundleDetails.Builder()
+ .workingDir(new File("target/work/extensions/mock-bundle"))
+ .coordinate(new BundleCoordinate("org.apache.nifi.mock", mockProcessorClass.getName(), "1.0.0"))
+ .build();
+
+ final Bundle mockBundle = new Bundle(bundleDetails, mockProcessorClass.getClassLoader());
+ discoverExtensions(Set.of(mockBundle));
+
+ mockComponentClassLoaders.put(mockProcessorClass.getName(), mockProcessorClass.getClassLoader());
+ registerExtensionClass(Processor.class, mockProcessorClass.getName(), mockBundle);
+ }
+
+ public synchronized void addControllerService(final Class<? extends ControllerService> mockControllerServiceClass) {
+ final BundleDetails bundleDetails = new BundleDetails.Builder()
+ .workingDir(new File("target/work/extensions/mock-bundle"))
+ .coordinate(new BundleCoordinate("org.apache.nifi.mock", mockControllerServiceClass.getName(), "1.0.0"))
+ .build();
+
+ final Bundle mockBundle = new Bundle(bundleDetails, mockControllerServiceClass.getClassLoader());
+ discoverExtensions(Set.of(mockBundle));
+
+ mockComponentClassLoaders.put(mockControllerServiceClass.getName(), mockControllerServiceClass.getClassLoader());
+ registerExtensionClass(ControllerService.class, mockControllerServiceClass.getName(), mockBundle);
+ }
+
+ @Override
+ public InstanceClassLoader createInstanceClassLoader(final String classType, final String instanceIdentifier, final Bundle bundle, final Set<URL> additionalUrls,
+ final boolean register, final String classloaderIsolationKey) {
+
+ final ClassLoader classLoader = mockComponentClassLoaders.get(classType);
+ if (classLoader != null) {
+ return new InstanceClassLoader(instanceIdentifier, classType, additionalUrls, Set.of(), classLoader);
+ }
+
+ return super.createInstanceClassLoader(classType, instanceIdentifier, bundle, additionalUrls, register, classloaderIsolationKey);
+ }
+}
diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockExtensionMapper.java b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockExtensionMapper.java
new file mode 100644
index 00000000000..edf81cdeaac
--- /dev/null
+++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockExtensionMapper.java
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.mock.connector.server;
+
+import org.apache.nifi.flow.Bundle;
+import org.apache.nifi.flow.VersionedControllerService;
+import org.apache.nifi.flow.VersionedProcessor;
+
+import java.util.HashMap;
+import java.util.Map;
+
+public class MockExtensionMapper {
+
+ private final Map<String, String> processorMocks = new HashMap<>();
+ private final Map<String, String> controllerServiceMocks = new HashMap<>();
+
+ public void mockProcessor(final String processorType, final String mockProcessorClassName) {
+ processorMocks.put(processorType, mockProcessorClassName);
+ }
+
+ public void mockControllerService(final String controllerServiceType, final String mockControllerServiceClassName) {
+ controllerServiceMocks.put(controllerServiceType, mockControllerServiceClassName);
+ }
+
+ public void mapProcessor(final VersionedProcessor processor) {
+ final String type = processor.getType();
+ final String implementationClassName = processorMocks.get(type);
+ if (implementationClassName == null) {
+ return;
+ }
+
+ processor.setType(implementationClassName);
+ processor.setBundle(new Bundle("org.apache.nifi.mock", implementationClassName, "1.0.0"));
+ }
+
+ public void mapControllerService(final VersionedControllerService controllerService) {
+ final String type = controllerService.getType();
+ final String implementationClassName = controllerServiceMocks.get(type);
+ if (implementationClassName == null) {
+ return;
+ }
+
+ controllerService.setType(implementationClassName);
+ controllerService.setBundle(new Bundle("org.apache.nifi.mock", implementationClassName, "1.0.0"));
+ }
+}
diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockNiFiConnectorWebContext.java b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockNiFiConnectorWebContext.java
new file mode 100644
index 00000000000..9e8891ee527
--- /dev/null
+++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockNiFiConnectorWebContext.java
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.mock.connector.server;
+
+import org.apache.nifi.components.connector.ConnectorNode;
+import org.apache.nifi.components.connector.ConnectorRepository;
+import org.apache.nifi.components.connector.components.FlowContext;
+import org.apache.nifi.web.NiFiConnectorWebContext;
+
+/**
+ * Mock implementation of {@link NiFiConnectorWebContext} for the connector test runner.
+ * Provides direct access to the connector instance and its flow contexts without
+ * authorization proxying, since the mock server uses a permit-all authorizer.
+ */
+public class MockNiFiConnectorWebContext implements NiFiConnectorWebContext {
+
+ private final ConnectorRepository connectorRepository;
+
+ public MockNiFiConnectorWebContext(final ConnectorRepository connectorRepository) {
+ this.connectorRepository = connectorRepository;
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ public <T> ConnectorWebContext<T> getConnectorWebContext(final String connectorId) throws IllegalArgumentException {
+ final ConnectorNode connectorNode = connectorRepository.getConnector(connectorId);
+ if (connectorNode == null) {
+ throw new IllegalArgumentException("Unable to find connector with id: " + connectorId);
+ }
+
+ final T connector = (T) connectorNode.getConnector();
+ final FlowContext workingFlowContext = connectorNode.getWorkingFlowContext();
+ final FlowContext activeFlowContext = connectorNode.getActiveFlowContext();
+
+ return new ConnectorWebContext<>(connector, workingFlowContext, activeFlowContext);
+ }
+}
diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockRuleViolationsManager.java b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockRuleViolationsManager.java
new file mode 100644
index 00000000000..bb0e2636ea5
--- /dev/null
+++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockRuleViolationsManager.java
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.mock.connector.server;
+
+import org.apache.nifi.flow.VersionedComponent;
+import org.apache.nifi.flow.VersionedProcessGroup;
+import org.apache.nifi.validation.RuleViolation;
+import org.apache.nifi.validation.RuleViolationsManager;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+
+public class MockRuleViolationsManager implements RuleViolationsManager {
+ @Override
+ public void upsertComponentViolations(final String subjectId, final Collection<RuleViolation> violations) {
+
+ }
+
+ @Override
+ public void upsertGroupViolations(final VersionedProcessGroup processGroup, final Collection<RuleViolation> violations,
+ final Map<VersionedComponent, Collection<RuleViolation>> componentToRuleViolations) {
+ }
+
+ @Override
+ public Collection<RuleViolation> getRuleViolationsForSubject(final String subjectId) {
+ return List.of();
+ }
+
+ @Override
+ public Collection<RuleViolation> getRuleViolationsForGroup(final String groupId) {
+ return List.of();
+ }
+
+ @Override
+ public Collection<RuleViolation> getRuleViolationsForGroups(final Collection<String> groupIds) {
+ return List.of();
+ }
+
+ @Override
+ public Collection<RuleViolation> getAllRuleViolations() {
+ return List.of();
+ }
+
+ @Override
+ public void removeRuleViolationsForSubject(final String subjectId) {
+
+ }
+
+ @Override
+ public void removeRuleViolationsForRule(final String ruleId) {
+
+ }
+
+ @Override
+ public void cleanUp() {
+
+ }
+
+ @Override
+ public boolean isEmpty() {
+ return false;
+ }
+}
diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockServerConfigVerificationResult.java b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockServerConfigVerificationResult.java
new file mode 100644
index 00000000000..fed82f92902
--- /dev/null
+++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockServerConfigVerificationResult.java
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.mock.connector.server;
+
+import org.apache.nifi.components.ConfigVerificationResult;
+import org.apache.nifi.components.ConfigVerificationResult.Outcome;
+
+import java.util.Collections;
+import java.util.List;
+
+public class MockServerConfigVerificationResult implements ConnectorConfigVerificationResult {
+ private final List<ConfigVerificationResult> results;
+
+ public MockServerConfigVerificationResult(final List<ConfigVerificationResult> results) {
+ this.results = results;
+ }
+
+ @Override
+ public List<ConfigVerificationResult> getAllResults() {
+ return Collections.unmodifiableList(results);
+ }
+
+ @Override
+ public List<ConfigVerificationResult> getFailedResults() {
+ return results.stream()
+ .filter(result -> result.getOutcome() == Outcome.FAILED)
+ .toList();
+ }
+
+ @Override
+ public void assertNoFailures() {
+ final List<ConfigVerificationResult> failedResults = getFailedResults();
+ if (!failedResults.isEmpty()) {
+ throw new AssertionError("Configuration verification failed: " + failedResults);
+ }
+ }
+}
diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockStateManager.java b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockStateManager.java
new file mode 100644
index 00000000000..f22269ceabf
--- /dev/null
+++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockStateManager.java
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.mock.connector.server;
+
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateManager;
+import org.apache.nifi.components.state.StateMap;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicLong;
+
+public class MockStateManager implements StateManager {
+ private final AtomicLong versionCounter = new AtomicLong(0L);
+ private final boolean dropStateKeySupported;
+
+ private volatile StateMap localStateMap = new MockStateMap(null, -1L);
+ private volatile StateMap clusterStateMap = new MockStateMap(null, -1L);
+
+ public MockStateManager(final boolean dropStateKeySupported) {
+ this.dropStateKeySupported = dropStateKeySupported;
+ }
+
+ @Override
+ public synchronized void setState(final Map<String, String> state, final Scope scope) throws IOException {
+ final StateMap newStateMap = new MockStateMap(state, versionCounter.incrementAndGet());
+
+ if (scope == Scope.CLUSTER) {
+ clusterStateMap = newStateMap;
+ } else {
+ localStateMap = newStateMap;
+ }
+ }
+
+ @Override
+ public synchronized StateMap getState(final Scope scope) throws IOException {
+ if (scope == Scope.CLUSTER) {
+ return clusterStateMap;
+ } else {
+ return localStateMap;
+ }
+ }
+
+ @Override
+ public synchronized boolean replace(final StateMap oldValue, final Map<String, String> newValue, final Scope scope) throws IOException {
+ final StateMap currentStateMap = scope == Scope.CLUSTER ? clusterStateMap : localStateMap;
+
+ if (currentStateMap == oldValue) {
+ final StateMap newStateMap = new MockStateMap(newValue, versionCounter.incrementAndGet());
+
+ if (scope == Scope.CLUSTER) {
+ clusterStateMap = newStateMap;
+ } else {
+ localStateMap = newStateMap;
+ }
+
+ return true;
+ }
+
+ return false;
+ }
+
+ @Override
+ public synchronized void clear(final Scope scope) throws IOException {
+ setState(Collections.emptyMap(), scope);
+ }
+
+ @Override
+ public boolean isStateKeyDropSupported() {
+ return dropStateKeySupported;
+ }
+}
+
diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockStateManagerProvider.java b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockStateManagerProvider.java
new file mode 100644
index 00000000000..77484904f25
--- /dev/null
+++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockStateManagerProvider.java
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.mock.connector.server;
+
+import org.apache.nifi.components.state.StateManager;
+import org.apache.nifi.components.state.StateManagerProvider;
+
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+
+public class MockStateManagerProvider implements StateManagerProvider {
+ private final ConcurrentMap<String, StateManager> stateManagers = new ConcurrentHashMap<>();
+ private volatile boolean clusterProviderEnabled = false;
+
+ @Override
+ public StateManager getStateManager(final String componentId, final boolean dropStateKeySupported) {
+ return stateManagers.computeIfAbsent(componentId, id -> new MockStateManager(dropStateKeySupported));
+ }
+
+ @Override
+ public void onComponentRemoved(final String componentId) {
+ stateManagers.remove(componentId);
+ }
+
+ @Override
+ public void shutdown() {
+ stateManagers.clear();
+ }
+
+ @Override
+ public void enableClusterProvider() {
+ clusterProviderEnabled = true;
+ }
+
+ @Override
+ public void disableClusterProvider() {
+ clusterProviderEnabled = false;
+ }
+
+ @Override
+ public boolean isClusterProviderEnabled() {
+ return clusterProviderEnabled;
+ }
+}
diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockStateMap.java b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockStateMap.java
new file mode 100644
index 00000000000..54b0980d616
--- /dev/null
+++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockStateMap.java
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.mock.connector.server;
+
+import org.apache.nifi.components.state.StateMap;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+
+public class MockStateMap implements StateMap {
+ private final Map<String, String> stateValues;
+ private final long version;
+
+ public MockStateMap(final Map<String, String> stateValues, final long version) {
+ this.stateValues = stateValues == null ? Collections.emptyMap() : new HashMap<>(stateValues);
+ this.version = version;
+ }
+
+ @Override
+ public Optional<String> getStateVersion() {
+ return version == -1L ? Optional.empty() : Optional.of(Long.toString(version));
+ }
+
+ @Override
+ public String get(final String key) {
+ return stateValues.get(key);
+ }
+
+ @Override
+ public Map<String, String> toMap() {
+ return Collections.unmodifiableMap(stateValues);
+ }
+
+ @Override
+ public String toString() {
+ return "MockStateMap[version=" + version + ", values=" + stateValues + "]";
+ }
+}
+
diff --git a/nifi-manifest/nifi-extension-manifest-model/src/main/java/org/apache/nifi/extension/manifest/ExtensionType.java b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/NopPropertyEncryptor.java
similarity index 68%
copy from nifi-manifest/nifi-extension-manifest-model/src/main/java/org/apache/nifi/extension/manifest/ExtensionType.java
copy to nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/NopPropertyEncryptor.java
index 0413e260ed1..56fbe816b7f 100644
--- a/nifi-manifest/nifi-extension-manifest-model/src/main/java/org/apache/nifi/extension/manifest/ExtensionType.java
+++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/NopPropertyEncryptor.java
@@ -14,23 +14,19 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.nifi.extension.manifest;
-/**
- * Possible types of extensions.
- */
-public enum ExtensionType {
-
- PROCESSOR,
-
- CONTROLLER_SERVICE,
-
- REPORTING_TASK,
-
- FLOW_ANALYSIS_RULE,
+package org.apache.nifi.mock.connector.server;
- PARAMETER_PROVIDER,
+import org.apache.nifi.encrypt.PropertyEncryptor;
- FLOW_REGISTRY_CLIENT;
+public class NopPropertyEncryptor implements PropertyEncryptor {
+ @Override
+ public String encrypt(final String property) {
+ return property;
+ }
+ @Override
+ public String decrypt(final String encryptedProperty) {
+ return encryptedProperty;
+ }
}
diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/PermitAllAuthorizer.java b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/PermitAllAuthorizer.java
new file mode 100644
index 00000000000..75b2577f25e
--- /dev/null
+++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/PermitAllAuthorizer.java
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.mock.connector.server;
+
+import org.apache.nifi.authorization.AuthorizationRequest;
+import org.apache.nifi.authorization.AuthorizationResult;
+import org.apache.nifi.authorization.Authorizer;
+import org.apache.nifi.authorization.AuthorizerConfigurationContext;
+import org.apache.nifi.authorization.AuthorizerInitializationContext;
+import org.apache.nifi.authorization.exception.AuthorizationAccessException;
+import org.apache.nifi.authorization.exception.AuthorizerCreationException;
+import org.apache.nifi.authorization.exception.AuthorizerDestructionException;
+
+public class PermitAllAuthorizer implements Authorizer {
+ @Override
+ public AuthorizationResult authorize(final AuthorizationRequest request) throws AuthorizationAccessException {
+ return AuthorizationResult.approved();
+ }
+
+ @Override
+ public void initialize(final AuthorizerInitializationContext initializationContext) throws AuthorizerCreationException {
+ }
+
+ @Override
+ public void onConfigured(final AuthorizerConfigurationContext configurationContext) throws AuthorizerCreationException {
+ }
+
+ @Override
+ public void preDestruction() throws AuthorizerDestructionException {
+ }
+}
diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/StandardConnectorMockServer.java b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/StandardConnectorMockServer.java
new file mode 100644
index 00000000000..d491d0e6c19
--- /dev/null
+++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/StandardConnectorMockServer.java
@@ -0,0 +1,629 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.mock.connector.server;
+
+import jakarta.servlet.ServletContext;
+import org.apache.nifi.admin.service.AuditService;
+import org.apache.nifi.asset.Asset;
+import org.apache.nifi.asset.AssetManager;
+import org.apache.nifi.authorization.Authorizer;
+import org.apache.nifi.bundle.Bundle;
+import org.apache.nifi.bundle.BundleCoordinate;
+import org.apache.nifi.bundle.BundleDetails;
+import org.apache.nifi.cluster.ClusterDetailsFactory;
+import org.apache.nifi.components.ConfigVerificationResult;
+import org.apache.nifi.components.DescribedValue;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.connector.AssetReference;
+import org.apache.nifi.components.connector.Connector;
+import org.apache.nifi.components.connector.ConnectorNode;
+import org.apache.nifi.components.connector.ConnectorRepository;
+import org.apache.nifi.components.connector.ConnectorState;
+import org.apache.nifi.components.connector.ConnectorValueReference;
+import org.apache.nifi.components.connector.FlowUpdateException;
+import org.apache.nifi.components.connector.FrameworkFlowContext;
+import org.apache.nifi.components.connector.GhostConnector;
+import org.apache.nifi.components.connector.SecretReference;
+import org.apache.nifi.components.connector.StandaloneConnectorRequestReplicator;
+import org.apache.nifi.components.connector.StepConfiguration;
+import org.apache.nifi.components.connector.StringLiteralValue;
+import org.apache.nifi.components.connector.secrets.SecretsManager;
+import org.apache.nifi.components.state.StateManagerProvider;
+import org.apache.nifi.components.validation.DisabledServiceValidationResult;
+import org.apache.nifi.components.validation.ValidationState;
+import org.apache.nifi.connectable.FlowFileTransferCounts;
+import org.apache.nifi.controller.ControllerService;
+import org.apache.nifi.controller.DecommissionTask;
+import org.apache.nifi.controller.FlowController;
+import org.apache.nifi.controller.metrics.DefaultComponentMetricReporter;
+import org.apache.nifi.controller.repository.FlowFileEventRepository;
+import org.apache.nifi.controller.repository.metrics.RingBufferEventRepository;
+import org.apache.nifi.controller.status.history.StatusHistoryDumpFactory;
+import org.apache.nifi.controller.status.history.StatusHistoryRepository;
+import org.apache.nifi.controller.status.history.VolatileComponentStatusRepository;
+import org.apache.nifi.diagnostics.DiagnosticsFactory;
+import org.apache.nifi.encrypt.PropertyEncryptor;
+import org.apache.nifi.engine.FlowEngine;
+import org.apache.nifi.events.VolatileBulletinRepository;
+import org.apache.nifi.flow.VersionedExternalFlow;
+import org.apache.nifi.flow.VersionedParameter;
+import org.apache.nifi.flow.VersionedParameterContext;
+import org.apache.nifi.flow.VersionedProcessGroup;
+import org.apache.nifi.groups.ProcessGroup;
+import org.apache.nifi.mock.connector.server.secrets.ConnectorTestRunnerSecretsManager;
+import org.apache.nifi.nar.ExtensionMapping;
+import org.apache.nifi.parameter.Parameter;
+import org.apache.nifi.parameter.ParameterContext;
+import org.apache.nifi.processor.Processor;
+import org.apache.nifi.registry.flow.mapping.ComponentIdLookup;
+import org.apache.nifi.registry.flow.mapping.FlowMappingOptions;
+import org.apache.nifi.registry.flow.mapping.VersionedComponentFlowMapper;
+import org.apache.nifi.registry.flow.mapping.VersionedComponentStateLookup;
+import org.apache.nifi.reporting.BulletinRepository;
+import org.apache.nifi.util.NiFiProperties;
+import org.apache.nifi.validation.RuleViolationsManager;
+import org.apache.nifi.web.NiFiConnectorWebContext;
+import org.eclipse.jetty.ee.webapp.WebAppClassLoader;
+import org.eclipse.jetty.ee11.webapp.WebAppContext;
+import org.eclipse.jetty.server.Server;
+import org.eclipse.jetty.server.ServerConnector;
+import org.eclipse.jetty.server.handler.ContextHandlerCollection;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import java.util.jar.JarFile;
+import java.util.stream.Stream;
+
+public class StandardConnectorMockServer implements ConnectorMockServer {
+ private static final Logger logger = LoggerFactory.getLogger(StandardConnectorMockServer.class);
+
+ private static final String CONNECTOR_ID = "test-connector";
+ private static final String NAR_DEPENDENCIES_PATH = "NAR-INF/bundled-dependencies";
+ private static final String CONNECTOR_WAR_MANIFEST_PATH = "META-INF/nifi-connector";
+ private static final String WAR_EXTENSION = ".war";
+
+ private Bundle systemBundle;
+ private Set<Bundle> bundles;
+ private NiFiProperties nifiProperties;
+ private FlowController flowController;
+ private MockExtensionDiscoveringManager extensionManager;
+ private ConnectorNode connectorNode;
+ private ConnectorRepository connectorRepository;
+ private FlowEngine flowEngine;
+ private MockExtensionMapper mockExtensionMapper;
+ private FlowFileTransferCounts initialFlowFileTransferCounts = new FlowFileTransferCounts(0L, 0L, 0L, 0L);
+ private Server jettyServer;
+
+ @Override
+ public void start() {
+ extensionManager = new MockExtensionDiscoveringManager();
+ extensionManager.discoverExtensions(systemBundle, bundles);
+ extensionManager.logClassLoaderMapping();
+
+ final FlowFileEventRepository flowFileEventRepository = new RingBufferEventRepository(5);
+ final Authorizer authorizer = new PermitAllAuthorizer();
+ final AuditService auditService = new MockAuditService();
+ final PropertyEncryptor propertyEncryptor = new NopPropertyEncryptor();
+ final BulletinRepository bulletinRepository = new VolatileBulletinRepository();
+ final StatusHistoryRepository statusHistoryRepository = new VolatileComponentStatusRepository(nifiProperties);
+ final RuleViolationsManager ruleViolationManager = new MockRuleViolationsManager();
+ final StateManagerProvider stateManagerProvider = new MockStateManagerProvider();
+
+ flowController = FlowController.createStandaloneInstance(
+ flowFileEventRepository,
+ null,
+ nifiProperties,
+ authorizer,
+ auditService,
+ new DefaultComponentMetricReporter(),
+ propertyEncryptor,
+ bulletinRepository,
+ extensionManager,
+ statusHistoryRepository,
+ ruleViolationManager,
+ stateManagerProvider,
+ new StandaloneConnectorRequestReplicator());
+
+ try {
+ flowController.getRepositoryContextFactory().getFlowFileRepository().loadFlowFiles(Collections::emptyList);
+ } catch (final IOException e) {
+ throw new RuntimeException("Failed to initialize FlowFile Repository", e);
+ }
+
+ connectorRepository = flowController.getConnectorRepository();
+ if (!(connectorRepository instanceof MockConnectorRepository)) {
+ throw new IllegalStateException("Connector Repository is not an instance of MockConnectorRepository");
+ }
+
+ mockExtensionMapper = new MockExtensionMapper();
+ ((MockConnectorRepository) connectorRepository).setMockExtensionMapper(mockExtensionMapper);
+
+ flowEngine = new FlowEngine(4, "Connector Threads");
+
+ startJettyServer();
+ }
+
+ @Override
+ public void initialize(final NiFiProperties properties, final Bundle systemBundle, final Set<Bundle> bundles, final ExtensionMapping extensionMapping) {
+ this.systemBundle = systemBundle;
+ this.bundles = bundles;
+ this.nifiProperties = properties;
+ }
+
+ @Override
+ public void stop() {
+ if (jettyServer != null) {
+ try {
+ jettyServer.stop();
+ logger.info("Jetty server stopped");
+ } catch (final Exception e) {
+ logger.warn("Failed to stop Jetty server", e);
+ }
+ }
+ if (flowEngine != null) {
+ flowEngine.shutdown();
+ }
+ if (flowController != null) {
+ flowController.shutdown(false);
+ }
+ }
+
+ @Override
+ public DiagnosticsFactory getDiagnosticsFactory() {
+ return null;
+ }
+
+ @Override
+ public DiagnosticsFactory getThreadDumpFactory() {
+ return null;
+ }
+
+ @Override
+ public DecommissionTask getDecommissionTask() {
+ return null;
+ }
+
+ @Override
+ public ClusterDetailsFactory getClusterDetailsFactory() {
+ return null;
+ }
+
+ @Override
+ public StatusHistoryDumpFactory getStatusHistoryDumpFactory() {
+ return null;
+ }
+
+ @Override
+ public void instantiateConnector(final String connectorClassName) {
+ final List<Bundle> bundles = extensionManager.getBundles(connectorClassName);
+ if (bundles.isEmpty()) {
+ throw new IllegalStateException("No bundles found for connector class: " + connectorClassName + " - ensure that you have included all relevant NARs in the configured lib directory");
+ }
+ if (bundles.size() > 1) {
+ throw new IllegalStateException("Multiple bundles found for connector class: " + connectorClassName + " - unable to determine which bundle to use. Ensure that only a single version of " +
+ "the Connector is included in the configured lib directory. Available bundles: " + bundles);
+ }
+
+ final BundleCoordinate bundleCoordinate = bundles.getFirst().getBundleDetails().getCoordinate();
+ connectorNode = flowController.getFlowManager().createConnector(connectorClassName, CONNECTOR_ID, bundleCoordinate, true, true);
+
+ if (connectorNode.isExtensionMissing()) {
+ final Connector connector = connectorNode.getConnector();
+ if (connector instanceof final GhostConnector ghostConnector) {
+ throw new IllegalStateException("Failed to create Connector of type " + connectorClassName, ghostConnector.getCauseOfGhost());
+ } else {
+ throw new IllegalStateException("Failed to create Connector of type " + connectorClassName);
+ }
+ }
+ }
+
+ @Override
+ public void registerMockBundle(final ClassLoader classLoader, final File workingDirectory) {
+ new BundleDetails.Builder()
+ .workingDir(workingDirectory)
+ .coordinate(new BundleCoordinate("org.apache.nifi.mock", "nifi-connector-mock-bundle", "1.0.0"))
+ .build();
+ }
+
+ @Override
+ public void applyUpdate() throws FlowUpdateException {
+ final ConnectorState initialDesiredState = connectorNode.getDesiredState();
+
+ connectorNode.transitionStateForUpdating();
+ connectorNode.prepareForUpdate();
+ connectorNode.applyUpdate();
+
+ if (initialDesiredState == ConnectorState.RUNNING) {
+ connectorNode.start(flowEngine);
+ } else {
+ connectorNode.stop(flowEngine);
+ }
+ }
+
+ @Override
+ public void configure(final String stepName, final StepConfiguration configuration) throws FlowUpdateException {
+ connectorNode.setConfiguration(stepName, configuration);
+ }
+
+ @Override
+ public void configure(final String stepName, final Map<String, String> propertyValues) throws FlowUpdateException {
+ configure(stepName, propertyValues, Collections.emptyMap());
+ }
+
+ @Override
+ public void configure(final String stepName, final Map<String, String> propertyValues, final Map<String, ConnectorValueReference> propertyReferences) throws FlowUpdateException {
+ final StepConfiguration stepConfiguration = createStepConfiguration(propertyValues, propertyReferences);
+ configure(stepName, stepConfiguration);
+ }
+
+ @Override
+ public ConnectorConfigVerificationResult verifyConfiguration(final String stepName, final Map<String, String> propertyValueOverrides) {
+ return verifyConfiguration(stepName, propertyValueOverrides, Collections.emptyMap());
+ }
+
+ @Override
+ public ConnectorConfigVerificationResult verifyConfiguration(final String stepName, final Map<String, String> propertyValueOverrides,
+ final Map<String, ConnectorValueReference> referenceOverrides) {
+
+ final StepConfiguration configuration = createStepConfiguration(propertyValueOverrides, referenceOverrides);
+ return verifyConfiguration(stepName, configuration);
+ }
+
+ private StepConfiguration createStepConfiguration(final Map<String, String> propertyValues, final Map<String, ConnectorValueReference> propertyReferences) {
+ final Map<String, ConnectorValueReference> references = new HashMap<>();
+ propertyValues.forEach((key, value) -> references.put(key, value == null ? null : new StringLiteralValue(value)));
+ references.putAll(propertyReferences);
+ return new StepConfiguration(references);
+ }
+
+ @Override
+ public ConnectorConfigVerificationResult verifyConfiguration(final String stepName, final StepConfiguration configurationOverrides) {
+ final List<ConfigVerificationResult> results = connectorNode.verifyConfigurationStep(stepName, configurationOverrides);
+ return new MockServerConfigVerificationResult(results);
+ }
+
+ @Override
+ public void addSecret(final String name, final String value) {
+ final SecretsManager secretsManager = connectorRepository.getSecretsManager();
+ if (!(secretsManager instanceof final ConnectorTestRunnerSecretsManager testRunnerSecretsManager)) {
+ throw new IllegalStateException("Secrets Manager is not an instance of ConnectorTestRunnerSecretsManager");
+ }
+
+ testRunnerSecretsManager.addSecret(name, value);
+ }
+
+ @Override
+ public SecretReference createSecretReference(final String secretName) {
+ return new SecretReference(ConnectorTestRunner.SECRET_PROVIDER_ID, ConnectorTestRunner.SECRET_PROVIDER_NAME, secretName, secretName);
+ }
+
+ @Override
+ public AssetReference addAsset(final File file) {
+ final AssetManager assetManager = flowController.getConnectorAssetManager();
+
+ try (final InputStream inputStream = new FileInputStream(file)) {
+ final Asset asset = assetManager.createAsset(CONNECTOR_ID, file.getName(), inputStream);
+ return new AssetReference(Set.of(asset.getIdentifier()));
+ } catch (final IOException e) {
+ throw new RuntimeException("Failed to add asset from file: " + file.getAbsolutePath(), e);
+ }
+ }
+
+ @Override
+ public AssetReference addAsset(final String assetName, final InputStream contents) {
+ final AssetManager assetManager = flowController.getConnectorAssetManager();
+
+ try {
+ final Asset asset = assetManager.createAsset(CONNECTOR_ID, assetName, contents);
+ return new AssetReference(Set.of(asset.getIdentifier()));
+ } catch (final IOException e) {
+ throw new RuntimeException("Failed to add asset: " + assetName, e);
+ }
+ }
+
+ @Override
+ public void startConnector() {
+ initialFlowFileTransferCounts = connectorNode.getFlowFileTransferCounts();
+
+ connectorNode.start(flowEngine);
+ }
+
+ @Override
+ public void stopConnector() {
+ try {
+ connectorNode.stop(flowEngine).get(10, TimeUnit.SECONDS);
+ } catch (final InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new RuntimeException("Interrupted while waiting for connector to stop", e);
+ } catch (final Exception e) {
+ throw new RuntimeException("Failed to stop Connector", e);
+ }
+ }
+
+ @Override
+ public void waitForDataIngested(final Duration maxWaitTime) {
+ final long startTime = System.currentTimeMillis();
+ final long expirationTime = startTime + maxWaitTime.toMillis();
+
+ while (getTransferCountsSinceStart().getReceivedCount() == 0L) {
+ if (System.currentTimeMillis() > expirationTime) {
+ throw new RuntimeException("Timed out waiting for data to be ingested by the Connector");
+ }
+
+ try {
+ Thread.sleep(100L);
+ } catch (final InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new RuntimeException("Interrupted while waiting for data to be ingested by the Connector", e);
+ }
+ }
+ }
+
+ private FlowFileTransferCounts getTransferCountsSinceStart() {
+ final FlowFileTransferCounts currentCounts = connectorNode.getFlowFileTransferCounts();
+ return currentCounts.minus(initialFlowFileTransferCounts);
+ }
+
+ @Override
+ public void waitForIdle(final Duration maxWaitTime) {
+ waitForIdle(Duration.ofMillis(0L), maxWaitTime);
+ }
+
+ @Override
+ public void waitForIdle(final Duration minimumIdleTime, final Duration maxWaitTime) {
+ Optional<Duration> idleTime = connectorNode.getIdleDuration();
+
+ // Wait until idleTime is not empty and is at least equal to minimumIdleTime
+ final long startTime = System.currentTimeMillis();
+ final long expirationTime = startTime + maxWaitTime.toMillis();
+
+ while (idleTime.isEmpty() || idleTime.get().compareTo(minimumIdleTime) <= 0) {
+ if (System.currentTimeMillis() > expirationTime) {
+ throw new RuntimeException("Timed out waiting for Connector to be idle");
+ }
+
+ try {
+ Thread.sleep(100L);
+ } catch (final InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new RuntimeException("Interrupted while waiting for Connector to be idle", e);
+ }
+
+ idleTime = connectorNode.getIdleDuration();
+ }
+ }
+
+ @Override
+ public List<DescribedValue> fetchAllowableValues(final String stepName, final String propertyName) {
+ return connectorNode.fetchAllowableValues(stepName, propertyName);
+ }
+
+ @Override
+ public List<ValidationResult> validate() {
+ final ValidationState validationState = connectorNode.performValidation();
+ return validationState.getValidationErrors().stream()
+ .filter(result -> !result.isValid())
+ .filter(result -> !DisabledServiceValidationResult.isMatch(result))
+ .toList();
+ }
+
+ @Override
+ public VersionedExternalFlow getActiveFlowSnapshot() {
+ final FrameworkFlowContext activeFlowContext = connectorNode.getActiveFlowContext();
+ if (activeFlowContext == null) {
+ throw new IllegalStateException("Active Flow Context is not available. The Connector may not have been initialized or may not have an initial flow.");
+ }
+ return createFlowSnapshot(activeFlowContext);
+ }
+
+ @Override
+ public VersionedExternalFlow getWorkingFlowSnapshot() {
+ final FrameworkFlowContext workingFlowContext = connectorNode.getWorkingFlowContext();
+ if (workingFlowContext == null) {
+ throw new IllegalStateException("Working Flow Context is not available. No configuration changes may have been made since the last update was applied.");
+ }
+ return createFlowSnapshot(workingFlowContext);
+ }
+
+ private VersionedExternalFlow createFlowSnapshot(final FrameworkFlowContext flowContext) {
+ final ProcessGroup processGroup = flowContext.getManagedProcessGroup();
+
+ final FlowMappingOptions flowMappingOptions = new FlowMappingOptions.Builder()
+ .mapSensitiveConfiguration(false)
+ .mapPropertyDescriptors(true)
+ .stateLookup(VersionedComponentStateLookup.ENABLED_OR_DISABLED)
+ .sensitiveValueEncryptor(value -> value)
+ .componentIdLookup(ComponentIdLookup.VERSIONED_OR_GENERATE)
+ .mapInstanceIdentifiers(true)
+ .mapControllerServiceReferencesToVersionedId(true)
+ .mapFlowRegistryClientId(true)
+ .mapAssetReferences(true)
+ .build();
+
+ final VersionedComponentFlowMapper flowMapper = new VersionedComponentFlowMapper(extensionManager, flowMappingOptions);
+ final VersionedProcessGroup versionedGroup = flowMapper.mapProcessGroup(
+ processGroup, flowController.getControllerServiceProvider(), flowController.getFlowManager(), true);
+
+ final VersionedExternalFlow externalFlow = new VersionedExternalFlow();
+ externalFlow.setFlowContents(versionedGroup);
+
+ final ParameterContext parameterContext = processGroup.getParameterContext();
+ if (parameterContext != null) {
+ final Map<String, VersionedParameterContext> parameterContexts = new HashMap<>();
+ final VersionedParameterContext versionedParameterContext = createVersionedParameterContext(parameterContext);
+ parameterContexts.put(versionedParameterContext.getName(), versionedParameterContext);
+ externalFlow.setParameterContexts(parameterContexts);
+ }
+
+ return externalFlow;
+ }
+
+ private VersionedParameterContext createVersionedParameterContext(final ParameterContext parameterContext) {
+ final VersionedParameterContext versionedParameterContext = new VersionedParameterContext();
+ versionedParameterContext.setName(parameterContext.getName());
+ versionedParameterContext.setDescription(parameterContext.getDescription());
+ versionedParameterContext.setIdentifier(parameterContext.getIdentifier());
+
+ final Set<VersionedParameter> versionedParameters = new LinkedHashSet<>();
+ for (final Parameter parameter : parameterContext.getParameters().values()) {
+ final VersionedParameter versionedParameter = new VersionedParameter();
+ versionedParameter.setName(parameter.getDescriptor().getName());
+ versionedParameter.setDescription(parameter.getDescriptor().getDescription());
+ versionedParameter.setSensitive(parameter.getDescriptor().isSensitive());
+ versionedParameter.setProvided(parameter.getDescriptor().isSensitive());
+ if (!parameter.getDescriptor().isSensitive()) {
+ versionedParameter.setValue(parameter.getValue());
+ }
+ versionedParameters.add(versionedParameter);
+ }
+ versionedParameterContext.setParameters(versionedParameters);
+
+ return versionedParameterContext;
+ }
+
+ @Override
+ public void mockProcessor(final String processorType, final Class<? extends Processor> mockProcessorClass) {
+ mockExtensionMapper.mockProcessor(processorType, mockProcessorClass.getName());
+ extensionManager.addProcessor(mockProcessorClass);
+ }
+
+ @Override
+ public void mockControllerService(final String controllerServiceType, final Class<? extends ControllerService> mockControllerServiceClass) {
+ mockExtensionMapper.mockControllerService(controllerServiceType, mockControllerServiceClass.getName());
+ extensionManager.addControllerService(mockControllerServiceClass);
+ }
+
+ @Override
+ public int getHttpPort() {
+ if (jettyServer == null) {
+ return -1;
+ }
+
+ final ServerConnector connector = (ServerConnector) jettyServer.getConnectors()[0];
+ return connector.getLocalPort();
+ }
+
+ @Override
+ public void close() {
+ stop();
+ }
+
+ private void startJettyServer() {
+ final String httpPortValue = nifiProperties.getProperty(NiFiProperties.WEB_HTTP_PORT);
+ if (httpPortValue == null || httpPortValue.isBlank()) {
+ logger.debug("No HTTP port configured; skipping Jetty server startup");
+ return;
+ }
+
+ final int httpPort = Integer.parseInt(httpPortValue);
+ final Map<File, Bundle> wars = findWars(bundles);
+ if (wars.isEmpty()) {
+ logger.debug("No WAR files found in NAR bundles; skipping Jetty server startup");
+ return;
+ }
+
+ jettyServer = new Server();
+
+ final ServerConnector serverConnector = new ServerConnector(jettyServer);
+ serverConnector.setPort(httpPort);
+ jettyServer.addConnector(serverConnector);
+
+ final List<WebAppContext> webAppContexts = new ArrayList<>();
+ final ContextHandlerCollection handlers = new ContextHandlerCollection();
+ for (final Map.Entry<File, Bundle> entry : wars.entrySet()) {
+ final File warFile = entry.getKey();
+ final Bundle bundle = entry.getValue();
+
+ final String warName = warFile.getName();
+ final String contextPath = "/" + warName.substring(0, warName.length() - WAR_EXTENSION.length());
+
+ final WebAppContext webAppContext = new WebAppContext(warFile.getPath(), contextPath);
+ webAppContext.setClassLoader(new WebAppClassLoader(bundle.getClassLoader(), webAppContext));
+
+ handlers.addHandler(webAppContext);
+ webAppContexts.add(webAppContext);
+ logger.info("Deploying WAR [{}] at context path [{}]", warFile.getAbsolutePath(), contextPath);
+ }
+
+ jettyServer.setHandler(handlers);
+
+ try {
+ jettyServer.start();
+ logger.info("Jetty server started on port [{}]", getHttpPort());
+ } catch (final Exception e) {
+ throw new RuntimeException("Failed to start Jetty server", e);
+ }
+
+ performInjectionForConnectorUis(webAppContexts);
+ }
+
+ private void performInjectionForConnectorUis(final List<WebAppContext> webAppContexts) {
+ final NiFiConnectorWebContext connectorWebContext = new MockNiFiConnectorWebContext(connectorRepository);
+ for (final WebAppContext webAppContext : webAppContexts) {
+ final ServletContext servletContext = webAppContext.getServletHandler().getServletContext();
+ servletContext.setAttribute("nifi-connector-web-context", connectorWebContext);
+ logger.info("Injected NiFiConnectorWebContext into WAR context [{}]", webAppContext.getContextPath());
+ }
+ }
+
+ public Map<File, Bundle> findWars(final Set<Bundle> bundles) {
+ final Map<File, Bundle> wars = new HashMap<>();
+
+ bundles.forEach(bundle -> {
+ final BundleDetails details = bundle.getBundleDetails();
+ final Path bundledDependencies = new File(details.getWorkingDirectory(), NAR_DEPENDENCIES_PATH).toPath();
+ if (Files.isDirectory(bundledDependencies)) {
+ try (final Stream<Path> dependencies = Files.list(bundledDependencies)) {
+ dependencies.filter(dependency -> dependency.getFileName().toString().endsWith(WAR_EXTENSION))
+ .map(Path::toFile)
+ .filter(this::isConnectorWar)
+ .forEach(dependency -> wars.put(dependency, bundle));
+ } catch (final IOException e) {
+ logger.warn("Failed to find WAR files in bundled-dependencies [{}]", bundledDependencies, e);
+ }
+ }
+ });
+
+ return wars;
+ }
+
+ private boolean isConnectorWar(final File warFile) {
+ try (final JarFile jarFile = new JarFile(warFile)) {
+ return jarFile.getJarEntry(CONNECTOR_WAR_MANIFEST_PATH) != null;
+ } catch (final IOException e) {
+ logger.warn("Unable to inspect WAR file [{}] for connector manifest", warFile, e);
+ return false;
+ }
+ }
+}
diff --git a/nifi-manifest/nifi-extension-manifest-model/src/main/java/org/apache/nifi/extension/manifest/ExtensionType.java b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/secrets/ConnectorTestRunnerAuthorizable.java
similarity index 61%
copy from nifi-manifest/nifi-extension-manifest-model/src/main/java/org/apache/nifi/extension/manifest/ExtensionType.java
copy to nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/secrets/ConnectorTestRunnerAuthorizable.java
index 0413e260ed1..5600c34f143 100644
--- a/nifi-manifest/nifi-extension-manifest-model/src/main/java/org/apache/nifi/extension/manifest/ExtensionType.java
+++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/secrets/ConnectorTestRunnerAuthorizable.java
@@ -14,23 +14,21 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.nifi.extension.manifest;
-/**
- * Possible types of extensions.
- */
-public enum ExtensionType {
-
- PROCESSOR,
-
- CONTROLLER_SERVICE,
-
- REPORTING_TASK,
-
- FLOW_ANALYSIS_RULE,
+package org.apache.nifi.mock.connector.server.secrets;
- PARAMETER_PROVIDER,
+import org.apache.nifi.authorization.Resource;
+import org.apache.nifi.authorization.resource.Authorizable;
+import org.apache.nifi.authorization.resource.ResourceFactory;
- FLOW_REGISTRY_CLIENT;
+public class ConnectorTestRunnerAuthorizable implements Authorizable {
+ @Override
+ public Authorizable getParentAuthorizable() {
+ return null;
+ }
+ @Override
+ public Resource getResource() {
+ return ResourceFactory.getControllerResource();
+ }
}
diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/secrets/ConnectorTestRunnerSecretProvider.java b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/secrets/ConnectorTestRunnerSecretProvider.java
new file mode 100644
index 00000000000..e12a3711bc1
--- /dev/null
+++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/secrets/ConnectorTestRunnerSecretProvider.java
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.mock.connector.server.secrets;
+
+import org.apache.nifi.authorization.resource.Authorizable;
+import org.apache.nifi.components.connector.Secret;
+import org.apache.nifi.components.connector.secrets.SecretProvider;
+import org.apache.nifi.components.connector.secrets.StandardSecret;
+import org.apache.nifi.mock.connector.server.ConnectorTestRunner;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+public class ConnectorTestRunnerSecretProvider implements SecretProvider {
+ public static final String GROUP_NAME = "Default";
+ private static final Authorizable AUTHORIZABLE = new ConnectorTestRunnerAuthorizable();
+
+ private final Map<String, String> secrets = new HashMap<>();
+
+ public void addSecret(final String key, final String value) {
+ this.secrets.put(key, value);
+ }
+
+ @Override
+ public String getProviderId() {
+ return ConnectorTestRunner.SECRET_PROVIDER_ID;
+ }
+
+ @Override
+ public String getProviderName() {
+ return ConnectorTestRunner.SECRET_PROVIDER_ID;
+ }
+
+ @Override
+ public List<Secret> getAllSecrets() {
+ final List<Secret> secrets = new ArrayList<>();
+ for (final Map.Entry<String, String> entry : this.secrets.entrySet()) {
+ final Secret secret = new StandardSecret.Builder()
+ .providerId(ConnectorTestRunner.SECRET_PROVIDER_ID)
+ .providerName(ConnectorTestRunner.SECRET_PROVIDER_NAME)
+ .groupName(GROUP_NAME)
+ .name(entry.getKey())
+ .value(entry.getValue())
+ .authorizable(AUTHORIZABLE)
+ .fullyQualifiedName(GROUP_NAME + "." + entry.getKey())
+ .build();
+
+ secrets.add(secret);
+ }
+
+ return secrets;
+ }
+
+ @Override
+ public List<Secret> getSecrets(final List<String> fullyQualifiedSecretNames) {
+ final List<Secret> matchingSecrets = new ArrayList<>();
+
+ for (final String secretName : fullyQualifiedSecretNames) {
+ final String value = secrets.get(secretName);
+
+ if (value != null) {
+ final Secret secret = new StandardSecret.Builder()
+ .providerId(ConnectorTestRunner.SECRET_PROVIDER_ID)
+ .providerName(ConnectorTestRunner.SECRET_PROVIDER_NAME)
+ .groupName(GROUP_NAME)
+ .name(secretName)
+ .fullyQualifiedName(GROUP_NAME + "." + secretName)
+ .value(value)
+ .authorizable(AUTHORIZABLE)
+ .build();
+
+ matchingSecrets.add(secret);
+ }
+ }
+ return matchingSecrets;
+ }
+}
diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/secrets/ConnectorTestRunnerSecretsManager.java b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/secrets/ConnectorTestRunnerSecretsManager.java
new file mode 100644
index 00000000000..e06f228454a
--- /dev/null
+++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/secrets/ConnectorTestRunnerSecretsManager.java
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.mock.connector.server.secrets;
+
+import org.apache.nifi.components.connector.Secret;
+import org.apache.nifi.components.connector.SecretReference;
+import org.apache.nifi.components.connector.secrets.SecretProvider;
+import org.apache.nifi.components.connector.secrets.SecretsManager;
+import org.apache.nifi.components.connector.secrets.SecretsManagerInitializationContext;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+public class ConnectorTestRunnerSecretsManager implements SecretsManager {
+ private final ConnectorTestRunnerSecretProvider secretProvider = new ConnectorTestRunnerSecretProvider();
+
+ @Override
+ public void initialize(final SecretsManagerInitializationContext initializationContext) {
+ }
+
+ public void addSecret(final String name, final String value) {
+ secretProvider.addSecret(name, value);
+ }
+
+ @Override
+ public List<Secret> getAllSecrets() {
+ return secretProvider.getAllSecrets();
+ }
+
+ @Override
+ public Set<SecretProvider> getSecretProviders() {
+ return Set.of(secretProvider);
+ }
+
+ @Override
+ public Optional<Secret> getSecret(final SecretReference secretReference) {
+ // Check that appropriate provider given
+ final SecretProvider provider = getProvider(secretReference);
+ if (provider == null) {
+ return Optional.empty();
+ }
+
+ final List<Secret> secrets = provider.getSecrets(List.of(secretReference.getFullyQualifiedName()));
+ return secrets.isEmpty() ? Optional.empty() : Optional.of(secrets.getFirst());
+ }
+
+ private SecretProvider getProvider(final SecretReference secretReference) {
+ final String providerId = secretReference.getProviderId();
+ if (providerId != null) {
+ for (final SecretProvider provider : getSecretProviders()) {
+ if (provider.getProviderId().equals(providerId)) {
+ return provider;
+ }
+ }
+
+ return null;
+ }
+
+ final String providerName = secretReference.getProviderName();
+ if (providerName != null) {
+ for (final SecretProvider provider : getSecretProviders()) {
+ if (provider.getProviderName().equals(providerName)) {
+ return provider;
+ }
+ }
+ }
+
+ return null;
+ }
+
+ @Override
+ public Map<SecretReference, Secret> getSecrets(final Set<SecretReference> secretReferences) {
+ final Map<SecretReference, Secret> secrets = new HashMap<>();
+ for (final SecretReference reference : secretReferences) {
+ final Secret secret = getSecret(reference).orElse(null);
+ secrets.put(reference, secret);
+ }
+
+ return secrets;
+ }
+}
diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/resources/META-INF/services/org.apache.nifi.asset.AssetManager b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/resources/META-INF/services/org.apache.nifi.NiFiServer
similarity index 92%
copy from nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/resources/META-INF/services/org.apache.nifi.asset.AssetManager
copy to nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/resources/META-INF/services/org.apache.nifi.NiFiServer
index 6fbf05d9497..70572a99eae 100644
--- a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/resources/META-INF/services/org.apache.nifi.asset.AssetManager
+++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/resources/META-INF/services/org.apache.nifi.NiFiServer
@@ -13,4 +13,4 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-org.apache.nifi.asset.StandardAssetManager
\ No newline at end of file
+org.apache.nifi.mock.connector.server.StandardConnectorMockServer
diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/resources/META-INF/services/org.apache.nifi.asset.AssetManager b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/resources/META-INF/services/org.apache.nifi.asset.AssetManager
similarity index 92%
copy from nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/resources/META-INF/services/org.apache.nifi.asset.AssetManager
copy to nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/resources/META-INF/services/org.apache.nifi.asset.AssetManager
index 6fbf05d9497..c23d0b17077 100644
--- a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/resources/META-INF/services/org.apache.nifi.asset.AssetManager
+++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/resources/META-INF/services/org.apache.nifi.asset.AssetManager
@@ -13,4 +13,4 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-org.apache.nifi.asset.StandardAssetManager
\ No newline at end of file
+org.apache.nifi.mock.connector.server.MockConnectorAssetManager
diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/resources/META-INF/services/org.apache.nifi.asset.AssetManager b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/resources/META-INF/services/org.apache.nifi.components.connector.ConnectorRepository
similarity index 92%
copy from nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/resources/META-INF/services/org.apache.nifi.asset.AssetManager
copy to nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/resources/META-INF/services/org.apache.nifi.components.connector.ConnectorRepository
index 6fbf05d9497..d9230afaec5 100644
--- a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/resources/META-INF/services/org.apache.nifi.asset.AssetManager
+++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/resources/META-INF/services/org.apache.nifi.components.connector.ConnectorRepository
@@ -13,4 +13,4 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-org.apache.nifi.asset.StandardAssetManager
\ No newline at end of file
+org.apache.nifi.mock.connector.server.MockConnectorRepository
diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/resources/META-INF/services/org.apache.nifi.asset.AssetManager b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/resources/META-INF/services/org.apache.nifi.components.connector.secrets.SecretsManager
similarity index 90%
copy from nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/resources/META-INF/services/org.apache.nifi.asset.AssetManager
copy to nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/resources/META-INF/services/org.apache.nifi.components.connector.secrets.SecretsManager
index 6fbf05d9497..a1c0de54d1f 100644
--- a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/resources/META-INF/services/org.apache.nifi.asset.AssetManager
+++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/resources/META-INF/services/org.apache.nifi.components.connector.secrets.SecretsManager
@@ -13,4 +13,4 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-org.apache.nifi.asset.StandardAssetManager
\ No newline at end of file
+org.apache.nifi.mock.connector.server.secrets.ConnectorTestRunnerSecretsManager
diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-server/src/test/java/org/apache/nifi/mock/connector/server/MockConnectorAssetManagerTest.java b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/test/java/org/apache/nifi/mock/connector/server/MockConnectorAssetManagerTest.java
new file mode 100644
index 00000000000..29521787641
--- /dev/null
+++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/test/java/org/apache/nifi/mock/connector/server/MockConnectorAssetManagerTest.java
@@ -0,0 +1,245 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.mock.connector.server;
+
+import org.apache.nifi.asset.Asset;
+import org.apache.nifi.asset.AssetManagerInitializationContext;
+import org.apache.nifi.asset.AssetReferenceLookup;
+import org.apache.nifi.controller.NodeTypeProvider;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+class MockConnectorAssetManagerTest {
+
+ private static final String CONNECTOR_ID = "test-connector";
+ private static final String ASSET_NAME = "test-asset.txt";
+ private static final String ASSET_CONTENTS = "test-contents";
+
+ @TempDir
+ private Path tempDir;
+
+ private MockConnectorAssetManager assetManager;
+
+ @BeforeEach
+ void setUp() {
+ assetManager = new MockConnectorAssetManager();
+ final AssetManagerInitializationContext context = createInitializationContext(tempDir);
+ assetManager.initialize(context);
+ }
+
+ @Test
+ void testCreateAsset() throws IOException {
+ final InputStream contents = new ByteArrayInputStream(ASSET_CONTENTS.getBytes(StandardCharsets.UTF_8));
+
+ final Asset asset = assetManager.createAsset(CONNECTOR_ID, ASSET_NAME, contents);
+
+ assertNotNull(asset);
+ assertNotNull(asset.getIdentifier());
+ assertEquals(CONNECTOR_ID, asset.getOwnerIdentifier());
+ assertEquals(ASSET_NAME, asset.getName());
+ assertTrue(asset.getFile().exists());
+ assertEquals(ASSET_CONTENTS, Files.readString(asset.getFile().toPath(), StandardCharsets.UTF_8));
+ }
+
+ @Test
+ void testCreateAssetGeneratesUniqueIdentifiers() throws IOException {
+ final InputStream contents1 = new ByteArrayInputStream(ASSET_CONTENTS.getBytes(StandardCharsets.UTF_8));
+ final InputStream contents2 = new ByteArrayInputStream(ASSET_CONTENTS.getBytes(StandardCharsets.UTF_8));
+
+ final Asset asset1 = assetManager.createAsset(CONNECTOR_ID, ASSET_NAME, contents1);
+ final Asset asset2 = assetManager.createAsset(CONNECTOR_ID, ASSET_NAME, contents2);
+
+ assertNotEquals(asset1.getIdentifier(), asset2.getIdentifier());
+ }
+
+ @Test
+ void testSaveAsset() throws IOException {
+ final String assetId = "specific-asset-id";
+ final InputStream contents = new ByteArrayInputStream(ASSET_CONTENTS.getBytes(StandardCharsets.UTF_8));
+
+ final Asset asset = assetManager.saveAsset(CONNECTOR_ID, assetId, ASSET_NAME, contents);
+
+ assertNotNull(asset);
+ assertEquals(assetId, asset.getIdentifier());
+ assertEquals(CONNECTOR_ID, asset.getOwnerIdentifier());
+ assertEquals(ASSET_NAME, asset.getName());
+ assertTrue(asset.getFile().exists());
+ assertEquals(ASSET_CONTENTS, Files.readString(asset.getFile().toPath(), StandardCharsets.UTF_8));
+ }
+
+ @Test
+ void testSaveAssetOverwritesExisting() throws IOException {
+ final String assetId = "overwrite-asset-id";
+ final String originalContents = "original";
+ final String updatedContents = "updated";
+
+ final InputStream originalStream = new ByteArrayInputStream(originalContents.getBytes(StandardCharsets.UTF_8));
+ final Asset originalAsset = assetManager.saveAsset(CONNECTOR_ID, assetId, ASSET_NAME, originalStream);
+ assertEquals(originalContents, Files.readString(originalAsset.getFile().toPath(), StandardCharsets.UTF_8));
+
+ final InputStream updatedStream = new ByteArrayInputStream(updatedContents.getBytes(StandardCharsets.UTF_8));
+ final Asset updatedAsset = assetManager.saveAsset(CONNECTOR_ID, assetId, ASSET_NAME, updatedStream);
+ assertEquals(updatedContents, Files.readString(updatedAsset.getFile().toPath(), StandardCharsets.UTF_8));
+ }
+
+ @Test
+ void testGetAsset() throws IOException {
+ final InputStream contents = new ByteArrayInputStream(ASSET_CONTENTS.getBytes(StandardCharsets.UTF_8));
+ final Asset createdAsset = assetManager.createAsset(CONNECTOR_ID, ASSET_NAME, contents);
+
+ final Optional<Asset> retrieved = assetManager.getAsset(createdAsset.getIdentifier());
+
+ assertTrue(retrieved.isPresent());
+ assertEquals(createdAsset.getIdentifier(), retrieved.get().getIdentifier());
+ assertEquals(createdAsset.getName(), retrieved.get().getName());
+ assertEquals(createdAsset.getOwnerIdentifier(), retrieved.get().getOwnerIdentifier());
+ }
+
+ @Test
+ void testGetAssetReturnsEmptyForNonExistent() {
+ final Optional<Asset> retrieved = assetManager.getAsset("non-existent-id");
+
+ assertTrue(retrieved.isEmpty());
+ }
+
+ @Test
+ void testGetAssets() throws IOException {
+ final InputStream contents1 = new ByteArrayInputStream("content1".getBytes(StandardCharsets.UTF_8));
+ final InputStream contents2 = new ByteArrayInputStream("content2".getBytes(StandardCharsets.UTF_8));
+ final InputStream contents3 = new ByteArrayInputStream("content3".getBytes(StandardCharsets.UTF_8));
+
+ assetManager.createAsset(CONNECTOR_ID, "asset1.txt", contents1);
+ assetManager.createAsset(CONNECTOR_ID, "asset2.txt", contents2);
+ assetManager.createAsset("other-connector", "asset3.txt", contents3);
+
+ final List<Asset> connectorAssets = assetManager.getAssets(CONNECTOR_ID);
+ assertEquals(2, connectorAssets.size());
+ assertTrue(connectorAssets.stream().allMatch(a -> a.getOwnerIdentifier().equals(CONNECTOR_ID)));
+
+ final List<Asset> otherAssets = assetManager.getAssets("other-connector");
+ assertEquals(1, otherAssets.size());
+ assertEquals("other-connector", otherAssets.get(0).getOwnerIdentifier());
+ }
+
+ @Test
+ void testGetAssetsReturnsEmptyListForUnknownOwner() {
+ final List<Asset> assets = assetManager.getAssets("unknown-owner");
+
+ assertNotNull(assets);
+ assertTrue(assets.isEmpty());
+ }
+
+ @Test
+ void testCreateMissingAsset() {
+ final Asset missingAsset = assetManager.createMissingAsset(CONNECTOR_ID, ASSET_NAME);
+
+ assertNotNull(missingAsset);
+ assertNotNull(missingAsset.getIdentifier());
+ assertEquals(CONNECTOR_ID, missingAsset.getOwnerIdentifier());
+ assertEquals(ASSET_NAME, missingAsset.getName());
+ assertFalse(missingAsset.getFile().exists());
+ assertTrue(missingAsset.getDigest().isEmpty());
+ }
+
+ @Test
+ void testCreateMissingAssetCanBeRetrieved() {
+ final Asset missingAsset = assetManager.createMissingAsset(CONNECTOR_ID, ASSET_NAME);
+
+ final Optional<Asset> retrieved = assetManager.getAsset(missingAsset.getIdentifier());
+ assertTrue(retrieved.isPresent());
+ assertEquals(missingAsset.getIdentifier(), retrieved.get().getIdentifier());
+
+ final List<Asset> ownerAssets = assetManager.getAssets(CONNECTOR_ID);
+ assertEquals(1, ownerAssets.size());
+ }
+
+ @Test
+ void testDeleteAsset() throws IOException {
+ final InputStream contents = new ByteArrayInputStream(ASSET_CONTENTS.getBytes(StandardCharsets.UTF_8));
+ final Asset createdAsset = assetManager.createAsset(CONNECTOR_ID, ASSET_NAME, contents);
+ final Path assetFilePath = createdAsset.getFile().toPath();
+ assertTrue(Files.exists(assetFilePath));
+
+ final Optional<Asset> deleted = assetManager.deleteAsset(createdAsset.getIdentifier());
+
+ assertTrue(deleted.isPresent());
+ assertEquals(createdAsset.getIdentifier(), deleted.get().getIdentifier());
+ assertFalse(Files.exists(assetFilePath));
+ assertTrue(assetManager.getAsset(createdAsset.getIdentifier()).isEmpty());
+ }
+
+ @Test
+ void testDeleteAssetReturnsEmptyForNonExistent() {
+ final Optional<Asset> deleted = assetManager.deleteAsset("non-existent-id");
+
+ assertTrue(deleted.isEmpty());
+ }
+
+ @Test
+ void testInitializeCreatesStorageDirectory() {
+ final Path newStorageDir = tempDir.resolve("new-storage");
+ assertFalse(Files.exists(newStorageDir));
+
+ final MockConnectorAssetManager newManager = new MockConnectorAssetManager();
+ final AssetManagerInitializationContext context = createInitializationContext(newStorageDir);
+ newManager.initialize(context);
+
+ assertTrue(Files.exists(newStorageDir));
+ assertTrue(Files.isDirectory(newStorageDir));
+ }
+
+ private AssetManagerInitializationContext createInitializationContext(final Path storageDirectory) {
+ final Map<String, String> properties = new HashMap<>();
+ properties.put("directory", storageDirectory.toAbsolutePath().toString());
+
+ return new AssetManagerInitializationContext() {
+ @Override
+ public AssetReferenceLookup getAssetReferenceLookup() {
+ return null;
+ }
+
+ @Override
+ public Map<String, String> getProperties() {
+ return properties;
+ }
+
+ @Override
+ public NodeTypeProvider getNodeTypeProvider() {
+ return null;
+ }
+ };
+ }
+}
diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-server/src/test/java/org/apache/nifi/mock/connector/server/MockNiFiConnectorWebContextTest.java b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/test/java/org/apache/nifi/mock/connector/server/MockNiFiConnectorWebContextTest.java
new file mode 100644
index 00000000000..659fe2e8fb7
--- /dev/null
+++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/test/java/org/apache/nifi/mock/connector/server/MockNiFiConnectorWebContextTest.java
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.mock.connector.server;
+
+import org.apache.nifi.components.connector.Connector;
+import org.apache.nifi.components.connector.ConnectorNode;
+import org.apache.nifi.components.connector.ConnectorRepository;
+import org.apache.nifi.components.connector.FrameworkFlowContext;
+import org.apache.nifi.web.NiFiConnectorWebContext;
+import org.apache.nifi.web.NiFiConnectorWebContext.ConnectorWebContext;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.mockito.Mock;
+import org.mockito.junit.jupiter.MockitoExtension;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.mockito.Mockito.when;
+
+@ExtendWith(MockitoExtension.class)
+class MockNiFiConnectorWebContextTest {
+
+ private static final String CONNECTOR_ID = "test-connector";
+
+ @Mock
+ private ConnectorRepository connectorRepository;
+
+ @Mock
+ private ConnectorNode connectorNode;
+
+ @Mock
+ private Connector connector;
+
+ @Mock
+ private FrameworkFlowContext workingFlowContext;
+
+ @Mock
+ private FrameworkFlowContext activeFlowContext;
+
+ @Test
+ void testGetConnectorWebContextReturnsConnectorAndFlowContexts() {
+ when(connectorRepository.getConnector(CONNECTOR_ID)).thenReturn(connectorNode);
+ when(connectorNode.getConnector()).thenReturn(connector);
+ when(connectorNode.getWorkingFlowContext()).thenReturn(workingFlowContext);
+ when(connectorNode.getActiveFlowContext()).thenReturn(activeFlowContext);
+
+ final NiFiConnectorWebContext context = new MockNiFiConnectorWebContext(connectorRepository);
+ final ConnectorWebContext<Connector> result = context.getConnectorWebContext(CONNECTOR_ID);
+
+ assertNotNull(result);
+ assertEquals(connector, result.connector());
+ assertEquals(workingFlowContext, result.workingFlowContext());
+ assertEquals(activeFlowContext, result.activeFlowContext());
+ }
+
+ @Test
+ void testGetConnectorWebContextThrowsForUnknownConnector() {
+ when(connectorRepository.getConnector("unknown-id")).thenReturn(null);
+
+ final NiFiConnectorWebContext context = new MockNiFiConnectorWebContext(connectorRepository);
+
+ assertThrows(IllegalArgumentException.class, () -> context.getConnectorWebContext("unknown-id"));
+ }
+}
diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-server/src/test/java/org/apache/nifi/mock/connector/server/MockStateManagerProviderTest.java b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/test/java/org/apache/nifi/mock/connector/server/MockStateManagerProviderTest.java
new file mode 100644
index 00000000000..d8ecf8e8729
--- /dev/null
+++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/test/java/org/apache/nifi/mock/connector/server/MockStateManagerProviderTest.java
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.mock.connector.server;
+
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateManager;
+import org.apache.nifi.components.state.StateMap;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNotSame;
+import static org.junit.jupiter.api.Assertions.assertSame;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+class MockStateManagerProviderTest {
+
+ @Test
+ void testGetStateManagerReturnsSameInstance() {
+ final MockStateManagerProvider provider = new MockStateManagerProvider();
+ final StateManager stateManager1 = provider.getStateManager("component1", false);
+ final StateManager stateManager2 = provider.getStateManager("component1", false);
+
+ assertNotNull(stateManager1);
+ assertSame(stateManager1, stateManager2);
+ }
+
+ @Test
+ void testGetStateManagerReturnsDifferentInstancesForDifferentComponents() {
+ final MockStateManagerProvider provider = new MockStateManagerProvider();
+ final StateManager stateManager1 = provider.getStateManager("component1", false);
+ final StateManager stateManager2 = provider.getStateManager("component2", false);
+
+ assertNotNull(stateManager1);
+ assertNotNull(stateManager2);
+ assertNotSame(stateManager1, stateManager2);
+ }
+
+ @Test
+ void testOnComponentRemoved() {
+ final MockStateManagerProvider provider = new MockStateManagerProvider();
+ final StateManager stateManager1 = provider.getStateManager("component1", false);
+ assertNotNull(stateManager1);
+
+ provider.onComponentRemoved("component1");
+
+ final StateManager stateManager2 = provider.getStateManager("component1", false);
+ assertNotNull(stateManager2);
+ assertNotSame(stateManager1, stateManager2);
+ }
+
+ @Test
+ void testShutdown() {
+ final MockStateManagerProvider provider = new MockStateManagerProvider();
+ provider.getStateManager("component1", false);
+ provider.getStateManager("component2", false);
+
+ provider.shutdown();
+
+ final StateManager stateManager = provider.getStateManager("component1", false);
+ assertNotNull(stateManager);
+ }
+
+ @Test
+ void testClusterProviderEnabled() {
+ final MockStateManagerProvider provider = new MockStateManagerProvider();
+ assertFalse(provider.isClusterProviderEnabled());
+
+ provider.enableClusterProvider();
+ assertTrue(provider.isClusterProviderEnabled());
+
+ provider.disableClusterProvider();
+ assertFalse(provider.isClusterProviderEnabled());
+ }
+
+ @Test
+ void testStateManagerSetAndGetLocalState() throws IOException {
+ final MockStateManagerProvider provider = new MockStateManagerProvider();
+ final StateManager stateManager = provider.getStateManager("component1", false);
+
+ final Map<String, String> state = new HashMap<>();
+ state.put("key1", "value1");
+ state.put("key2", "value2");
+
+ stateManager.setState(state, Scope.LOCAL);
+
+ final StateMap stateMap = stateManager.getState(Scope.LOCAL);
+ assertNotNull(stateMap);
+ assertTrue(stateMap.getStateVersion().isPresent());
+ assertEquals("value1", stateMap.get("key1"));
+ assertEquals("value2", stateMap.get("key2"));
+ assertEquals(2, stateMap.toMap().size());
+ }
+
+ @Test
+ void testStateManagerSetAndGetClusterState() throws IOException {
+ final MockStateManagerProvider provider = new MockStateManagerProvider();
+ final StateManager stateManager = provider.getStateManager("component1", false);
+
+ final Map<String, String> state = new HashMap<>();
+ state.put("key1", "value1");
+ state.put("key2", "value2");
+
+ stateManager.setState(state, Scope.CLUSTER);
+
+ final StateMap stateMap = stateManager.getState(Scope.CLUSTER);
+ assertNotNull(stateMap);
+ assertTrue(stateMap.getStateVersion().isPresent());
+ assertEquals("value1", stateMap.get("key1"));
+ assertEquals("value2", stateMap.get("key2"));
+ }
+
+ @Test
+ void testStateManagerReplace() throws IOException {
+ final MockStateManagerProvider provider = new MockStateManagerProvider();
+ final StateManager stateManager = provider.getStateManager("component1", false);
+
+ final Map<String, String> initialState = new HashMap<>();
+ initialState.put("key1", "value1");
+ stateManager.setState(initialState, Scope.LOCAL);
+
+ final StateMap oldStateMap = stateManager.getState(Scope.LOCAL);
+
+ final Map<String, String> newState = new HashMap<>();
+ newState.put("key1", "value2");
+
+ final boolean replaced = stateManager.replace(oldStateMap, newState, Scope.LOCAL);
+ assertTrue(replaced);
+
+ final StateMap updatedStateMap = stateManager.getState(Scope.LOCAL);
+ assertEquals("value2", updatedStateMap.get("key1"));
+
+ final boolean replacedAgain = stateManager.replace(oldStateMap, initialState, Scope.LOCAL);
+ assertFalse(replacedAgain);
+ }
+
+ @Test
+ void testStateManagerClear() throws IOException {
+ final MockStateManagerProvider provider = new MockStateManagerProvider();
+ final StateManager stateManager = provider.getStateManager("component1", false);
+
+ final Map<String, String> state = new HashMap<>();
+ state.put("key1", "value1");
+ stateManager.setState(state, Scope.LOCAL);
+
+ stateManager.clear(Scope.LOCAL);
+
+ final StateMap stateMap = stateManager.getState(Scope.LOCAL);
+ assertNotNull(stateMap);
+ assertTrue(stateMap.toMap().isEmpty());
+ }
+
+ @Test
+ void testStateKeyDropSupported() {
+ final MockStateManagerProvider provider = new MockStateManagerProvider();
+ final StateManager stateManager1 = provider.getStateManager("component1", false);
+ assertFalse(stateManager1.isStateKeyDropSupported());
+
+ final StateManager stateManager2 = provider.getStateManager("component2", true);
+ assertTrue(stateManager2.isStateKeyDropSupported());
+ }
+
+ @Test
+ void testThreadSafety() throws InterruptedException, IOException {
+ final MockStateManagerProvider provider = new MockStateManagerProvider();
+ final StateManager stateManager = provider.getStateManager("component1", false);
+ final int threadCount = 10;
+ final int operationsPerThread = 100;
+ final ExecutorService executorService = Executors.newFixedThreadPool(threadCount);
+ final CountDownLatch startLatch = new CountDownLatch(1);
+ final CountDownLatch completionLatch = new CountDownLatch(threadCount);
+ final AtomicInteger successCount = new AtomicInteger(0);
+
+ for (int i = 0; i < threadCount; i++) {
+ final int threadId = i;
+ executorService.submit(() -> {
+ try {
+ startLatch.await();
+ for (int j = 0; j < operationsPerThread; j++) {
+ final Map<String, String> state = new HashMap<>();
+ state.put("thread", String.valueOf(threadId));
+ state.put("operation", String.valueOf(j));
+ stateManager.setState(state, Scope.LOCAL);
+
+ final StateMap stateMap = stateManager.getState(Scope.LOCAL);
+ if (stateMap != null && stateMap.toMap().size() == 2) {
+ successCount.incrementAndGet();
+ }
+ }
+ } catch (final Exception e) {
+ e.printStackTrace();
+ } finally {
+ completionLatch.countDown();
+ }
+ });
+ }
+
+ startLatch.countDown();
+ completionLatch.await(30, TimeUnit.SECONDS);
+ executorService.shutdown();
+
+ assertEquals(threadCount * operationsPerThread, successCount.get());
+ final StateMap finalState = stateManager.getState(Scope.LOCAL);
+ assertNotNull(finalState);
+ assertEquals(2, finalState.toMap().size());
+ }
+
+ @Test
+ void testEmptyStateMap() throws IOException {
+ final MockStateManagerProvider provider = new MockStateManagerProvider();
+ final StateManager stateManager = provider.getStateManager("component1", false);
+
+ final StateMap stateMap = stateManager.getState(Scope.LOCAL);
+ assertNotNull(stateMap);
+ assertFalse(stateMap.getStateVersion().isPresent());
+ assertTrue(stateMap.toMap().isEmpty());
+ }
+
+ @Test
+ void testVersionIncrementing() throws IOException {
+ final MockStateManagerProvider provider = new MockStateManagerProvider();
+ final StateManager stateManager = provider.getStateManager("component1", false);
+
+ final Map<String, String> state1 = new HashMap<>();
+ state1.put("key", "value1");
+ stateManager.setState(state1, Scope.LOCAL);
+ final StateMap stateMap1 = stateManager.getState(Scope.LOCAL);
+
+ final Map<String, String> state2 = new HashMap<>();
+ state2.put("key", "value2");
+ stateManager.setState(state2, Scope.LOCAL);
+ final StateMap stateMap2 = stateManager.getState(Scope.LOCAL);
+
+ assertTrue(stateMap1.getStateVersion().isPresent());
+ assertTrue(stateMap2.getStateVersion().isPresent());
+
+ final long version1 = Long.parseLong(stateMap1.getStateVersion().get());
+ final long version2 = Long.parseLong(stateMap2.getStateVersion().get());
+ assertTrue(version2 > version1);
+ }
+}
+
diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-server/src/test/java/org/apache/nifi/mock/connector/server/StandardConnectorMockServerJettyTest.java b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/test/java/org/apache/nifi/mock/connector/server/StandardConnectorMockServerJettyTest.java
new file mode 100644
index 00000000000..980d8341c97
--- /dev/null
+++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/test/java/org/apache/nifi/mock/connector/server/StandardConnectorMockServerJettyTest.java
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.mock.connector.server;
+
+import org.apache.nifi.bundle.Bundle;
+import org.apache.nifi.bundle.BundleCoordinate;
+import org.apache.nifi.bundle.BundleDetails;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Map;
+import java.util.Set;
+import java.util.jar.JarEntry;
+import java.util.jar.JarOutputStream;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+class StandardConnectorMockServerJettyTest {
+
+ private static final String NAR_DEPENDENCIES_PATH = "NAR-INF/bundled-dependencies";
+
+ @TempDir
+ private Path tempDir;
+
+ @Test
+ void testFindWarsDiscoversSingleWarFile() throws Exception {
+ final Path bundleWorkingDir = tempDir.resolve("test-bundle");
+ final Path depsDir = bundleWorkingDir.resolve(NAR_DEPENDENCIES_PATH);
+ Files.createDirectories(depsDir);
+
+ final Path warFile = depsDir.resolve("my-app.war");
+ createConnectorWar(warFile);
+
+ final Bundle bundle = createBundle(bundleWorkingDir);
+ final StandardConnectorMockServer server = new StandardConnectorMockServer();
+ final Map<File, Bundle> wars = server.findWars(Set.of(bundle));
+
+ assertEquals(1, wars.size());
+ assertTrue(wars.containsKey(warFile.toFile()));
+ assertEquals(bundle, wars.get(warFile.toFile()));
+ }
+
+ @Test
+ void testFindWarsIgnoresNonWarFiles() throws Exception {
+ final Path bundleWorkingDir = tempDir.resolve("test-bundle");
+ final Path depsDir = bundleWorkingDir.resolve(NAR_DEPENDENCIES_PATH);
+ Files.createDirectories(depsDir);
+
+ Files.createFile(depsDir.resolve("some-lib.jar"));
+ Files.createFile(depsDir.resolve("config.xml"));
+
+ final Bundle bundle = createBundle(bundleWorkingDir);
+ final StandardConnectorMockServer server = new StandardConnectorMockServer();
+ final Map<File, Bundle> wars = server.findWars(Set.of(bundle));
+
+ assertTrue(wars.isEmpty());
+ }
+
+ @Test
+ void testFindWarsIgnoresWarWithoutConnectorManifest() throws Exception {
+ final Path bundleWorkingDir = tempDir.resolve("test-bundle");
+ final Path depsDir = bundleWorkingDir.resolve(NAR_DEPENDENCIES_PATH);
+ Files.createDirectories(depsDir);
+
+ final Path warFile = depsDir.resolve("non-connector.war");
+ createWarWithoutConnectorManifest(warFile);
+
+ final Bundle bundle = createBundle(bundleWorkingDir);
+ final StandardConnectorMockServer server = new StandardConnectorMockServer();
+ final Map<File, Bundle> wars = server.findWars(Set.of(bundle));
+
+ assertTrue(wars.isEmpty());
+ }
+
+ @Test
+ void testFindWarsHandlesMissingDependenciesDirectory() throws Exception {
+ final Path bundleWorkingDir = tempDir.resolve("empty-bundle");
+ Files.createDirectories(bundleWorkingDir);
+
+ final Bundle bundle = createBundle(bundleWorkingDir);
+ final StandardConnectorMockServer server = new StandardConnectorMockServer();
+ final Map<File, Bundle> wars = server.findWars(Set.of(bundle));
+
+ assertTrue(wars.isEmpty());
+ }
+
+ @Test
+ void testFindWarsDiscoversMultipleWarFiles() throws Exception {
+ final Path bundleWorkingDir = tempDir.resolve("multi-war-bundle");
+ final Path depsDir = bundleWorkingDir.resolve(NAR_DEPENDENCIES_PATH);
+ Files.createDirectories(depsDir);
+
+ createConnectorWar(depsDir.resolve("app-one.war"));
+ createConnectorWar(depsDir.resolve("app-two.war"));
+ Files.createFile(depsDir.resolve("some-lib.jar"));
+
+ final Bundle bundle = createBundle(bundleWorkingDir);
+ final StandardConnectorMockServer server = new StandardConnectorMockServer();
+ final Map<File, Bundle> wars = server.findWars(Set.of(bundle));
+
+ assertEquals(2, wars.size());
+ }
+
+ @Test
+ void testFindWarsFromMultipleBundles() throws Exception {
+ final Path bundleDir1 = tempDir.resolve("bundle-1");
+ final Path depsDir1 = bundleDir1.resolve(NAR_DEPENDENCIES_PATH);
+ Files.createDirectories(depsDir1);
+ createConnectorWar(depsDir1.resolve("first-app.war"));
+
+ final Path bundleDir2 = tempDir.resolve("bundle-2");
+ final Path depsDir2 = bundleDir2.resolve(NAR_DEPENDENCIES_PATH);
+ Files.createDirectories(depsDir2);
+ createConnectorWar(depsDir2.resolve("second-app.war"));
+
+ final Bundle bundle1 = createBundle(bundleDir1, "test-bundle-1");
+ final Bundle bundle2 = createBundle(bundleDir2, "test-bundle-2");
+
+ final StandardConnectorMockServer server = new StandardConnectorMockServer();
+ final Map<File, Bundle> wars = server.findWars(Set.of(bundle1, bundle2));
+
+ assertEquals(2, wars.size());
+ }
+
+ @Test
+ void testGetHttpPortReturnsNegativeOneWhenNoServer() {
+ final StandardConnectorMockServer server = new StandardConnectorMockServer();
+ assertEquals(-1, server.getHttpPort());
+ }
+
+ @Test
+ void testFindWarsReturnsEmptyForEmptyBundleSet() throws Exception {
+ final StandardConnectorMockServer server = new StandardConnectorMockServer();
+ final Map<File, Bundle> wars = server.findWars(Set.of());
+ assertTrue(wars.isEmpty());
+ }
+
+ private static void createConnectorWar(final Path warPath) throws IOException {
+ try (final JarOutputStream jarOut = new JarOutputStream(new FileOutputStream(warPath.toFile()))) {
+ jarOut.putNextEntry(new JarEntry("META-INF/nifi-connector"));
+ jarOut.closeEntry();
+ }
+ }
+
+ private static void createWarWithoutConnectorManifest(final Path warPath) throws IOException {
+ try (final JarOutputStream jarOut = new JarOutputStream(new FileOutputStream(warPath.toFile()))) {
+ jarOut.putNextEntry(new JarEntry("WEB-INF/web.xml"));
+ jarOut.closeEntry();
+ }
+ }
+
+ private Bundle createBundle(final Path workingDir) {
+ return createBundle(workingDir, "test-bundle");
+ }
+
+ private Bundle createBundle(final Path workingDir, final String artifactId) {
+ final BundleDetails details = new BundleDetails.Builder()
+ .workingDir(workingDir.toFile())
+ .coordinate(new BundleCoordinate("org.test", artifactId, "1.0.0"))
+ .build();
+
+ return new Bundle(details, ClassLoader.getSystemClassLoader());
+ }
+
+}
diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-integration-tests/pom.xml b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-integration-tests/pom.xml
new file mode 100644
index 00000000000..88ab6070315
--- /dev/null
+++ b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-integration-tests/pom.xml
@@ -0,0 +1,130 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-connector-mock-test-bundle</artifactId>
+ <version>2.9.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-connector-mock-integration-tests</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-connector-mock</artifactId>
+ <version>2.9.0-SNAPSHOT</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-connector-mock-api</artifactId>
+ <version>2.9.0-SNAPSHOT</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-lookup-service-api</artifactId>
+ <version>2.9.0-SNAPSHOT</version>
+ <scope>provided</scope>
+ </dependency>
+
+ <!-- Test Dependencies -->
+ <dependency>
+ <groupId>org.junit.jupiter</groupId>
+ <artifactId>junit-jupiter</artifactId>
+ <scope>test</scope>
+ </dependency>
+
+ <!-- NARs required for integration testing -->
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-connector-mock-server-nar</artifactId>
+ <version>2.9.0-SNAPSHOT</version>
+ <type>nar</type>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-connector-mock-test-connectors-nar</artifactId>
+ <version>2.9.0-SNAPSHOT</version>
+ <type>nar</type>
+ </dependency>
+
+ <!-- NARs for components referenced in the Connectors -->
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-standard-nar</artifactId>
+ <version>2.9.0-SNAPSHOT</version>
+ <type>nar</type>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-standard-services-api-nar</artifactId>
+ <version>2.9.0-SNAPSHOT</version>
+ <type>nar</type>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-standard-shared-nar</artifactId>
+ <version>2.9.0-SNAPSHOT</version>
+ <type>nar</type>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-update-attribute-nar</artifactId>
+ <version>2.9.0-SNAPSHOT</version>
+ <type>nar</type>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-lookup-services-nar</artifactId>
+ <version>2.9.0-SNAPSHOT</version>
+ <type>nar</type>
+ </dependency>
+ </dependencies>
+
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-dependency-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>copy-nar</id>
+ <phase>generate-resources</phase>
+ <goals>
+ <goal>copy-dependencies</goal>
+ </goals>
+ <configuration>
+ <outputDirectory>${project.build.directory}/libDir</outputDirectory>
+ <includeTypes>jar,nar</includeTypes>
+ <includeScope>compile</includeScope>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+</project>
+
diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-integration-tests/src/test/java/org/apache/nifi/mock/connectors/tests/AllowableValuesIT.java b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-integration-tests/src/test/java/org/apache/nifi/mock/connectors/tests/AllowableValuesIT.java
new file mode 100644
index 00000000000..a350730b722
--- /dev/null
+++ b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-integration-tests/src/test/java/org/apache/nifi/mock/connectors/tests/AllowableValuesIT.java
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.mock.connectors.tests;
+
+import org.apache.nifi.components.ConfigVerificationResult;
+import org.apache.nifi.mock.connector.StandardConnectorTestRunner;
+import org.apache.nifi.mock.connector.server.ConnectorConfigVerificationResult;
+import org.apache.nifi.mock.connector.server.ConnectorTestRunner;
+import org.junit.jupiter.api.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Integration tests that verify fetchable allowable values are properly validated
+ * during configuration verification. These tests exercise the fix for NIFI-15258,
+ * where {@code StandardConnectorNode.fetchAllowableValues} incorrectly used the
+ * active flow context instead of the working flow context passed as a parameter.
+ *
+ * <p>The {@code AllowableValuesConnector} returns allowable values only when the
+ * flow context type is WORKING. If the framework incorrectly passes the ACTIVE
+ * context, the connector returns an empty list, causing all values to be accepted.</p>
+ */
+public class AllowableValuesIT {
+
+ private static final String CONNECTOR_CLASS = "org.apache.nifi.mock.connectors.AllowableValuesConnector";
+
+ @Test
+ public void testVerifyConfigurationRejectsInvalidAllowableValue() throws IOException {
+ try (final ConnectorTestRunner runner = createRunner()) {
+ final ConnectorConfigVerificationResult result = runner.verifyConfiguration("Selection", Map.of("Color", "purple"));
+
+ final List<ConfigVerificationResult> failedResults = result.getFailedResults();
+ assertFalse(failedResults.isEmpty(), "Expected at least one failed result for invalid allowable value 'purple'");
+ assertEquals(1, failedResults.size());
+
+ final String explanation = failedResults.getFirst().getExplanation();
+ assertTrue(explanation.contains("allowable values"), "Expected explanation to mention allowable values but was: " + explanation);
+ }
+ }
+
+ @Test
+ public void testVerifyConfigurationAcceptsValidAllowableValue() throws IOException {
+ try (final ConnectorTestRunner runner = createRunner()) {
+ final ConnectorConfigVerificationResult result = runner.verifyConfiguration("Selection", Map.of("Color", "red"));
+ result.assertNoFailures();
+ }
+ }
+
+ @Test
+ public void testVerifyConfigurationAcceptsAllValidAllowableValues() throws IOException {
+ try (final ConnectorTestRunner runner = createRunner()) {
+ for (final String validColor : List.of("red", "green", "blue")) {
+ final ConnectorConfigVerificationResult result = runner.verifyConfiguration("Selection", Map.of("Color", validColor));
+ result.assertNoFailures();
+ }
+ }
+ }
+
+ private ConnectorTestRunner createRunner() {
+ return new StandardConnectorTestRunner.Builder()
+ .connectorClassName(CONNECTOR_CLASS)
+ .narLibraryDirectory(new File("target/libDir"))
+ .build();
+ }
+}
diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-integration-tests/src/test/java/org/apache/nifi/mock/connectors/tests/CreateConnectorIT.java b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-integration-tests/src/test/java/org/apache/nifi/mock/connectors/tests/CreateConnectorIT.java
new file mode 100644
index 00000000000..18884a8d830
--- /dev/null
+++ b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-integration-tests/src/test/java/org/apache/nifi/mock/connectors/tests/CreateConnectorIT.java
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.mock.connectors.tests;
+
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flow.VersionedControllerService;
+import org.apache.nifi.flow.VersionedExternalFlow;
+import org.apache.nifi.flow.VersionedProcessGroup;
+import org.apache.nifi.flow.VersionedProcessor;
+import org.apache.nifi.mock.connector.StandardConnectorTestRunner;
+import org.apache.nifi.mock.connector.server.ConnectorTestRunner;
+import org.junit.jupiter.api.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class CreateConnectorIT {
+
+ @Test
+ public void testCreateStartAndStopGenerateAndUpdateConnector() throws IOException {
+ try (final ConnectorTestRunner testRunner = new StandardConnectorTestRunner.Builder()
+ .connectorClassName("org.apache.nifi.mock.connectors.GenerateAndLog")
+ .narLibraryDirectory(new File("target/libDir"))
+ .build()) {
+
+ // Verify the active flow snapshot reflects the initial flow loaded from Generate_and_Update.json
+ final VersionedExternalFlow activeFlow = testRunner.getActiveFlowSnapshot();
+ final VersionedProcessGroup rootGroup = activeFlow.getFlowContents();
+
+ final Set<VersionedProcessor> processors = rootGroup.getProcessors();
+ assertEquals(3, processors.size());
+ assertTrue(findProcessorByType(processors, "org.apache.nifi.processors.standard.GenerateFlowFile").isPresent());
+ assertTrue(findProcessorByType(processors, "org.apache.nifi.processors.standard.LookupAttribute").isPresent());
+ assertTrue(findProcessorByType(processors, "org.apache.nifi.processors.attributes.UpdateAttribute").isPresent());
+
+ assertEquals(2, rootGroup.getConnections().size());
+
+ final Set<VersionedControllerService> controllerServices = rootGroup.getControllerServices();
+ assertEquals(1, controllerServices.size());
+ assertEquals("org.apache.nifi.lookup.SimpleKeyValueLookupService", controllerServices.iterator().next().getType());
+
+ testRunner.startConnector();
+ testRunner.stopConnector();
+ }
+ }
+
+ @Test
+ public void testConnectorWithMissingBundleFailsValidate() throws IOException {
+
+ try (final ConnectorTestRunner testRunner = new StandardConnectorTestRunner.Builder()
+ .connectorClassName("org.apache.nifi.mock.connectors.MissingBundleConnector")
+ .narLibraryDirectory(new File("target/libDir"))
+ .build()) {
+
+ final List<ValidationResult> results = testRunner.validate();
+ assertEquals(results.size(), 1);
+ final String message = results.getFirst().getExplanation();
+ assertTrue(message.contains("com.example.nonexistent:missing-nar:1.0.0"), "Expected exception message to contain missing bundle coordinates but was: " + message);
+ assertTrue(message.contains("com.example.nonexistent.MissingProcessor"), "Expected exception message to contain missing processor type but was: " + message);
+ }
+ }
+
+ private Optional<VersionedProcessor> findProcessorByType(final Set<VersionedProcessor> processors, final String type) {
+ for (final VersionedProcessor processor : processors) {
+ if (type.equals(processor.getType())) {
+ return Optional.of(processor);
+ }
+ }
+ return Optional.empty();
+ }
+}
diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-integration-tests/src/test/java/org/apache/nifi/mock/connectors/tests/CronScheduleIT.java b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-integration-tests/src/test/java/org/apache/nifi/mock/connectors/tests/CronScheduleIT.java
new file mode 100644
index 00000000000..10dea97a432
--- /dev/null
+++ b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-integration-tests/src/test/java/org/apache/nifi/mock/connectors/tests/CronScheduleIT.java
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.mock.connectors.tests;
+
+import org.apache.nifi.components.ConfigVerificationResult;
+import org.apache.nifi.mock.connector.StandardConnectorTestRunner;
+import org.apache.nifi.mock.connector.server.ConnectorConfigVerificationResult;
+import org.apache.nifi.mock.connector.server.ConnectorTestRunner;
+import org.junit.jupiter.api.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+
+import static org.junit.jupiter.api.Assertions.assertFalse;
+
+/**
+ * Integration tests that verify CRON expression validation during configuration verification.
+ * The {@code CronScheduleConnector} maps the "Trigger Schedule" connector property to a
+ * CRON-driven GenerateFlowFile processor's scheduling period parameter, and verification
+ * delegates to the processor's validation which checks the CRON expression.
+ */
+public class CronScheduleIT {
+
+ private static final String CONNECTOR_CLASS = "org.apache.nifi.mock.connectors.CronScheduleConnector";
+
+ @Test
+ public void testValidCronExpression() throws IOException {
+ try (final ConnectorTestRunner runner = createRunner()) {
+ final ConnectorConfigVerificationResult result = runner.verifyConfiguration("Schedule",
+ Map.of("Trigger Schedule", "0 0 * * * *"));
+ result.assertNoFailures();
+ }
+ }
+
+ @Test
+ public void testInvalidCronExpression() throws IOException {
+ try (final ConnectorTestRunner runner = createRunner()) {
+ final ConnectorConfigVerificationResult result = runner.verifyConfiguration("Schedule",
+ Map.of("Trigger Schedule", "invalid-cron"));
+ final List<ConfigVerificationResult> failedResults = result.getFailedResults();
+ assertFalse(failedResults.isEmpty());
+ }
+ }
+
+ private ConnectorTestRunner createRunner() {
+ return new StandardConnectorTestRunner.Builder()
+ .connectorClassName(CONNECTOR_CLASS)
+ .narLibraryDirectory(new File("target/libDir"))
+ .build();
+ }
+}
diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-integration-tests/src/test/java/org/apache/nifi/mock/connectors/tests/MockControllerServiceIT.java b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-integration-tests/src/test/java/org/apache/nifi/mock/connectors/tests/MockControllerServiceIT.java
new file mode 100644
index 00000000000..1f044ccf21c
--- /dev/null
+++ b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-integration-tests/src/test/java/org/apache/nifi/mock/connectors/tests/MockControllerServiceIT.java
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.mock.connectors.tests;
+
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.lookup.LookupFailureException;
+import org.apache.nifi.lookup.StringLookupService;
+import org.apache.nifi.mock.connector.StandardConnectorTestRunner;
+import org.apache.nifi.mock.connector.server.ConnectorTestRunner;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.Timeout;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class MockControllerServiceIT {
+
+ @Test
+ @Timeout(10)
+ public void testMockControllerService() throws IOException {
+ try (final ConnectorTestRunner runner = new StandardConnectorTestRunner.Builder()
+ .narLibraryDirectory(new File("target/libDir"))
+ .connectorClassName("org.apache.nifi.mock.connectors.GenerateAndLog")
+ .mockControllerService("org.apache.nifi.lookup.SimpleKeyValueLookupService", MockStringLookupService.class)
+ .build()) {
+
+ runner.startConnector();
+
+ // Wait until MockStringLookupService.lookup is invoked at least once.
+ // We use @Timeout on the test to avoid hanging indefinitely in case of failure.
+ while (MockStringLookupService.lookupCounter.get() < 1) {
+ Thread.yield();
+ }
+
+ assertTrue(MockStringLookupService.lookupCounter.get() >= 1);
+
+ runner.stopConnector();
+ }
+ }
+
+ public static class MockStringLookupService extends AbstractControllerService implements StringLookupService {
+ private static final AtomicLong lookupCounter = new AtomicLong(0L);
+
+ @Override
+ protected PropertyDescriptor getSupportedDynamicPropertyDescriptor(final String propertyDescriptorName) {
+ return new PropertyDescriptor.Builder()
+ .name(propertyDescriptorName)
+ .addValidator(Validator.VALID)
+ .dynamic(true)
+ .build();
+ }
+
+ @Override
+ public Optional<String> lookup(final Map<String, Object> coordinates) throws LookupFailureException {
+ lookupCounter.incrementAndGet();
+ return Optional.of("mock-value");
+ }
+
+ @Override
+ public Set<String> getRequiredKeys() {
+ return Set.of("key");
+ }
+ }
+}
diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-integration-tests/src/test/java/org/apache/nifi/mock/connectors/tests/MockProcessorIT.java b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-integration-tests/src/test/java/org/apache/nifi/mock/connectors/tests/MockProcessorIT.java
new file mode 100644
index 00000000000..465d0340002
--- /dev/null
+++ b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-integration-tests/src/test/java/org/apache/nifi/mock/connectors/tests/MockProcessorIT.java
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.mock.connectors.tests;
+
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.mock.connector.StandardConnectorTestRunner;
+import org.apache.nifi.mock.connector.server.ConnectorTestRunner;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.Timeout;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+
+public class MockProcessorIT {
+
+ @Test
+ @Timeout(10)
+ public void testMockProcessor() throws IOException {
+ try (final ConnectorTestRunner runner = new StandardConnectorTestRunner.Builder()
+ .narLibraryDirectory(new File("target/libDir"))
+ .connectorClassName("org.apache.nifi.mock.connectors.GenerateAndLog")
+ .mockProcessor("org.apache.nifi.processors.attributes.UpdateAttribute", MockProcessor.class)
+ .build()) {
+
+ runner.startConnector();
+
+ // Wait until MockProcessor is triggered at least once. We use @Timeout on the test to avoid
+ // hanging indefinitely in case of failure.
+ while (MockProcessor.invocationCounter.get() < 1) {
+ Thread.yield();
+ }
+ runner.stopConnector();
+ }
+ }
+
+ public static class MockProcessor extends AbstractProcessor {
+ private static final AtomicLong invocationCounter = new AtomicLong(0L);
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .build();
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return Set.of(REL_SUCCESS);
+ }
+
+ // Support any properties so that properties of UpdateAttribute can be set without the Processor becoming invalid
+ @Override
+ protected PropertyDescriptor getSupportedDynamicPropertyDescriptor(final String propertyDescriptorName) {
+ return new PropertyDescriptor.Builder()
+ .name(propertyDescriptorName)
+ .addValidator(Validator.VALID)
+ .build();
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ invocationCounter.incrementAndGet();
+
+ final FlowFile flowFile = session.get();
+ if (flowFile != null) {
+ session.transfer(flowFile, REL_SUCCESS);
+ }
+ }
+ }
+}
diff --git a/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/pom.xml b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors-nar/pom.xml
similarity index 58%
copy from nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/pom.xml
copy to nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors-nar/pom.xml
index f877db14d04..c2d17f8b288 100644
--- a/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/pom.xml
+++ b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors-nar/pom.xml
@@ -14,40 +14,22 @@
limitations under the License.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
<parent>
- <artifactId>nifi-system-test-extensions-bundle</artifactId>
<groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-connector-mock-test-bundle</artifactId>
<version>2.9.0-SNAPSHOT</version>
</parent>
- <modelVersion>4.0.0</modelVersion>
- <artifactId>nifi-system-test-extensions</artifactId>
+ <artifactId>nifi-connector-mock-test-connectors-nar</artifactId>
+ <packaging>nar</packaging>
<dependencies>
<dependency>
<groupId>org.apache.nifi</groupId>
- <artifactId>nifi-stateless-api</artifactId>
+ <artifactId>nifi-connector-mock-test-connectors</artifactId>
<version>2.9.0-SNAPSHOT</version>
</dependency>
- <dependency>
- <groupId>org.apache.nifi</groupId>
- <artifactId>nifi-utils</artifactId>
- <version>2.9.0-SNAPSHOT</version>
- </dependency>
- <dependency>
- <groupId>org.apache.nifi</groupId>
- <artifactId>nifi-bin-manager</artifactId>
- <version>2.9.0-SNAPSHOT</version>
- </dependency>
- <dependency>
- <groupId>com.fasterxml.jackson.core</groupId>
- <artifactId>jackson-databind</artifactId>
- </dependency>
- <dependency>
- <groupId>org.apache.nifi</groupId>
- <artifactId>nifi-system-test-extensions-services-api</artifactId>
- <version>2.9.0-SNAPSHOT</version>
- <scope>provided</scope>
- </dependency>
</dependencies>
-</project>
\ No newline at end of file
+</project>
+
diff --git a/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/pom.xml b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors/pom.xml
similarity index 64%
copy from nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/pom.xml
copy to nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors/pom.xml
index f877db14d04..479fa9db078 100644
--- a/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/pom.xml
+++ b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors/pom.xml
@@ -14,40 +14,35 @@
limitations under the License.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
<parent>
- <artifactId>nifi-system-test-extensions-bundle</artifactId>
<groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-connector-mock-test-bundle</artifactId>
<version>2.9.0-SNAPSHOT</version>
</parent>
- <modelVersion>4.0.0</modelVersion>
- <artifactId>nifi-system-test-extensions</artifactId>
+ <artifactId>nifi-connector-mock-test-connectors</artifactId>
+ <packaging>jar</packaging>
<dependencies>
<dependency>
<groupId>org.apache.nifi</groupId>
- <artifactId>nifi-stateless-api</artifactId>
- <version>2.9.0-SNAPSHOT</version>
- </dependency>
- <dependency>
- <groupId>org.apache.nifi</groupId>
- <artifactId>nifi-utils</artifactId>
- <version>2.9.0-SNAPSHOT</version>
+ <artifactId>nifi-api</artifactId>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
- <artifactId>nifi-bin-manager</artifactId>
+ <artifactId>nifi-connector-utils</artifactId>
<version>2.9.0-SNAPSHOT</version>
</dependency>
+
+
+ <!-- Test Dependencies -->
<dependency>
- <groupId>com.fasterxml.jackson.core</groupId>
- <artifactId>jackson-databind</artifactId>
- </dependency>
- <dependency>
- <groupId>org.apache.nifi</groupId>
- <artifactId>nifi-system-test-extensions-services-api</artifactId>
- <version>2.9.0-SNAPSHOT</version>
- <scope>provided</scope>
+ <groupId>org.junit.jupiter</groupId>
+ <artifactId>junit-jupiter</artifactId>
+ <scope>test</scope>
</dependency>
</dependencies>
-</project>
\ No newline at end of file
+
+</project>
+
diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors/src/main/java/org/apache/nifi/mock/connectors/AllowableValuesConnector.java b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors/src/main/java/org/apache/nifi/mock/connectors/AllowableValuesConnector.java
new file mode 100644
index 00000000000..a263482eda2
--- /dev/null
+++ b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors/src/main/java/org/apache/nifi/mock/connectors/AllowableValuesConnector.java
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.mock.connectors;
+
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.ConfigVerificationResult;
+import org.apache.nifi.components.DescribedValue;
+import org.apache.nifi.components.connector.AbstractConnector;
+import org.apache.nifi.components.connector.ConfigurationStep;
+import org.apache.nifi.components.connector.ConnectorPropertyDescriptor;
+import org.apache.nifi.components.connector.ConnectorPropertyGroup;
+import org.apache.nifi.components.connector.FlowUpdateException;
+import org.apache.nifi.components.connector.components.FlowContext;
+import org.apache.nifi.components.connector.components.FlowContextType;
+import org.apache.nifi.components.connector.util.VersionedFlowUtils;
+import org.apache.nifi.flow.VersionedExternalFlow;
+
+import java.util.List;
+import java.util.Map;
+
+/**
+ * A test connector with a configuration step that has fetchable allowable values.
+ * The {@link #fetchAllowableValues} method returns allowable values only when invoked
+ * with a WORKING {@link FlowContextType}. If invoked with an ACTIVE context, it returns
+ * an empty list. This design verifies that the framework passes the correct (working)
+ * context during configuration verification, reproducing the scenario fixed in NIFI-15258.
+ */
+public class AllowableValuesConnector extends AbstractConnector {
+
+ static final ConnectorPropertyDescriptor COLOR = new ConnectorPropertyDescriptor.Builder()
+ .name("Color")
+ .description("The color to select")
+ .allowableValuesFetchable(true)
+ .required(true)
+ .build();
+
+ static final ConnectorPropertyGroup SELECTION_GROUP = new ConnectorPropertyGroup.Builder()
+ .name("Color Selection")
+ .addProperty(COLOR)
+ .build();
+
+ static final ConfigurationStep SELECTION_STEP = new ConfigurationStep.Builder()
+ .name("Selection")
+ .propertyGroups(List.of(SELECTION_GROUP))
+ .build();
+
+ private static final List<ConfigurationStep> CONFIGURATION_STEPS = List.of(SELECTION_STEP);
+
+ @Override
+ public VersionedExternalFlow getInitialFlow() {
+ return VersionedFlowUtils.loadFlowFromResource("flows/Generate_and_Update.json");
+ }
+
+ @Override
+ public List<ConfigurationStep> getConfigurationSteps() {
+ return CONFIGURATION_STEPS;
+ }
+
+ @Override
+ protected void onStepConfigured(final String stepName, final FlowContext flowContext) {
+ }
+
+ @Override
+ public void applyUpdate(final FlowContext workingContext, final FlowContext activeContext) throws FlowUpdateException {
+ }
+
+ @Override
+ public List<ConfigVerificationResult> verifyConfigurationStep(final String stepName, final Map<String, String> overrides, final FlowContext workingContext) {
+ return List.of();
+ }
+
+ @Override
+ public List<DescribedValue> fetchAllowableValues(final String stepName, final String propertyName, final FlowContext flowContext) {
+ if ("Selection".equals(stepName) && "Color".equals(propertyName)) {
+ if (flowContext.getType() == FlowContextType.WORKING) {
+ return List.of(
+ new AllowableValue("red", "Red"),
+ new AllowableValue("green", "Green"),
+ new AllowableValue("blue", "Blue")
+ );
+ }
+
+ // Return empty when the context is ACTIVE, simulating the pre-fix behavior
+ // where using the wrong context caused allowable values to be empty and
+ // any value to be incorrectly accepted.
+ return List.of();
+ }
+
+ return super.fetchAllowableValues(stepName, propertyName, flowContext);
+ }
+}
diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors/src/main/java/org/apache/nifi/mock/connectors/CronScheduleConnector.java b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors/src/main/java/org/apache/nifi/mock/connectors/CronScheduleConnector.java
new file mode 100644
index 00000000000..1e257c245c6
--- /dev/null
+++ b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors/src/main/java/org/apache/nifi/mock/connectors/CronScheduleConnector.java
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.mock.connectors;
+
+import org.apache.nifi.components.ConfigVerificationResult;
+import org.apache.nifi.components.connector.AbstractConnector;
+import org.apache.nifi.components.connector.ConfigurationStep;
+import org.apache.nifi.components.connector.ConnectorConfigurationContext;
+import org.apache.nifi.components.connector.ConnectorPropertyDescriptor;
+import org.apache.nifi.components.connector.ConnectorPropertyGroup;
+import org.apache.nifi.components.connector.FlowUpdateException;
+import org.apache.nifi.components.connector.components.FlowContext;
+import org.apache.nifi.components.connector.components.ProcessorFacade;
+import org.apache.nifi.components.connector.util.VersionedFlowUtils;
+import org.apache.nifi.flow.VersionedExternalFlow;
+
+import java.util.List;
+import java.util.Map;
+
+/**
+ * A test connector that validates CRON scheduling expressions via the "Trigger Schedule" parameter.
+ * The underlying flow ({@code Cron_Schedule_Connector.json}) contains a CRON-driven
+ * GenerateFlowFile processor whose scheduling period references the {@code #{Trigger Schedule}} parameter.
+ * Verification delegates to the processor's validate/verify cycle, which checks the CRON expression.
+ */
+public class CronScheduleConnector extends AbstractConnector {
+
+ static final String SCHEDULE_STEP_NAME = "Schedule";
+ static final String TRIGGER_SCHEDULE_PARAM = "Trigger Schedule";
+
+ static final ConnectorPropertyDescriptor TRIGGER_SCHEDULE = new ConnectorPropertyDescriptor.Builder()
+ .name(TRIGGER_SCHEDULE_PARAM)
+ .description("CRON expression for the GenerateFlowFile trigger schedule")
+ .required(true)
+ .build();
+
+ static final ConnectorPropertyGroup SCHEDULE_GROUP = new ConnectorPropertyGroup.Builder()
+ .name("Schedule Settings")
+ .addProperty(TRIGGER_SCHEDULE)
+ .build();
+
+ static final ConfigurationStep SCHEDULE_STEP = new ConfigurationStep.Builder()
+ .name(SCHEDULE_STEP_NAME)
+ .propertyGroups(List.of(SCHEDULE_GROUP))
+ .build();
+
+ @Override
+ public VersionedExternalFlow getInitialFlow() {
+ return VersionedFlowUtils.loadFlowFromResource("flows/Cron_Schedule_Connector.json");
+ }
+
+ @Override
+ public List<ConfigurationStep> getConfigurationSteps() {
+ return List.of(SCHEDULE_STEP);
+ }
+
+ @Override
+ protected void onStepConfigured(final String stepName, final FlowContext flowContext) throws FlowUpdateException {
+ if (SCHEDULE_STEP_NAME.equals(stepName)) {
+ final String triggerSchedule = flowContext.getConfigurationContext()
+ .getProperty(SCHEDULE_STEP_NAME, TRIGGER_SCHEDULE_PARAM).getValue();
+ final VersionedExternalFlow flow = getInitialFlow();
+ VersionedFlowUtils.setParameterValue(flow, TRIGGER_SCHEDULE_PARAM, triggerSchedule);
+ getInitializationContext().updateFlow(flowContext, flow);
+ }
+ }
+
+ @Override
+ public void applyUpdate(final FlowContext workingContext, final FlowContext activeContext) throws FlowUpdateException {
+ }
+
+ @Override
+ public List<ConfigVerificationResult> verifyConfigurationStep(final String stepName, final Map<String, String> overrides, final FlowContext flowContext) {
+ if (SCHEDULE_STEP_NAME.equals(stepName)) {
+ final ConnectorConfigurationContext configContext = flowContext.getConfigurationContext().createWithOverrides(stepName, overrides);
+ final String triggerSchedule = configContext.getProperty(SCHEDULE_STEP_NAME, TRIGGER_SCHEDULE_PARAM).getValue();
+
+ final VersionedExternalFlow flow = getInitialFlow();
+ VersionedFlowUtils.setParameterValue(flow, TRIGGER_SCHEDULE_PARAM, triggerSchedule);
+
+ final ProcessorFacade generateFlowFile = flowContext.getRootGroup().getProcessors().stream()
+ .filter(p -> p.getDefinition().getType().endsWith("GenerateFlowFile"))
+ .findFirst()
+ .orElseThrow();
+
+ return generateFlowFile.verify(flow, Map.of());
+ }
+
+ return List.of();
+ }
+}
diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors/src/main/java/org/apache/nifi/mock/connectors/GenerateAndLog.java b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors/src/main/java/org/apache/nifi/mock/connectors/GenerateAndLog.java
new file mode 100644
index 00000000000..96b9fa63c38
--- /dev/null
+++ b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors/src/main/java/org/apache/nifi/mock/connectors/GenerateAndLog.java
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.mock.connectors;
+
+import org.apache.nifi.components.ConfigVerificationResult;
+import org.apache.nifi.components.connector.AbstractConnector;
+import org.apache.nifi.components.connector.ConfigurationStep;
+import org.apache.nifi.components.connector.FlowUpdateException;
+import org.apache.nifi.components.connector.components.FlowContext;
+import org.apache.nifi.components.connector.util.VersionedFlowUtils;
+import org.apache.nifi.flow.VersionedExternalFlow;
+
+import java.util.List;
+import java.util.Map;
+
+public class GenerateAndLog extends AbstractConnector {
+
+ @Override
+ public VersionedExternalFlow getInitialFlow() {
+ return VersionedFlowUtils.loadFlowFromResource("flows/Generate_and_Update.json");
+ }
+
+ @Override
+ protected void onStepConfigured(final String stepName, final FlowContext flowContext) {
+ }
+
+ @Override
+ public List<ConfigurationStep> getConfigurationSteps() {
+ return List.of();
+ }
+
+ @Override
+ public void applyUpdate(final FlowContext workingContext, final FlowContext activeContext) throws FlowUpdateException {
+ }
+
+ @Override
+ public List<ConfigVerificationResult> verifyConfigurationStep(final String stepName, final Map<String, String> overrides, final FlowContext workingContext) {
+ return List.of();
+ }
+}
diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors/src/main/java/org/apache/nifi/mock/connectors/MissingBundleConnector.java b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors/src/main/java/org/apache/nifi/mock/connectors/MissingBundleConnector.java
new file mode 100644
index 00000000000..d49674ffd03
--- /dev/null
+++ b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors/src/main/java/org/apache/nifi/mock/connectors/MissingBundleConnector.java
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.mock.connectors;
+
+import org.apache.nifi.components.ConfigVerificationResult;
+import org.apache.nifi.components.connector.AbstractConnector;
+import org.apache.nifi.components.connector.ConfigurationStep;
+import org.apache.nifi.components.connector.FlowUpdateException;
+import org.apache.nifi.components.connector.components.FlowContext;
+import org.apache.nifi.flow.Bundle;
+import org.apache.nifi.flow.Position;
+import org.apache.nifi.flow.ScheduledState;
+import org.apache.nifi.flow.VersionedExternalFlow;
+import org.apache.nifi.flow.VersionedProcessGroup;
+import org.apache.nifi.flow.VersionedProcessor;
+
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+
+/**
+ * A test connector that returns an initial flow containing a processor with a bundle that does not exist.
+ * This is used to test the behavior when a connector's initial flow references unavailable components.
+ */
+public class MissingBundleConnector extends AbstractConnector {
+
+ @Override
+ public VersionedExternalFlow getInitialFlow() {
+ final VersionedProcessGroup rootGroup = new VersionedProcessGroup();
+ rootGroup.setIdentifier(UUID.randomUUID().toString());
+ rootGroup.setInstanceIdentifier(UUID.randomUUID().toString());
+ rootGroup.setName("Missing Bundle Connector Flow");
+ rootGroup.setPosition(new Position(0.0, 0.0));
+ rootGroup.setProcessGroups(new HashSet<>());
+ rootGroup.setConnections(new HashSet<>());
+ rootGroup.setInputPorts(new HashSet<>());
+ rootGroup.setOutputPorts(new HashSet<>());
+ rootGroup.setControllerServices(new HashSet<>());
+ rootGroup.setFunnels(new HashSet<>());
+ rootGroup.setLabels(new HashSet<>());
+
+ final VersionedProcessor missingProcessor = new VersionedProcessor();
+ missingProcessor.setIdentifier(UUID.randomUUID().toString());
+ missingProcessor.setInstanceIdentifier(UUID.randomUUID().toString());
+ missingProcessor.setName("Missing Processor");
+ missingProcessor.setType("com.example.nonexistent.MissingProcessor");
+ missingProcessor.setPosition(new Position(100.0, 100.0));
+ missingProcessor.setScheduledState(ScheduledState.ENABLED);
+ missingProcessor.setSchedulingPeriod("0 sec");
+ missingProcessor.setSchedulingStrategy("TIMER_DRIVEN");
+ missingProcessor.setExecutionNode("ALL");
+ missingProcessor.setPenaltyDuration("30 sec");
+ missingProcessor.setYieldDuration("1 sec");
+ missingProcessor.setBulletinLevel("WARN");
+ missingProcessor.setRunDurationMillis(0L);
+ missingProcessor.setConcurrentlySchedulableTaskCount(1);
+ missingProcessor.setAutoTerminatedRelationships(new HashSet<>());
+ missingProcessor.setProperties(Map.of());
+ missingProcessor.setPropertyDescriptors(Map.of());
+ missingProcessor.setGroupIdentifier(rootGroup.getIdentifier());
+
+ final Bundle missingBundle = new Bundle();
+ missingBundle.setGroup("com.example.nonexistent");
+ missingBundle.setArtifact("missing-nar");
+ missingBundle.setVersion("1.0.0");
+ missingProcessor.setBundle(missingBundle);
+
+ rootGroup.setProcessors(new HashSet<>(List.of(missingProcessor)));
+
+ final VersionedExternalFlow externalFlow = new VersionedExternalFlow();
+ externalFlow.setFlowContents(rootGroup);
+ return externalFlow;
+ }
+
+ @Override
+ protected void onStepConfigured(final String stepName, final FlowContext flowContext) {
+ }
+
+ @Override
+ public List<ConfigurationStep> getConfigurationSteps() {
+ return List.of();
+ }
+
+ @Override
+ public void applyUpdate(final FlowContext workingContext, final FlowContext activeContext) throws FlowUpdateException {
+ }
+
+ @Override
+ public List<ConfigVerificationResult> verifyConfigurationStep(final String stepName, final Map<String, String> overrides, final FlowContext workingContext) {
+ return List.of();
+ }
+}
diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/resources/META-INF/services/org.apache.nifi.asset.AssetManager b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors/src/main/resources/META-INF/services/org.apache.nifi.components.connector.Connector
similarity index 78%
copy from nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/resources/META-INF/services/org.apache.nifi.asset.AssetManager
copy to nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors/src/main/resources/META-INF/services/org.apache.nifi.components.connector.Connector
index 6fbf05d9497..379d9026452 100644
--- a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/resources/META-INF/services/org.apache.nifi.asset.AssetManager
+++ b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors/src/main/resources/META-INF/services/org.apache.nifi.components.connector.Connector
@@ -13,4 +13,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-org.apache.nifi.asset.StandardAssetManager
\ No newline at end of file
+org.apache.nifi.mock.connectors.AllowableValuesConnector
+org.apache.nifi.mock.connectors.GenerateAndLog
+org.apache.nifi.mock.connectors.MissingBundleConnector
+org.apache.nifi.mock.connectors.CronScheduleConnector
\ No newline at end of file
diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors/src/main/resources/flows/Cron_Schedule_Connector.json b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors/src/main/resources/flows/Cron_Schedule_Connector.json
new file mode 100644
index 00000000000..d078d4d6fa8
--- /dev/null
+++ b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors/src/main/resources/flows/Cron_Schedule_Connector.json
@@ -0,0 +1,410 @@
+{
+ "flowContents": {
+ "identifier": "1800c04e-f9b9-3293-bfc7-b35f43e0706c",
+ "instanceIdentifier": "4f8ca484-019c-1000-955f-68c5defeb22b",
+ "name": "Cron_Schedule_Connector",
+ "comments": "",
+ "position": {
+ "x": -1254.0,
+ "y": -437.70139741897583
+ },
+ "processGroups": [],
+ "remoteProcessGroups": [],
+ "processors": [
+ {
+ "identifier": "4d4160b1-736c-3be4-bc3a-84fc6eb4ad2a",
+ "instanceIdentifier": "499f9781-71c5-367f-aa17-5441bff29de9",
+ "name": "UpdateAttribute",
+ "comments": "",
+ "position": {
+ "x": -360.0,
+ "y": 24.0
+ },
+ "type": "org.apache.nifi.processors.attributes.UpdateAttribute",
+ "bundle": {
+ "group": "org.apache.nifi",
+ "artifact": "nifi-update-attribute-nar",
+ "version": "2026.1.20.21-SNAPSHOT"
+ },
+ "properties": {
+ "Delete Attributes Expression": null,
+ "Store State": "Do not store state",
+ "Cache Value Lookup Cache Size": "100",
+ "Stateful Variables Initial Value": null
+ },
+ "propertyDescriptors": {
+ "Delete Attributes Expression": {
+ "name": "Delete Attributes Expression",
+ "displayName": "Delete Attributes Expression",
+ "identifiesControllerService": false,
+ "sensitive": false,
+ "dynamic": false
+ },
+ "Store State": {
+ "name": "Store State",
+ "displayName": "Store State",
+ "identifiesControllerService": false,
+ "sensitive": false,
+ "dynamic": false
+ },
+ "Cache Value Lookup Cache Size": {
+ "name": "Cache Value Lookup Cache Size",
+ "displayName": "Cache Value Lookup Cache Size",
+ "identifiesControllerService": false,
+ "sensitive": false,
+ "dynamic": false
+ },
+ "Stateful Variables Initial Value": {
+ "name": "Stateful Variables Initial Value",
+ "displayName": "Stateful Variables Initial Value",
+ "identifiesControllerService": false,
+ "sensitive": false,
+ "dynamic": false
+ }
+ },
+ "style": {},
+ "schedulingPeriod": "0 sec",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "executionNode": "ALL",
+ "penaltyDuration": "30 sec",
+ "yieldDuration": "1 sec",
+ "bulletinLevel": "WARN",
+ "runDurationMillis": 25,
+ "concurrentlySchedulableTaskCount": 1,
+ "autoTerminatedRelationships": [
+ "success"
+ ],
+ "scheduledState": "ENABLED",
+ "retryCount": 10,
+ "retriedRelationships": [],
+ "backoffMechanism": "PENALIZE_FLOWFILE",
+ "maxBackoffPeriod": "10 mins",
+ "componentType": "PROCESSOR",
+ "groupIdentifier": "1800c04e-f9b9-3293-bfc7-b35f43e0706c"
+ },
+ {
+ "identifier": "54ab8572-0e10-39a5-b553-931f9c253023",
+ "instanceIdentifier": "6d6b9cd3-6d31-330a-40f9-185959ad1c78",
+ "name": "GenerateFlowFile",
+ "comments": "",
+ "position": {
+ "x": -360.0,
+ "y": -371.5
+ },
+ "type": "org.apache.nifi.processors.standard.GenerateFlowFile",
+ "bundle": {
+ "group": "org.apache.nifi",
+ "artifact": "nifi-standard-nar",
+ "version": "2026.1.20.21-SNAPSHOT"
+ },
+ "properties": {
+ "File Size": "0B",
+ "Batch Size": "1",
+ "Unique FlowFiles": "false",
+ "Mime Type": null,
+ "Custom Text": null,
+ "Character Set": "UTF-8",
+ "Data Format": "Text",
+ "key": "test.key"
+ },
+ "propertyDescriptors": {
+ "File Size": {
+ "name": "File Size",
+ "displayName": "File Size",
+ "identifiesControllerService": false,
+ "sensitive": false,
+ "dynamic": false
+ },
+ "Batch Size": {
+ "name": "Batch Size",
+ "displayName": "Batch Size",
+ "identifiesControllerService": false,
+ "sensitive": false,
+ "dynamic": false
+ },
+ "Unique FlowFiles": {
+ "name": "Unique FlowFiles",
+ "displayName": "Unique FlowFiles",
+ "identifiesControllerService": false,
+ "sensitive": false,
+ "dynamic": false
+ },
+ "Mime Type": {
+ "name": "Mime Type",
+ "displayName": "Mime Type",
+ "identifiesControllerService": false,
+ "sensitive": false,
+ "dynamic": false
+ },
+ "Custom Text": {
+ "name": "Custom Text",
+ "displayName": "Custom Text",
+ "identifiesControllerService": false,
+ "sensitive": false,
+ "dynamic": false
+ },
+ "Character Set": {
+ "name": "Character Set",
+ "displayName": "Character Set",
+ "identifiesControllerService": false,
+ "sensitive": false,
+ "dynamic": false
+ },
+ "Data Format": {
+ "name": "Data Format",
+ "displayName": "Data Format",
+ "identifiesControllerService": false,
+ "sensitive": false,
+ "dynamic": false
+ },
+ "key": {
+ "name": "key",
+ "displayName": "key",
+ "identifiesControllerService": false,
+ "sensitive": false,
+ "dynamic": true
+ }
+ },
+ "style": {},
+ "schedulingPeriod": "#{Trigger Schedule}",
+ "schedulingStrategy": "CRON_DRIVEN",
+ "executionNode": "ALL",
+ "penaltyDuration": "30 sec",
+ "yieldDuration": "1 sec",
+ "bulletinLevel": "WARN",
+ "runDurationMillis": 0,
+ "concurrentlySchedulableTaskCount": 1,
+ "autoTerminatedRelationships": [],
+ "scheduledState": "ENABLED",
+ "retryCount": 10,
+ "retriedRelationships": [],
+ "backoffMechanism": "PENALIZE_FLOWFILE",
+ "maxBackoffPeriod": "10 mins",
+ "componentType": "PROCESSOR",
+ "groupIdentifier": "1800c04e-f9b9-3293-bfc7-b35f43e0706c"
+ },
+ {
+ "identifier": "34c1ac1b-1f21-3fd6-a734-726d5b142b7a",
+ "instanceIdentifier": "4f8d0670-019c-1000-1ac6-c81ef75a70d0",
+ "name": "LookupAttribute",
+ "comments": "",
+ "position": {
+ "x": -360.0,
+ "y": -168.0
+ },
+ "type": "org.apache.nifi.processors.standard.LookupAttribute",
+ "bundle": {
+ "group": "org.apache.nifi",
+ "artifact": "nifi-standard-nar",
+ "version": "2026.1.20.21-SNAPSHOT"
+ },
+ "properties": {
+ "Lookup Service": "b013f870-aee8-3cc4-b022-ac385ded928d",
+ "test.attribute": "${key}",
+ "Include Empty Values": "true"
+ },
+ "propertyDescriptors": {
+ "Lookup Service": {
+ "name": "Lookup Service",
+ "displayName": "Lookup Service",
+ "identifiesControllerService": true,
+ "sensitive": false,
+ "dynamic": false
+ },
+ "test.attribute": {
+ "name": "test.attribute",
+ "displayName": "test.attribute",
+ "identifiesControllerService": false,
+ "sensitive": false,
+ "dynamic": true
+ },
+ "Include Empty Values": {
+ "name": "Include Empty Values",
+ "displayName": "Include Empty Values",
+ "identifiesControllerService": false,
+ "sensitive": false,
+ "dynamic": false
+ }
+ },
+ "style": {},
+ "schedulingPeriod": "0 sec",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "executionNode": "ALL",
+ "penaltyDuration": "30 sec",
+ "yieldDuration": "1 sec",
+ "bulletinLevel": "WARN",
+ "runDurationMillis": 0,
+ "concurrentlySchedulableTaskCount": 1,
+ "autoTerminatedRelationships": [
+ "failure"
+ ],
+ "scheduledState": "ENABLED",
+ "retryCount": 10,
+ "retriedRelationships": [],
+ "backoffMechanism": "PENALIZE_FLOWFILE",
+ "maxBackoffPeriod": "10 mins",
+ "componentType": "PROCESSOR",
+ "groupIdentifier": "1800c04e-f9b9-3293-bfc7-b35f43e0706c"
+ }
+ ],
+ "inputPorts": [],
+ "outputPorts": [],
+ "connections": [
+ {
+ "identifier": "893ad2c9-4a07-3447-b772-6b7149cfd6c1",
+ "instanceIdentifier": "5b9c96a4-7982-3746-2937-45511fb20c96",
+ "name": "",
+ "source": {
+ "id": "54ab8572-0e10-39a5-b553-931f9c253023",
+ "type": "PROCESSOR",
+ "groupId": "1800c04e-f9b9-3293-bfc7-b35f43e0706c",
+ "name": "GenerateFlowFile",
+ "comments": "",
+ "instanceIdentifier": "6d6b9cd3-6d31-330a-40f9-185959ad1c78"
+ },
+ "destination": {
+ "id": "34c1ac1b-1f21-3fd6-a734-726d5b142b7a",
+ "type": "PROCESSOR",
+ "groupId": "1800c04e-f9b9-3293-bfc7-b35f43e0706c",
+ "name": "LookupAttribute",
+ "comments": "",
+ "instanceIdentifier": "4f8d0670-019c-1000-1ac6-c81ef75a70d0"
+ },
+ "labelIndex": 0,
+ "zIndex": 1,
+ "selectedRelationships": [
+ "success"
+ ],
+ "backPressureObjectThreshold": 10000,
+ "backPressureDataSizeThreshold": "1 GB",
+ "flowFileExpiration": "0 sec",
+ "prioritizers": [],
+ "bends": [],
+ "loadBalanceStrategy": "DO_NOT_LOAD_BALANCE",
+ "partitioningAttribute": "",
+ "loadBalanceCompression": "DO_NOT_COMPRESS",
+ "componentType": "CONNECTION",
+ "groupIdentifier": "1800c04e-f9b9-3293-bfc7-b35f43e0706c"
+ },
+ {
+ "identifier": "f63459de-cf1f-3773-8f93-405518e085e4",
+ "instanceIdentifier": "4f93c5e2-019c-1000-2bea-06d8d611d5f6",
+ "name": "",
+ "source": {
+ "id": "34c1ac1b-1f21-3fd6-a734-726d5b142b7a",
+ "type": "PROCESSOR",
+ "groupId": "1800c04e-f9b9-3293-bfc7-b35f43e0706c",
+ "name": "LookupAttribute",
+ "comments": "",
+ "instanceIdentifier": "4f8d0670-019c-1000-1ac6-c81ef75a70d0"
+ },
+ "destination": {
+ "id": "4d4160b1-736c-3be4-bc3a-84fc6eb4ad2a",
+ "type": "PROCESSOR",
+ "groupId": "1800c04e-f9b9-3293-bfc7-b35f43e0706c",
+ "name": "UpdateAttribute",
+ "comments": "",
+ "instanceIdentifier": "499f9781-71c5-367f-aa17-5441bff29de9"
+ },
+ "labelIndex": 0,
+ "zIndex": 2,
+ "selectedRelationships": [
+ "matched",
+ "unmatched"
+ ],
+ "backPressureObjectThreshold": 10000,
+ "backPressureDataSizeThreshold": "1 GB",
+ "flowFileExpiration": "0 sec",
+ "prioritizers": [],
+ "bends": [],
+ "loadBalanceStrategy": "DO_NOT_LOAD_BALANCE",
+ "partitioningAttribute": "",
+ "loadBalanceCompression": "DO_NOT_COMPRESS",
+ "componentType": "CONNECTION",
+ "groupIdentifier": "1800c04e-f9b9-3293-bfc7-b35f43e0706c"
+ }
+ ],
+ "labels": [],
+ "funnels": [],
+ "controllerServices": [
+ {
+ "identifier": "b013f870-aee8-3cc4-b022-ac385ded928d",
+ "instanceIdentifier": "4f8d53d4-019c-1000-d376-abfaa091dc37",
+ "name": "SimpleKeyValueLookupService",
+ "comments": "",
+ "type": "org.apache.nifi.lookup.SimpleKeyValueLookupService",
+ "bundle": {
+ "group": "org.apache.nifi",
+ "artifact": "nifi-lookup-services-nar",
+ "version": "2026.1.20.21-SNAPSHOT"
+ },
+ "properties": {
+ "test.key": "Test Value"
+ },
+ "propertyDescriptors": {
+ "test.key": {
+ "name": "test.key",
+ "displayName": "test.key",
+ "identifiesControllerService": false,
+ "sensitive": false,
+ "dynamic": true
+ }
+ },
+ "controllerServiceApis": [
+ {
+ "type": "org.apache.nifi.lookup.StringLookupService",
+ "bundle": {
+ "group": "org.apache.nifi",
+ "artifact": "nifi-standard-services-api-nar",
+ "version": "2026.1.20.21-SNAPSHOT"
+ }
+ },
+ {
+ "type": "org.apache.nifi.lookup.LookupService",
+ "bundle": {
+ "group": "org.apache.nifi",
+ "artifact": "nifi-standard-services-api-nar",
+ "version": "2026.1.20.21-SNAPSHOT"
+ }
+ }
+ ],
+ "scheduledState": "DISABLED",
+ "bulletinLevel": "WARN",
+ "componentType": "CONTROLLER_SERVICE",
+ "groupIdentifier": "1800c04e-f9b9-3293-bfc7-b35f43e0706c"
+ }
+ ],
+ "parameterContextName": "Generate Parameters",
+ "defaultFlowFileExpiration": "0 sec",
+ "defaultBackPressureObjectThreshold": 10000,
+ "defaultBackPressureDataSizeThreshold": "1 GB",
+ "scheduledState": "ENABLED",
+ "executionEngine": "INHERITED",
+ "maxConcurrentTasks": 1,
+ "statelessFlowTimeout": "1 min",
+ "flowFileConcurrency": "UNBOUNDED",
+ "flowFileOutboundPolicy": "STREAM_WHEN_AVAILABLE",
+ "componentType": "PROCESS_GROUP"
+ },
+ "externalControllerServices": {},
+ "parameterContexts": {
+ "Generate Parameters": {
+ "name": "Generate Parameters",
+ "parameters": [
+ {
+ "name": "Trigger Schedule",
+ "description": "",
+ "sensitive": false,
+ "provided": false,
+ "value": ""
+ }
+ ],
+ "inheritedParameterContexts": [],
+ "description": "",
+ "componentType": "PARAMETER_CONTEXT"
+ }
+ },
+ "flowEncodingVersion": "1.0",
+ "parameterProviders": {},
+ "latest": false
+}
\ No newline at end of file
diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors/src/main/resources/flows/Generate_and_Update.json b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors/src/main/resources/flows/Generate_and_Update.json
new file mode 100644
index 00000000000..d4f5fc798ed
--- /dev/null
+++ b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors/src/main/resources/flows/Generate_and_Update.json
@@ -0,0 +1,393 @@
+{
+ "flowContents": {
+ "identifier": "1800c04e-f9b9-3293-bfc7-b35f43e0706c",
+ "instanceIdentifier": "4f8ca484-019c-1000-955f-68c5defeb22b",
+ "name": "Generate_and_Update",
+ "comments": "",
+ "position": {
+ "x": -1254.0,
+ "y": -437.70139741897583
+ },
+ "processGroups": [],
+ "remoteProcessGroups": [],
+ "processors": [
+ {
+ "identifier": "4d4160b1-736c-3be4-bc3a-84fc6eb4ad2a",
+ "instanceIdentifier": "499f9781-71c5-367f-aa17-5441bff29de9",
+ "name": "UpdateAttribute",
+ "comments": "",
+ "position": {
+ "x": -360.0,
+ "y": 24.0
+ },
+ "type": "org.apache.nifi.processors.attributes.UpdateAttribute",
+ "bundle": {
+ "group": "org.apache.nifi",
+ "artifact": "nifi-update-attribute-nar",
+ "version": "2026.1.20.21-SNAPSHOT"
+ },
+ "properties": {
+ "Delete Attributes Expression": null,
+ "Store State": "Do not store state",
+ "Cache Value Lookup Cache Size": "100",
+ "Stateful Variables Initial Value": null
+ },
+ "propertyDescriptors": {
+ "Delete Attributes Expression": {
+ "name": "Delete Attributes Expression",
+ "displayName": "Delete Attributes Expression",
+ "identifiesControllerService": false,
+ "sensitive": false,
+ "dynamic": false
+ },
+ "Store State": {
+ "name": "Store State",
+ "displayName": "Store State",
+ "identifiesControllerService": false,
+ "sensitive": false,
+ "dynamic": false
+ },
+ "Cache Value Lookup Cache Size": {
+ "name": "Cache Value Lookup Cache Size",
+ "displayName": "Cache Value Lookup Cache Size",
+ "identifiesControllerService": false,
+ "sensitive": false,
+ "dynamic": false
+ },
+ "Stateful Variables Initial Value": {
+ "name": "Stateful Variables Initial Value",
+ "displayName": "Stateful Variables Initial Value",
+ "identifiesControllerService": false,
+ "sensitive": false,
+ "dynamic": false
+ }
+ },
+ "style": {},
+ "schedulingPeriod": "0 sec",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "executionNode": "ALL",
+ "penaltyDuration": "30 sec",
+ "yieldDuration": "1 sec",
+ "bulletinLevel": "WARN",
+ "runDurationMillis": 25,
+ "concurrentlySchedulableTaskCount": 1,
+ "autoTerminatedRelationships": [
+ "success"
+ ],
+ "scheduledState": "ENABLED",
+ "retryCount": 10,
+ "retriedRelationships": [],
+ "backoffMechanism": "PENALIZE_FLOWFILE",
+ "maxBackoffPeriod": "10 mins",
+ "componentType": "PROCESSOR",
+ "groupIdentifier": "1800c04e-f9b9-3293-bfc7-b35f43e0706c"
+ },
+ {
+ "identifier": "54ab8572-0e10-39a5-b553-931f9c253023",
+ "instanceIdentifier": "6d6b9cd3-6d31-330a-40f9-185959ad1c78",
+ "name": "GenerateFlowFile",
+ "comments": "",
+ "position": {
+ "x": -360.0,
+ "y": -371.5
+ },
+ "type": "org.apache.nifi.processors.standard.GenerateFlowFile",
+ "bundle": {
+ "group": "org.apache.nifi",
+ "artifact": "nifi-standard-nar",
+ "version": "2026.1.20.21-SNAPSHOT"
+ },
+ "properties": {
+ "File Size": "0B",
+ "Batch Size": "1",
+ "Unique FlowFiles": "false",
+ "Mime Type": null,
+ "Custom Text": null,
+ "Character Set": "UTF-8",
+ "Data Format": "Text",
+ "key": "test.key"
+ },
+ "propertyDescriptors": {
+ "File Size": {
+ "name": "File Size",
+ "displayName": "File Size",
+ "identifiesControllerService": false,
+ "sensitive": false,
+ "dynamic": false
+ },
+ "Batch Size": {
+ "name": "Batch Size",
+ "displayName": "Batch Size",
+ "identifiesControllerService": false,
+ "sensitive": false,
+ "dynamic": false
+ },
+ "Unique FlowFiles": {
+ "name": "Unique FlowFiles",
+ "displayName": "Unique FlowFiles",
+ "identifiesControllerService": false,
+ "sensitive": false,
+ "dynamic": false
+ },
+ "Mime Type": {
+ "name": "Mime Type",
+ "displayName": "Mime Type",
+ "identifiesControllerService": false,
+ "sensitive": false,
+ "dynamic": false
+ },
+ "Custom Text": {
+ "name": "Custom Text",
+ "displayName": "Custom Text",
+ "identifiesControllerService": false,
+ "sensitive": false,
+ "dynamic": false
+ },
+ "Character Set": {
+ "name": "Character Set",
+ "displayName": "Character Set",
+ "identifiesControllerService": false,
+ "sensitive": false,
+ "dynamic": false
+ },
+ "Data Format": {
+ "name": "Data Format",
+ "displayName": "Data Format",
+ "identifiesControllerService": false,
+ "sensitive": false,
+ "dynamic": false
+ },
+ "key": {
+ "name": "key",
+ "displayName": "key",
+ "identifiesControllerService": false,
+ "sensitive": false,
+ "dynamic": true
+ }
+ },
+ "style": {},
+ "schedulingPeriod": "0 sec",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "executionNode": "ALL",
+ "penaltyDuration": "30 sec",
+ "yieldDuration": "1 sec",
+ "bulletinLevel": "WARN",
+ "runDurationMillis": 0,
+ "concurrentlySchedulableTaskCount": 1,
+ "autoTerminatedRelationships": [],
+ "scheduledState": "ENABLED",
+ "retryCount": 10,
+ "retriedRelationships": [],
+ "backoffMechanism": "PENALIZE_FLOWFILE",
+ "maxBackoffPeriod": "10 mins",
+ "componentType": "PROCESSOR",
+ "groupIdentifier": "1800c04e-f9b9-3293-bfc7-b35f43e0706c"
+ },
+ {
+ "identifier": "34c1ac1b-1f21-3fd6-a734-726d5b142b7a",
+ "instanceIdentifier": "4f8d0670-019c-1000-1ac6-c81ef75a70d0",
+ "name": "LookupAttribute",
+ "comments": "",
+ "position": {
+ "x": -360.0,
+ "y": -168.0
+ },
+ "type": "org.apache.nifi.processors.standard.LookupAttribute",
+ "bundle": {
+ "group": "org.apache.nifi",
+ "artifact": "nifi-standard-nar",
+ "version": "2026.1.20.21-SNAPSHOT"
+ },
+ "properties": {
+ "Lookup Service": "b013f870-aee8-3cc4-b022-ac385ded928d",
+ "test.attribute": "${key}",
+ "Include Empty Values": "true"
+ },
+ "propertyDescriptors": {
+ "Lookup Service": {
+ "name": "Lookup Service",
+ "displayName": "Lookup Service",
+ "identifiesControllerService": true,
+ "sensitive": false,
+ "dynamic": false
+ },
+ "test.attribute": {
+ "name": "test.attribute",
+ "displayName": "test.attribute",
+ "identifiesControllerService": false,
+ "sensitive": false,
+ "dynamic": true
+ },
+ "Include Empty Values": {
+ "name": "Include Empty Values",
+ "displayName": "Include Empty Values",
+ "identifiesControllerService": false,
+ "sensitive": false,
+ "dynamic": false
+ }
+ },
+ "style": {},
+ "schedulingPeriod": "0 sec",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "executionNode": "ALL",
+ "penaltyDuration": "30 sec",
+ "yieldDuration": "1 sec",
+ "bulletinLevel": "WARN",
+ "runDurationMillis": 0,
+ "concurrentlySchedulableTaskCount": 1,
+ "autoTerminatedRelationships": [
+ "failure"
+ ],
+ "scheduledState": "ENABLED",
+ "retryCount": 10,
+ "retriedRelationships": [],
+ "backoffMechanism": "PENALIZE_FLOWFILE",
+ "maxBackoffPeriod": "10 mins",
+ "componentType": "PROCESSOR",
+ "groupIdentifier": "1800c04e-f9b9-3293-bfc7-b35f43e0706c"
+ }
+ ],
+ "inputPorts": [],
+ "outputPorts": [],
+ "connections": [
+ {
+ "identifier": "893ad2c9-4a07-3447-b772-6b7149cfd6c1",
+ "instanceIdentifier": "5b9c96a4-7982-3746-2937-45511fb20c96",
+ "name": "",
+ "source": {
+ "id": "54ab8572-0e10-39a5-b553-931f9c253023",
+ "type": "PROCESSOR",
+ "groupId": "1800c04e-f9b9-3293-bfc7-b35f43e0706c",
+ "name": "GenerateFlowFile",
+ "comments": "",
+ "instanceIdentifier": "6d6b9cd3-6d31-330a-40f9-185959ad1c78"
+ },
+ "destination": {
+ "id": "34c1ac1b-1f21-3fd6-a734-726d5b142b7a",
+ "type": "PROCESSOR",
+ "groupId": "1800c04e-f9b9-3293-bfc7-b35f43e0706c",
+ "name": "LookupAttribute",
+ "comments": "",
+ "instanceIdentifier": "4f8d0670-019c-1000-1ac6-c81ef75a70d0"
+ },
+ "labelIndex": 0,
+ "zIndex": 1,
+ "selectedRelationships": [
+ "success"
+ ],
+ "backPressureObjectThreshold": 10000,
+ "backPressureDataSizeThreshold": "1 GB",
+ "flowFileExpiration": "0 sec",
+ "prioritizers": [],
+ "bends": [],
+ "loadBalanceStrategy": "DO_NOT_LOAD_BALANCE",
+ "partitioningAttribute": "",
+ "loadBalanceCompression": "DO_NOT_COMPRESS",
+ "componentType": "CONNECTION",
+ "groupIdentifier": "1800c04e-f9b9-3293-bfc7-b35f43e0706c"
+ },
+ {
+ "identifier": "f63459de-cf1f-3773-8f93-405518e085e4",
+ "instanceIdentifier": "4f93c5e2-019c-1000-2bea-06d8d611d5f6",
+ "name": "",
+ "source": {
+ "id": "34c1ac1b-1f21-3fd6-a734-726d5b142b7a",
+ "type": "PROCESSOR",
+ "groupId": "1800c04e-f9b9-3293-bfc7-b35f43e0706c",
+ "name": "LookupAttribute",
+ "comments": "",
+ "instanceIdentifier": "4f8d0670-019c-1000-1ac6-c81ef75a70d0"
+ },
+ "destination": {
+ "id": "4d4160b1-736c-3be4-bc3a-84fc6eb4ad2a",
+ "type": "PROCESSOR",
+ "groupId": "1800c04e-f9b9-3293-bfc7-b35f43e0706c",
+ "name": "UpdateAttribute",
+ "comments": "",
+ "instanceIdentifier": "499f9781-71c5-367f-aa17-5441bff29de9"
+ },
+ "labelIndex": 0,
+ "zIndex": 2,
+ "selectedRelationships": [
+ "matched",
+ "unmatched"
+ ],
+ "backPressureObjectThreshold": 10000,
+ "backPressureDataSizeThreshold": "1 GB",
+ "flowFileExpiration": "0 sec",
+ "prioritizers": [],
+ "bends": [],
+ "loadBalanceStrategy": "DO_NOT_LOAD_BALANCE",
+ "partitioningAttribute": "",
+ "loadBalanceCompression": "DO_NOT_COMPRESS",
+ "componentType": "CONNECTION",
+ "groupIdentifier": "1800c04e-f9b9-3293-bfc7-b35f43e0706c"
+ }
+ ],
+ "labels": [],
+ "funnels": [],
+ "controllerServices": [
+ {
+ "identifier": "b013f870-aee8-3cc4-b022-ac385ded928d",
+ "instanceIdentifier": "4f8d53d4-019c-1000-d376-abfaa091dc37",
+ "name": "SimpleKeyValueLookupService",
+ "comments": "",
+ "type": "org.apache.nifi.lookup.SimpleKeyValueLookupService",
+ "bundle": {
+ "group": "org.apache.nifi",
+ "artifact": "nifi-lookup-services-nar",
+ "version": "2026.1.20.21-SNAPSHOT"
+ },
+ "properties": {
+ "test.key": "Test Value"
+ },
+ "propertyDescriptors": {
+ "test.key": {
+ "name": "test.key",
+ "displayName": "test.key",
+ "identifiesControllerService": false,
+ "sensitive": false,
+ "dynamic": true
+ }
+ },
+ "controllerServiceApis": [
+ {
+ "type": "org.apache.nifi.lookup.LookupService",
+ "bundle": {
+ "group": "org.apache.nifi",
+ "artifact": "nifi-standard-services-api-nar",
+ "version": "2026.1.20.21-SNAPSHOT"
+ }
+ },
+ {
+ "type": "org.apache.nifi.lookup.StringLookupService",
+ "bundle": {
+ "group": "org.apache.nifi",
+ "artifact": "nifi-standard-services-api-nar",
+ "version": "2026.1.20.21-SNAPSHOT"
+ }
+ }
+ ],
+ "scheduledState": "DISABLED",
+ "bulletinLevel": "WARN",
+ "componentType": "CONTROLLER_SERVICE",
+ "groupIdentifier": "1800c04e-f9b9-3293-bfc7-b35f43e0706c"
+ }
+ ],
+ "defaultFlowFileExpiration": "0 sec",
+ "defaultBackPressureObjectThreshold": 10000,
+ "defaultBackPressureDataSizeThreshold": "1 GB",
+ "scheduledState": "ENABLED",
+ "executionEngine": "INHERITED",
+ "maxConcurrentTasks": 1,
+ "statelessFlowTimeout": "1 min",
+ "flowFileConcurrency": "UNBOUNDED",
+ "flowFileOutboundPolicy": "STREAM_WHEN_AVAILABLE",
+ "componentType": "PROCESS_GROUP"
+ },
+ "externalControllerServices": {},
+ "parameterContexts": {},
+ "flowEncodingVersion": "1.0",
+ "parameterProviders": {},
+ "latest": false
+}
\ No newline at end of file
diff --git a/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/pom.xml b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/pom.xml
similarity index 50%
copy from nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/pom.xml
copy to nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/pom.xml
index f877db14d04..bc7d2ebc1a8 100644
--- a/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/pom.xml
+++ b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/pom.xml
@@ -14,40 +14,20 @@
limitations under the License.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
<parent>
- <artifactId>nifi-system-test-extensions-bundle</artifactId>
<groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-connector-mock-bundle</artifactId>
<version>2.9.0-SNAPSHOT</version>
</parent>
- <modelVersion>4.0.0</modelVersion>
- <artifactId>nifi-system-test-extensions</artifactId>
+ <artifactId>nifi-connector-mock-test-bundle</artifactId>
+ <packaging>pom</packaging>
+
+ <modules>
+ <module>nifi-connector-mock-test-connectors</module>
+ <module>nifi-connector-mock-test-connectors-nar</module>
+ <module>nifi-connector-mock-integration-tests</module>
+ </modules>
+</project>
- <dependencies>
- <dependency>
- <groupId>org.apache.nifi</groupId>
- <artifactId>nifi-stateless-api</artifactId>
- <version>2.9.0-SNAPSHOT</version>
- </dependency>
- <dependency>
- <groupId>org.apache.nifi</groupId>
- <artifactId>nifi-utils</artifactId>
- <version>2.9.0-SNAPSHOT</version>
- </dependency>
- <dependency>
- <groupId>org.apache.nifi</groupId>
- <artifactId>nifi-bin-manager</artifactId>
- <version>2.9.0-SNAPSHOT</version>
- </dependency>
- <dependency>
- <groupId>com.fasterxml.jackson.core</groupId>
- <artifactId>jackson-databind</artifactId>
- </dependency>
- <dependency>
- <groupId>org.apache.nifi</groupId>
- <artifactId>nifi-system-test-extensions-services-api</artifactId>
- <version>2.9.0-SNAPSHOT</version>
- <scope>provided</scope>
- </dependency>
- </dependencies>
-</project>
\ No newline at end of file
diff --git a/nifi-connector-mock-bundle/nifi-connector-mock/pom.xml b/nifi-connector-mock-bundle/nifi-connector-mock/pom.xml
new file mode 100644
index 00000000000..e32af9beacc
--- /dev/null
+++ b/nifi-connector-mock-bundle/nifi-connector-mock/pom.xml
@@ -0,0 +1,147 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-connector-mock-bundle</artifactId>
+ <version>2.9.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-connector-mock</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-nar-utils</artifactId>
+ <version>2.9.0-SNAPSHOT</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-properties</artifactId>
+ <version>2.9.0-SNAPSHOT</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-connector-mock-api</artifactId>
+ <version>2.9.0-SNAPSHOT</version>
+ </dependency>
+
+ <!-- Dependencies necessary to bootstrap NiFi -->
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-framework-nar-utils</artifactId>
+ <version>2.9.0-SNAPSHOT</version>
+ </dependency>
+
+
+ <!-- Core NiFi API dependencies -->
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-framework-api</artifactId>
+ <version>2.9.0-SNAPSHOT</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-property-utils</artifactId>
+ <version>2.9.0-SNAPSHOT</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-python-framework-api</artifactId>
+ <version>2.9.0-SNAPSHOT</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-stateless-api</artifactId>
+ <version>2.9.0-SNAPSHOT</version>
+ </dependency>
+
+ <!-- Logging dependencies to ensure that everything funnels through SLF4J -->
+ <dependency>
+ <groupId>org.slf4j</groupId>
+ <artifactId>log4j-over-slf4j</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.slf4j</groupId>
+ <artifactId>jul-to-slf4j</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.slf4j</groupId>
+ <artifactId>jcl-over-slf4j</artifactId>
+ </dependency>
+
+ <!-- Test Dependencies -->
+ <dependency>
+ <groupId>org.junit.jupiter</groupId>
+ <artifactId>junit-jupiter</artifactId>
+ <scope>test</scope>
+ </dependency>
+
+ <!-- NARs required -->
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-framework-nar</artifactId>
+ <version>2.9.0-SNAPSHOT</version>
+ <type>nar</type>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-jetty-nar</artifactId>
+ <version>2.9.0-SNAPSHOT</version>
+ <type>nar</type>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-provenance-repository-nar</artifactId>
+ <version>2.9.0-SNAPSHOT</version>
+ <type>nar</type>
+ </dependency>
+ </dependencies>
+
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-dependency-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>copy-nar</id>
+ <phase>generate-resources</phase>
+ <goals>
+ <goal>copy-dependencies</goal>
+ </goals>
+ <configuration>
+ <outputDirectory>${project.build.directory}/libDir</outputDirectory>
+ <includeTypes>jar,nar</includeTypes>
+ <includeScope>compile</includeScope>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+</project>
\ No newline at end of file
diff --git a/nifi-connector-mock-bundle/nifi-connector-mock/src/main/java/org/apache/nifi/mock/connector/StandardConnectorTestRunner.java b/nifi-connector-mock-bundle/nifi-connector-mock/src/main/java/org/apache/nifi/mock/connector/StandardConnectorTestRunner.java
new file mode 100644
index 00000000000..7153f8d868b
--- /dev/null
+++ b/nifi-connector-mock-bundle/nifi-connector-mock/src/main/java/org/apache/nifi/mock/connector/StandardConnectorTestRunner.java
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.mock.connector;
+
+import org.apache.nifi.NiFiServer;
+import org.apache.nifi.bundle.Bundle;
+import org.apache.nifi.components.DescribedValue;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.connector.AssetReference;
+import org.apache.nifi.components.connector.ConnectorValueReference;
+import org.apache.nifi.components.connector.FlowUpdateException;
+import org.apache.nifi.components.connector.SecretReference;
+import org.apache.nifi.components.connector.StepConfiguration;
+import org.apache.nifi.controller.ControllerService;
+import org.apache.nifi.flow.VersionedExternalFlow;
+import org.apache.nifi.mock.connector.server.ConnectorConfigVerificationResult;
+import org.apache.nifi.mock.connector.server.ConnectorMockServer;
+import org.apache.nifi.mock.connector.server.ConnectorTestRunner;
+import org.apache.nifi.nar.ExtensionMapping;
+import org.apache.nifi.nar.NarClassLoaders;
+import org.apache.nifi.nar.NarUnpackMode;
+import org.apache.nifi.nar.NarUnpacker;
+import org.apache.nifi.nar.SystemBundle;
+import org.apache.nifi.processor.Processor;
+import org.apache.nifi.util.NiFiProperties;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Path;
+import java.time.Duration;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+
+public class StandardConnectorTestRunner implements ConnectorTestRunner, Closeable {
+ private final File narLibraryDirectory;
+ private final int httpPort;
+
+ private ConnectorMockServer mockServer;
+
+ private StandardConnectorTestRunner(final Builder builder) {
+ this.narLibraryDirectory = builder.narLibraryDirectory;
+ this.httpPort = builder.httpPort;
+
+ try {
+ bootstrapInstance();
+ } catch (final Exception e) {
+ throw new RuntimeException("Failed to bootstrap ConnectorTestRunner", e);
+ }
+
+ // It is important that we register the processor and controller service mocks before instantiating the connector.
+ // Otherwise, the call to instantiateConnector will initialize the Connector, which may update the flow.
+ // If the flow is updated before the mocks are registered, the components will be created without
+ // using the mocks. Subsequent updates to the flow will not replace the components already created because
+ // these are not recognized as updates to the flow, since the framework assumes that the type of a component
+ // with a given ID does not change.
+ builder.processorMocks.forEach(mockServer::mockProcessor);
+ builder.controllerServiceMocks.forEach(mockServer::mockControllerService);
+
+ mockServer.instantiateConnector(builder.connectorClassName);
+ }
+
+ private void bootstrapInstance() throws IOException, ClassNotFoundException {
+ final List<Path> libDirectoryPaths = List.of(narLibraryDirectory.toPath());
+ final File extensionsWorkingDir = new File("target/work/extensions");
+ final File frameworkWorkingDir = new File("target/work/framework");
+
+ final Bundle systemBundle = SystemBundle.create(narLibraryDirectory.getAbsolutePath(), ClassLoader.getSystemClassLoader());
+
+ final ExtensionMapping extensionMapping = NarUnpacker.unpackNars(systemBundle, frameworkWorkingDir, extensionsWorkingDir, libDirectoryPaths, true,
+ NarClassLoaders.FRAMEWORK_NAR_ID, true, false, NarUnpackMode.UNPACK_INDIVIDUAL_JARS, bundleCoordinate -> true);
+
+ final NarClassLoaders narClassLoaders = new NarClassLoaders();
+ narClassLoaders.init(frameworkWorkingDir, extensionsWorkingDir);
+
+ final NiFiServer nifiServer = narClassLoaders.getServer();
+ if (nifiServer == null) {
+ throw new RuntimeException("Could not find NiFiServer instance");
+ }
+ if (!(nifiServer instanceof ConnectorMockServer)) {
+ throw new RuntimeException("Test ClassPath does not contain ConnectorMockServer. " +
+ "Ensure that the appropriate module is packaged in the NAR library directory: " + narLibraryDirectory.getAbsolutePath());
+ }
+
+ // Set Application Server Class Loader for subsequent operations
+ final ClassLoader applicationServerClassLoader = narClassLoaders.getServer().getClass().getClassLoader();
+ Thread.currentThread().setContextClassLoader(applicationServerClassLoader);
+
+ final Set<Bundle> narBundles = narClassLoaders.getBundles();
+
+ final Properties additionalProperties = new Properties();
+ if (httpPort >= 0) {
+ additionalProperties.setProperty(NiFiProperties.WEB_HTTP_PORT, String.valueOf(httpPort));
+ }
+
+ final NiFiProperties properties;
+ try (final InputStream propertiesIn = getClass().getClassLoader().getResourceAsStream("nifi.properties")) {
+ properties = NiFiProperties.createBasicNiFiProperties(propertiesIn, additionalProperties);
+ }
+
+ nifiServer.initialize(properties, systemBundle, narBundles, extensionMapping);
+ nifiServer.start();
+
+ mockServer = (ConnectorMockServer) nifiServer;
+ mockServer.registerMockBundle(getClass().getClassLoader(), new File(extensionsWorkingDir, "mock-implementations-bundle"));
+ }
+
+ @Override
+ public void close() {
+ if (mockServer != null) {
+ mockServer.stop();
+ }
+ }
+
+ @Override
+ public void applyUpdate() throws FlowUpdateException {
+ mockServer.applyUpdate();
+ }
+
+ @Override
+ public void configure(final String stepName, final StepConfiguration configuration) throws FlowUpdateException {
+ mockServer.configure(stepName, configuration);
+ }
+
+ @Override
+ public void configure(final String stepName, final Map<String, String> propertyValues) throws FlowUpdateException {
+ mockServer.configure(stepName, propertyValues);
+ }
+
+ @Override
+ public void configure(final String stepName, final Map<String, String> propertyValues, final Map<String, ConnectorValueReference> propertyReferences) throws FlowUpdateException {
+ mockServer.configure(stepName, propertyValues, propertyReferences);
+ }
+
+ @Override
+ public SecretReference createSecretReference(final String secretName) {
+ return mockServer.createSecretReference(secretName);
+ }
+
+ @Override
+ public ConnectorConfigVerificationResult verifyConfiguration(final String stepName, final Map<String, String> propertyValueOverrides) {
+ return mockServer.verifyConfiguration(stepName, propertyValueOverrides);
+ }
+
+ @Override
+ public ConnectorConfigVerificationResult verifyConfiguration(final String stepName, final Map<String, String> propertyValueOverrides,
+ final Map<String, ConnectorValueReference> referenceOverrides) {
+
+ return mockServer.verifyConfiguration(stepName, propertyValueOverrides, referenceOverrides);
+ }
+
+ @Override
+ public ConnectorConfigVerificationResult verifyConfiguration(final String stepName, final StepConfiguration configurationOverrides) {
+ return mockServer.verifyConfiguration(stepName, configurationOverrides);
+ }
+
+ @Override
+ public void addSecret(final String name, final String value) {
+ mockServer.addSecret(name, value);
+ }
+
+ @Override
+ public AssetReference addAsset(final File file) {
+ return mockServer.addAsset(file);
+ }
+
+ @Override
+ public AssetReference addAsset(final String assetName, final InputStream contents) {
+ return mockServer.addAsset(assetName, contents);
+ }
+
+ @Override
+ public void startConnector() {
+ mockServer.startConnector();
+ }
+
+ @Override
+ public void stopConnector() {
+ mockServer.stopConnector();
+ }
+
+ @Override
+ public void waitForDataIngested(final Duration maxWaitTime) {
+ mockServer.waitForDataIngested(maxWaitTime);
+ }
+
+ @Override
+ public void waitForIdle(final Duration maxWaitTime) {
+ mockServer.waitForIdle(maxWaitTime);
+ }
+
+ @Override
+ public void waitForIdle(final Duration minIdleTime, final Duration maxWaitTime) {
+ mockServer.waitForIdle(minIdleTime, maxWaitTime);
+ }
+
+ @Override
+ public List<ValidationResult> validate() {
+ return mockServer.validate();
+ }
+
+ @Override
+ public int getHttpPort() {
+ return mockServer.getHttpPort();
+ }
+
+ @Override
+ public List<DescribedValue> fetchAllowableValues(final String stepName, final String propertyName) {
+ return mockServer.fetchAllowableValues(stepName, propertyName);
+ }
+
+ @Override
+ public VersionedExternalFlow getActiveFlowSnapshot() {
+ return mockServer.getActiveFlowSnapshot();
+ }
+
+ @Override
+ public VersionedExternalFlow getWorkingFlowSnapshot() {
+ return mockServer.getWorkingFlowSnapshot();
+ }
+
+
+ public static class Builder {
+ private String connectorClassName;
+ private File narLibraryDirectory;
+ private int httpPort = -1;
+ private final Map<String, Class<? extends Processor>> processorMocks = new HashMap<>();
+ private final Map<String, Class<? extends ControllerService>> controllerServiceMocks = new HashMap<>();
+
+ public Builder connectorClassName(final String connectorClassName) {
+ this.connectorClassName = connectorClassName;
+ return this;
+ }
+
+ public Builder narLibraryDirectory(final File libDirectory) {
+ this.narLibraryDirectory = libDirectory;
+ return this;
+ }
+
+ public Builder httpPort(final int httpPort) {
+ this.httpPort = httpPort;
+ return this;
+ }
+
+ public Builder mockProcessor(final String processorType, final Class<? extends Processor> mockProcessorClass) {
+ processorMocks.put(processorType, mockProcessorClass);
+ return this;
+ }
+
+ public Builder mockControllerService(final String controllerServiceType, final Class<? extends ControllerService> mockControllerServiceClass) {
+ controllerServiceMocks.put(controllerServiceType, mockControllerServiceClass);
+ return this;
+ }
+
+ public StandardConnectorTestRunner build() {
+ if (!narLibraryDirectory.exists() || !narLibraryDirectory.isDirectory()) {
+ throw new IllegalArgumentException("NAR file does not exist or is not a directory: " + narLibraryDirectory.getAbsolutePath());
+ }
+
+ return new StandardConnectorTestRunner(this);
+ }
+ }
+}
diff --git a/nifi-connector-mock-bundle/nifi-connector-mock/src/main/resources/nifi.properties b/nifi-connector-mock-bundle/nifi-connector-mock/src/main/resources/nifi.properties
new file mode 100644
index 00000000000..44deabc71ad
--- /dev/null
+++ b/nifi-connector-mock-bundle/nifi-connector-mock/src/main/resources/nifi.properties
@@ -0,0 +1,165 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Core Properties #
+nifi.flow.configuration.file=target/nifi-storage/conf/flow.json.gz
+nifi.flow.configuration.archive.enabled=true
+nifi.flow.configuration.archive.dir=target/nifi-storage/conf/archive/
+nifi.flow.configuration.archive.max.time=30 days
+nifi.flow.configuration.archive.max.storage=500 MB
+nifi.flow.configuration.archive.max.count=
+nifi.flowcontroller.autoResumeState=true
+nifi.flowcontroller.graceful.shutdown.period=10 sec
+nifi.flowservice.writedelay.interval=500 ms
+nifi.administrative.yield.duration=3 sec
+# If a component has no work to do (is "bored"), how long should we wait before checking again for work?
+nifi.bored.yield.duration=10 millis
+nifi.queue.backpressure.count=10000
+nifi.queue.backpressure.size=1 GB
+
+nifi.nar.library.directory=./lib
+nifi.nar.library.autoload.directory=./extensions
+nifi.nar.working.directory=./work/nar/
+
+#####################
+# Python Extensions #
+#####################
+# Uncomment in order to enable Python Extensions.
+#nifi.python.command=python
+nifi.python.framework.source.directory=./python/framework
+nifi.python.extensions.source.directory.default=./python/extensions
+nifi.python.working.directory=./work/python
+nifi.python.max.processes=100
+nifi.python.max.processes.per.extension.type=10
+
+####################
+# State Management #
+####################
+nifi.state.management.configuration.file=target/nifi-storage/state-management.xml
+# The ID of the local state provider
+nifi.state.management.provider.local=local-provider
+
+# Database Settings
+nifi.database.directory=target/nifi-storage/database_repository
+
+# FlowFile Repository
+nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository
+nifi.flowfile.repository.wal.implementation=org.apache.nifi.wali.SequentialAccessWriteAheadLog
+nifi.flowfile.repository.directory=target/nifi-storage/flowfile_repository
+nifi.flowfile.repository.checkpoint.interval=1 mins
+nifi.flowfile.repository.always.sync=false
+
+nifi.swap.manager.implementation=org.apache.nifi.controller.FileSystemSwapManager
+nifi.queue.swap.threshold=20000
+nifi.swap.in.period=5 sec
+nifi.swap.in.threads=1
+nifi.swap.out.period=5 sec
+nifi.swap.out.threads=4
+
+# Content Repository
+nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
+nifi.content.claim.max.appendable.size=50 KB
+nifi.content.repository.directory.default=target/nifi-storage/content_repository
+nifi.content.repository.archive.max.retention.period=12 hours
+nifi.content.repository.archive.max.usage.percentage=90%
+nifi.content.repository.archive.enabled=false
+nifi.content.repository.always.sync=false
+
+# Provenance Repository Properties
+nifi.provenance.repository.implementation=org.apache.nifi.provenance.VolatileProvenanceRepository
+
+# Volatile Provenance Respository Properties
+nifi.provenance.repository.buffer.size=1000
+
+# Component Status Repository
+nifi.components.status.repository.implementation=org.apache.nifi.controller.status.history.VolatileComponentStatusRepository
+nifi.components.status.repository.buffer.size=1440
+nifi.components.status.snapshot.frequency=1 min
+
+# NAR Persistence Provider Properties
+nifi.nar.persistence.provider.properties.directory=target/nifi-storage/nar_repository
+
+# Asset Management
+nifi.asset.manager.properties.directory=target/nifi-storage/assets
+
+# Connector Asset Manager
+nifi.connector.asset.manager.implementation=org.apache.nifi.mock.connector.server.MockConnectorAssetManager
+
+# Secrets Manager
+nifi.secrets.manager.implementation=org.apache.nifi.mock.connector.server.secrets.ConnectorTestRunnerSecretsManager
+
+# Site to Site properties
+nifi.remote.input.host=localhost
+nifi.remote.input.secure=true
+nifi.remote.input.socket.port=7780
+nifi.remote.input.http.enabled=true
+nifi.remote.input.http.transaction.ttl=30 sec
+nifi.remote.contents.cache.expiration=30 secs
+
+# web properties #
+nifi.web.war.directory=./lib
+nifi.web.http.host=
+nifi.web.http.port=
+nifi.web.http.network.interface.default=
+nifi.web.https.host=localhost
+nifi.web.https.port=0
+nifi.web.https.network.interface.default=
+nifi.web.jetty.working.directory=target/work/jetty
+nifi.web.jetty.threads=200
+nifi.web.max.header.size=16 KB
+nifi.web.proxy.context.path=
+nifi.web.proxy.host=
+
+# security properties #
+nifi.sensitive.props.key=nifi-connector-mock-tests
+nifi.sensitive.props.algorithm=NIFI_PBKDF2_AES_GCM_256
+
+nifi.security.keystore=
+nifi.security.keystoreType=PKCS12
+nifi.security.keystorePasswd=
+nifi.security.keyPasswd=
+nifi.security.truststore=
+nifi.security.truststoreType=PKCS12
+nifi.security.truststorePasswd=
+nifi.security.user.authorizer=system-test-authorizer
+nifi.security.user.login.identity.provider=
+
+# cluster common properties (all nodes must have same values) #
+nifi.cluster.protocol.heartbeat.interval=5 sec
+nifi.cluster.protocol.is.secure=true
+
+# cluster node properties (only configure for cluster nodes) #
+nifi.cluster.is.node=false
+nifi.cluster.node.address=
+nifi.cluster.node.protocol.port=
+nifi.cluster.node.protocol.threads=10
+nifi.cluster.node.protocol.max.threads=50
+nifi.cluster.node.event.history.size=25
+nifi.cluster.node.connection.timeout=5 sec
+nifi.cluster.node.read.timeout=5 sec
+nifi.cluster.node.max.concurrent.requests=100
+nifi.cluster.firewall.file=
+nifi.cluster.flow.election.max.wait.time=5 mins
+nifi.cluster.flow.election.max.candidates=
+
+# cluster load balancing properties #
+nifi.cluster.load.balance.host=
+nifi.cluster.load.balance.port=6342
+nifi.cluster.load.balance.connections.per.node=4
+nifi.cluster.load.balance.max.thread.count=8
+nifi.cluster.load.balance.comms.timeout=30 sec
+
+# Connector Repository
+nifi.components.connectors.repository.implementation=org.apache.nifi.mock.connector.server.MockConnectorRepository
diff --git a/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/pom.xml b/nifi-connector-mock-bundle/pom.xml
similarity index 50%
copy from nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/pom.xml
copy to nifi-connector-mock-bundle/pom.xml
index f877db14d04..64fd76bf604 100644
--- a/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/pom.xml
+++ b/nifi-connector-mock-bundle/pom.xml
@@ -14,40 +14,21 @@
limitations under the License.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
<parent>
- <artifactId>nifi-system-test-extensions-bundle</artifactId>
<groupId>org.apache.nifi</groupId>
+ <artifactId>nifi</artifactId>
<version>2.9.0-SNAPSHOT</version>
</parent>
- <modelVersion>4.0.0</modelVersion>
- <artifactId>nifi-system-test-extensions</artifactId>
+ <artifactId>nifi-connector-mock-bundle</artifactId>
+ <packaging>pom</packaging>
- <dependencies>
- <dependency>
- <groupId>org.apache.nifi</groupId>
- <artifactId>nifi-stateless-api</artifactId>
- <version>2.9.0-SNAPSHOT</version>
- </dependency>
- <dependency>
- <groupId>org.apache.nifi</groupId>
- <artifactId>nifi-utils</artifactId>
- <version>2.9.0-SNAPSHOT</version>
- </dependency>
- <dependency>
- <groupId>org.apache.nifi</groupId>
- <artifactId>nifi-bin-manager</artifactId>
- <version>2.9.0-SNAPSHOT</version>
- </dependency>
- <dependency>
- <groupId>com.fasterxml.jackson.core</groupId>
- <artifactId>jackson-databind</artifactId>
- </dependency>
- <dependency>
- <groupId>org.apache.nifi</groupId>
- <artifactId>nifi-system-test-extensions-services-api</artifactId>
- <version>2.9.0-SNAPSHOT</version>
- <scope>provided</scope>
- </dependency>
- </dependencies>
+ <modules>
+ <module>nifi-connector-mock</module>
+ <module>nifi-connector-mock-api</module>
+ <module>nifi-connector-mock-server</module>
+ <module>nifi-connector-mock-server-nar</module>
+ <module>nifi-connector-mock-test-bundle</module>
+ </modules>
</project>
\ No newline at end of file
diff --git a/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/pom.xml b/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-connector/pom.xml
similarity index 60%
copy from nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/pom.xml
copy to nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-connector/pom.xml
index f877db14d04..a58046afaf9 100644
--- a/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/pom.xml
+++ b/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-connector/pom.xml
@@ -1,4 +1,4 @@
-<?xml version="1.0" encoding="UTF-8"?>
+<?xml version="1.0"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -14,40 +14,27 @@
limitations under the License.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
<parent>
- <artifactId>nifi-system-test-extensions-bundle</artifactId>
<groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-kafka-to-s3-bundle</artifactId>
<version>2.9.0-SNAPSHOT</version>
</parent>
- <modelVersion>4.0.0</modelVersion>
- <artifactId>nifi-system-test-extensions</artifactId>
+ <artifactId>nifi-kafka-to-s3-connector</artifactId>
<dependencies>
<dependency>
- <groupId>org.apache.nifi</groupId>
- <artifactId>nifi-stateless-api</artifactId>
- <version>2.9.0-SNAPSHOT</version>
- </dependency>
- <dependency>
- <groupId>org.apache.nifi</groupId>
- <artifactId>nifi-utils</artifactId>
- <version>2.9.0-SNAPSHOT</version>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-api</artifactId>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
- <artifactId>nifi-bin-manager</artifactId>
- <version>2.9.0-SNAPSHOT</version>
- </dependency>
- <dependency>
- <groupId>com.fasterxml.jackson.core</groupId>
- <artifactId>jackson-databind</artifactId>
+ <artifactId>nifi-api</artifactId>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
- <artifactId>nifi-system-test-extensions-services-api</artifactId>
- <version>2.9.0-SNAPSHOT</version>
- <scope>provided</scope>
+ <artifactId>nifi-connector-utils</artifactId>
</dependency>
</dependencies>
-</project>
\ No newline at end of file
+</project>
diff --git a/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-connector/src/main/java/org/apache/nifi/connectors/kafkas3/KafkaConnectionStep.java b/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-connector/src/main/java/org/apache/nifi/connectors/kafkas3/KafkaConnectionStep.java
new file mode 100644
index 00000000000..793b1e9be0d
--- /dev/null
+++ b/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-connector/src/main/java/org/apache/nifi/connectors/kafkas3/KafkaConnectionStep.java
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.connectors.kafkas3;
+
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.components.connector.ConfigurationStep;
+import org.apache.nifi.components.connector.ConnectorPropertyDescriptor;
+import org.apache.nifi.components.connector.ConnectorPropertyGroup;
+import org.apache.nifi.components.connector.PropertyType;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.List;
+
+public class KafkaConnectionStep {
+ public static final String STEP_NAME = "Kafka Connection";
+
+ public static final ConnectorPropertyDescriptor KAFKA_BROKERS = new ConnectorPropertyDescriptor.Builder()
+ .name("Kafka Brokers")
+ .description("A comma-separated list of Kafka brokers to connect to.")
+ .required(true)
+ .validators(StandardValidators.HOSTNAME_PORT_LIST_VALIDATOR)
+ .type(PropertyType.STRING_LIST)
+ .build();
+
+ public static final ConnectorPropertyDescriptor SECURITY_PROTOCOL = new ConnectorPropertyDescriptor.Builder()
+ .name("Security Protocol")
+ .description("The security protocol to use when connecting to Kafka brokers.")
+ .required(true)
+ .type(PropertyType.STRING)
+ .defaultValue("SASL_PLAINTEXT")
+ .allowableValues("PLAINTEXT", "SSL", "SASL_PLAINTEXT", "SASL_SSL")
+ .build();
+
+ public static final ConnectorPropertyDescriptor SASL_MECHANISM = new ConnectorPropertyDescriptor.Builder()
+ .name("SASL Mechanism")
+ .description("The SASL mechanism to use for authentication.")
+ .required(true)
+ .type(PropertyType.STRING)
+ .allowableValues("PLAIN", "SCRAM-SHA-256", "SCRAM-SHA-512", "GSSAPI", "OAUTHBEARER")
+ .dependsOn(SECURITY_PROTOCOL, "SASL_PLAINTEXT", "SASL_SSL")
+ .defaultValue("SCRAM-SHA-512")
+ .build();
+
+ public static final ConnectorPropertyDescriptor USERNAME = new ConnectorPropertyDescriptor.Builder()
+ .name("Username")
+ .description("The username for SASL authentication.")
+ .required(true)
+ .type(PropertyType.STRING)
+ .validators(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(SECURITY_PROTOCOL, "SASL_PLAINTEXT", "SASL_SSL")
+ .build();
+
+ public static final ConnectorPropertyDescriptor PASSWORD = new ConnectorPropertyDescriptor.Builder()
+ .name("Password")
+ .description("The password for SASL authentication.")
+ .required(true)
+ .type(PropertyType.SECRET)
+ .validators(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(SECURITY_PROTOCOL, "SASL_PLAINTEXT", "SASL_SSL")
+ .build();
+
+ public static final ConnectorPropertyGroup KAFKA_SERVER_GROUP = new ConnectorPropertyGroup.Builder()
+ .name("Kafka Server Settings")
+ .description("Settings for connecting to the Kafka server")
+ .properties(List.of(
+ KAFKA_BROKERS,
+ SECURITY_PROTOCOL,
+ SASL_MECHANISM,
+ USERNAME,
+ PASSWORD
+ ))
+ .build();
+
+ public static final ConnectorPropertyDescriptor SCHEMA_REGISTRY_URL = new ConnectorPropertyDescriptor.Builder()
+ .name("Schema Registry URL")
+ .description("The URL of the Schema Registry.")
+ .required(false)
+ .type(PropertyType.STRING)
+ .validators(StandardValidators.URL_VALIDATOR)
+ .validators(Validator.VALID)
+ .build();
+
+ public static final ConnectorPropertyDescriptor SCHEMA_REGISTRY_USERNAME = new ConnectorPropertyDescriptor.Builder()
+ .name("Schema Registry Username")
+ .description("The username for Schema Registry authentication.")
+ .required(false)
+ .type(PropertyType.STRING)
+ .validators(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(SCHEMA_REGISTRY_URL)
+ .build();
+
+ public static final ConnectorPropertyDescriptor SCHEMA_REGISTRY_PASSWORD = new ConnectorPropertyDescriptor.Builder()
+ .name("Schema Registry Password")
+ .description("The password for Schema Registry authentication.")
+ .required(false)
+ .type(PropertyType.SECRET)
+ .validators(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(SCHEMA_REGISTRY_URL)
+ .build();
+
+
+ public static final ConnectorPropertyGroup SCHEMA_REGISTRY_GROUP = new ConnectorPropertyGroup.Builder()
+ .name("Schema Registry Settings")
+ .description("Settings for connecting to the Schema Registry")
+ .properties(List.of(
+ SCHEMA_REGISTRY_URL,
+ SCHEMA_REGISTRY_USERNAME,
+ SCHEMA_REGISTRY_PASSWORD
+ ))
+ .build();
+
+ public static final ConfigurationStep KAFKA_CONNECTION_STEP = new ConfigurationStep.Builder()
+ .name(STEP_NAME)
+ .description("Configure Kafka connection settings")
+ .propertyGroups(List.of(
+ KAFKA_SERVER_GROUP,
+ SCHEMA_REGISTRY_GROUP
+ ))
+ .build();
+}
diff --git a/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-connector/src/main/java/org/apache/nifi/connectors/kafkas3/KafkaToS3.java b/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-connector/src/main/java/org/apache/nifi/connectors/kafkas3/KafkaToS3.java
new file mode 100644
index 00000000000..9c4bc1b7d38
--- /dev/null
+++ b/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-connector/src/main/java/org/apache/nifi/connectors/kafkas3/KafkaToS3.java
@@ -0,0 +1,297 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.connectors.kafkas3;
+
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.ConfigVerificationResult;
+import org.apache.nifi.components.ConfigVerificationResult.Outcome;
+import org.apache.nifi.components.DescribedValue;
+import org.apache.nifi.components.connector.AbstractConnector;
+import org.apache.nifi.components.connector.ConfigurationStep;
+import org.apache.nifi.components.connector.ConnectorConfigurationContext;
+import org.apache.nifi.components.connector.FlowUpdateException;
+import org.apache.nifi.components.connector.InvocationFailedException;
+import org.apache.nifi.components.connector.components.ControllerServiceFacade;
+import org.apache.nifi.components.connector.components.FlowContext;
+import org.apache.nifi.components.connector.components.ProcessGroupFacade;
+import org.apache.nifi.components.connector.components.ProcessorFacade;
+import org.apache.nifi.components.connector.util.VersionedFlowUtils;
+import org.apache.nifi.flow.VersionedControllerService;
+import org.apache.nifi.flow.VersionedExternalFlow;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.stream.Collectors;
+
+@CapabilityDescription("Provides the ability to ingest data from Apache Kafka topics, merge it together into an object of reasonable " +
+ "size, and write that data to Amazon S3.")
+@Tags({"kafka", "s3"})
+public class KafkaToS3 extends AbstractConnector {
+
+ private static final List<ConfigurationStep> configurationSteps = List.of(
+ KafkaConnectionStep.KAFKA_CONNECTION_STEP,
+ KafkaTopicsStep.KAFKA_TOPICS_STEP,
+ S3Step.S3_STEP
+ );
+
+ private volatile CompletableFuture<Void> drainFlowFileFuture = null;
+
+
+ @Override
+ public List<ConfigurationStep> getConfigurationSteps() {
+ return configurationSteps;
+ }
+
+ @Override
+ public void prepareForUpdate(final FlowContext workingContext, final FlowContext activeContext) throws FlowUpdateException {
+ final String activeS3DataFormat = activeContext.getConfigurationContext().getProperty(
+ S3Step.S3_STEP_NAME, S3Step.S3_DATA_FORMAT.getName()).getValue();
+ final String workingS3DataFormat = workingContext.getConfigurationContext().getProperty(
+ S3Step.S3_STEP_NAME, S3Step.S3_DATA_FORMAT.getName()).getValue();
+
+ if (!activeS3DataFormat.equals(workingS3DataFormat)) {
+ getLogger().info("S3 Data Format changed from {} to {}; draining flow before updating it", activeS3DataFormat, workingS3DataFormat);
+
+ drainFlowFileFuture = drainFlowFiles(activeContext);
+ try {
+ drainFlowFileFuture.get(5, TimeUnit.MINUTES);
+ } catch (final InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new FlowUpdateException("Interrupted while waiting for all FlowFiles to drain from the flow", e);
+ } catch (final TimeoutException e) {
+ throw new FlowUpdateException("Timed out waiting for all FlowFiles to drain from the flow", e);
+ } catch (final ExecutionException e) {
+ throw new FlowUpdateException("Failed to drain FlowFiles from flow", e.getCause());
+ }
+
+ getLogger().info("All FlowFiles drained from the flow; proceeding with flow update");
+ }
+ }
+
+ @Override
+ public void abortUpdate(final FlowContext workingContext, final Throwable throwable) {
+ if (drainFlowFileFuture != null) {
+ drainFlowFileFuture.cancel(true);
+ drainFlowFileFuture = null;
+ }
+ }
+
+ @Override
+ public void applyUpdate(final FlowContext workingContext, final FlowContext activeContext) throws FlowUpdateException {
+ final VersionedExternalFlow flow = buildFlow(workingContext.getConfigurationContext());
+ getInitializationContext().updateFlow(activeContext, flow);
+ }
+
+ @Override
+ public VersionedExternalFlow getInitialFlow() {
+ return KafkaToS3FlowBuilder.loadInitialFlow();
+ }
+
+ @Override
+ public void onStepConfigured(final String stepName, final FlowContext workingContext) throws FlowUpdateException {
+ final VersionedExternalFlow flow = buildFlow(workingContext.getConfigurationContext());
+ getInitializationContext().updateFlow(workingContext, flow);
+ }
+
+ @Override
+ public List<ConfigVerificationResult> verifyConfigurationStep(final String stepName, final Map<String, String> propertyValueOverrides, final FlowContext workingFlowContext) {
+ // Get the current ConfigurationContext and then create a new one that contains the provided property values
+ final ConnectorConfigurationContext configurationContext = workingFlowContext.getConfigurationContext().createWithOverrides(stepName, propertyValueOverrides);
+ final VersionedExternalFlow flow = buildFlow(configurationContext);
+
+ // Validate Connectivity
+ if (stepName.equals(KafkaConnectionStep.STEP_NAME)) {
+ return verifyKafkaConnectivity(workingFlowContext, flow);
+ }
+ if (stepName.equals(KafkaTopicsStep.STEP_NAME)) {
+ final List<ConfigVerificationResult> results = new ArrayList<>();
+ results.addAll(verifyTopicsExists(workingFlowContext));
+ results.addAll(verifyKafkaParsability(workingFlowContext, flow));
+ return results;
+ }
+
+ return Collections.emptyList();
+ }
+
+ private List<ConfigVerificationResult> verifyKafkaParsability(final FlowContext workingFlowContext, final VersionedExternalFlow flow) {
+ // Enable Controller Services necessary for parsing records.
+ // We determine which Controller Services are referenced by the flow and enable them, but we do not use
+ // getRootGroup().getLifecycle().enableReferencedControllerServices(ControllerServiceReferenceScope.INCLUDE_REFERENCED_SERVICES_ONLY)
+ // because that would include the Controller Services that are referenced by the currently configured flow, and it's possible that the
+ // what is being verified uses a different set of Controller Services (e.g., the verified flow may use a JSON Reader while the current
+ // flow uses an Avro Reader).
+ final ProcessGroupFacade rootGroup = workingFlowContext.getRootGroup();
+ final Set<VersionedControllerService> referencedServices = VersionedFlowUtils.getReferencedControllerServices(flow.getFlowContents());
+ final Set<String> serviceIds = referencedServices.stream()
+ .map(VersionedControllerService::getIdentifier)
+ .collect(Collectors.toSet());
+
+ try {
+ rootGroup.getLifecycle().enableControllerServices(serviceIds).get(10, TimeUnit.SECONDS);
+ } catch (final Exception e) {
+ return List.of(new ConfigVerificationResult.Builder()
+ .verificationStepName("Record Parsing")
+ .outcome(Outcome.FAILED)
+ .explanation("Failed to enable Controller Services due to " + e)
+ .build());
+ }
+
+ try {
+ final ProcessorFacade consumeKafkaFacade = findProcessors(rootGroup,
+ processor -> processor.getDefinition().getType().endsWith("ConsumeKafka")).getFirst();
+
+ final List<ConfigVerificationResult> configVerificationResults = consumeKafkaFacade.verify(flow, Map.of());
+ for (final ConfigVerificationResult result : configVerificationResults) {
+ if (result.getOutcome() == Outcome.FAILED) {
+ return List.of(result);
+ }
+ }
+
+ return Collections.emptyList();
+ } finally {
+ rootGroup.getLifecycle().disableControllerServices(serviceIds);
+ }
+ }
+
+
+ private List<ConfigVerificationResult> verifyTopicsExists(final FlowContext workingFlowContext) {
+ final List<String> topicsAvailable;
+ try {
+ topicsAvailable = getAvailableTopics(workingFlowContext);
+ } catch (final Exception e) {
+ return List.of(new ConfigVerificationResult.Builder()
+ .verificationStepName("Verify Kafka topics exist")
+ .outcome(Outcome.SKIPPED)
+ .explanation("Unable to validate that topics exist due to " + e)
+ .build());
+ }
+
+ final Set<String> topicNames = new HashSet<>(topicsAvailable);
+ final List<String> specifiedTopics = workingFlowContext.getConfigurationContext().getProperty(KafkaTopicsStep.STEP_NAME,
+ KafkaTopicsStep.TOPIC_NAMES.getName()).asList();
+ final String missingTopics = specifiedTopics.stream()
+ .filter(topic -> !topicNames.contains(topic))
+ .collect(Collectors.joining(", "));
+
+ if (!missingTopics.isEmpty()) {
+ return List.of(new ConfigVerificationResult.Builder()
+ .verificationStepName("Verify Kafka topics exist")
+ .outcome(Outcome.FAILED)
+ .explanation("The following topics do not exist in the Kafka cluster: " + missingTopics)
+ .build());
+ } else {
+ return List.of(new ConfigVerificationResult.Builder()
+ .verificationStepName("Verify Kafka topics exist")
+ .outcome(Outcome.SUCCESSFUL)
+ .explanation("All specified topics exist in the Kafka cluster")
+ .build());
+ }
+ }
+
+ private List<ConfigVerificationResult> verifyKafkaConnectivity(final FlowContext workingFlowContext, final VersionedExternalFlow flow) {
+ // Build a new version of the flow so that we can get the relevant properties of the Kafka Connection Service
+ final ControllerServiceFacade connectionService = getKafkaConnectionService(workingFlowContext);
+ final List<ConfigVerificationResult> configVerificationResults = connectionService.verify(flow, Map.of());
+
+ for (final ConfigVerificationResult result : configVerificationResults) {
+ if (result.getOutcome() == Outcome.FAILED) {
+ return List.of(new ConfigVerificationResult.Builder()
+ .verificationStepName("Verify Kafka connectivity")
+ .outcome(Outcome.FAILED)
+ .explanation(result.getExplanation())
+ .build());
+ }
+ }
+
+ return Collections.emptyList();
+ }
+
+ private VersionedExternalFlow buildFlow(final ConnectorConfigurationContext configurationContext) {
+ final KafkaToS3FlowBuilder flowBuilder = new KafkaToS3FlowBuilder(configurationContext);
+ return flowBuilder.buildFlow();
+ }
+
+ @Override
+ public List<DescribedValue> fetchAllowableValues(final String stepName, final String propertyName, final FlowContext flowContext) {
+ if (stepName.equals(KafkaTopicsStep.STEP_NAME) && propertyName.equals(KafkaTopicsStep.TOPIC_NAMES.getName())) {
+ return createAllowableValues(getAvailableTopics(flowContext));
+ } else if (stepName.equals(S3Step.S3_STEP_NAME) && propertyName.equals(S3Step.S3_REGION.getName())) {
+ return createAllowableValues(getPossibleS3Regions(flowContext));
+ }
+
+ return super.fetchAllowableValues(stepName, propertyName, flowContext);
+ }
+
+ private List<DescribedValue> createAllowableValues(final List<String> values) {
+ return values.stream().map(this::createAllowableValue).collect(Collectors.toList());
+ }
+
+ private DescribedValue createAllowableValue(final String value) {
+ return new AllowableValue(value, value, value);
+ }
+
+ @SuppressWarnings("unchecked")
+ private List<String> getAvailableTopics(final FlowContext flowContext) {
+ // If Kafka Brokers not yet set, return empty list
+ final ConnectorConfigurationContext config = flowContext.getConfigurationContext();
+ if (!config.getProperty(KafkaConnectionStep.KAFKA_CONNECTION_STEP, KafkaConnectionStep.KAFKA_BROKERS).isSet()) {
+ return List.of();
+ }
+
+ final ControllerServiceFacade kafkaConnectionService = getKafkaConnectionService(flowContext);
+
+ try {
+ return (List<String>) kafkaConnectionService.invokeConnectorMethod("listTopicNames", Map.of());
+ } catch (final Exception e) {
+ getLogger().warn("Failed to retrieve available Kafka topics", e);
+ return List.of();
+ }
+ }
+
+ private ControllerServiceFacade getKafkaConnectionService(final FlowContext flowContext) {
+ return flowContext.getRootGroup().getControllerServices().stream()
+ .filter(service -> service.getDefinition().getType().endsWith("Kafka3ConnectionService"))
+ .findFirst()
+ .orElseThrow();
+ }
+
+ @SuppressWarnings("unchecked")
+ private List<String> getPossibleS3Regions(final FlowContext flowContext) {
+ final ProcessorFacade processorFacade = flowContext.getRootGroup().getProcessors().stream()
+ .filter(proc -> proc.getDefinition().getType().endsWith("PutS3Object"))
+ .findFirst()
+ .orElseThrow();
+
+ try {
+ return (List<String>) processorFacade.invokeConnectorMethod("getAvailableRegions", Map.of());
+ } catch (final InvocationFailedException e) {
+ getLogger().error("Failed to obtain list of available S3 regions", e);
+ return Collections.emptyList();
+ }
+ }
+}
diff --git a/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-connector/src/main/java/org/apache/nifi/connectors/kafkas3/KafkaToS3FlowBuilder.java b/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-connector/src/main/java/org/apache/nifi/connectors/kafkas3/KafkaToS3FlowBuilder.java
new file mode 100644
index 00000000000..e7d5a75c333
--- /dev/null
+++ b/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-connector/src/main/java/org/apache/nifi/connectors/kafkas3/KafkaToS3FlowBuilder.java
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.connectors.kafkas3;
+
+import org.apache.nifi.components.connector.ConnectorConfigurationContext;
+import org.apache.nifi.components.connector.StepConfigurationContext;
+import org.apache.nifi.components.connector.util.VersionedFlowUtils;
+import org.apache.nifi.flow.VersionedControllerService;
+import org.apache.nifi.flow.VersionedExternalFlow;
+import org.apache.nifi.flow.VersionedProcessGroup;
+import org.apache.nifi.processor.DataUnit;
+
+import java.util.HashMap;
+import java.util.Map;
+
+public class KafkaToS3FlowBuilder {
+ private static final String FLOW_JSON_PATH = "flows/Kafka_to_S3.json";
+
+ private final ConnectorConfigurationContext configContext;
+
+ public KafkaToS3FlowBuilder(final ConnectorConfigurationContext configurationContext) {
+ this.configContext = configurationContext;
+ }
+
+ public static VersionedExternalFlow loadInitialFlow() {
+ return VersionedFlowUtils.loadFlowFromResource(FLOW_JSON_PATH);
+ }
+
+ public VersionedExternalFlow buildFlow() {
+ final VersionedExternalFlow externalFlow = VersionedFlowUtils.loadFlowFromResource(FLOW_JSON_PATH);
+ configureSchemaRegistry(externalFlow);
+
+ updateKafkaConnectionParameters(externalFlow);
+ updateSchemaRegistryParameters(externalFlow);
+ updateReaderWriter(externalFlow);
+ updateKafkaTopicsParameters(externalFlow);
+ updateS3Config(externalFlow);
+
+ return externalFlow;
+ }
+
+ private void configureSchemaRegistry(final VersionedExternalFlow externalFlow) {
+ final String schemaRegistryUrl = configContext.getProperty(KafkaConnectionStep.KAFKA_CONNECTION_STEP, KafkaConnectionStep.SCHEMA_REGISTRY_URL).getValue();
+
+ if (schemaRegistryUrl == null) {
+ final VersionedProcessGroup processGroup = externalFlow.getFlowContents();
+
+ // Remove any references to the Schema Registry service.
+ final VersionedControllerService schemaRegistryService = processGroup.getControllerServices().stream()
+ .filter(service -> service.getType().endsWith("ConfluentSchemaRegistry"))
+ .findFirst()
+ .orElseThrow();
+ VersionedFlowUtils.removeControllerServiceReferences(processGroup, schemaRegistryService.getIdentifier());
+
+ final VersionedControllerService schemaReferenceReader = processGroup.getControllerServices().stream()
+ .filter(service -> service.getType().endsWith("ConfluentEncodedSchemaReferenceReader"))
+ .findFirst()
+ .orElseThrow();
+ VersionedFlowUtils.removeControllerServiceReferences(processGroup, schemaReferenceReader.getIdentifier());
+
+ processGroup.getControllerServices().stream()
+ .filter(service -> service.getType().endsWith("JsonTreeReader"))
+ .forEach(service -> service.getProperties().put("Schema Access Strategy", "infer-schema"));
+
+ processGroup.getControllerServices().stream()
+ .filter(service -> service.getType().endsWith("JsonRecordSetWriter"))
+ .forEach(service -> service.getProperties().put("Schema Write Strategy", "no-schema"));
+ }
+ }
+
+ private void updateSchemaRegistryParameters(final VersionedExternalFlow externalFlow) {
+ final StepConfigurationContext stepContext = configContext.scopedToStep(KafkaConnectionStep.KAFKA_CONNECTION_STEP);
+
+ final String schemaRegistryUrl = stepContext.getProperty(KafkaConnectionStep.SCHEMA_REGISTRY_URL).getValue();
+ VersionedFlowUtils.setParameterValue(externalFlow, "Schema Registry URLs", schemaRegistryUrl);
+
+ if (schemaRegistryUrl == null) {
+ final Map<String, String> properties = new HashMap<>();
+ properties.put("schema-access-strategy", "infer-schema");
+ properties.put("schema-registry", null);
+ properties.put("schema-reference-reader", null);
+
+ externalFlow.getFlowContents().getControllerServices().stream()
+ .filter(service -> service.getType().endsWith("JsonTreeReader"))
+ .findFirst()
+ .ifPresent(service -> service.setProperties(properties));
+ } else {
+ final String username = stepContext.getProperty(KafkaConnectionStep.SCHEMA_REGISTRY_USERNAME).getValue();
+ final String password = stepContext.getProperty(KafkaConnectionStep.SCHEMA_REGISTRY_PASSWORD).getValue();
+
+ VersionedFlowUtils.setParameterValue(externalFlow, "Schema Registry Username", username);
+ VersionedFlowUtils.setParameterValue(externalFlow, "Schema Registry Password", password);
+
+ final String authenticationType = (username == null || username.isEmpty()) ? "NONE" : "BASIC";
+ VersionedFlowUtils.setParameterValue(externalFlow, "Schema Registry Authentication Type", authenticationType);
+ }
+ }
+
+ private void updateKafkaConnectionParameters(final VersionedExternalFlow externalFlow) {
+ final StepConfigurationContext stepContext = configContext.scopedToStep(KafkaConnectionStep.KAFKA_CONNECTION_STEP);
+
+ final String kafkaBrokers = stepContext.getProperty(KafkaConnectionStep.KAFKA_BROKERS).getValue();
+ VersionedFlowUtils.setParameterValue(externalFlow, "Kafka Bootstrap Servers", kafkaBrokers);
+
+ final String securityProtocol = stepContext.getProperty(KafkaConnectionStep.SECURITY_PROTOCOL).getValue();
+ VersionedFlowUtils.setParameterValue(externalFlow, "Kafka Security Protocol", securityProtocol);
+
+ if (securityProtocol.contains("SASL")) {
+ final String saslMechanism = stepContext.getProperty(KafkaConnectionStep.SASL_MECHANISM).getValue();
+ VersionedFlowUtils.setParameterValue(externalFlow, "Kafka SASL Mechanism", saslMechanism);
+
+ final String username = stepContext.getProperty(KafkaConnectionStep.USERNAME).getValue();
+ VersionedFlowUtils.setParameterValue(externalFlow, "Kafka SASL Username", username);
+
+ final String password = stepContext.getProperty(KafkaConnectionStep.PASSWORD).getValue();
+ VersionedFlowUtils.setParameterValue(externalFlow, "Kafka SASL Password", password);
+ }
+ }
+
+ private void updateReaderWriter(final VersionedExternalFlow externalFlow) {
+ final VersionedProcessGroup rootGroup = externalFlow.getFlowContents();
+
+ final VersionedControllerService avroService = rootGroup.getControllerServices().stream()
+ .filter(service -> service.getType().endsWith("AvroReader"))
+ .findFirst()
+ .orElseThrow();
+
+ final String kafkaDataFormat = configContext.getProperty(KafkaTopicsStep.STEP_NAME, KafkaTopicsStep.KAFKA_DATA_FORMAT.getName()).getValue();
+ if (!kafkaDataFormat.equalsIgnoreCase("JSON")) {
+ // Update ConsumeKafka processor to use Avro Reader
+ rootGroup.getProcessors().stream()
+ .filter(versionedProcessor -> versionedProcessor.getType().endsWith("ConsumeKafka"))
+ .findFirst()
+ .ifPresent(processor -> processor.getProperties().put("Record Reader", avroService.getIdentifier()));
+ }
+
+ VersionedFlowUtils.setParameterValue(externalFlow, "Kafka Data Format", kafkaDataFormat);
+
+ final String s3DataFormat = configContext.getProperty(S3Step.S3_STEP, S3Step.S3_DATA_FORMAT).getValue();
+ VersionedFlowUtils.setParameterValue(externalFlow, "S3 Data Format", s3DataFormat);
+ }
+
+ private void updateKafkaTopicsParameters(final VersionedExternalFlow externalFlow) {
+ final StepConfigurationContext stepContext = configContext.scopedToStep(KafkaTopicsStep.STEP_NAME);
+
+ final String topics = stepContext.getProperty(KafkaTopicsStep.TOPIC_NAMES.getName()).getValue();
+ VersionedFlowUtils.setParameterValue(externalFlow, "Topic Names", topics);
+
+ final String groupId = stepContext.getProperty(KafkaTopicsStep.CONSUMER_GROUP_ID.getName()).getValue();
+ VersionedFlowUtils.setParameterValue(externalFlow, "Consumer Group ID", groupId);
+
+ final String offsetReset = stepContext.getProperty(KafkaTopicsStep.OFFSET_RESET.getName()).getValue();
+ VersionedFlowUtils.setParameterValue(externalFlow, "Kafka Auto Offset Reset", offsetReset);
+ }
+
+ private void updateS3Config(final VersionedExternalFlow externalFlow) {
+ final StepConfigurationContext stepContext = configContext.scopedToStep(S3Step.S3_STEP);
+
+ final String region = stepContext.getProperty(S3Step.S3_REGION).getValue();
+ VersionedFlowUtils.setParameterValue(externalFlow, "S3 Region", region);
+
+ final String bucket = stepContext.getProperty(S3Step.S3_BUCKET).getValue();
+ VersionedFlowUtils.setParameterValue(externalFlow, "S3 Bucket", bucket);
+
+ final String prefix = stepContext.getProperty(S3Step.S3_PREFIX).getValue();
+ VersionedFlowUtils.setParameterValue(externalFlow, "S3 Prefix", prefix);
+
+ final String endpointOverrideUrl = stepContext.getProperty(S3Step.S3_ENDPOINT_OVERRIDE_URL).getValue();
+ VersionedFlowUtils.setParameterValue(externalFlow, "S3 Endpoint Override URL", endpointOverrideUrl);
+
+ final String authStrategy = stepContext.getProperty(S3Step.S3_AUTHENTICATION_STRATEGY).getValue();
+ if (authStrategy.equals(S3Step.DEFAULT_CREDENTIALS)) {
+ final VersionedControllerService credentialsService = externalFlow.getFlowContents().getControllerServices().stream()
+ .filter(service -> service.getType().endsWith("AWSCredentialsProviderControllerService"))
+ .findFirst()
+ .orElseThrow();
+
+ credentialsService.setProperties(Map.of("default-credentials", "true"));
+ } else {
+ final String accessKey = stepContext.getProperty(S3Step.S3_ACCESS_KEY_ID).getValue();
+ VersionedFlowUtils.setParameterValue(externalFlow, "S3 Access Key ID", accessKey);
+
+ final String secretKey = stepContext.getProperty(S3Step.S3_SECRET_ACCESS_KEY).getValue();
+ VersionedFlowUtils.setParameterValue(externalFlow, "S3 Secret Access Key", secretKey);
+ }
+
+ final long mergeBytes = stepContext.getProperty(S3Step.TARGET_OBJECT_SIZE).asDataSize(DataUnit.B).longValue();
+ final String mergeSize = stepContext.getProperty(S3Step.TARGET_OBJECT_SIZE).getValue();
+ VersionedFlowUtils.setParameterValue(externalFlow, "Target Object Size", mergeSize);
+
+ // Max Bin size will be either 10% more than target size or target size + 100MB, whichever is smaller
+ final long maxBinSize = (long) Math.min(mergeBytes + 100_000_000, mergeBytes * 1.1D);
+ VersionedFlowUtils.setParameterValue(externalFlow, "Maximum Object Size", maxBinSize + " B");
+
+ final String mergeLatency = stepContext.getProperty(S3Step.MERGE_LATENCY).getValue();
+ VersionedFlowUtils.setParameterValue(externalFlow, "Merge Latency", mergeLatency);
+ }
+}
diff --git a/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-connector/src/main/java/org/apache/nifi/connectors/kafkas3/KafkaTopicsStep.java b/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-connector/src/main/java/org/apache/nifi/connectors/kafkas3/KafkaTopicsStep.java
new file mode 100644
index 00000000000..754a708c395
--- /dev/null
+++ b/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-connector/src/main/java/org/apache/nifi/connectors/kafkas3/KafkaTopicsStep.java
@@ -0,0 +1,85 @@
+/*
... 58878 lines suppressed ...