[jira] [Updated] (NIFI-12735) Execute flow analysis before committing flow into Registry

["Simon Bence (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/NIFI-12735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Simon Bence updated NIFI-12735:
-------------------------------
    Description: 
When committing a flow into a Registry, it would be beneficial to have the opportunity to execute the existing flow analysis rules and potentially interrupt the commit.

The change aims for:
- Execute existing flow analysis rules out of the scheduled times
- Gracefully resject commit when the flow does not meet the requirements
- Providing a toggle to turn on or off the feature
- In case of the feature is inactive or no rules are set, the Registry handling behaviour should not change*

The change does not aim for:
- RegistryClient level adjustments
- Using different rules set or other kind of validation methods
- Providing support for the 1.x line

{*}
Expect this following branch: https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/StandardNiFiServiceFacade.java#L5028
As with the usage of this feature, failing snapshot commits will be more frequent and I consider "lingering" flow definitions without snapshot misleading for users.

The following test cases describe the proposed behaviour in detail:

TC1.1 Adding flow with rules violation
GIVEN a `DisallowComponentType` flow analysis rule, disallowing `GenerateFlowFile`
AND NiFi property `nifi.registry.check.for.rules.violation` is true
AND process group G1 containing a `GenerateFlowFile`
AND violation appears as validation failure
WHEN the user tries to add G1 under version control, using registry R1
THEN the UI displays the following error message "*Cannot store flow version to registry due to rules violations*"
AND the flow will not appear in registry R1
AND the UI will show G1 as un-versioned
AND NiFi node will see G1 as un-versioned

TC1.2 Adding flow with rules violation before scheduled
GIVEN a `DisallowComponentType` flow analysis rule, disallowing `GenerateFlowFile`
AND NiFi property `nifi.registry.check.for.rules.violation` is true
AND process group G1 containing a `GenerateFlowFile`
AND violation does not appear in UI
WHEN the user tries to add G1 under version control, using registry R1
THEN the UI displays the following error message "*Cannot store flow version to registry due to rules violations*"
AND the flow will not appear in registry R1
AND the UI will show G1 as un-versioned
AND NiFi node will see G1 as un-versioned

TC1.3 When turned off violating flows will be added
GIVEN a `DisallowComponentType` flow analysis rule, disallowing `GenerateFlowFile`
AND NiFi property `nifi.registry.check.for.rules.violation` is fals
AND process group G1 containing a `GenerateFlowFile`
AND violation appears as validation failure
WHEN the user tries to add G1 under version control, using registry R1
THEN the flow is committed normally

TC1.4 Adding version with rules violation (when the rule is new)
GIVEN NiFi property `nifi.registry.check.for.rules.violation` is true
AND process group G1 containing a `GenerateFlowFile`
AND process group G1 is under version control
AND `LogAttribute` processor is added (not committed to the registry)
GIVEN adding a `DisallowComponentType` flow analysis rule, disallowing `GenerateFlowFile`
AND  the user tries to add G1 under version control, using registry R1
THEN the UI displays the following error message "*Cannot store flow version to registry due to rules violations*"
AND the flow will not appear in registry R1
AND the UI will show G1 as un-versioned
AND NiFi node will see G1 as un-versioned

TC1.5 Adding version with rules violation (when the violation is new)
GIVEN a `DisallowComponentType` flow analysis rule, disallowing `GenerateFlowFile`
AND NiFi property `nifi.registry.check.for.rules.violation` is true
AND process group G1 is under version control
AND `GenerateFlowFile` processor is added (not committed to the registry)
WHEN the user tries to add new version of G1 to registry R1
THEN the UI displays the following error message "*Cannot store flow version to registry due to rules violations*"
AND the flow will not appear in registry R1
AND the UI will show G1 as un-versioned
AND NiFi node will see G1 as un-versioned

TC2.1 Adding flow after fixing violation (by removing violating processor)
CONTINUING TC1.1
WHEN `GenerateFlowFile` is deleted from process group G1
WHEN the user tries to add G1 under version control, using registry R1
THEN the flow is committed normally

TC2.2 Adding flow after fixing violation (by disabling violated rule)
CONTINUING TC1.5
WHEN `DisallowComponentType` is disabled
WHEN the user tries to add G1 under version control, using registry R1
THEN the flow is committed normally

TC2.3 Adding version after fixing violation (by removing violating processor)
CONTINUING TC1.5
WHEN `GenerateFlowFile` is deleted from process group G1
WHEN the user tries to add G1 under version control, using registry R1
THEN the flow is committed normally

TC2.4 Adding version after fixing violation (by disabling violated rule)
CONTINUING TC1.5
WHEN `DisallowComponentType` is disabled
WHEN the user tries to add G1 under version control, using registry R1
THEN the flow is committed normally

TC3.1 Adding violating flow after disabling rule
GIVEN a `DisallowComponentType` flow analysis rule, disallowing `GenerateFlowFile`
AND NiFi property `nifi.registry.check.for.rules.violation` is true
AND process group G1 containing a `GenerateFlowFile`
AND violation appears as validation failure
WHEN `DisallowComponentType` is disabled
AND the user tries to add G1 under version control, using registry R1
THEN the flow is committed normally

TC3.2 Adding violating version after turning disabling rule
GIVEN a `DisallowComponentType` flow analysis rule, disallowing `GenerateFlowFile`
AND NiFi property `nifi.registry.check.for.rules.violation` is true
AND process group G1 is under version control
AND `GenerateFlowFile` processor is added (not committed to the registry)
AND violation appears as validation failure
WHEN `DisallowComponentType` is disabled
AND the user tries to add new version of G1 to registry R1
THEN the flow is committed normally

  was:
When committing a flow into a Registry, it would be beneficial to have the opportunity to execute the existing flow analysis rules and potentially interrupt the commit.

The change aims for:
- Execute existing flow analysis rules out of the scheduled times
- Gracefully resject commit when the flow does not meet the requirements
- Providing a toggle to turn on or off the feature
- In case of the feature is inactive or no rules are set, the Registry handling behaviour should not change*

The change does not aim for:
- RegistryClient level adjustments
- Using different rules set or other kind of validation methods
- Providing support for the 1.x line

[*]
Expect this following branch: https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/StandardNiFiServiceFacade.java#L5028
As with the usage of this feature, failing snapshot commits will be more frequent and I consider "lingering" flow definitions without snapshot misleading for users.

The following test cases describe the proposed behaviour in detail:

## TC1.1 Adding flow with rules violation

GIVEN a `DisallowComponentType` flow analysis rule, disallowing `GenerateFlowFile`
AND NiFi property `nifi.registry.check.for.rules.violation` is true
AND process group G1 containing a `GenerateFlowFile`
AND violation appears as validation failure
WHEN the user tries to add G1 under version control, using registry R1
THEN the UI displays the following error message "*Cannot store flow version to registry due to rules violations*"
AND the flow will not appear in registry R1
AND the UI will show G1 as un-versioned
AND NiFi node will see G1 as un-versioned

## TC1.2 Adding flow with rules violation before scheduled
GIVEN a `DisallowComponentType` flow analysis rule, disallowing `GenerateFlowFile`
AND NiFi property `nifi.registry.check.for.rules.violation` is true
AND process group G1 containing a `GenerateFlowFile`
AND violation does not appear in UI
WHEN the user tries to add G1 under version control, using registry R1
THEN the UI displays the following error message "*Cannot store flow version to registry due to rules violations*"
AND the flow will not appear in registry R1
AND the UI will show G1 as un-versioned
AND NiFi node will see G1 as un-versioned

## TC1.3 When turned off violating flows will be added
GIVEN a `DisallowComponentType` flow analysis rule, disallowing `GenerateFlowFile`
AND NiFi property `nifi.registry.check.for.rules.violation` is fals
AND process group G1 containing a `GenerateFlowFile`
AND violation appears as validation failure
WHEN the user tries to add G1 under version control, using registry R1
THEN the flow is committed normally

## TC1.4 Adding version with rules violation (when the rule is new)
GIVEN NiFi property `nifi.registry.check.for.rules.violation` is true
AND process group G1 containing a `GenerateFlowFile`
AND process group G1 is under version control
AND `LogAttribute` processor is added (not committed to the registry)
GIVEN adding a `DisallowComponentType` flow analysis rule, disallowing `GenerateFlowFile`
AND  the user tries to add G1 under version control, using registry R1
THEN the UI displays the following error message "*Cannot store flow version to registry due to rules violations*"
AND the flow will not appear in registry R1
AND the UI will show G1 as un-versioned
AND NiFi node will see G1 as un-versioned

## TC1.5 Adding version with rules violation (when the violation is new)
GIVEN a `DisallowComponentType` flow analysis rule, disallowing `GenerateFlowFile`
AND NiFi property `nifi.registry.check.for.rules.violation` is true
AND process group G1 is under version control
AND `GenerateFlowFile` processor is added (not committed to the registry)
WHEN the user tries to add new version of G1 to registry R1
THEN the UI displays the following error message "*Cannot store flow version to registry due to rules violations*"
AND the flow will not appear in registry R1
AND the UI will show G1 as un-versioned
AND NiFi node will see G1 as un-versioned

## TC2.1 Adding flow after fixing violation (by removing violating processor)
CONTINUING TC1.1
WHEN `GenerateFlowFile` is deleted from process group G1
WHEN the user tries to add G1 under version control, using registry R1
THEN the flow is committed normally

## TC2.2 Adding flow after fixing violation (by disabling violated rule)
CONTINUING TC1.5
WHEN `DisallowComponentType` is disabled
WHEN the user tries to add G1 under version control, using registry R1
THEN the flow is committed normally

## TC2.3 Adding version after fixing violation (by removing violating processor)
CONTINUING TC1.5
WHEN `GenerateFlowFile` is deleted from process group G1
WHEN the user tries to add G1 under version control, using registry R1
THEN the flow is committed normally

## TC2.4 Adding version after fixing violation (by disabling violated rule)
CONTINUING TC1.5
WHEN `DisallowComponentType` is disabled
WHEN the user tries to add G1 under version control, using registry R1
THEN the flow is committed normally

## TC3.1 Adding violating flow after disabling rule
GIVEN a `DisallowComponentType` flow analysis rule, disallowing `GenerateFlowFile`
AND NiFi property `nifi.registry.check.for.rules.violation` is true
AND process group G1 containing a `GenerateFlowFile`
AND violation appears as validation failure
WHEN `DisallowComponentType` is disabled
AND the user tries to add G1 under version control, using registry R1
THEN the flow is committed normally

## TC3.2 Adding violating version after turning disabling rule
GIVEN a `DisallowComponentType` flow analysis rule, disallowing `GenerateFlowFile`
AND NiFi property `nifi.registry.check.for.rules.violation` is true
AND process group G1 is under version control
AND `GenerateFlowFile` processor is added (not committed to the registry)
AND violation appears as validation failure
WHEN `DisallowComponentType` is disabled
AND the user tries to add new version of G1 to registry R1
THEN the flow is committed normally


> Execute flow analysis before committing flow into Registry
> ----------------------------------------------------------
>
>                 Key: NIFI-12735
>                 URL: https://issues.apache.org/jira/browse/NIFI-12735
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Core Framework
>            Reporter: Simon Bence
>            Assignee: Simon Bence
>            Priority: Major
>
> When committing a flow into a Registry, it would be beneficial to have the opportunity to execute the existing flow analysis rules and potentially interrupt the commit.
> The change aims for:
> - Execute existing flow analysis rules out of the scheduled times
> - Gracefully resject commit when the flow does not meet the requirements
> - Providing a toggle to turn on or off the feature
> - In case of the feature is inactive or no rules are set, the Registry handling behaviour should not change*
> The change does not aim for:
> - RegistryClient level adjustments
> - Using different rules set or other kind of validation methods
> - Providing support for the 1.x line
> {*}
> Expect this following branch: https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/StandardNiFiServiceFacade.java#L5028
> As with the usage of this feature, failing snapshot commits will be more frequent and I consider "lingering" flow definitions without snapshot misleading for users.
> The following test cases describe the proposed behaviour in detail:
> TC1.1 Adding flow with rules violation
> GIVEN a `DisallowComponentType` flow analysis rule, disallowing `GenerateFlowFile`
> AND NiFi property `nifi.registry.check.for.rules.violation` is true
> AND process group G1 containing a `GenerateFlowFile`
> AND violation appears as validation failure
> WHEN the user tries to add G1 under version control, using registry R1
> THEN the UI displays the following error message "*Cannot store flow version to registry due to rules violations*"
> AND the flow will not appear in registry R1
> AND the UI will show G1 as un-versioned
> AND NiFi node will see G1 as un-versioned
> TC1.2 Adding flow with rules violation before scheduled
> GIVEN a `DisallowComponentType` flow analysis rule, disallowing `GenerateFlowFile`
> AND NiFi property `nifi.registry.check.for.rules.violation` is true
> AND process group G1 containing a `GenerateFlowFile`
> AND violation does not appear in UI
> WHEN the user tries to add G1 under version control, using registry R1
> THEN the UI displays the following error message "*Cannot store flow version to registry due to rules violations*"
> AND the flow will not appear in registry R1
> AND the UI will show G1 as un-versioned
> AND NiFi node will see G1 as un-versioned
> TC1.3 When turned off violating flows will be added
> GIVEN a `DisallowComponentType` flow analysis rule, disallowing `GenerateFlowFile`
> AND NiFi property `nifi.registry.check.for.rules.violation` is fals
> AND process group G1 containing a `GenerateFlowFile`
> AND violation appears as validation failure
> WHEN the user tries to add G1 under version control, using registry R1
> THEN the flow is committed normally
> TC1.4 Adding version with rules violation (when the rule is new)
> GIVEN NiFi property `nifi.registry.check.for.rules.violation` is true
> AND process group G1 containing a `GenerateFlowFile`
> AND process group G1 is under version control
> AND `LogAttribute` processor is added (not committed to the registry)
> GIVEN adding a `DisallowComponentType` flow analysis rule, disallowing `GenerateFlowFile`
> AND  the user tries to add G1 under version control, using registry R1
> THEN the UI displays the following error message "*Cannot store flow version to registry due to rules violations*"
> AND the flow will not appear in registry R1
> AND the UI will show G1 as un-versioned
> AND NiFi node will see G1 as un-versioned
> TC1.5 Adding version with rules violation (when the violation is new)
> GIVEN a `DisallowComponentType` flow analysis rule, disallowing `GenerateFlowFile`
> AND NiFi property `nifi.registry.check.for.rules.violation` is true
> AND process group G1 is under version control
> AND `GenerateFlowFile` processor is added (not committed to the registry)
> WHEN the user tries to add new version of G1 to registry R1
> THEN the UI displays the following error message "*Cannot store flow version to registry due to rules violations*"
> AND the flow will not appear in registry R1
> AND the UI will show G1 as un-versioned
> AND NiFi node will see G1 as un-versioned
> TC2.1 Adding flow after fixing violation (by removing violating processor)
> CONTINUING TC1.1
> WHEN `GenerateFlowFile` is deleted from process group G1
> WHEN the user tries to add G1 under version control, using registry R1
> THEN the flow is committed normally
> TC2.2 Adding flow after fixing violation (by disabling violated rule)
> CONTINUING TC1.5
> WHEN `DisallowComponentType` is disabled
> WHEN the user tries to add G1 under version control, using registry R1
> THEN the flow is committed normally
> TC2.3 Adding version after fixing violation (by removing violating processor)
> CONTINUING TC1.5
> WHEN `GenerateFlowFile` is deleted from process group G1
> WHEN the user tries to add G1 under version control, using registry R1
> THEN the flow is committed normally
> TC2.4 Adding version after fixing violation (by disabling violated rule)
> CONTINUING TC1.5
> WHEN `DisallowComponentType` is disabled
> WHEN the user tries to add G1 under version control, using registry R1
> THEN the flow is committed normally
> TC3.1 Adding violating flow after disabling rule
> GIVEN a `DisallowComponentType` flow analysis rule, disallowing `GenerateFlowFile`
> AND NiFi property `nifi.registry.check.for.rules.violation` is true
> AND process group G1 containing a `GenerateFlowFile`
> AND violation appears as validation failure
> WHEN `DisallowComponentType` is disabled
> AND the user tries to add G1 under version control, using registry R1
> THEN the flow is committed normally
> TC3.2 Adding violating version after turning disabling rule
> GIVEN a `DisallowComponentType` flow analysis rule, disallowing `GenerateFlowFile`
> AND NiFi property `nifi.registry.check.for.rules.violation` is true
> AND process group G1 is under version control
> AND `GenerateFlowFile` processor is added (not committed to the registry)
> AND violation appears as validation failure
> WHEN `DisallowComponentType` is disabled
> AND the user tries to add new version of G1 to registry R1
> THEN the flow is committed normally



--
This message was sent by Atlassian Jira
(v8.20.10#820010)