[jira] [Updated] (MINIFICPP-1288) Implement static flow configuration validation

["Adam Hunyadi (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/MINIFICPP-1288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Adam Hunyadi updated MINIFICPP-1288:
------------------------------------
    Description: 
*Background:*

MiNiFi flows are represented by yaml configuration files. When these are published to the FlowController in MiNiFi, it handles this as a “reconfigure and restart”. The current logic is:
 # Backup the old process group (in memory representation of the configuration).

 # Stop the FlowController.

 # Try to build up a new process group based on the new configuration.

 # If we fail during the construction of the process group nodes we should keep the backup, otherwise we let the new flow run and consider it validated.

This is not awful, but expects the checks implemented in terms of processor construction-time checks and separated cheks on the connection network. [[The implementation is also not very robust]|https://github.com/apache/nifi-minifi-cpp/blob/rel/minifi-cpp-0.7.0/libminifi/src/core/ProcessGroup.cpp#L343-L359], so there is not much extra value in keeping it.

*Proposal:*

We should try and validate the configuration files before trying to reload using them. Not only would this make us fail earlier, but would have the checks a bit more decoupled from the processors. We already send processor configuratin requirement data via the heartbeat to EFM, so we have every piece of information needed to statically validate the configuration files.

This way the new logic should simplify to:
 # Validate the configuration

 # If it is valid load it, otherwise load it and any subsequent error should be considered runtime exception.

As this would potentially break configuration-file backward compatibilty, all the changes should be implemented behind a feature flag (build-time).

 

  was:
*Background:*

MiNiFi flows are represented by yaml configuration files. When these are published to the FlowController in MiNiFi, it handles this as a “reconfigure and restart”. The current logic is:
 # Backup the old process group (in memory representation of the configuration).

 # Stop the FlowController.

 # Try to build up a new process group based on the new configuration.

 # If we fail during the construction of the process group nodes we should keep the backup, otherwise we let the new flow run and consider it validated.

This is not awful, but expects the checks implemented in terms of processor construction-time checks and separated cheks on the connection network.[ [The implementation is also not very robust]|https://github.com/apache/nifi-minifi-cpp/blob/rel/minifi-cpp-0.7.0/libminifi/src/core/ProcessGroup.cpp#L343-L359], so there is not much extra value in keeping it.

 

*Proposal:*

We should try and validate the configuration files before trying to reload using them. Not only would this make us fail earlier, but would have the checks a bit more decoupled from the processors. We already send processor configuratin requirement data via the heartbeat to EFM, so we have every piece of information needed to statically validate the configuration files.

This way the new logic should simplify to:
 # Validate the configuration

 # If it is valid load it, otherwise load it and any subsequent error should be considered runtime exception.

As this would potentially break configuration-file backward compatibilty, all the changes should be implemented behind a feature flag (build-time).

 


> Implement static flow configuration validation
> ----------------------------------------------
>
>                 Key: MINIFICPP-1288
>                 URL: https://issues.apache.org/jira/browse/MINIFICPP-1288
>             Project: Apache NiFi MiNiFi C++
>          Issue Type: Task
>            Reporter: Adam Hunyadi
>            Assignee: Adam Hunyadi
>            Priority: Major
>
> *Background:*
> MiNiFi flows are represented by yaml configuration files. When these are published to the FlowController in MiNiFi, it handles this as a “reconfigure and restart”. The current logic is:
>  # Backup the old process group (in memory representation of the configuration).
>  # Stop the FlowController.
>  # Try to build up a new process group based on the new configuration.
>  # If we fail during the construction of the process group nodes we should keep the backup, otherwise we let the new flow run and consider it validated.
> This is not awful, but expects the checks implemented in terms of processor construction-time checks and separated cheks on the connection network. [[The implementation is also not very robust]|https://github.com/apache/nifi-minifi-cpp/blob/rel/minifi-cpp-0.7.0/libminifi/src/core/ProcessGroup.cpp#L343-L359], so there is not much extra value in keeping it.
> *Proposal:*
> We should try and validate the configuration files before trying to reload using them. Not only would this make us fail earlier, but would have the checks a bit more decoupled from the processors. We already send processor configuratin requirement data via the heartbeat to EFM, so we have every piece of information needed to statically validate the configuration files.
> This way the new logic should simplify to:
>  # Validate the configuration
>  # If it is valid load it, otherwise load it and any subsequent error should be considered runtime exception.
> As this would potentially break configuration-file backward compatibilty, all the changes should be implemented behind a feature flag (build-time).
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)