[jira] [Comment Edited] (MINIFICPP-2304) Clean up Python processor initialization

["Gábor Gyimesi (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MINIFICPP-2304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17828351#comment-17828351 ] 

Gábor Gyimesi edited comment on MINIFICPP-2304 at 3/19/24 2:06 PM:
-------------------------------------------------------------------

h1. Python processor initialization cleanup

Issue: All Python processors are initialized multiple times, for the following reasons:
1. Initializing for Script File property setting
2. Initializing for registering processor for agent manifest
3. Initilizing when creating processor from flow config. -> This is the only initialization that should be done.

The first two initializations are done for all processors, which involves evaluating the python processor script for which all the dependencies need to be installed.

Goal: Python processors should only be initialized if they are used by the flow configuration, this would allow us to only install the dependencies for the processors that are actually used in the flow configuration.
h2. Initialization suggested changes
h2. Script File property setting

When the `Script File` property value is set in the `PythonObjectFactory`, the processor is initialized to set the supported properties. In this case it shouldn't be necessary to evaluate the script file, the initialization should only take place when the processor is first used.

Possible solution: We can change this to only set the supported properties in this scenario, so the script evaluation won't be run.
h3. Registering processor for agent manifest

When the `PythonCreator` registers the python processor classes, it also retrieves the description and the properties of the python processors for the manifest using the `ExternalBuildDescription::addExternalComponent` method. This requires the eval of the script file and running the describe and onInitialize functions of the processor. This must be done for all processors to create the manifest for C2.
 - For MiNiFi python processors we need to call the `onInitialize` and `describe` method to get the supported properties and the description of the processor, this cannot be avoided.
 - For NiFi python processors we can avoid calling the `describe` method by using the ast module to get the description from ProcessorDetails class.
 - We cannot really avoid calling the `onInitialize` method for NiFi processors, as it is more complex to find and parse the PropertyDescriptor objects from the script file.
 

The goal of this ticket is to only initialize the python processors that are in use, but this does not seem to be viable, so the initialization will not be changed at the moment.


was (Author: lordgamez):
# Python processor initialization cleanup

Issue: All Python processors are initialized multiple times, for the following reasons:
1. Initializing for Script File property setting
2. Initializing for registering processor for agent manifest
3. Initilizing when creating processor from flow config. -> This is the only initialization that should be done.

The first two initializations are done for all processors, which involves evaluating the python processor script for which all the dependencies need to be installed.

Goal: Python processors should only be initialized if they are used by the flow configuration, this would allow us to only install the dependencies for the processors that are actually used in the flow configuration.

## Initialization suggested changes

### Script File property setting

When the `Script File` property value is set in the `PythonObjectFactory`, the processor is initialized to set the supported properties. In this case it shouldn't be necessary to evaluate the script file, the initialization should only take place when the processor is first used.

Possible solution: We can change this to only set the supported properties in this scenario, so the script evaluation won't be run.

### Registering processor for agent manifest

When the `PythonCreator` registers the python processor classes, it also retrieves the description and the properties of the python processors for the manifest using the `ExternalBuildDescription::addExternalComponent` method. This requires the eval of the script file and running the describe and onInitialize functions of the processor. This must be done for all processors to create the manifest for C2.

- For MiNiFi python processors we need to call the `onInitialize` and `describe` method to get the supported properties and the description of the processor, this cannot be avoided.
- For NiFi python processors we can avoid calling the `describe` method by using the ast module to get the description from ProcessorDetails class.
- We cannot really avoid calling the `onInitialize` method for NiFi processors, as it is more complex to find and parse the PropertyDescriptor objects from the script file.
 
The goal of this ticket is to only initialize the python processors that are in use, but this does not seem to be viable, so the initialization will not be changed at the moment.

> Clean up Python processor initialization
> ----------------------------------------
>
>                 Key: MINIFICPP-2304
>                 URL: https://issues.apache.org/jira/browse/MINIFICPP-2304
>             Project: Apache NiFi MiNiFi C++
>          Issue Type: Improvement
>            Reporter: Gábor Gyimesi
>            Assignee: Gábor Gyimesi
>            Priority: Major
>
> Python processor initialization should be refactored to be cleaner. We instantiate the Python processors twice:
> We instantiate the Python processors that are used in the configured MiNiFi flow. This is straightforward and not problematic.
> The problem is that before that we also instantiate all python processors in the PythonCreator::registerScriptDescription method for getting the class description for all available python processors for the agent manifest
>  * In this scenario we call the Python processors' initialize method twice:
>  ** Once the PythonObjectFactory::create method calls it to initialize the supported properties to set the ScriptFile property to the path of the Python processor
>  ** After this the PythonCreator::registerScriptDescription also calls it explicitly to load the python processor from the set path
>  ** This should be circumvented to not need double initialization and have a telling warning message in ExecutePythonProcessor::initialize() if the loadScript method fails
>  * We should also find a way to avoid initializing all the Python processors and retrieve the processor data without it. With NiFi Python processors a way for this could be to use the "ast" python module to retrieve the processor details which does not require loading the python module



--
This message was sent by Atlassian Jira
(v8.20.10#820010)