You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "Fabian Reiber (Jira)" <ji...@apache.org> on 2025/06/04 15:07:00 UTC

[jira] [Commented] (NIFI-14621) Failed to import external python dependencies in nar extensions managed by nifi registry

    [ https://issues.apache.org/jira/browse/NIFI-14621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17956133#comment-17956133 ] 

Fabian Reiber commented on NIFI-14621:
--------------------------------------

Additional information: It only happens with third party dependencies. Now I tried it with {{{{import sys}}}} and I could use the processor by downloading it from nifi registry. It was stored with the colons in the name of nar file in the filesystem.

> Failed to import external python dependencies in nar extensions managed by nifi registry
> ----------------------------------------------------------------------------------------
>
>                 Key: NIFI-14621
>                 URL: https://issues.apache.org/jira/browse/NIFI-14621
>             Project: Apache NiFi
>          Issue Type: Bug
>          Components: Core Framework
>    Affects Versions: 2.3.0
>         Environment: Podman 5.5.0
> Debian 12
> Nifi 2.3.0
> Nifi-Registry 1.26.0
>            Reporter: Fabian Reiber
>            Priority: Major
>
> I wrote a simple Python processor with an external dependency, created a nar file with hatch, uploaded the file to a Nifi Registry and configured the [Nifi nar registry property|https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#nifi-registry-nar-provider] in {{nifi.properties}} to use this registry. After a restart of Nifi, it logs an error that the dependency can't be found:
> {code:python}
> 2025-06-02 12:35:18,000 ERROR [python-log-212] py4j.java_gateway There was an exception while executing the Python Proxy on the Python Side.
> Traceback (most recent call last):
>   File "/opt/nifi/nifi-current/python/framework/py4j/java_gateway.py", line 2506, in _call_proxy
>     return_value = getattr(self.pool[obj_id], method)(*params)
>                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/opt/nifi/nifi-current/./python/framework/Controller.py", line 71, in createProcessor
>     processorClass = self.extensionManager.getProcessorClass(processorType, version, work_dir)
>                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/opt/nifi/nifi-current/python/framework/ExtensionManager.py", line 105, in getProcessorClass
>     processor_class = self.__load_extension_module(module_file, details.local_dependencies)
>                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/opt/nifi/nifi-current/python/framework/ExtensionManager.py", line 365, in __load_extension_module
>     module_spec.loader.exec_module(module)
>   File "<frozen importlib._bootstrap_external>", line 940, in exec_module
>   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
>   File "/opt/nifi/nifi-current/work/nar/extensions/processors::processors-nar::0.0.1::444c6774-eb67-4bf2-abab-25beb1631736.nar-unpacked/processors/hello_world.py", line 1, in <module>
>     import redis
> ModuleNotFoundError: No module named 'redis'
> {code}
> Regardless of whether it makes sense to use redis in a separate python module, it does not work with pandas or any other third party dependency as well.
> But if I replace the colons of the filename of the automatically downloaded nar file {{processors::processors-nar::0.0.1::444c6774-eb67-4bf2-abab-25beb1631736.nar}} with dashes {{processors-processors-nar-0.0.1-444c6774-eb67-4bf2-abab-25beb1631736.nar}} it works.
> It does not explain why the python environment does not find the external dependencies. I would suggest, that the loader fails parsing the nar file due to the colons. I tested it with Nifi 2.3.0 but would expect, that 2.4.0 is affected as well, because I did not find any release note which indicates a fix.
> The workaround is not suitable. For any new version or new processor, I have to rename it, after all  nar files were downloaded.
> I found out that the [NiFiRegistryExtensionBundleMetadata|https://github.com/apache/nifi/blob/rel/nifi-2.3.0/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/registry/extension/NiFiRegistryExtensionBundleMetadata.java#L27] class uses the colons as a filename separator.
> I used [this how to|https://datavolo.io/2024/05/how-to-package-and-deploy-python-processors-for-apache-nifi/] to deploy a nar file with hatch. After the nar file was created with {{hatch build --target nar}}, I uploaded it to the registry with {{curl -F file=@dist/processors-0.0.1.nar http://localhost/nifi-registry-api/buckets/4a7579d3-c41e-4a9b-a3db-9f57340c9e29/bundles/nifi-nar}}. 
> If you need more information, please let me know :)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)