[jira] [Closed] (FLINK-38161) PyFlink: Unable to find Pemja library in thread mode when Python dependencies are installed in a separate layer

["Dian Fu (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-38161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dian Fu closed FLINK-38161.
---------------------------
    Resolution: Duplicate

This issue should have been addressed in FLINK-38332 which bumped PEMJA to 0.5.5

> PyFlink: Unable to find Pemja library in thread mode when Python dependencies are installed in a separate layer
> ---------------------------------------------------------------------------------------------------------------
>
>                 Key: FLINK-38161
>                 URL: https://issues.apache.org/jira/browse/FLINK-38161
>             Project: Flink
>          Issue Type: Bug
>          Components: API / Python
>    Affects Versions: 1.19.3, 1.20.2
>            Reporter: Jaydeep Karia
>            Priority: Major
>
> *Problem Summary*
> PyFlink applications running in *thread mode* fail at startup with a {{RuntimeException: Failed to find PemJa Library}} when Python dependencies are installed into a non-standard {{site-packages}} directory. This is a common scenario in modern containerized environments, particularly when using tools like Cloud Native Buildpacks, which create separate layers for dependencies.
> h3. Environment & Steps to Reproduce
> This issue occurs in environments where {{pip}} installs packages into a directory that is different from the system's primary Python {{site-packages}} folder.
>  # *Set up an environment* with layered {{site-packages}} directories. For example, using buildpacks might result in a file structure like this:
>  ** *System {{{}site-packages{}}}:* {{/layers/<python-buildpack>/runtime-depends/opt/bb/lib/python3.11/site-packages}}
>  ** *Pip-installed {{{}site-packages{}}}:* {{/layers/<pip-buildpack>/requirements/lib/python3.11/site-packages/}}
>  # *Install {{apache-flink and pemja}}* into the pip layer. The current {{flink-python}} distributions pin the {{pemja}} dependency to version {{{}0.4.1{}}}.
>  # *Execute a PyFlink job* that is configured to run in {{{}THREAD_MODE{}}}.
>  # {*}Observe the failure{*}. The job fails immediately on startup with a {{{}RuntimeException{}}}, as Flink cannot locate the Pemja native library.
> h3. Root Cause Analysis 🔍
> The version of {{pemja}} currently pinned by {{flink-python}} ({{{}pemja==0.4.1{}}}) attempts to locate its native library by executing the following Python code to find the {{site-packages}} path:
>  
> {{"import sysconfig; print(sysconfig.get_paths()[\"purelib\"])";}}
> This implementation can be seen in the [pemja 0.4.1 source code on GitHub|https://github.com/alibaba/pemja/blob/release-0.4/src/main/java/pemja/utils/CommonUtils.java#L39].
> The core problem is that {{sysconfig.get_paths()["purelib"]}} {*}only returns the standard library's installation path{*}. It does not search or account for other paths where pip might install packages, such as a separate buildpack layer or a virtual environment. As a result, when {{pemja}} is installed in any location other than the system's default {{purelib}} directory, Flink is unable to find its library files and fails.
> h3. Proposed Solution ✅
> This path discovery issue has been addressed and resolved in *{{pemja}} version {{0.6.0}} and newer: [https://github.com/alibaba/pemja/pull/65]*
> The {{pemja}} dependency version specified in {{flink-python}} needs to be upgraded to allow for a more recent version that incorporates this fix.
> *Required Action:*
> Upgrade the {{install_requires}} for {{pemja}} within the {{flink-python/setup.py}} file.
>  * *Current Line:* {{pemja==0.4.1}}
>  * *Proposed Change:* {{pemja>=0.6.0}}
>  * *File Location:* [{{flink-python/setup.py}} on GitHub|https://github.com/apache/flink/blob/release-1.20.2/flink-python/setup.py#L327]
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)