You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Dian Fu (Jira)" <ji...@apache.org> on 2023/06/17 10:04:00 UTC

[jira] [Closed] (FLINK-32136) Pyflink gateway server launch fails when purelib != platlib

     [ https://issues.apache.org/jira/browse/FLINK-32136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dian Fu closed FLINK-32136.
---------------------------
    Fix Version/s: 1.18.0
                   1.16.3
                   1.17.2
         Assignee: Dian Fu
       Resolution: Fixed

Fixed in:
- master via f64563bc1a7ff698acd708b61e9e80ae9c3e848f
- release-1.17 via 1b3f25432a29005370b0f51aaa7d4ee79a5edd58
- release-1.16 via f9394025fb756c844ec5f2615971f227d40b9244

> Pyflink gateway server launch fails when purelib != platlib
> -----------------------------------------------------------
>
>                 Key: FLINK-32136
>                 URL: https://issues.apache.org/jira/browse/FLINK-32136
>             Project: Flink
>          Issue Type: Bug
>          Components: API / Python
>    Affects Versions: 1.13.3
>            Reporter: William Ashley
>            Assignee: Dian Fu
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 1.18.0, 1.16.3, 1.17.2
>
>
> On distros where python's {{purelib}} is different than {{platlib}} (e.g. Amazon Linux 2, but from my research it's all of the Redhat-based ones), you wind up with components of packages being installed across two different locations (e.g. {{/usr/local/lib/python3.7/site-packages/pyflink}} and {{{}/usr/local/lib64/python3.7/site-packages/pyflink{}}}).
> {{_find_flink_home}} [handles|https://github.com/apache/flink/blob/06688f345f6793a8964ec00002175f44cda13c33/flink-python/pyflink/find_flink_home.py#L58C63-L60] this, and in flink releases <= 1.13.2 its setting of the {{FLINK_LIB_DIR}} environment variable was the one being used. However, from 1.13.3, a refactoring of {{launch_gateway_server_process}} ([1.13.2,|https://github.com/apache/flink/blob/release-1.13.2/flink-python/pyflink/pyflink_gateway_server.py#L200] [1.13.3|https://github.com/apache/flink/blob/release-1.13.3/flink-python/pyflink/pyflink_gateway_server.py#L280]) re-ordered some method calls. {{{}prepare_environment_variable{}}}'s [non-awareness|https://github.com/apache/flink/blob/release-1.13.3/flink-python/pyflink/pyflink_gateway_server.py#L94C67-L95] of multiple homes and setting of {{FLINK_LIB_DIR}} now is the one that matters, and it is the incorrect location.
> I've confirmed this problem on Amazon Linux 2 and 2023. The problem does not exist on, for example, Ubuntu 20 and 22 (for which {{platlib}} == {{{}purelib{}}}).
> Repro steps on Amazon Linux 2
> {quote}{{yum -y install python3 java-11}}
> {{pip3 install apache-flink==1.13.3}}
> {{python3 -c 'from pyflink.table import EnvironmentSettings ; EnvironmentSettings.new_instance()'}}
> {quote}
> The resulting error is
> {quote}{{The flink-python jar is not found in the opt folder of the FLINK_HOME: /usr/local/lib64/python3.7/site-packages/pyflink}}
> {{Error: Could not find or load main class org.apache.flink.client.python.PythonGatewayServer}}
> {{Caused by: java.lang.ClassNotFoundException: org.apache.flink.client.python.PythonGatewayServer}}
> {{Traceback (most recent call last):}}
> {{  File "<string>", line 1, in <module>}}
> {{  File "/usr/local/lib64/python3.7/site-packages/pyflink/table/environment_settings.py", line 214, in new_instance}}
> {{    return EnvironmentSettings.Builder()}}
> {{  File "/usr/local/lib64/python3.7/site-packages/pyflink/table/environment_settings.py", line 48, in {_}{{_}}init{{_}}{_}}}
> {{    gateway = get_gateway()}}
> {{  File "/usr/local/lib64/python3.7/site-packages/pyflink/java_gateway.py", line 62, in get_gateway}}
> {{    _gateway = launch_gateway()}}
> {{  File "/usr/local/lib64/python3.7/site-packages/pyflink/java_gateway.py", line 112, in launch_gateway}}
> {{    raise Exception("Java gateway process exited before sending its port number")}}
> {{Exception: Java gateway process exited before sending its port number}}
> {quote}
> The flink home under /lib64/ does not contain the jar, but it is in the /lib/ location
> {quote}{{bash-4.2# find /usr/local/lib64/python3.7/site-packages/pyflink -name "flink-python*.jar"}}
> {{bash-4.2# find /usr/local/lib/python3.7/site-packages/pyflink -name "flink-python*.jar"}}
> {{/usr/local/lib/python3.7/site-packages/pyflink/opt/flink-python_2.11-1.13.3.jar}}
> {quote}
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)