[jira] [Created] (FLINK-32758) PyFlink bounds are overly restrictive and outdated

["Deepyaman Datta (Jira)" <ji...@apache.org>]

Deepyaman Datta created FLINK-32758:
---------------------------------------

             Summary: PyFlink bounds are overly restrictive and outdated
                 Key: FLINK-32758
                 URL: https://issues.apache.org/jira/browse/FLINK-32758
             Project: Flink
          Issue Type: Improvement
          Components: API / Python
    Affects Versions: 1.17.1
            Reporter: Deepyaman Datta


Hi! I am part of a team building the Flink backend for Ibis ([https://github.com/ibis-project/ibis]). We would like to leverage PyFlink under the hood for execution; however, PyFlink's requirements are incompatible with several other Ibis requirements. Beyond Ibis, PyFlink's outdated and restrictive requirements prevent it from being used alongside most recent releases of Python data libraries.

Some of the major libraries we (and likely others in the Python community interested in using PyFlink alongside other libraries) need compatibility with:
 * PyArrow (at least >=10.0.0, but there's no reason not to be also be compatible with latest)
 * pandas (should be compatible with 2.x series, but also probably with 1.4.x, released January 2022, and 1.5.x)
 * numpy (1.22 was released in December 2022)
 * Newer releases of Apache Beam
 * Newer releases of cython

Furthermore, uncapped dependencies could be more generally preferable, as they avoid the need for frequent PyFlink releases as newer versions of libraries are released. A common (and great) argument for not upper-bounding dependencies, especially for libraries: [https://iscinumpy.dev/post/bound-version-constraints/]

I am currently testing removing upper bounds in [https://github.com/apache/flink/pull/23141]; so far, builds pass without issue in [b65c072|https://github.com/apache/flink/pull/23141/commits/b65c0723ed66e01e83d718f770aa916f41f34581], and I'm currently waiting on [c8eb15c|https://github.com/apache/flink/pull/23141/commits/c8eb15cbc371dc259fb4fda5395f0f55e08ea9c6] to see if I can get PyArrow to resolve >=10.0.0. Solving the proposed dependencies results in:



{{#}}
{{# This file is autogenerated by pip-compile with Python 3.8}}
{{# by the following command:}}
{{#}}
{{#    pip-compile --config=pyproject.toml --output-file=dev/compiled-requirements.txt dev/dev-requirements.txt}}
{{#}}
{{apache-beam==2.49.0}}
{{    # via -r dev/dev-requirements.txt}}
{{avro-python3==1.10.2}}
{{    # via -r dev/dev-requirements.txt}}
{{certifi==2023.7.22}}
{{    # via requests}}
{{charset-normalizer==3.2.0}}
{{    # via requests}}
{{cloudpickle==2.2.1}}
{{    # via}}
{{    #   -r dev/dev-requirements.txt}}
{{    #   apache-beam}}
{{crcmod==1.7}}
{{    # via apache-beam}}
{{cython==3.0.0}}
{{    # via -r dev/dev-requirements.txt}}
{{dill==0.3.1.1}}
{{    # via apache-beam}}
{{dnspython==2.4.1}}
{{    # via pymongo}}
{{docopt==0.6.2}}
{{    # via hdfs}}
{{exceptiongroup==1.1.2}}
{{    # via pytest}}
{{fastavro==1.8.2}}
{{    # via}}
{{    #   -r dev/dev-requirements.txt}}
{{    #   apache-beam}}
{{fasteners==0.18}}
{{    # via apache-beam}}
{{find-libpython==0.3.0}}
{{    # via pemja}}
{{grpcio==1.56.2}}
{{    # via}}
{{    #   -r dev/dev-requirements.txt}}
{{    #   apache-beam}}
{{    #   grpcio-tools}}
{{grpcio-tools==1.56.2}}
{{    # via -r dev/dev-requirements.txt}}
{{hdfs==2.7.0}}
{{    # via apache-beam}}
{{httplib2==0.22.0}}
{{    # via}}
{{    #   -r dev/dev-requirements.txt}}
{{    #   apache-beam}}
{{idna==3.4}}
{{    # via requests}}
{{iniconfig==2.0.0}}
{{    # via pytest}}
{{numpy==1.24.4}}
{{    # via}}
{{    #   -r dev/dev-requirements.txt}}
{{    #   apache-beam}}
{{    #   pandas}}
{{    #   pyarrow}}
{{objsize==0.6.1}}
{{    # via apache-beam}}
{{orjson==3.9.2}}
{{    # via apache-beam}}
{{packaging==23.1}}
{{    # via pytest}}
{{pandas==2.0.3}}
{{    # via -r dev/dev-requirements.txt}}
{{pemja==0.3.0 ; platform_system != "Windows"}}
{{    # via -r dev/dev-requirements.txt}}
{{pluggy==1.2.0}}
{{    # via pytest}}
{{proto-plus==1.22.3}}
{{    # via apache-beam}}
{{protobuf==4.23.4}}
{{    # via}}
{{    #   -r dev/dev-requirements.txt}}
{{    #   apache-beam}}
{{    #   grpcio-tools}}
{{    #   proto-plus}}
{{py4j==0.10.9.7}}
{{    # via -r dev/dev-requirements.txt}}
{{pyarrow==11.0.0}}
{{    # via}}
{{    #   -r dev/dev-requirements.txt}}
{{    #   apache-beam}}
{{pydot==1.4.2}}
{{    # via apache-beam}}
{{pymongo==4.4.1}}
{{    # via apache-beam}}
{{pyparsing==3.1.1}}
{{    # via}}
{{    #   httplib2}}
{{    #   pydot}}
{{pytest==7.4.0}}
{{    # via -r dev/dev-requirements.txt}}
{{python-dateutil==2.8.2}}
{{    # via}}
{{    #   -r dev/dev-requirements.txt}}
{{    #   apache-beam}}
{{    #   pandas}}
{{pytz==2023.3}}
{{    # via}}
{{    #   -r dev/dev-requirements.txt}}
{{    #   apache-beam}}
{{    #   pandas}}
{{regex==2023.6.3}}
{{    # via apache-beam}}
{{requests==2.31.0}}
{{    # via}}
{{    #   apache-beam}}
{{    #   hdfs}}
{{six==1.16.0}}
{{    # via}}
{{    #   hdfs}}
{{    #   python-dateutil}}
{{tomli==2.0.1}}
{{    # via pytest}}
{{typing-extensions==4.7.1}}
{{    # via apache-beam}}
{{tzdata==2023.3}}
{{    # via pandas}}
{{urllib3==2.0.4}}
{{    # via requests}}
{{wheel==0.41.0}}
{{    # via -r dev/dev-requirements.txt}}
{{zstandard==0.21.0}}
{{    # via apache-beam}}
{{# The following packages are considered to be unsafe in a requirements file:}}
{{# pip}}
{{# setuptools}}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)