[jira] [Created] (FLINK-39895) PyFlink tests fail on release-1.20: setuptools 82 removed pkg_resources, breaking the Beam Python harness

["Martijn Visser (Jira)" <ji...@apache.org>]

Martijn Visser created FLINK-39895:
--------------------------------------

             Summary: PyFlink tests fail on release-1.20: setuptools 82 removed pkg_resources, breaking the Beam Python harness
                 Key: FLINK-39895
                 URL: https://issues.apache.org/jira/browse/FLINK-39895
             Project: Flink
          Issue Type: Bug
          Components: Build System / CI, Tests
    Affects Versions: 1.20.5
            Reporter: Martijn Visser
            Assignee: Martijn Visser


All PyFlink unit tests (e.g. {{pyflink/datastream/tests/test_window.py}}, {{test_stream_execution_environment.py}}) and the PyFlink end-to-end test fail on the {{release-1.20}} branch nightly. Example: https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=75796

The Java side fails creating the Beam stage bundle factory:

{noformat}
java.lang.RuntimeException: Failed to create stage bundle factory! ...
Caused by: java.lang.IllegalStateException: Process died with exit code 0
{noformat}

The Beam Python harness ({{beam_boot.py}}) dies on import:

{noformat}
File ".../apache_beam/runners/portability/stager.py", line 63, in <module>
    import pkg_resources
ModuleNotFoundError: No module named 'pkg_resources'
{noformat}

{{setuptools}} 82.0.0 removed the {{pkg_resources}} module. apache-beam 2.48.0
(the version embedded on release-1.20, pinned {{apache-beam>=2.43.0,<2.49.0}})
still imports {{pkg_resources}}. The pins in {{flink-python/dev/dev-requirements.txt}}
and {{flink-python/pyproject.toml}} were open-ended ({{setuptools>=75.3}}), so pip
resolved {{setuptools-82.0.1}} and the harness exits before serving the gRPC
FnHarness, hence "Process died with exit code 0".



--
This message was sent by Atlassian Jira
(v8.20.10#820010)