[jira] [Created] (FLINK-30803) PyFlink mishandles script dependencies

["Nuno Afonso (Jira)" <ji...@apache.org>]

Nuno Afonso created FLINK-30803:
-----------------------------------

             Summary: PyFlink mishandles script dependencies
                 Key: FLINK-30803
                 URL: https://issues.apache.org/jira/browse/FLINK-30803
             Project: Flink
          Issue Type: Bug
          Components: API / Python
    Affects Versions: 1.15.3, 1.15.2, 1.16.0
            Reporter: Nuno Afonso
         Attachments: word_count_split.zip

h2. Summary

Since Flink 1.15, PyFlink is unable to run scripts that import other scripts under other directories. For instance, if _main.py_ imports {_}job/word_count.py{_}, PyFlink will fail due to not finding the _job_ directory.

The issue seems to have started after a [refactoring of _PythonDriver_|https://github.com/apache/flink/commit/330aae0c6e0811f50888d17830f10f7a29efe7d7] to address FLINK-26847. The path to the Python script is removed, which forces PyFlink to use the copy in its temporary directory. When files are copied to this directory, the original directory structure is not maintained and ends up breaking the imports.
h2. Testing

To confirm the regression, I ran the attached application in both a Flink 1.14.6 and 1.15.3 clusters.
h3. Flink 1.14.6

Application was able to start after being submitted via CLI:

 
{code:java}
% ./bin/flink run --python ~/sandbox/word_count_split/main.py
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.flink.api.java.ClosureCleaner (file:/.../flink-1.14.6/lib/flink-dist_2.12-1.14.6.jar) to field java.lang.String.value
WARNING: Please consider reporting this to the maintainers of org.apache.flink.api.java.ClosureCleaner
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Job has been submitted with JobID 6f7be21072384ca3a314af10860c4ba8 {code}
 
h3. Flink 1.15.3

Application did not start due to not finding the _job_ directory:

 
{code:java}
% ./bin/flink run --python ~/sandbox/word_count_split/main.py
Traceback (most recent call last):
  File "/usr/lib64/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib64/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmp/pyflink/40c649c3-24af-46ef-ae27-e0019cb55769/3673dd18-adff-40e0-bb11-06a3f00ba29c/main.py", line 5, in <module>
    from job.word_count import word_count
ModuleNotFoundError: No module named 'job'
org.apache.flink.client.program.ProgramAbortException: java.lang.RuntimeException: Python process exits with code: 1
        at org.apache.flink.client.python.PythonDriver.main(PythonDriver.java:140)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
        at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
        at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
        at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:841)
        at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:240)
        at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1085)
        at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1163)
        at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
        at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1163)
Caused by: java.lang.RuntimeException: Python process exits with code: 1
        at org.apache.flink.client.python.PythonDriver.main(PythonDriver.java:130)
        ... 13 more {code}
 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)