You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Joska H <ha...@gmail.com> on 2025/04/01 06:42:51 UTC
PyFink Application Mode Dependency Resolution
When submitting PyFink jobs via Application Mode:
./bin/flink run-application -t yarn-application \
-Djobmanager.memory.process.size=1024m \
-Dtaskmanager.memory.process.size=1024m \
-Dyarn.application.name=test_applicationmode \
-pyclientexec /usr/bin/python \
-pyexec /usr/bin/python \
-python.pythonpath /usr/bin/python \
-py /data/module/flink-1.18.1/work_space/scripts/test.py
We encounter ModuleNotFoundError:pyflink despite all worker nodes having the required Python packages. However, these jobs work properly in YARN Per-Job mode. Could you give me some advise.
We would appreciate any code samples or configuration references you could provide. Please find attached our full submission script and error logs for reference.
Best regards,
Shiquan Song.
Re: PyFink Application Mode Dependency Resolution
Posted by Dian Fu <di...@gmail.com>.
From the exception stack, I suspect that it's not true that all the
cluster nodes have installed PyFlink. Could you check the node where
the application master executes and check if it's true?
Regards,
Dian
On Wed, Apr 9, 2025 at 9:52 AM Joska H <ha...@gmail.com> wrote:
>
> We have installed PyFlink on all nodes of the cluster. Each node can successfully execute python /example/word_count.py and can also execute jobs in yarn-per-job mode. However, when submitting jobs in application mode, an error occurs indicating that the package cannot be found.
> The submission command is:
>
> flink run-application --target yarn-application \
> -py /data/module/flink-1.18.1/work_space/scripts/test.py \
> -pyexec /usr/bin/python \
> -Dyarn.application.name=test_applicationmode \
> -Dpython.client.executable=/usr/bin/python
>
> The error message is:
>
> 2025-04-09 09:45:14,846 INFO org.apache.flink.client.python.PythonDriver [] - --------------------------- Python Process Started --------------------------
> 2025-04-09 09:45:14,858 INFO org.apache.flink.client.python.PythonDriver [] - Traceback (most recent call last):
> File "/data/module/flink-1.18.1/work_space/scripts/test.py", line 1, in <module>
> from pyflink.common import Row
> ModuleNotFoundError: No module named 'pyflink'
>
> We hope this information helps clarify the situation. Looking forward to your advice.
> Best regards，
> Shiquan
>
> ________________________________
> From: Dian Fu <di...@gmail.com>
> Sent: Wednesday, April 9, 2025 1:31 AM
> To: Joska H <ha...@gmail.com>
> Cc: user@flink.apache.org <us...@flink.apache.org>
> Subject: Re: PyFink Application Mode Dependency Resolution
>
> Have you installed PyFlink in the used Python environment?
>
> Regards,
> Dian
>
> On Tue, Apr 1, 2025 at 2:43 PM Joska H <ha...@gmail.com> wrote:
> >
> > When submitting PyFink jobs via Application Mode:
> >
> > ./bin/flink run-application -t yarn-application \
> > -Djobmanager.memory.process.size=1024m \
> > -Dtaskmanager.memory.process.size=1024m \
> > -Dyarn.application.name=test_applicationmode \
> > -pyclientexec /usr/bin/python \
> > -pyexec /usr/bin/python \
> > -python.pythonpath /usr/bin/python \
> > -py /data/module/flink-1.18.1/work_space/scripts/test.py
> >
> > We encounter ModuleNotFoundError:pyflink despite all worker nodes having the required Python packages. However, these jobs work properly in YARN Per-Job mode. Could you give me some advise.
> >
> > We would appreciate any code samples or configuration references you could provide. Please find attached our full submission script and error logs for reference.
> >
> > Best regards,
> > Shiquan Song.
> >
> >
Re: PyFink Application Mode Dependency Resolution
Posted by Dian Fu <di...@gmail.com>.
Have you installed PyFlink in the used Python environment?
Regards,
Dian
On Tue, Apr 1, 2025 at 2:43 PM Joska H <ha...@gmail.com> wrote:
>
> When submitting PyFink jobs via Application Mode:
>
> ./bin/flink run-application -t yarn-application \
> -Djobmanager.memory.process.size=1024m \
> -Dtaskmanager.memory.process.size=1024m \
> -Dyarn.application.name=test_applicationmode \
> -pyclientexec /usr/bin/python \
> -pyexec /usr/bin/python \
> -python.pythonpath /usr/bin/python \
> -py /data/module/flink-1.18.1/work_space/scripts/test.py
>
> We encounter ModuleNotFoundError:pyflink despite all worker nodes having the required Python packages. However, these jobs work properly in YARN Per-Job mode. Could you give me some advise.
>
> We would appreciate any code samples or configuration references you could provide. Please find attached our full submission script and error logs for reference.
>
> Best regards,
> Shiquan Song.
>
>