You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user-zh@flink.apache.org by 程龙 <13...@163.com> on 2022/11/25 08:01:37 UTC
Pyflink提交
在使用pyflink提交任务时，部署模式onyarn
1 在不使用Map等算子下如下参数 能够提交成功 并且运行
.flink run -ynm pytest -m yarn-cluster -pyclientexec ***/python3 -pyexec ***/python3 -pyarch *** /python3.6.8.zip -py demo.py
2 在使用到map算子时 提交没有问题，但是运行报错，报错日志如下：
.flink run -ynm pytest -m yarn-cluster -pyclientexec ***/python3 -pyexec ***/python3 -pyarch *** /python3.6.8.zip -py demo.py
报错内容：
Caused by: java.io.IOException: Failed to execute the command: ./python3.6.8.zip/bin/python3 -c import pyflink;import os;print(os.path.join(os.path.abspath(os.path.dirname(pyflink.__file__)), 'bin'))
output: Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'pyflink'
at org.apache.flink.python.util.PythonEnvironmentManagerUtils.execute(PythonEnvironmentManagerUtils.java:211)
at org.apache.flink.python.util.PythonEnvironmentManagerUtils.getPythonUdfRunnerScript(PythonEnvironmentManagerUtils.java:154)
at org.apache.flink.python.env.beam.ProcessPythonEnvironmentManager.createEnvironment(ProcessPythonEnvironmentManager.java:156)
at org.apache.flink.streaming.api.runners.python.beam.BeamPythonFunctionRunner.createPythonExecutionEnvironment(BeamPythonFunctionRunner.java:398)
at org.apache.flink.streaming.api.runners.python.beam.BeamPythonFunctionRunner.lambda$open$0(BeamPythonFunctionRunner.java:246)
at org.apache.flink.runtime.memory.MemoryManager.lambda$getSharedMemoryResourceForManagedMemory$5(MemoryManager.java:539)
请问环境变量如何设置才能正常运行
Re: Pyflink提交
Posted by Dian Fu <di...@gmail.com>.
集群端的 Python 环境中没有安装 PyFlink： ***/python3 这个环境
On Fri, Nov 25, 2022 at 4:02 PM 程龙 <13...@163.com> wrote:
> 在使用pyflink提交任务时，部署模式onyarn
> 1 在不使用Map等算子下如下参数 能够提交成功 并且运行
> .flink run -ynm pytest -m yarn-cluster -pyclientexec ***/python3
> -pyexec ***/python3 -pyarch *** /python3.6.8.zip -py demo.py
>
>
> 2 在使用到map算子时 提交没有问题，但是运行报错，报错日志如下：
> .flink run -ynm pytest -m yarn-cluster -pyclientexec ***/python3
> -pyexec ***/python3 -pyarch *** /python3.6.8.zip -py demo.py
> 报错内容：
> Caused by: java.io.IOException: Failed to execute the command:
> ./python3.6.8.zip/bin/python3 -c import pyflink;import
> os;print(os.path.join(os.path.abspath(os.path.dirname(pyflink.__file__)),
> 'bin'))
> output: Traceback (most recent call last):
> File "<string>", line 1, in <module>
> ModuleNotFoundError: No module named 'pyflink'
> at
> org.apache.flink.python.util.PythonEnvironmentManagerUtils.execute(PythonEnvironmentManagerUtils.java:211)
> at
> org.apache.flink.python.util.PythonEnvironmentManagerUtils.getPythonUdfRunnerScript(PythonEnvironmentManagerUtils.java:154)
> at
> org.apache.flink.python.env.beam.ProcessPythonEnvironmentManager.createEnvironment(ProcessPythonEnvironmentManager.java:156)
> at
> org.apache.flink.streaming.api.runners.python.beam.BeamPythonFunctionRunner.createPythonExecutionEnvironment(BeamPythonFunctionRunner.java:398)
> at
> org.apache.flink.streaming.api.runners.python.beam.BeamPythonFunctionRunner.lambda$open$0(BeamPythonFunctionRunner.java:246)
> at
> org.apache.flink.runtime.memory.MemoryManager.lambda$getSharedMemoryResourceForManagedMemory$5(MemoryManager.java:539)
>
>
> 请问环境变量如何设置才能正常运行
Re: Pyflink提交
Posted by Xingbo Huang <hx...@gmail.com>.
Hi,
根据报错的提示，执行命令./python3.6.8.zip/bin/python3时没法导入pyflink，你可以在本地检查一下你的这个虚拟环境是不是没有成功安上pyflink
Best,
Xingbo
程龙 <13...@163.com> 于2022年11月25日周五 16:02写道：
> 在使用pyflink提交任务时，部署模式onyarn
> 1 在不使用Map等算子下如下参数 能够提交成功 并且运行
> .flink run -ynm pytest -m yarn-cluster -pyclientexec ***/python3
> -pyexec ***/python3 -pyarch *** /python3.6.8.zip -py demo.py
>
>
> 2 在使用到map算子时 提交没有问题，但是运行报错，报错日志如下：
> .flink run -ynm pytest -m yarn-cluster -pyclientexec ***/python3
> -pyexec ***/python3 -pyarch *** /python3.6.8.zip -py demo.py
> 报错内容：
> Caused by: java.io.IOException: Failed to execute the command:
> ./python3.6.8.zip/bin/python3 -c import pyflink;import
> os;print(os.path.join(os.path.abspath(os.path.dirname(pyflink.__file__)),
> 'bin'))
> output: Traceback (most recent call last):
> File "<string>", line 1, in <module>
> ModuleNotFoundError: No module named 'pyflink'
> at
> org.apache.flink.python.util.PythonEnvironmentManagerUtils.execute(PythonEnvironmentManagerUtils.java:211)
> at
> org.apache.flink.python.util.PythonEnvironmentManagerUtils.getPythonUdfRunnerScript(PythonEnvironmentManagerUtils.java:154)
> at
> org.apache.flink.python.env.beam.ProcessPythonEnvironmentManager.createEnvironment(ProcessPythonEnvironmentManager.java:156)
> at
> org.apache.flink.streaming.api.runners.python.beam.BeamPythonFunctionRunner.createPythonExecutionEnvironment(BeamPythonFunctionRunner.java:398)
> at
> org.apache.flink.streaming.api.runners.python.beam.BeamPythonFunctionRunner.lambda$open$0(BeamPythonFunctionRunner.java:246)
> at
> org.apache.flink.runtime.memory.MemoryManager.lambda$getSharedMemoryResourceForManagedMemory$5(MemoryManager.java:539)
>
>
> 请问环境变量如何设置才能正常运行