You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user-zh@flink.apache.org by RS <ti...@163.com> on 2022/06/07 03:23:42 UTC
sql-client pyexec参数生效疑问
Hi，
环境：
- flink-1.14.3， 单机集群
- 服务器上默认python2，也存在python3.6.8
- /xxx/bin/python3是python3生成的虚拟环境
使用sql-client测试pyflink的udf，自定义了一个函数f1，/xxx/p.py
启动命令：
./bin/sql-client.sh -pyfs file:///xxx/p.py -pyexec /xxx/bin/python3
配置pyexec指定了使用的python为python3
执行命令报错，报错信息如下：
Flink SQL> create temporary function fun1 as 'p.f1' language python;
[INFO] Execute statement succeed.
Flink SQL> select fun1('a',1,'s');
Traceback (most recent call last):
File "/usr/lib64/python2.7/runpy.py", line 151, in _run_module_as_main
mod_name, loader, code, fname = _get_module_details(mod_name)
File "/usr/lib64/python2.7/runpy.py", line 101, in _get_module_details
loader = get_loader(mod_name)
File "/usr/lib64/python2.7/pkgutil.py", line 464, in get_loader
return find_loader(fullname)
File "/usr/lib64/python2.7/pkgutil.py", line 474, in find_loader
for importer in iter_importers(fullname):
File "/usr/lib64/python2.7/pkgutil.py", line 430, in iter_importers
__import__(pkg)
File "/home/flink-1.14.3/opt/python/pyflink.zip/pyflink/__init__.py", line 26, in <module>
RuntimeError: Python versions prior to 3.6 are not supported for PyFlink [sys.version_info(major=2, minor=7, micro=5, releaselevel='final', serial=0)].
[ERROR] Could not execute SQL statement. Reason:
java.lang.IllegalStateException: Instantiating python function 'p.f1' failed.
报错提示中使用到的是python2，不是参数里面配置的python3，如何让pyexec生效？
Thx
Re:Re: sql-client pyexec参数生效疑问
Posted by RS <ti...@163.com>.
Hi，
是这个问题了，成功了，不清楚为什么要把UDF的解释器分开配置
Thx
在 2022-06-08 13:29:48，"Dian Fu" <di...@gmail.com> 写道：
>有两个参数指定Python解释器：
>1）-pyexec，指定的是作业执行过程中，用来运行Python UDF的Python解释器路径
>2）-pyclientexec，指定客户端编译作业的时候，用到的Python解释器路径，具体信息可以看一下：
>https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/python/dependency_management/#python-interpreter-of-client
>
>可以把这个参数-pyclientexec 也加上试试。
>
>On Tue, Jun 7, 2022 at 11:24 AM RS <ti...@163.com> wrote:
>
>> Hi，
>>
>>
>> 环境：
>> - flink-1.14.3， 单机集群
>> - 服务器上默认python2，也存在python3.6.8
>> - /xxx/bin/python3是python3生成的虚拟环境
>>
>>
>> 使用sql-client测试pyflink的udf，自定义了一个函数f1，/xxx/p.py
>> 启动命令：
>> ./bin/sql-client.sh -pyfs file:///xxx/p.py -pyexec /xxx/bin/python3
>> 配置pyexec指定了使用的python为python3
>>
>>
>> 执行命令报错，报错信息如下：
>> Flink SQL> create temporary function fun1 as 'p.f1' language python;
>> [INFO] Execute statement succeed.
>> Flink SQL> select fun1('a',1,'s');
>> Traceback (most recent call last):
>> File "/usr/lib64/python2.7/runpy.py", line 151, in _run_module_as_main
>> mod_name, loader, code, fname = _get_module_details(mod_name)
>> File "/usr/lib64/python2.7/runpy.py", line 101, in _get_module_details
>> loader = get_loader(mod_name)
>> File "/usr/lib64/python2.7/pkgutil.py", line 464, in get_loader
>> return find_loader(fullname)
>> File "/usr/lib64/python2.7/pkgutil.py", line 474, in find_loader
>> for importer in iter_importers(fullname):
>> File "/usr/lib64/python2.7/pkgutil.py", line 430, in iter_importers
>> __import__(pkg)
>> File "/home/flink-1.14.3/opt/python/pyflink.zip/pyflink/__init__.py",
>> line 26, in <module>
>> RuntimeError: Python versions prior to 3.6 are not supported for PyFlink
>> [sys.version_info(major=2, minor=7, micro=5, releaselevel='final',
>> serial=0)].
>> [ERROR] Could not execute SQL statement. Reason:
>> java.lang.IllegalStateException: Instantiating python function 'p.f1'
>> failed.
>>
>>
>> 报错提示中使用到的是python2，不是参数里面配置的python3，如何让pyexec生效？
>>
>>
>> Thx
Re: sql-client pyexec参数生效疑问
Posted by Dian Fu <di...@gmail.com>.
有两个参数指定Python解释器：
1）-pyexec，指定的是作业执行过程中，用来运行Python UDF的Python解释器路径
2）-pyclientexec，指定客户端编译作业的时候，用到的Python解释器路径，具体信息可以看一下：
https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/python/dependency_management/#python-interpreter-of-client
可以把这个参数-pyclientexec 也加上试试。
On Tue, Jun 7, 2022 at 11:24 AM RS <ti...@163.com> wrote:
> Hi，
>
>
> 环境：
> - flink-1.14.3， 单机集群
> - 服务器上默认python2，也存在python3.6.8
> - /xxx/bin/python3是python3生成的虚拟环境
>
>
> 使用sql-client测试pyflink的udf，自定义了一个函数f1，/xxx/p.py
> 启动命令：
> ./bin/sql-client.sh -pyfs file:///xxx/p.py -pyexec /xxx/bin/python3
> 配置pyexec指定了使用的python为python3
>
>
> 执行命令报错，报错信息如下：
> Flink SQL> create temporary function fun1 as 'p.f1' language python;
> [INFO] Execute statement succeed.
> Flink SQL> select fun1('a',1,'s');
> Traceback (most recent call last):
> File "/usr/lib64/python2.7/runpy.py", line 151, in _run_module_as_main
> mod_name, loader, code, fname = _get_module_details(mod_name)
> File "/usr/lib64/python2.7/runpy.py", line 101, in _get_module_details
> loader = get_loader(mod_name)
> File "/usr/lib64/python2.7/pkgutil.py", line 464, in get_loader
> return find_loader(fullname)
> File "/usr/lib64/python2.7/pkgutil.py", line 474, in find_loader
> for importer in iter_importers(fullname):
> File "/usr/lib64/python2.7/pkgutil.py", line 430, in iter_importers
> __import__(pkg)
> File "/home/flink-1.14.3/opt/python/pyflink.zip/pyflink/__init__.py",
> line 26, in <module>
> RuntimeError: Python versions prior to 3.6 are not supported for PyFlink
> [sys.version_info(major=2, minor=7, micro=5, releaselevel='final',
> serial=0)].
> [ERROR] Could not execute SQL statement. Reason:
> java.lang.IllegalStateException: Instantiating python function 'p.f1'
> failed.
>
>
> 报错提示中使用到的是python2，不是参数里面配置的python3，如何让pyexec生效？
>
>
> Thx