You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "jackwangcs (Jira)" <ji...@apache.org> on 2021/10/29 15:10:00 UTC
[jira] [Commented] (FLINK-24663) PyFlink failed to get the site
packege path because of SyntaxError in shell command
[ https://issues.apache.org/jira/browse/FLINK-24663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17436028#comment-17436028 ]
jackwangcs commented on FLINK-24663:
------------------------------------
In fact, we can only quote the long python command part like below:
{code:java}
python3 -c 'import sys;from distutils.dist import Distribution;install_obj = Distribution().get_command_obj('\''install'\'', create=True);print(sys.argv[1]);install_obj.prefix = sys.argv[1];install_obj.finalize_options();installed_dir = [install_obj.install_purelib];install_obj.install_purelib != install_obj.install_platlib and installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir) > 1 and print(installed_dir[1])' /tmp/requirements
/tmp/requirements
/tmp/requirements/lib/python3.6/site-packages {code}
I will try to fix this by quoting the python argument.
> PyFlink failed to get the site packege path because of SyntaxError in shell command
> -----------------------------------------------------------------------------------
>
> Key: FLINK-24663
> URL: https://issues.apache.org/jira/browse/FLINK-24663
> Project: Flink
> Issue Type: Bug
> Components: API / Python
> Affects Versions: 1.12.3
> Reporter: jackwangcs
> Priority: Major
>
> Flink throws an exception when it tries to install 3rd party dependencies:
> {code:java}
> Caused by: java.io.IOException: Failed to execute the command: python -c import sys;from distutils.dist import Distribution;install_obj = Distribution().get_command_obj('install', create=True);install_obj.prefix = sys.argv[1];install_obj.finalize_options();installed_dir = [install_obj.install_purelib];install_obj.install_purelib != install_obj.install_platlib and installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir) > 1 and print(installed_dir[1]) /mnt/yarn/usercache/hadoop/appcache/application_1629776785656_0100/python-dist-fb549cea-0857-4b11-9eb7-7818eaa3f561/python-requirements
> output: File "<string>", line 1
> import sys;from distutils.dist import Distribution;install_obj = Distribution().get_command_obj('install', create=True);install_obj.prefix = sys.argv[1];install_obj.finalize_options();installed_dir = [install_obj.install_purelib];install_obj.install_purelib != install_obj.install_platlib and installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir) > 1 and print(installed_dir[1]) ^SyntaxError: invalid syntax
> at org.apache.flink.python.util.PythonEnvironmentManagerUtils.execute(PythonEnvironmentManagerUtils.java:211) at org.apache.flink.python.util.PythonEnvironmentManagerUtils.getSitePackagesPath(PythonEnvironmentManagerUtils.java:171) at org.apache.flink.python.util.PythonEnvironmentManagerUtils.pipInstallRequirements(PythonEnvironmentManagerUtils.java:99) at org.apache.flink.python.env.beam.ProcessPythonEnvironmentManager.createEnvironment(ProcessPythonEnvironmentManager.java:169) at org.apache.flink.streaming.api.runners.python.beam.BeamPythonFunctionRunner.createPythonExecutionEnvironment(BeamPythonFunctionRunner.java:339)
> ....{code}
> This can be reproduced by running the python script in a bash shell:
> {code:java}
> python3 -c import sys;from distutils.dist import Distribution;install_obj = Distribution().get_command_obj('install', create=True);print(sys.argv[1]);install_obj.prefix = sys.argv[1];install_obj.finalize_options();installed_dir = [install_obj.install_purelib];install_obj.install_purelib != install_obj.install_platlib and installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir) > 1 and print(installed_dir[1]) /tmp/requirements
> -bash: syntax error near unexpected token `(' {code}
> The solution is to quote all argements to execute:
> {code:java}
> python3 "-c" "import sys;from distutils.dist import Distribution;install_obj = Distribution().get_command_obj('install', create=True);print(sys.argv[1]);install_obj.prefix = sys.argv[1];install_obj.finalize_options();installed_dir = [install_obj.install_purelib];install_obj.install_purelib != install_obj.install_platlib and installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir) > 1 and print(installed_dir[1])" "/tmp/requirements"
> /tmp/requirements
> /tmp/requirements/lib/python3.6/site-packages{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)