[jira] [Created] (FLINK-24663) PyFlink failed to get the site packege path because of no quote in shell command

["jackwangcs (Jira)" <ji...@apache.org>]

jackwangcs created FLINK-24663:
----------------------------------

             Summary: PyFlink failed to get the site packege path because of no quote in shell command
                 Key: FLINK-24663
                 URL: https://issues.apache.org/jira/browse/FLINK-24663
             Project: Flink
          Issue Type: Bug
          Components: API / Python
    Affects Versions: 1.12.3
            Reporter: jackwangcs


Flink throws an exception when it tries to install 3rd party dependencies:
{code:java}
Caused by: java.io.IOException: Failed to execute the command: python -c import sys;from distutils.dist import Distribution;install_obj = Distribution().get_command_obj('install', create=True);install_obj.prefix = sys.argv[1];install_obj.finalize_options();installed_dir = [install_obj.install_purelib];install_obj.install_purelib != install_obj.install_platlib and installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir) > 1 and print(installed_dir[1]) /mnt/yarn/usercache/hadoop/appcache/application_1629776785656_0100/python-dist-fb549cea-0857-4b11-9eb7-7818eaa3f561/python-requirementsCaused by: java.io.IOException: Failed to execute the command: python -c import sys;from distutils.dist import Distribution;install_obj = Distribution().get_command_obj('install', create=True);install_obj.prefix = sys.argv[1];install_obj.finalize_options();installed_dir = [install_obj.install_purelib];install_obj.install_purelib != install_obj.install_platlib and installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir) > 1 and print(installed_dir[1]) /mnt/yarn/usercache/hadoop/appcache/application_1629776785656_0100/python-dist-fb549cea-0857-4b11-9eb7-7818eaa3f561/python-requirementsoutput:   File "<string>", line 1    import sys;from distutils.dist import Distribution;install_obj = Distribution().get_command_obj('install', create=True);install_obj.prefix = sys.argv[1];install_obj.finalize_options();installed_dir = [install_obj.install_purelib];install_obj.install_purelib != install_obj.install_platlib and installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir) > 1 and print(installed_dir[1])                                                                                                                                                                                                                                                                                                                                                                                                                  ^SyntaxError: invalid syntax
 at org.apache.flink.python.util.PythonEnvironmentManagerUtils.execute(PythonEnvironmentManagerUtils.java:211) at org.apache.flink.python.util.PythonEnvironmentManagerUtils.getSitePackagesPath(PythonEnvironmentManagerUtils.java:171) at org.apache.flink.python.util.PythonEnvironmentManagerUtils.pipInstallRequirements(PythonEnvironmentManagerUtils.java:99) at org.apache.flink.python.env.beam.ProcessPythonEnvironmentManager.createEnvironment(ProcessPythonEnvironmentManager.java:169) at org.apache.flink.streaming.api.runners.python.beam.BeamPythonFunctionRunner.createPythonExecutionEnvironment(BeamPythonFunctionRunner.java:339)
....{code}
This can be reproduced by running the python script in a bash shell:
{code:java}
python3 -c import sys;from distutils.dist import Distribution;install_obj = Distribution().get_command_obj('install', create=True);print(sys.argv[1]);install_obj.prefix = sys.argv[1];install_obj.finalize_options();installed_dir = [install_obj.install_purelib];install_obj.install_purelib != install_obj.install_platlib and installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir) > 1 and print(installed_dir[1]) /tmp/requirements
-bash: syntax error near unexpected token `(' {code}
The solution is to quota all argements to execute:
{code:java}
python3 -c "import sys;from distutils.dist import Distribution;install_obj = Distribution().get_command_obj('install', create=True);print(sys.argv[1]);install_obj.prefix = sys.argv[1];install_obj.finalize_options();installed_dir = [install_obj.install_purelib];install_obj.install_purelib != install_obj.install_platlib and installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir) > 1 and print(installed_dir[1])" "/tmp/requirements"
/tmp/requirements
/tmp/requirements/lib/python3.6/site-packages {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)