[jira] [Created] (IMPALA-12434) Isolate shell/pkg_resources.py to keep it off of the PYTHONPATH

["Joe McDonnell (Jira)" <ji...@apache.org>]

Joe McDonnell created IMPALA-12434:
--------------------------------------

             Summary: Isolate shell/pkg_resources.py to keep it off of the PYTHONPATH
                 Key: IMPALA-12434
                 URL: https://issues.apache.org/jira/browse/IMPALA-12434
             Project: IMPALA
          Issue Type: Task
          Components: Infrastructure
    Affects Versions: Impala 4.3.0
            Reporter: Joe McDonnell


In some internal builds on Redhat 8, the pip install of impala-shell into the Python 3 virtualenv can fail with this error:
{noformat}
Exception:
Traceback (most recent call last):
  File "impala/shell/build/py3_venv/lib64/python3.6/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "impala/shell/build/py3_venv/lib64/python3.6/site-packages/pip/commands/install.py", line 357, in run
    wb.build(autobuilding=True)
  File "impala/shell/build/py3_venv/lib64/python3.6/site-packages/pip/wheel.py", line 753, in build
    self.requirement_set.prepare_files(self.finder)
  File "impala/shell/build/py3_venv/lib64/python3.6/site-packages/pip/req/req_set.py", line 381, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "impala/shell/build/py3_venv/lib64/python3.6/site-packages/pip/req/req_set.py", line 637, in _prepare_file
    abstract_dist.prep_for_dist()
  File "impala/shell/build/py3_venv/lib64/python3.6/site-packages/pip/req/req_set.py", line 130, in prep_for_dist
    self.req_to_install.run_egg_info()
  File "impala/shell/build/py3_venv/lib64/python3.6/site-packages/pip/req/req_install.py", line 417, in run_egg_info
    self.setup_py, self.link,
  File "impala/shell/build/py3_venv/lib64/python3.6/site-packages/pip/req/req_install.py", line 387, in setup_py
    import setuptools  # noqa
  File "impala/shell/build/py3_venv/lib64/python3.6/site-packages/setuptools/__init__.py", line 12, in <module>
    import setuptools.version
  File "impala/shell/build/py3_venv/lib64/python3.6/site-packages/setuptools/version.py", line 1, in <module>
    import pkg_resources
  File "impala/shell/pkg_resources.py", line 50
    def _bypass_ensure_directory(name, mode=0777):
                                              ^
SyntaxError: invalid token
make[3]: *** [shell/CMakeFiles/shell_python3_install.dir/build.make:70: shell/CMakeFiles/shell_python3_install] Error 2
{noformat}
What seems to happen is shell/pkg_resources.py is on the PYTHONPATH and our pkg_resources.py doesn't work with Python 3. If I move pkg_resources.py to a different directory (like a legacy subdirectory) or if I add a WORKING_DIRECTORY for the CMake command, it works fine. In general, keeping pkg_resources.py away from our PYTHONPATH (which includes $IMPALA_HOME/shell) seems like a good idea.

This problem doesn't reproduce on the regular upstream Impala build or on my local Ubuntu 20 workstation build, but it reliably reproduces on an internal build.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)