[jira] [Created] (IMPALA-11849) Improve python environment stability around impala_shell packaging

["Gergely Fürnstáhl (Jira)" <ji...@apache.org>]

Gergely Fürnstáhl created IMPALA-11849:
------------------------------------------

             Summary: Improve python environment stability around impala_shell packaging
                 Key: IMPALA-11849
                 URL: https://issues.apache.org/jira/browse/IMPALA-11849
             Project: IMPALA
          Issue Type: Improvement
            Reporter: Gergely Fürnstáhl


How do we use python in Impala currently:
 * custom built python2 for ABI compatibility reasons to use by impala_shell
 * During build, we test the installability of impala_shell package by:
 ** Distributing by source to a tar.gz
 ** Creating virtualenv using the custom built impala-virtualenv using the system interpreter (/usr/bin/python*) both for python2 and python3
 *** This automatically installs pip and setuptools, by default a version it chooses.
 *** This version of impala-virtualenv only support --no-pip and --no-setuptools, a moderner version support --setuptools <version> too, both could be used to fix the version of the packaging environment regardless of the interpreters version (to an extent, they need to be compatible obviously)
 ** Packaging it to a wheel
 *** This uses an isolated build, where with modernising the packaging and introducing pyproject.toml we could specify setuptools' version
 ** Install it to the virtual env
 *** This uses pip, with the possibility to fallback to setuptools legacy method. 
 * During impala-shell release, we probably do something similar, but need to investigate it.

Problems:
 * We use 3 different pythons during the build, system python's version could vary depending on the OS/image.
 * pip and setuptools' version is chosen by the interpreter, which makes the build not reproducible, a new release could break the install ([IMPALA-11846|https://issues.apache.org/jira/browse/IMPALA-11846])
 * custom built python2 is not getting any younger/better

Possibilities:
 * Specify exact version of pip and setuptools used during build to make it reproducible, specify setuptools version used for packaging impala_shell e.g. by using a pyproject.toml
 * Custom build python3 too, a newer version could make our tests running faster too and drop dependency on system python.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org