You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Gergely Fürnstáhl (Jira)" <ji...@apache.org> on 2023/01/18 15:49:00 UTC
[jira] [Updated] (IMPALA-11849) Improve python environment stability around impala_shell packaging
[ https://issues.apache.org/jira/browse/IMPALA-11849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gergely Fürnstáhl updated IMPALA-11849:
---------------------------------------
Description:
How do we use python in Impala currently:
* custom built python2 for ABI compatibility reasons to use by impala_shell
* During build, we test the installability of impala_shell package by:
** Distributing by source to a tar.gz
** Creating virtualenv using the custom built impala-virtualenv using the system interpreter (/usr/bin/python*) both for python2 and python3
*** This automatically installs pip and setuptools, by default a version it chooses.
*** This version of impala-virtualenv only support --no-pip and --no-setuptools, a moderner version support --setuptools <version> too, both could be used to fix the version of the packaging environment regardless of the interpreters version (to an extent, they need to be compatible obviously)
** Packaging it to a wheel
*** This uses an isolated build, where with modernising the packaging and introducing pyproject.toml we could specify setuptools' version
** Install it to the virtual env
*** This uses pip, with the possibility to fallback to setuptools legacy method.
* During impala-shell release, we probably do something similar, but need to investigate it.
Problems:
* We use 3 different pythons during the build, system python's version could vary depending on the OS/image.
* pip and setuptools' version is chosen by the interpreter, which makes the build not reproducible, a new release could break the install (IMPALA-11846)
* custom built python2 is not getting any younger/better
Possibilities:
* Specify exact version of pip and setuptools used during build to make it reproducible, specify setuptools version used for packaging impala_shell e.g. by using a pyproject.toml
* Custom build python3 too, a newer version could make our tests running faster too and drop dependency on system python.
* Test installing impala_shell with a more realistic BUILD_VERSION instead of a hardcoded one.
was:
How do we use python in Impala currently:
* custom built python2 for ABI compatibility reasons to use by impala_shell
* During build, we test the installability of impala_shell package by:
** Distributing by source to a tar.gz
** Creating virtualenv using the custom built impala-virtualenv using the system interpreter (/usr/bin/python*) both for python2 and python3
*** This automatically installs pip and setuptools, by default a version it chooses.
*** This version of impala-virtualenv only support --no-pip and --no-setuptools, a moderner version support --setuptools <version> too, both could be used to fix the version of the packaging environment regardless of the interpreters version (to an extent, they need to be compatible obviously)
** Packaging it to a wheel
*** This uses an isolated build, where with modernising the packaging and introducing pyproject.toml we could specify setuptools' version
** Install it to the virtual env
*** This uses pip, with the possibility to fallback to setuptools legacy method.
* During impala-shell release, we probably do something similar, but need to investigate it.
Problems:
* We use 3 different pythons during the build, system python's version could vary depending on the OS/image.
* pip and setuptools' version is chosen by the interpreter, which makes the build not reproducible, a new release could break the install ([IMPALA-11846|https://issues.apache.org/jira/browse/IMPALA-11846])
* custom built python2 is not getting any younger/better
Possibilities:
* Specify exact version of pip and setuptools used during build to make it reproducible, specify setuptools version used for packaging impala_shell e.g. by using a pyproject.toml
* Custom build python3 too, a newer version could make our tests running faster too and drop dependency on system python.
> Improve python environment stability around impala_shell packaging
> ------------------------------------------------------------------
>
> Key: IMPALA-11849
> URL: https://issues.apache.org/jira/browse/IMPALA-11849
> Project: IMPALA
> Issue Type: Improvement
> Reporter: Gergely Fürnstáhl
> Priority: Major
>
> How do we use python in Impala currently:
> * custom built python2 for ABI compatibility reasons to use by impala_shell
> * During build, we test the installability of impala_shell package by:
> ** Distributing by source to a tar.gz
> ** Creating virtualenv using the custom built impala-virtualenv using the system interpreter (/usr/bin/python*) both for python2 and python3
> *** This automatically installs pip and setuptools, by default a version it chooses.
> *** This version of impala-virtualenv only support --no-pip and --no-setuptools, a moderner version support --setuptools <version> too, both could be used to fix the version of the packaging environment regardless of the interpreters version (to an extent, they need to be compatible obviously)
> ** Packaging it to a wheel
> *** This uses an isolated build, where with modernising the packaging and introducing pyproject.toml we could specify setuptools' version
> ** Install it to the virtual env
> *** This uses pip, with the possibility to fallback to setuptools legacy method.
> * During impala-shell release, we probably do something similar, but need to investigate it.
> Problems:
> * We use 3 different pythons during the build, system python's version could vary depending on the OS/image.
> * pip and setuptools' version is chosen by the interpreter, which makes the build not reproducible, a new release could break the install (IMPALA-11846)
> * custom built python2 is not getting any younger/better
> Possibilities:
> * Specify exact version of pip and setuptools used during build to make it reproducible, specify setuptools version used for packaging impala_shell e.g. by using a pyproject.toml
> * Custom build python3 too, a newer version could make our tests running faster too and drop dependency on system python.
> * Test installing impala_shell with a more realistic BUILD_VERSION instead of a hardcoded one.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org