You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Valentyn Tymofieiev (Jira)" <ji...@apache.org> on 2021/10/04 18:28:00 UTC
[jira] [Commented] (BEAM-8152) Provide a way to better control
minor+patch versions of Python 3.x interpreters used to run Beam tests
locally and on Jenkins.
[ https://issues.apache.org/jira/browse/BEAM-8152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17424098#comment-17424098 ]
Valentyn Tymofieiev commented on BEAM-8152:
-------------------------------------------
Looked a bit into various ways installing Python 3 on Jenkins.
Compiling a desired version from sources and installing it, worked reasonably well on Ubuntu 16:
Steps were like this:
{noformat}
# see: https://devguide.python.org/setup/
sudo apt-get install -y build-essential gdb lcov libbz2-dev libffi-dev \
libgdbm-dev liblzma-dev libncurses5-dev libreadline6-dev \
libsqlite3-dev libssl-dev lzma lzma-dev tk-dev uuid-dev zlib1g-dev
wget https://www.python.org/ftp/python/3.9.7/Python-3.9.7.tgz
tar -xvf Python-3.9.7.tgz
cd ./Python-3.9.7
./configure
make
sudo make install
{noformat}
Ideally we could have a setup where we could switch to a different Python version without having to reimage Jenkins. Also running a one-time script on all nodes perhaps can help. For this I looked into
pyenv. It also works well on ubunutu and can create environments for a particular python version. One complicaiton is, pyenv seems to environments in a particular location where pyenv is installed.
On Jenkins, it would be better to install virtual environments under /workspace folder that can be cleaned up. Some people were discussing this and other frictions of using pyenv on Jenkins on:
https://github.com/pyenv/pyenv-virtualenv/issues/408#issuecomment-783478626.
Another discussion of managing multiple python versions further in this thread: https://github.com/python-poetry/poetry/issues/522
In case of Ubuntu, installing a Python version may be possible and easy from https://launchpad.net/~deadsnakes/+archive/ubuntu/ppa repo via:
{noformat}
sudo apt install python3.9 python3.9-dev python3.9-distutils python3.9-venv
{noformat}
Ultimately, the easiest UX for users may be perhaps to assume as few tools as possible, and expect that installations of python3.6, python3.7, python3.8 are available in $PATH, and create virtual environments with venv or virtualenv. Note that according to https://stackoverflow.com/a/47559925/5153670 , using virtualenv does not work well if Python installation is not provided in a system directory, seems that venv is becoming a preferred recommendation.
> Provide a way to better control minor+patch versions of Python 3.x interpreters used to run Beam tests locally and on Jenkins.
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: BEAM-8152
> URL: https://issues.apache.org/jira/browse/BEAM-8152
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core, testing
> Reporter: Valentyn Tymofieiev
> Priority: P3
>
> Currently, Beam Python test infrastructure does not provide fine-grained way to control Python interpreter version. The major+minor version are typically selected by virtual environment, and the patch version of interpreter is defined by the version of python package available on machine that is running the tests.
> For example, Jenkins ubuntu-based machines, use Python 3.5.2 for python 3.5 test suites, while debian-based SDK harness containers for Python 3.5 come with Python 3.5.6, and the python3.5 package available on my dev machine is Python 3.5.4.
> Throughout development of Python 3.5.x, Cpython implementation details that have changed in and these changes affect certain codepaths in Beam, such as type inference.
>
> When we encounter such issues, it is difficult for Beam developers to test their changes against a particular patch version of Python interpreter both remotely and locally. Opening this issue to make it simpler.
> cc: [~markflyhigh] [~yifanzou] [~udim] [~altay] who may have opinions and ideas about how to make this simpler.
> Note that there are separate questions:
> 1) which patch versions of Python we should test against on Jenkins
> 2) which patch versions of Python Beam should claim to support.
> Regardless of the answers to those questions, we may want to make it easier for an engineer to run a test suite against a particular patch version of Python, and/or make it easier to switch which patch version is used by Jenkins.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)