You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Valentyn Tymofieiev (Jira)" <ji...@apache.org> on 2021/10/05 01:01:17 UTC
[jira] [Commented] (BEAM-12000) Support Python 3.9 in Apache Beam
[ https://issues.apache.org/jira/browse/BEAM-12000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17424242#comment-17424242 ]
Valentyn Tymofieiev commented on BEAM-12000:
--------------------------------------------
Preparing the upgrade of Jenkins image to include Python 3.9.
There was also an ask to increase the Jenkins boot disk size, so will create the image based off 1TB disk.
gcloud compute instances create valentyn-py39-install-1tb --project=apache-beam-testing --zone us-central1-b --image-family=jenkins-slave-boot-image --machine-type=n1-highmem-16 --boot-disk-size 1000
{noformat}
Created
[https://www.googleapis.com/compute/v1/projects/apache-beam-testing/zones/us-central1-b/instances/valentyn-py39-install-1tb].
WARNING: Some requests generated warnings:
- Disk size: '1000 GB' is larger than image size: '500 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
{noformat}
# wait for a few min for machine to boot.
gcloud compute sshy--project=apache-beam-testing --zone us-central1-b
Looks like partition is sized properly:
{noformat}
$ sudo df -Th
Filesystem Type Size Used Avail Use% Mounted on
/dev/sda1 ext4 970G 17G 953G 2% /
{noformat}
Note that some process is may be holding the apt lock right after first boot:
{noformat}
$ ps aux | grep -i apt
root 3205 0.0 0.0 4504 708 ? Ss 18:19 0:00 /bin/sh /usr/lib/apt/apt.systemd.daily install
root 3217 0.0 0.0 4504 1676 ? S 18:19 0:00 /bin/sh /usr/lib/apt/apt.systemd.daily lock_is_held install
valentyn 3390 0.0 0.0 12944 932 pts/0 S+ 18:20 0:00 grep --color=auto -i apt
{noformat}
So before running more apt commands, we may need to wait a few min until that is over.
After ~5 min we are clear:
{noformat}
$ ps aux | grep -i apt
valentyn 31897 0.0 0.0 12944 932 pts/0 S+ 18:23 0:00 grep --color=auto -i apt
{noformat}
Install Python 3.9 from deadsnakes-ppa (which was already added to repo list).
{noformat}
sudo apt install python3.9 python3.9-dev python3.9-distutils python3.9-venv
{noformat}
Also installing python3.7-venv, to explore if we can stop using virtualenv on jenkins, and instead use python3.x -m venv …, see [discussion in BEAM-8152|https://issues.apache.org/jira/browse/BEAM-8152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17424228#comment-17424228].
{noformat}
sudo apt install python3.7-venv
{noformat}
As a side effect, the last command updates python3.7 to 3.7.10.
After that, exit the session, stopped the VM created a new image as described in [1], then start a new from a newly-created image and do some validation to make sure versions installed and available
{noformat}
:~$ gcloud compute instances create valentyn-py39-verify --project=apache-beam-testing --zone us-central1-b --image-family=jenkins-slave-boot-image --machine-type=n1-highmem-16
Created [https://www.googleapis.com/compute/v1/projects/apache-beam-testing/zones/us-central1-b/instances/valentyn-py39-verify].
NAME ZONE MACHINE_TYPE PREEMPTIBLE INTERNAL_IP EXTERNAL_IP STATUS
valentyn-py39-verify us-central1-b n1-highmem-16 10.128.0.99 34.122.67.197 RUNNING
:~$ gcloud compute ssh valentyn-py39-verify --project=apache-beam-testing --zone us-central1-b
Last login: Tue Oct 5 00:03:36 2021 from 98.42.137.166
valentyn@valentyn-py39-verify:~$ python3.9
Python 3.9.4 (default, Apr 9 2021, 01:15:05)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
valentyn@valentyn-py39-verify:~$ python3.7 -m venv /tmp/abc
source ./valentyn@valentyn-py39-verify:~$ source ./tmp/abc/bin/activate
-bash: ./tmp/abc/bin/activate: No such file or directory
valentyn@valentyn-py39-verify:~$ source /tmp/abc/bin/activate
(abc) valentyn@valentyn-py39-verify:~$ python
Python 3.7.10 (default, Feb 20 2021, 21:21:24)
valentyn@valentyn-py39-verify:~$ virtualenv -p python3.9 /tmp/venv1
Running virtualenv with interpreter /usr/bin/python3.9
Using base prefix '/usr'
New python executable in /tmp/venv1/bin/python3.9
Also creating executable in /tmp/venv1/bin/python
. /tmp/vInstalling setuptools, pip, wheel...
env1/bin/acdone.
valentyn@valentyn-py39-verify:~$ . /tmp/venv1/bin/activate
(venv1) valentyn@valentyn-py39-verify:~$ python
Python 3.9.4 (default, Apr 9 2021, 01:15:05)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
{noformat}
Finally, clean up leftover VMs.
Next, we should upgrade the Jenkins nodes to use the new image. Current plan is to combine this with Go update.
[1] https://cwiki.apache.org/confluence/display/BEAM/Jenkins+Tips#JenkinsTips-HowtoinstallandupgradesoftwareonJenkinsworkers
> Support Python 3.9 in Apache Beam
> ---------------------------------
>
> Key: BEAM-12000
> URL: https://issues.apache.org/jira/browse/BEAM-12000
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
> Reporter: Valentyn Tymofieiev
> Assignee: Valentyn Tymofieiev
> Priority: P2
> Time Spent: 8h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)