Using different python version in PythonVirtualEnv operator

[Malte Brunst <Br...@schoenberger-group.com>]

Hi fellow Airflow users,

in one of my DAGs I am executing a python function from a python module I developed via the PythonVirtualEnv operator.
I want to be able to develop this module independently of Airflow, which means I also want to be able to use a different python version than the one from the Airflow version I am using.
I am currently running the Docker Airflow image apache/airflow:2.1.1-python3.8
When I try to install python 3.9 as a dependency into the PythonVirtualEnv operator it throws an error saying it cannot find the python version:
[cid:image001.png@01D7ABC0.84A9F3C0]


How can I install an additional python version into my Airflow image that can be used in the PythonVirtualEnv operator but does not interfere with the python version that Airflow itself is using?
Thank you very much for any help or input!

Best
Robert

Re: Using different python version in PythonVirtualEnv operator

[Jarek Potiuk <ja...@potiuk.com>]

Just create your own image by extending the image as explained here:
https://airflow.apache.org/docs/docker-stack/build.html.

Specifically this chapter
https://airflow.apache.org/docs/docker-stack/build.html#extending-the-image
is followed by some examples. You just need to install python 3.9 in the
way that you'd usually install it in debian.

J.

On Fri, Sep 17, 2021 at 12:35 PM Malte Brunst <
Brunst.Malte@schoenberger-group.com> wrote:

> Hi fellow Airflow users,
>
>
>
> in one of my DAGs I am executing a python function from a python module I
> developed via the PythonVirtualEnv operator.
>
> I want to be able to develop this module independently of Airflow, which
> means I also want to be able to use a different python version than the one
> from the Airflow version I am using.
>
> I am currently running the Docker Airflow image
> apache/airflow:2.1.1-python3.8
>
> When I try to install python 3.9 as a dependency into the PythonVirtualEnv
> operator it throws an error saying it cannot find the python version:
>
>
>
>
>
> How can I install an additional python version into my Airflow image that
> can be used in the PythonVirtualEnv operator but does not interfere with
> the python version that Airflow itself is using?
>
> Thank you very much for any help or input!
>
>
>
> Best
>
> Robert
>
>
>
>
>
>
>
>