[jira] [Commented] (AIRFLOW-5520) DataflowPythonOperator dependency management requires side effects

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-5520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17018933#comment-17018933 ] 

ASF subversion and git services commented on AIRFLOW-5520:
----------------------------------------------------------

Commit 4dcce7be9995129c66da171c274acbafceb4e77b in airflow's branch refs/heads/master from Kamil Breguła
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=4dcce7b ]

[AIRFLOW-5520] Add options to run Dataflow in a virtual environment (#6590)



> DataflowPythonOperator dependency management requires side effects
> ------------------------------------------------------------------
>
>                 Key: AIRFLOW-5520
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-5520
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: gcp
>    Affects Versions: 1.10.2
>            Reporter: Jacob Ferriero
>            Priority: Major
>
> When using DataflowPythonOperator it is difficult to manage apache beam version, (and other python dependencies) without affecting your entire airflow environment. It seems the Dataflow hook just submits a subprocess and python 
> The operator / hook should be improved to isolate python dependencies for running run py_file.
> Perhaps this could be achieved in a virtual environment (similar to PythonVirtualEnvOperator).
> For beam it's often customary to specify a --requirements_file or --setup_file to manage python dependencies, we could run one of these in the venv to get it setup. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)