You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Iuliia Volkova (JIRA)" <ji...@apache.org> on 2018/12/10 12:46:00 UTC
[jira] [Comment Edited] (AIRFLOW-2774) DataFlowPythonOperator needs
to support DirectRunner to speed up end-to-end testing of Airflow dag
[ https://issues.apache.org/jira/browse/AIRFLOW-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16714671#comment-16714671 ]
Iuliia Volkova edited comment on AIRFLOW-2774 at 12/10/18 12:45 PM:
--------------------------------------------------------------------
but DirectRunner is not a 'local' version of DataflowRunner, it's just a different engine that supported by Apache Beam, as SparkRunner or ApexRunner, and others.
was (Author: xnuinside):
but DirectRunner is not a 'local' version of DataflowRunner, it's just a different engine that supported by Apache Beam, as SparkRunner or ApexRunner, or others.
> DataFlowPythonOperator needs to support DirectRunner to speed up end-to-end testing of Airflow dag
> --------------------------------------------------------------------------------------------------
>
> Key: AIRFLOW-2774
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2774
> Project: Apache Airflow
> Issue Type: Improvement
> Components: Dataflow
> Affects Versions: 1.9.0
> Reporter: Evgeny Podlepaev
> Priority: Minor
>
> **DataFlowPythonOperator needs to support DirectRunner as a runner option to facilitate local end-to-end testing of the entire Airflow dag. Right now if DirectRunner is set via job options, the DataFlowHook will wait infinitely trying to get status of the remote job which does not exist:
> _DataflowJob(self.get_conn(), variables['project'], name,
> variables['region'], self.poll_sleep).wait_for_done()
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)