[jira] [Commented] (AIRFLOW-2016) Add support for Dataproc Workflow Templates

["Kevin McHale (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16470746#comment-16470746 ] 

Kevin McHale commented on AIRFLOW-2016:
---------------------------------------

hi [~DanSedov] I recently tried using these operators (or, at least, the instantiate-inline variant) and I received errors due to the {{template_id}} field being missing.  I think that the issue comes from the airflow {{template_fields}} settings that are set on [this line|https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/operators/dataproc_operator.py#L1034] in the base class.  Airflow's jinja templater tries to populate these fields on the derived classes, but {{template_id}} does not apply for the instantiate-inline operator, and likewise {{template}} should not apply for the instantiate operator, so an error is thrown.

I have put together a quick fix that looks like [this|https://github.com/apache/incubator-airflow/compare/master...mchalek:mchalek-fix-inline-workflow-template-operator] and I am running that successfully on our Airflow deployments at Etsy, but I just wanted to check with you before submitting it to make sure I'm not missing something?  In particular we are using Airflow 1.9.0 and I suppose it's possible that the jinja templater behaves differently in the latest airflow versions?

> Add support for Dataproc Workflow Templates
> -------------------------------------------
>
>                 Key: AIRFLOW-2016
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2016
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: contrib
>            Reporter: Dan Sedov
>            Assignee: Dan Sedov
>            Priority: Minor
>             Fix For: 1.10.0
>
>
> Add new operators to support instantiation of Google Cloud Dataproc Workflow Templates.
> See: https://cloud.google.com/dataproc/docs/reference/rest/v1beta2/projects.regions.workflowTemplates



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)