You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Daniel Cooper (JIRA)" <ji...@apache.org> on 2019/08/12 10:40:00 UTC

[jira] [Created] (AIRFLOW-5168) Dataproc operators fail on 1.10.4

Daniel Cooper created AIRFLOW-5168:
--------------------------------------

             Summary: Dataproc operators fail on 1.10.4
                 Key: AIRFLOW-5168
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-5168
             Project: Apache Airflow
          Issue Type: Bug
          Components: worker
    Affects Versions: 1.10.4
            Reporter: Daniel Cooper


We've found a problem with DataprocOperators introduced in 1.10.4.

There is a commit to master which adds label support, this has a change in the hook & base operator: [https://github.com/apache/airflow/commit/4d58f36df3b533edfce873799dc161ee34571d63#diff-bc55ad8b749b7a136f51ffabfdbaf13dL684|https://slack-redir.net/link?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fcommit%2F4d58f36df3b533edfce873799dc161ee34571d63%23diff-bc55ad8b749b7a136f51ffabfdbaf13dL684]

Then in the 1.10.4 branch there is a massive commit for GCP DLP: [https://github.com/apache/airflow/commit/222c6ac45de54ed0398645c1d456a592e194325b|https://slack-redir.net/link?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fcommit%2F222c6ac45de54ed0398645c1d456a592e194325b]

This seems to have leaked the operator part of the label change only: [https://github.com/apache/airflow/commit/222c6ac45de54ed0398645c1d456a592e194325b#diff-bc55ad8b749b7a136f51ffabfdbaf13dR693|https://slack-redir.net/link?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fcommit%2F222c6ac45de54ed0398645c1d456a592e194325b%23diff-bc55ad8b749b7a136f51ffabfdbaf13dR693]

This then manifests as this error when trying to execute a Dataproc job: {{[2019-08-09 14:32:02,971] \{taskinstance.py:1047} ERROR - _DataProcJobBuilder instance has no attribute 'add_labels'}}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)