[jira] [Comment Edited] (AIRFLOW-3035) gcp_dataproc_hook should treat CANCELLED job state consistently

["Jeffrey Payne (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-3035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16611350#comment-16611350 ] 

Jeffrey Payne edited comment on AIRFLOW-3035 at 9/11/18 10:44 PM:
------------------------------------------------------------------

I also removed some redundant {{print()}} statements from {{_DataProcJob.wait_for_done()}}.


was (Author: jeffkpayne@gmail.com):
I also removed some redundant {{print()}} statements.

> gcp_dataproc_hook should treat CANCELLED job state consistently
> ---------------------------------------------------------------
>
>                 Key: AIRFLOW-3035
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-3035
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: contrib
>    Affects Versions: 1.10.0, 2.0.0, 1.10.1
>            Reporter: Jeffrey Payne
>            Assignee: Jeffrey Payne
>            Priority: Minor
>              Labels: dataproc
>
> When a DP job is cancelled, {{gcp_dataproc_hook.py}} does not treat the {{CANCELLED}} state in a consistent and non-intuitive manner:
> # The API internal to {{gcp_dataproc_hook.py}} returns {{False}} from {{_DataProcJob.wait_for_done()}}, resulting in {{raise_error()}} being called for cancelled jobs, yet {{raise_error()}} only raises {{Exception}} if the job state is {{ERROR}}.
> # The end result from the perspective of the {{dataproc_operator.py}} for a cancelled job is that the job succeeded, which results in the success callback being called.  This seems strange to me, as a "cancelled" job is rarely considered successful, in my experience.
> Simply changing {{raise_error()}} from:
> {code:python}
>         if 'ERROR' == self.job['status']['state']:
> {code}
> to
> {code:python}
>         if self.job['status']['state'] in ('ERROR', 'CANCELLED'):
> {code}
> would fix both of these...
> Another, perhaps better, option would be to have the dataproc job operators accept a list of {{error_states}} that could be passed into {{raise_error()}}, allowing the caller to determine which states should result in "failure" of the task.  I would lean towards that option.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)