You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by "Juan M George (Jira)" <ji...@apache.org> on 2019/09/23 13:34:00 UTC

[jira] [Created] (AIRFLOW-5540) task with SparkSubmitOperator does not fail if the spark job it executes fails.

Juan M George created AIRFLOW-5540:
--------------------------------------

             Summary: task with SparkSubmitOperator  does not  fail if the  spark job it executes fails.
                 Key: AIRFLOW-5540
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-5540
             Project: Apache Airflow
          Issue Type: Wish
          Components: operators
    Affects Versions: 1.10.3
         Environment: RHEL 7.3 with default airflow  installation.
            Reporter: Juan M George
         Attachments: airfow_issues

In  my  test Dag, I have a task that uses SparkSubmitOperator operator to execute a spark job  that  reads from a table in a database and  does some processing and write into a file.   In  a scenario, where source table from where  I read data does not exist, the spark job fails but the  task  is still be shown as  executed successfully.  I don't see any other way to handle this business logic failure from the operator side.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)