You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Julie Chien (JIRA)" <ji...@apache.org> on 2018/10/15 19:15:00 UTC

[jira] [Created] (AIRFLOW-3211) Airflow losing track of running GCP Dataproc jobs upon Airflow restart

Julie Chien created AIRFLOW-3211:
------------------------------------

             Summary: Airflow losing track of running GCP Dataproc jobs upon Airflow restart
                 Key: AIRFLOW-3211
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-3211
             Project: Apache Airflow
          Issue Type: Improvement
          Components: gcp
    Affects Versions: 1.10.0, 1.9.0
            Reporter: Julie Chien
            Assignee: Julie Chien
             Fix For: 1.10.0, 1.9.0


If Airflow restarts (say, due to deployments, system updates, or regular machine restarts such as the weekly restarts in GCP App Engine) while it's running a job on GCP Dataproc, it'll lose track of that job, mark the task as failed, and eventually retry. However, the jobs may still be running on Dataproc and maybe even finish successfully. So when Airflow retries and reruns the job, the same job will run twice.
 
To reproduce:
1. Create a DAG in Airflow that runs a Dataproc job that sleeps for 10 minutes. Wait for the cluster to spin up and the job to start running on Dataproc.
2. SSH into the machine that hosts Airflow and run the following commands to simulate restarting Airflow:
{{supervisorctl stop airflow-webserver supervisorctl stop airflow-scheduler ps aux | grep python | grep airflow | awk '\{print $2}' | xargs -r kill -9 supervisorctl restart airflow-scheduler supervisorctl restart airflow-webserver}}
3. Wait for Airflow to retry the task after Supervisor respawns the Airflow processes. Click on the cluster in Dataproc to observe that the job will have been resubmitted, even though the first job is still running without error.
 
At Etsy, we've customized the Dataproc operators to allow for the new Airflow task to pick up where the old one left off upon Airflow restarts, and have been happily using our solution for the past 6 months. I'd like to submit a PR to merge this change upstream.
 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)