You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Julie Chien (JIRA)" <ji...@apache.org> on 2018/10/15 19:15:00 UTC
[jira] [Created] (AIRFLOW-3211) Airflow losing track of running GCP
Dataproc jobs upon Airflow restart
Julie Chien created AIRFLOW-3211:
------------------------------------
Summary: Airflow losing track of running GCP Dataproc jobs upon Airflow restart
Key: AIRFLOW-3211
URL: https://issues.apache.org/jira/browse/AIRFLOW-3211
Project: Apache Airflow
Issue Type: Improvement
Components: gcp
Affects Versions: 1.10.0, 1.9.0
Reporter: Julie Chien
Assignee: Julie Chien
Fix For: 1.10.0, 1.9.0
If Airflow restarts (say, due to deployments, system updates, or regular machine restarts such as the weekly restarts in GCP App Engine) while it's running a job on GCP Dataproc, it'll lose track of that job, mark the task as failed, and eventually retry. However, the jobs may still be running on Dataproc and maybe even finish successfully. So when Airflow retries and reruns the job, the same job will run twice.
To reproduce:
1. Create a DAG in Airflow that runs a Dataproc job that sleeps for 10 minutes. Wait for the cluster to spin up and the job to start running on Dataproc.
2. SSH into the machine that hosts Airflow and run the following commands to simulate restarting Airflow:
{{supervisorctl stop airflow-webserver supervisorctl stop airflow-scheduler ps aux | grep python | grep airflow | awk '\{print $2}' | xargs -r kill -9 supervisorctl restart airflow-scheduler supervisorctl restart airflow-webserver}}
3. Wait for Airflow to retry the task after Supervisor respawns the Airflow processes. Click on the cluster in Dataproc to observe that the job will have been resubmitted, even though the first job is still running without error.
At Etsy, we've customized the Dataproc operators to allow for the new Airflow task to pick up where the old one left off upon Airflow restarts, and have been happily using our solution for the past 6 months. I'd like to submit a PR to merge this change upstream.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)