You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/10/26 07:13:47 UTC

[GitHub] [airflow] jimduden6804 opened a new issue #19218: Orphaned Tasks are Retried Leading to Multiple Running Aws Batch Jobs

jimduden6804 opened a new issue #19218:
URL: https://github.com/apache/airflow/issues/19218

### Apache Airflow version

2.1.4

### Operating System

Debian GNU/Linux 11 (bullseye)

### Versions of Apache Airflow Providers

apache-airflow-providers-amazon==2.2.0
apache-airflow-providers-ftp==2.0.1
apache-airflow-providers-http==2.0.1
apache-airflow-providers-imap==2.0.1
apache-airflow-providers-sqlite==2.0.1

### Deployment

Virtualenv installation

### Deployment details

Pip installation.
Scheduler and worker running in different containers in same AWS EKS pod.

### What happened

We use AwsBatchOperator to trigger an AWS batch job.
While the job is running, the Airflow is deployed.
This leads to the task being retriggered.
This is quite expensive in our case.

### What you expected to happen

We would expect that the task is not retried.
Failing would be fine.

### How to reproduce

Create a AWS-Batch job with sufficient runtime.
Trigger it in Airflow with AwsBatchOperator
Restart/redeploy Airflow

### Anything else

-----
default_task_retries is not set (default 0)
-----
We see a similar problem using TriggerDagRunOperator.
-----
It looks like the the job becomes orphaned and is retried.
In the scheduler logs I find:
Reset the following 5 orphaned TaskInstances:
<TaskInstance: my_aws_batch_dag.task 2021-10-25 08:26:47.993733+00:00 [running]>
...
[2021-10-25 12:04:31,538] {scheduler_job.py:353} INFO - 5 tasks up for execution:
<TaskInstance: my_aws_batch_dag.task 2021-10-25 08:26:47.993733+00:00 [running]>
...

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #19218: Orphaned Tasks are Retried Leading to Multiple Running Aws Batch Jobs

Posted by GitBox <gi...@apache.org>.

potiuk commented on issue #19218:
URL: https://github.com/apache/airflow/issues/19218#issuecomment-963519286


   This is correct behaviour. When you redeploy airflow, it gets restarted and tasks are failed abruptly. So they did not fail, they simply did not complete - and since we do not know if the task completed or not we re-run it. This is expected.
   
   The right way of redeploying airflow is to drain all workers before restarting (you need to send SIGTERM to those workers - that puts them in offline mode - which means that they will not take any more tasks but they shutdown when all the tasks complete. Only then you should redeploy airflow.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk closed issue #19218: Orphaned Tasks are Retried Leading to Multiple Running Aws Batch Jobs

Posted by GitBox <gi...@apache.org>.

potiuk closed issue #19218:
URL: https://github.com/apache/airflow/issues/19218


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #19218: Orphaned Tasks are Retried Leading to Multiple Running Aws Batch Jobs

Posted by GitBox <gi...@apache.org>.

boring-cyborg[bot] commented on issue #19218:
URL: https://github.com/apache/airflow/issues/19218#issuecomment-951629209


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org