You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/09/07 09:18:25 UTC
[GitHub] [airflow] cesar-vermeulen opened a new issue, #26197: Dynamic tasks are randomly put in removed state
cesar-vermeulen opened a new issue, #26197:
URL: https://github.com/apache/airflow/issues/26197
### Apache Airflow version
Other Airflow 2 version
### What happened
We have a DAG running with a cronjob schedule every 4 minutes that checks whether there is a new file present in a file system or not. If there is one available, it lists its path and passes it as a dynamic task via XCOM (one task = one file).
We have experienced a couple of times already that this task is never started, nor does it appear in the logs (not in scheduler logs, nor in AKS logs). It simply disappears and the status of the task is removed.
The DAG has following configurations:
Schedule interval: "*/4 5-22 * * 1-5"
Catchup: False
End Date: None
Max Active Runs: 1
Concurrency: 1
DAG Run Timeout: 0:10:00
### What you think should happen instead
The task should be triggered and started. Instead, it gets the status removed without any logs or trace
### How to reproduce
This happens in random moments, so exact steps to reproduce are hard to determine
### Operating System
Airflow base docker image
### Versions of Apache Airflow Providers
* apache-airflow-providers-microsoft-azure: 4.2.0
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
We are running on AKS with a postgresql backend, with the airflow helm chart of 2.3.3. Docker image is the airflow base image.
Logs are being sent to an azure file share.
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #26197: Dynamic tasks are randomly put in removed state
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #26197:
URL: https://github.com/apache/airflow/issues/26197#issuecomment-1239130710
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] cesar-vermeulen commented on issue #26197: Dynamic tasks are randomly put in removed state
Posted by GitBox <gi...@apache.org>.
cesar-vermeulen commented on issue #26197:
URL: https://github.com/apache/airflow/issues/26197#issuecomment-1240380609
Hey @uranusjr,
Thank you for your reply.
I noticed #25744, however, in this situation, it does not happen during backfill.
Would this behaviour also be expected when not using backfill?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #26197: Dynamic tasks are randomly put in removed state
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #26197:
URL: https://github.com/apache/airflow/issues/26197#issuecomment-1240856832
The linked issue is slightly misleading; it describes backfilling, but the cause has nothing to do with backfilling specifically, as you can read from the PR fixing it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #26197: Dynamic tasks are randomly put in removed state
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #26197:
URL: https://github.com/apache/airflow/issues/26197#issuecomment-1240155752
duplicate of #25744
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr closed issue #26197: Dynamic tasks are randomly put in removed state
Posted by GitBox <gi...@apache.org>.
uranusjr closed issue #26197: Dynamic tasks are randomly put in removed state
URL: https://github.com/apache/airflow/issues/26197
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org