You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/08/11 13:39:15 UTC
[GitHub] [airflow] lssatvik opened a new issue, #25672: Tasks not executing, getting skipped after dag timeout
lssatvik opened a new issue, #25672:
URL: https://github.com/apache/airflow/issues/25672
### Apache Airflow version
2.3.3
### What happened
I have multiple dags running regularly. Dags fail randomly for no reason. When I look at the grid view, one task is in skipped state due to dag timeout. But that task did not get executed during the entire period until timeout.
This happens randomly to any dag, to any task, at any time.
<img width="1721" alt="image" src="https://user-images.githubusercontent.com/3456652/184146382-95571378-ca4e-4227-9609-3cc372b6da62.png">
### What you think should happen instead
_No response_
### How to reproduce
_No response_
### Operating System
Ubuntu 20.04.3 LTS
### Versions of Apache Airflow Providers
_No response_
### Deployment
Virtualenv installation
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] malthe commented on issue #25672: Tasks get stuck at random, never execute, get skipped after dag timeout
Posted by GitBox <gi...@apache.org>.
malthe commented on issue #25672:
URL: https://github.com/apache/airflow/issues/25672#issuecomment-1226853251
@lssatvik it might also be that this is a duplicate of #24731. There we have some logs that hint at this being some sort of bad internal state after a redis disconnect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] lssatvik commented on issue #25672: Tasks get stuck at random, never execute, get skipped after dag timeout
Posted by GitBox <gi...@apache.org>.
lssatvik commented on issue #25672:
URL: https://github.com/apache/airflow/issues/25672#issuecomment-1226844275
I shall endeavour to obtain the logs from a feasible case. Will update once I obtain them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #25672: Tasks get stuck at random, never execute, get skipped after dag timeout
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #25672:
URL: https://github.com/apache/airflow/issues/25672#issuecomment-1569308797
This issue has been closed because it has not received response from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #25672: Tasks get stuck at random, never execute, get skipped after dag timeout
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #25672:
URL: https://github.com/apache/airflow/issues/25672#issuecomment-1223962651
Any logs. Withhout trying to see if there are some logs around the problem you are mentioning, we are not able to help. just try to correlate the logs (there are few of them - including celery, redis, schedulers, worker) and try to see if you can find any anomalies there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] closed issue #25672: Tasks get stuck at random, never execute, get skipped after dag timeout
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #25672: Tasks get stuck at random, never execute, get skipped after dag timeout
URL: https://github.com/apache/airflow/issues/25672
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] malthe commented on issue #25672: Tasks get stuck at random, never execute, get skipped after dag timeout
Posted by GitBox <gi...@apache.org>.
malthe commented on issue #25672:
URL: https://github.com/apache/airflow/issues/25672#issuecomment-1217735597
Can you check the logs to see if you see something similar to this:
Here we lose contact to redis, but after reconnecting, the worker is in a stuck state – this would explain how your "tasks get stuck at random".
What's your setup?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] lssatvik commented on issue #25672: Tasks get stuck at random, never execute, get skipped after dag timeout
Posted by GitBox <gi...@apache.org>.
lssatvik commented on issue #25672:
URL: https://github.com/apache/airflow/issues/25672#issuecomment-1220733091
@malthe What are those logs?
I use celery and redis.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #25672: Tasks get stuck at random, never execute, get skipped after dag timeout
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #25672:
URL: https://github.com/apache/airflow/issues/25672#issuecomment-1556339531
This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org