You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/08/21 22:08:30 UTC
[GitHub] [airflow] potiuk opened a new issue, #25859: Task stuck as deferred
potiuk opened a new issue, #25859:
URL: https://github.com/apache/airflow/issues/25859
### Discussed in https://github.com/apache/airflow/discussions/25630
<div type='discussions-op-text'>
<sup>Originally posted by **Limess** August 10, 2022</sup>
On Airflow version 2.3.3:
A task has become stuck in the deferred state.
Upon clearing the task, nothing happens.
Upon marking the task as a success/failure, the task is marked correctly, but after clearing it immediately goes back to deferred with the same retry number.
I'd expect the task to restart it's execution entirely when cleared.
I haven't raised a bug as I don't have any idea what to do to reproduce this - this hasn't happened to previous deferred tasks I've tried to clear
.</div>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #25859: Task stuck as deferred
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #25859:
URL: https://github.com/apache/airflow/issues/25859#issuecomment-1385797312
> I will try to produce logs, issue however was transient and I'm not sure why worried
That calls even more for more logs and details. If it was transient for you, there is no sane way no-one will be able to reproduce it (and the only way to fix the problem is to reproduce it first)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #25859: Task stuck as deferred
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #25859:
URL: https://github.com/apache/airflow/issues/25859#issuecomment-1385062082
Logs from around clearing the tasks and why it failed are necessary to solve that issue I believe @notatallshaw-gts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] notatallshaw-gts commented on issue #25859: Task stuck as deferred
Posted by GitBox <gi...@apache.org>.
notatallshaw-gts commented on issue #25859:
URL: https://github.com/apache/airflow/issues/25859#issuecomment-1346755223
We had this issue as you've described it this morning, we just moved a lot of tasks to async using Astronomer's FileSensorAsync. Did anyone else report further issues/workarounds?
We're going to turn on debug logs and see if we have it again tomorrow morning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #25859: Task stuck as deferred
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #25859:
URL: https://github.com/apache/airflow/issues/25859#issuecomment-1666590368
I'm closing this issue as we don't have enough information to take action / reproduce it and significant time has passed without new reports for problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] notatallshaw-gts commented on issue #25859: Task stuck as deferred
Posted by GitBox <gi...@apache.org>.
notatallshaw-gts commented on issue #25859:
URL: https://github.com/apache/airflow/issues/25859#issuecomment-1385594629
> Logs from around clearing the tasks and why it failed are necessary to solve that issue I believe @notatallshaw-gts
I will try to produce logs, issue however was transient and I'm not sure why 😟
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal closed issue #25859: Task stuck as deferred
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal closed issue #25859: Task stuck as deferred
URL: https://github.com/apache/airflow/issues/25859
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] notatallshaw-gts commented on issue #25859: Task stuck as deferred
Posted by GitBox <gi...@apache.org>.
notatallshaw-gts commented on issue #25859:
URL: https://github.com/apache/airflow/issues/25859#issuecomment-1349096500
Okay, we seem to got to the bottom of how this was being triggered for us, but I don't have any further information of why clearing the task didn't work.
Our steps to reproduce:
1. Create a task with https://registry.astronomer.io/providers/astronomer-providers/modules/filesensorasync
2. Provide a filepath that will never be found (this happened because we were using a user macro to generate the filepath, and it turned out we were relying on branching conditions in this user macro, so the filepath that we sent at defer time was not the ultimate filepath that is normally detected)
3. After several hours try to clear task, does not work, where as marking as success did
This was all in our prod environment and we didn't see any errors in Scheduler, Triggerer, or Worker logs, we are using Airflow 2.3.4 on Docker with Celery/Redis backend.
I am looking to set-up a dev reproduction of our prod environment later today, let me know if there's anything you would like me to check if I can reproduce in dev.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org