You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/07/29 09:04:11 UTC
[GitHub] [airflow] huozhanfeng opened a new pull request #17305: [Bug] Backfill job fails to run when there are tasks run into rescheduling state.
huozhanfeng opened a new pull request #17305:
URL: https://github.com/apache/airflow/pull/17305
Backfill job fails to run when there are tasks run into rescheduling state.
The error log as follows in issue #13322
<pre>
Traceback (most recent call last):
File "/opt/conda/bin/airflow", line 8, in <module>
sys.exit(main())
File "/opt/conda/lib/python3.8/site-packages/airflow/__main__.py", line 40, in main
args.func(args)
File "/opt/conda/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/opt/conda/lib/python3.8/site-packages/airflow/utils/cli.py", line 89, in wrapper
return f(*args, **kwargs)
File "/opt/conda/lib/python3.8/site-packages/airflow/cli/commands/dag_command.py", line 103, in dag_backfill
dag.run(
File "/opt/conda/lib/python3.8/site-packages/airflow/models/dag.py", line 1701, in run
job.run()
File "/opt/conda/lib/python3.8/site-packages/airflow/jobs/base_job.py", line 237, in run
self._execute()
File "/opt/conda/lib/python3.8/site-packages/airflow/utils/session.py", line 65, in wrapper
return func(*args, session=session, **kwargs)
File "/opt/conda/lib/python3.8/site-packages/airflow/jobs/backfill_job.py", line 799, in _execute
self._execute_for_run_dates(
File "/opt/conda/lib/python3.8/site-packages/airflow/utils/session.py", line 62, in wrapper
return func(*args, **kwargs)
File "/opt/conda/lib/python3.8/site-packages/airflow/jobs/backfill_job.py", line 722, in _execute_for_run_dates
processed_dag_run_dates = self._process_backfill_task_instances(
File "/opt/conda/lib/python3.8/site-packages/airflow/utils/session.py", line 62, in wrapper
return func(*args, **kwargs)
File "/opt/conda/lib/python3.8/site-packages/airflow/jobs/backfill_job.py", line 620, in _process_backfill_task_instances
self._update_counters(ti_status=ti_status)
File "/opt/conda/lib/python3.8/site-packages/airflow/utils/session.py", line 65, in wrapper
return func(*args, session=session, **kwargs)
File "/opt/conda/lib/python3.8/site-packages/airflow/jobs/backfill_job.py", line 211, in _update_counters
ti_status.running.pop(key)
KeyError: TaskInstanceKey(dag_id='dag_id', task_id='task_name', execution_date=datetime.datetime(2020, 12, 15, 0, 0, tzinfo=Timezone('UTC')), try_number=2)
</pre>
The root cause is that the filed `try_number` doesn't Increase when the task runs into rescheduling stage, but there is a reduce operation on `try_number`.
Currently, I can't think out a good ut to test it, only post the code here to help the one who is affected by it to solve the problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] huozhanfeng commented on pull request #17305: [Bug] Backfill job fails to run when there are tasks run into rescheduling state.
Posted by GitBox <gi...@apache.org>.
huozhanfeng commented on pull request #17305:
URL: https://github.com/apache/airflow/pull/17305#issuecomment-890311534
> Could you rebase this to the latest main? We had some disruptions in CI that caused the tests to fail in this PR.
Sure, I will rebase it. Thanks for your kind reminder.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] galuszkak commented on pull request #17305: [Bug] Backfill job fails to run when there are tasks run into rescheduling state.
Posted by GitBox <gi...@apache.org>.
galuszkak commented on pull request #17305:
URL: https://github.com/apache/airflow/pull/17305#issuecomment-911595088
Hey,
It looks like we hit same problem described in #13322 . We would be really grateful if anyone from reviewers could look into this patch? (cc @ashb @kaxil @XD-DENG )
Basically backfill feature is heavily used in our production release of Airflow, and it's blocking us with moving into Airflow 2.x.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] huozhanfeng commented on pull request #17305: [Bug] Backfill job fails to run when there are tasks run into rescheduling state.
Posted by GitBox <gi...@apache.org>.
huozhanfeng commented on pull request #17305:
URL: https://github.com/apache/airflow/pull/17305#issuecomment-890990367
> CI seems to pass (the failure is due to resource insufficiency and not related to this PR). I have no idea if this is a right change, so let’s see what people more familiar with this part of code have to say…
Sure, thank you a lot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pgodek commented on pull request #17305: [Bug] Backfill job fails to run when there are tasks run into rescheduling state.
Posted by GitBox <gi...@apache.org>.
pgodek commented on pull request #17305:
URL: https://github.com/apache/airflow/pull/17305#issuecomment-919906845
> Maybe there are other cases to cause the issue... You can print the content of `ti.key` and `ti_status.running` in function `_update_counters` to get the details.
Hi, so I was not clear. Fix from PR resolves issue with `KeyError: TaskInstanceKey ...`,
but task is looped until timeout.
Attempt counter is not updated, task logs contains sequence of info like below
```
--------------------------------------------------------------------------------
[2021-09-13 11:16:50,930] {taskinstance.py:1088} INFO - Starting attempt 1 of 4
[2021-09-13 11:16:50,930] {taskinstance.py:1089} INFO -
--------------------------------------------------------------------------------
...
--------------------------------------------------------------------------------
[2021-09-15 10:33:37,692] {taskinstance.py:1088} INFO - Starting attempt 1 of 4
[2021-09-15 10:33:37,692] {taskinstance.py:1089} INFO -
--------------------------------------------------------------------------------
...
```
until timeout is reached.
It is expected that after reaching attempts threshold it will fail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil merged pull request #17305: [Bug] Backfill job fails to run when there are tasks run into rescheduling state.
Posted by GitBox <gi...@apache.org>.
kaxil merged pull request #17305:
URL: https://github.com/apache/airflow/pull/17305
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] huozhanfeng edited a comment on pull request #17305: [Bug] Backfill job fails to run when there are tasks run into rescheduling state.
Posted by GitBox <gi...@apache.org>.
huozhanfeng edited a comment on pull request #17305:
URL: https://github.com/apache/airflow/pull/17305#issuecomment-919938155
> > Maybe there are other cases to cause the issue... You can print the content of `ti.key` and `ti_status.running` in function `_update_counters` to get the details.
>
> Hi, so I was not clear. Fix from PR resolves issue with `KeyError: TaskInstanceKey ...`,
> but task is looped until timeout.
> Attempt counter is not updated, task logs contains sequence of info like below
>
> ```
>
> --------------------------------------------------------------------------------
> [2021-09-13 11:16:50,930] {taskinstance.py:1088} INFO - Starting attempt 1 of 4
> [2021-09-13 11:16:50,930] {taskinstance.py:1089} INFO -
> --------------------------------------------------------------------------------
> ...
> --------------------------------------------------------------------------------
> [2021-09-15 10:33:37,692] {taskinstance.py:1088} INFO - Starting attempt 1 of 4
> [2021-09-15 10:33:37,692] {taskinstance.py:1089} INFO -
> --------------------------------------------------------------------------------
> ...
> ```
>
> until timeout is reached.
> It is expected that after reaching attempts threshold it will fail.
Is it in a rescheduling state? if so, it's reasonable because the attempt sequence num only is updated when the task runs into failure. And In rescheduling state, it only means the upstream is not met and will try again until upstream is done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] galuszkak edited a comment on pull request #17305: [Bug] Backfill job fails to run when there are tasks run into rescheduling state.
Posted by GitBox <gi...@apache.org>.
galuszkak edited a comment on pull request #17305:
URL: https://github.com/apache/airflow/pull/17305#issuecomment-911595088
Hey,
It looks like we hit same problem described in #13322 . We would be really grateful if anyone from reviewers could look into this patch? (cc @ashb @kaxil @XD-DENG )
Basically backfill feature is heavily used in our production release of Airflow, and it's blocking us with moving into Airflow 2.x.
Thanks for all contributors that are working really hard on this project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] huozhanfeng commented on pull request #17305: [Bug] Backfill job fails to run when there are tasks run into rescheduling state.
Posted by GitBox <gi...@apache.org>.
huozhanfeng commented on pull request #17305:
URL: https://github.com/apache/airflow/pull/17305#issuecomment-913386050
> Since this is a bug fix, please could you add a unit test to prevent future regressions of this?
Ehh...I have tried, but it's difficult to make a UT for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #17305: [Bug] Backfill job fails to run when there are tasks run into rescheduling state.
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #17305:
URL: https://github.com/apache/airflow/pull/17305#issuecomment-911718407
Since this is a bug fix, please could you add a unit test to prevent future regressions of this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] huozhanfeng commented on pull request #17305: [Bug] Backfill job fails to run when there are tasks run into rescheduling state.
Posted by GitBox <gi...@apache.org>.
huozhanfeng commented on pull request #17305:
URL: https://github.com/apache/airflow/pull/17305#issuecomment-919938155
> > Maybe there are other cases to cause the issue... You can print the content of `ti.key` and `ti_status.running` in function `_update_counters` to get the details.
>
> Hi, so I was not clear. Fix from PR resolves issue with `KeyError: TaskInstanceKey ...`,
> but task is looped until timeout.
> Attempt counter is not updated, task logs contains sequence of info like below
>
> ```
>
> --------------------------------------------------------------------------------
> [2021-09-13 11:16:50,930] {taskinstance.py:1088} INFO - Starting attempt 1 of 4
> [2021-09-13 11:16:50,930] {taskinstance.py:1089} INFO -
> --------------------------------------------------------------------------------
> ...
> --------------------------------------------------------------------------------
> [2021-09-15 10:33:37,692] {taskinstance.py:1088} INFO - Starting attempt 1 of 4
> [2021-09-15 10:33:37,692] {taskinstance.py:1089} INFO -
> --------------------------------------------------------------------------------
> ...
> ```
>
> until timeout is reached.
> It is expected that after reaching attempts threshold it will fail.
Is it in a rescheduling state? if so, it's reasonable because the attempt sequence num only is updated when the task runs into failed. And In rescheduling state, it only means the upstream is not met and will try again until upstream is done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] huozhanfeng edited a comment on pull request #17305: [Bug] Backfill job fails to run when there are tasks run into rescheduling state.
Posted by GitBox <gi...@apache.org>.
huozhanfeng edited a comment on pull request #17305:
URL: https://github.com/apache/airflow/pull/17305#issuecomment-889838682
Welcome anyone who is familiar with this code block to help to improve the UT. The code logic has run for two weeks in our production env and the stability is ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pgodek commented on pull request #17305: [Bug] Backfill job fails to run when there are tasks run into rescheduling state.
Posted by GitBox <gi...@apache.org>.
pgodek commented on pull request #17305:
URL: https://github.com/apache/airflow/pull/17305#issuecomment-920039312
> Is it in a rescheduling state? if so, it's reasonable because the attempt sequence num only is updated when the task runs into failure. And In rescheduling state, it only means the upstream is not met and will try again until upstream is done.
You are right, I messed up some concepts.
So I do confirm that this PR looks like solution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] huozhanfeng edited a comment on pull request #17305: [Bug] Backfill job fails to run when there are tasks run into rescheduling state.
Posted by GitBox <gi...@apache.org>.
huozhanfeng edited a comment on pull request #17305:
URL: https://github.com/apache/airflow/pull/17305#issuecomment-913570741
> Hi,
> we have tested this fix and it does not seems to solve issue.
Maybe there are other cases to cause the issue... You can print the content of `ti.key` and `ti_status.running` in function `_update_counters` to get the details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] huozhanfeng commented on pull request #17305: [Bug] Backfill job fails to run when there are tasks run into rescheduling state.
Posted by GitBox <gi...@apache.org>.
huozhanfeng commented on pull request #17305:
URL: https://github.com/apache/airflow/pull/17305#issuecomment-913570454
> Hi,
> we have tested this fix and it does not seems to solve issue.
Maybe there are other cases to cause the issue... You can print the content of `ti.key` and `ti_status.running` in function `_update_counters ` to get the details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #17305: [Bug] Backfill job fails to run when there are tasks run into rescheduling state.
Posted by GitBox <gi...@apache.org>.
kaxil commented on pull request #17305:
URL: https://github.com/apache/airflow/pull/17305#issuecomment-937635847
I am merging this to get it in 2.2.0rc1 end of today but we should write a test for it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil merged pull request #17305: [Bug] Backfill job fails to run when there are tasks run into rescheduling state.
Posted by GitBox <gi...@apache.org>.
kaxil merged pull request #17305:
URL: https://github.com/apache/airflow/pull/17305
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] huozhanfeng commented on pull request #17305: [Bug] Backfill job fails to run when there are tasks run into rescheduling state.
Posted by GitBox <gi...@apache.org>.
huozhanfeng commented on pull request #17305:
URL: https://github.com/apache/airflow/pull/17305#issuecomment-913570741
> Hi,
> we have tested this fix and it does not seems to solve issue.
Maybe there are other cases to cause the issue... You can print the content of ti.key and ti_status.running in function _update_counters to get the details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pgodek commented on pull request #17305: [Bug] Backfill job fails to run when there are tasks run into rescheduling state.
Posted by GitBox <gi...@apache.org>.
pgodek commented on pull request #17305:
URL: https://github.com/apache/airflow/pull/17305#issuecomment-913441664
Hi,
we have tested this fix and it does not seems to solve issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] huozhanfeng commented on pull request #17305: [Bug] Backfill job fails to run when there are tasks run into rescheduling state.
Posted by GitBox <gi...@apache.org>.
huozhanfeng commented on pull request #17305:
URL: https://github.com/apache/airflow/pull/17305#issuecomment-889838682
Welcome anyone who is familiar with this code block can help improve the UT. The code logic has run for two weeks in our production env and the stability is ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #17305: [Bug] Backfill job fails to run when there are tasks run into rescheduling state.
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #17305:
URL: https://github.com/apache/airflow/pull/17305#issuecomment-937402224
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #17305: [Bug] Backfill job fails to run when there are tasks run into rescheduling state.
Posted by GitBox <gi...@apache.org>.
kaxil commented on pull request #17305:
URL: https://github.com/apache/airflow/pull/17305#issuecomment-937635847
I am merging this to get it in 2.2.0rc1 end of today but we should write a test for it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #17305: [Bug] Backfill job fails to run when there are tasks run into rescheduling state.
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #17305:
URL: https://github.com/apache/airflow/pull/17305#issuecomment-937402224
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] huozhanfeng removed a comment on pull request #17305: [Bug] Backfill job fails to run when there are tasks run into rescheduling state.
Posted by GitBox <gi...@apache.org>.
huozhanfeng removed a comment on pull request #17305:
URL: https://github.com/apache/airflow/pull/17305#issuecomment-913570454
> Hi,
> we have tested this fix and it does not seems to solve issue.
Maybe there are other cases to cause the issue... You can print the content of `ti.key` and `ti_status.running` in function `_update_counters ` to get the details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on pull request #17305: [Bug] Backfill job fails to run when there are tasks run into rescheduling state.
Posted by GitBox <gi...@apache.org>.
uranusjr commented on pull request #17305:
URL: https://github.com/apache/airflow/pull/17305#issuecomment-890294285
Could you rebase this to the latest main? We had some disruptions in CI that caused the tests to fail in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on pull request #17305: [Bug] Backfill job fails to run when there are tasks run into rescheduling state.
Posted by GitBox <gi...@apache.org>.
uranusjr commented on pull request #17305:
URL: https://github.com/apache/airflow/pull/17305#issuecomment-890810717
CI seems to pass (the failure is due to resource insufficiency and not related to this PR). I have no idea if this is a right change, so let’s see what people more familiar with this part of code have to say…
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org