You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/01/22 03:34:40 UTC
[GitHub] [airflow] jpradass opened a new issue #19767: dag callbacks are missed sometimes
jpradass opened a new issue #19767:
URL: https://github.com/apache/airflow/issues/19767
### Apache Airflow version
2.0.1
### Operating System
RedHat ubi8/ubi-minimal
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon==1.1.0
apache-airflow-providers-celery==1.0.1
apache-airflow-providers-ftp==1.0.1
apache-airflow-providers-http==1.1.0
apache-airflow-providers-imap==1.0.1
apache-airflow-providers-postgres==1.0.1
apache-airflow-providers-redis==1.0.1
apache-airflow-providers-sqlite==1.0.1
### Deployment
Other Docker-based deployment
### Deployment details
I have set a PostgreSQL as database for airflow to keep its data and Redis as message broker with a custom backend with consumers to handle messages sent from airflow. So Airflow communicates with this backend when events happens as when jobs are running or have finished.
### What happened
I've noticed that scheduler aren't sending all events when they have happened. I have defined some custom functions as callbacks when a Dag is created, so when a job is finished the scheduler should make an event and put it to Redis to notify this backend I have built.
Reviewing logs I've seen callbacks are usually invoked in order but if there's one of them out of place, scheduler skip one notification even if job has finished successfully. Attached are the scheduler logs.
```
{ "kind": "LOG", "level": "DEBUG", "message": "[2021-11-22 13:20:02,228] scheduler_job.py:557 - Processing Callback Request: {'full_filepath': '/home/user/airflow/dags/my.namespace/my.namespace_514.py', 'msg': 'success', 'dag_id': 'my.namespace_514', 'execution_date': datetime.datetime(2021, 11, 22, 13, 15, tzinfo=Timezone('UTC')), 'is_failure_callback': False}" }
{ "kind": "LOG", "level": "INFO", "message": "[2021-11-22 13:20:02,302] logging_mixin.py:104 - { "kind": "LOG", "level": "INFO", "message": "[2021-11-22 13:20:02,302] dag.py:868 - Executing dag callback function: <function on_success_callback_job at 0x7f869fb93670>" }" }
{ "kind": "LOG", "level": "INFO", "message": "[2021-11-22 13:20:02,314] logging_mixin.py:104 - { "kind": "LOG", "level": "DEBUG", "message": "[2021-11-22 13:20:02,313] plugins_manager.py:264 - Plugins are already loaded. Skipping." }" }
{ "kind": "LOG", "level": "INFO", "message": "[2021-11-22 13:20:02,314] logging_mixin.py:104 - { "kind": "LOG", "level": "DEBUG", "message": "[2021-11-22 13:20:02,314] plugins_manager.py:414 - Integrate DAG plugins" }" }
{ "kind": "LOG", "level": "INFO", "message": "[2021-11-22 13:20:02,327] logging_mixin.py:104 - { "kind": "LOG", "level": "DEBUG", "message": "[2021-11-22 13:20:02,327] taskinstance.py:1742 - Updating task params ({'namespace': 'my.namespace', 'runtimeName': 'airflow-2-0', 'job': '60e5f71edd72863533db23fb', 'runtimeType': 'AIRFLOW', 'version': 1}) with DagRun.conf ({'executedBy': '60e5f71edd72863533db23fb'})" }" }
{ "kind": "LOG", "level": "INFO", "message": "[2021-11-22 13:20:02,327] logging_mixin.py:104 - { "kind": "LOG", "level": "INFO", "message": "[2021-11-22 13:20:02,327] functions.py:13 - on_success_callback_job - namespace: my.namespace, runtimeJobExecId: 944102, runId: scheduled__2021-11-22T13:15:00+00:00" }" }
```
The next one to be notified is scheduled__2021-11-22T13:25:00+00:00 but should have been scheduled__2021-11-22T13:20:00+00:00 so, this last one will be out of place.
```
{ "kind": "LOG", "level": "DEBUG", "message": "[2021-11-22 13:30:04,522] scheduler_job.py:557 - Processing Callback Request: {'full_filepath': '/home/user/airflow/dags/my.namespace/my.namespace_514.py', 'msg': 'success', 'dag_id': 'my.namespace_514', 'execution_date': datetime.datetime(2021, 11, 22, 13, 25, tzinfo=Timezone('UTC')), 'is_failure_callback': False}" }
{ "kind": "LOG", "level": "INFO", "message": "[2021-11-22 13:30:04,636] logging_mixin.py:104 - { "kind": "LOG", "level": "INFO", "message": "[2021-11-22 13:30:04,635] dag.py:868 - Executing dag callback function: <function on_success_callback_job at 0x7fe01d0c0a60>" }" }
{ "kind": "LOG", "level": "INFO", "message": "[2021-11-22 13:30:04,653] logging_mixin.py:104 - { "kind": "LOG", "level": "DEBUG", "message": "[2021-11-22 13:30:04,653] plugins_manager.py:264 - Plugins are already loaded. Skipping." }" }
{ "kind": "LOG", "level": "INFO", "message": "[2021-11-22 13:30:04,654] logging_mixin.py:104 - { "kind": "LOG", "level": "DEBUG", "message": "[2021-11-22 13:30:04,654] plugins_manager.py:414 - Integrate DAG plugins" }" }
{ "kind": "LOG", "level": "INFO", "message": "[2021-11-22 13:30:04,712] logging_mixin.py:104 - { "kind": "LOG", "level": "DEBUG", "message": "[2021-11-22 13:30:04,711] taskinstance.py:1742 - Updating task params ({'namespace': 'my.namespace', 'runtimeName': 'airflow-2-0', 'job': '60e5f71edd72863533db23fb', 'runtimeType': 'AIRFLOW', 'version': 1}) with DagRun.conf ({'executedBy': '60e5f71edd72863533db23fb'})" }" }
{ "kind": "LOG", "level": "INFO", "message": "[2021-11-22 13:30:04,712] logging_mixin.py:104 - { "kind": "LOG", "level": "INFO", "message": "[2021-11-22 13:30:04,712] functions.py:13 - on_success_callback_job - namespace: my.namespace, runtimeJobExecId: 944168, runId: scheduled__2021-11-22T13:25:00+00:00" }" }
```
Now it executes callbacks for scheduled__2021-11-22T13:20:00+00:00 and the next one that should be scheduled__2021-11-22T13:30:00+00:00 is skipped.
```
{ "kind": "LOG", "level": "DEBUG", "message": "[2021-11-22 13:30:05,485] scheduler_job.py:557 - Processing Callback Request: {'full_filepath': '/home/user/airflow/dags/my.namespace/my.namespace_514.py', 'msg': 'success', 'dag_id': 'my.namespace_514', 'execution_date': datetime.datetime(2021, 11, 22, 13, 20, tzinfo=Timezone('UTC')), 'is_failure_callback': False}" }
{ "kind": "LOG", "level": "INFO", "message": "[2021-11-22 13:30:05,575] logging_mixin.py:104 - { "kind": "LOG", "level": "INFO", "message": "[2021-11-22 13:30:05,575] dag.py:868 - Executing dag callback function: <function on_success_callback_job at 0x7f869fa01ee0>" }" }
{ "kind": "LOG", "level": "INFO", "message": "[2021-11-22 13:30:05,583] logging_mixin.py:104 - { "kind": "LOG", "level": "DEBUG", "message": "[2021-11-22 13:30:05,583] plugins_manager.py:264 - Plugins are already loaded. Skipping." }" }
{ "kind": "LOG", "level": "INFO", "message": "[2021-11-22 13:30:05,584] logging_mixin.py:104 - { "kind": "LOG", "level": "DEBUG", "message": "[2021-11-22 13:30:05,584] plugins_manager.py:414 - Integrate DAG plugins" }" }
{ "kind": "LOG", "level": "INFO", "message": "[2021-11-22 13:30:05,599] logging_mixin.py:104 - { "kind": "LOG", "level": "DEBUG", "message": "[2021-11-22 13:30:05,599] taskinstance.py:1742 - Updating task params ({'namespace': 'my.namespace', 'runtimeName': 'airflow-2-0', 'job': '60e5f71edd72863533db23fb', 'runtimeType': 'AIRFLOW', 'version': 1}) with DagRun.conf ({'executedBy': '60e5f71edd72863533db23fb'})" }" }
{ "kind": "LOG", "level": "INFO", "message": "[2021-11-22 13:30:05,600] logging_mixin.py:104 - { "kind": "LOG", "level": "INFO", "message": "[2021-11-22 13:30:05,600] functions.py:13 - on_success_callback_job - namespace: my.namespace, runtimeJobExecId: 944137, runId: scheduled__2021-11-22T13:20:00+00:00" }" }
```
Now should be executed scheduled__2021-11-22T13:30:00+00:00 but, as mentioned before, is skipped so the next callbacks scheduler invokes is scheduled__2021-11-22T13:35:00+00:00.
```
{ "kind": "LOG", "level": "INFO", "message": "[2021-11-22 13:40:02,419] logging_mixin.py:104 - { "kind": "LOG", "level": "INFO", "message": "[2021-11-22 13:40:02,419] functions.py:13 - on_success_callback_job - namespace: gl.ether.devops, runtimeJobExecId: 944218, runId: scheduled__2021-11-22T13:35:00+00:00" }" }
```
### What you expected to happen
I expect that even if there's some notification out of place, the scheduler doesn't skip that one, so my backend can keep the data updated and not corrupted.
### How to reproduce
You could make a dag with a dummy job scheduled every 5 mins. Sometimes, if there's some kind of delay invoking callbacks or something, the scheduler could skip to notify using callbacks defined.
### Anything else
This problem usually happens once a day.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] closed issue #19767: dag callbacks are missed sometimes
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed issue #19767:
URL: https://github.com/apache/airflow/issues/19767
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ghostbody edited a comment on issue #19767: dag callbacks are missed sometimes
Posted by GitBox <gi...@apache.org>.
ghostbody edited a comment on issue #19767:
URL: https://github.com/apache/airflow/issues/19767#issuecomment-989709988
I had a look at the airflow sheduler logs and found this:
```
./logs/dag_processor_manager/dag_processor_manager.log:[2021-12-08 22:15:52,278] {manager.py:1092} ERROR - Processor for some_dag.py with PID 250 started at 2021-12-08T14:15:01.525929+00:00 has timed out, killing it.
./logs/dag_processor_manager/dag_processor_manager.log:[2021-12-08 22:16:43,632] {manager.py:1092} ERROR - Processor for some_dag.py with PID 581 started at 2021-12-08T14:15:53.491981+00:00 has timed out, killing it.
./logs/dag_processor_manager/dag_processor_manager.log:[2021-12-08 22:17:35,850] {manager.py:1092} ERROR - Processor for some_dag.py with PID 898 started at 2021-12-08T14:16:45.152570+00:00 has timed out, killing it.
./logs/dag_processor_manager/dag_processor_manager.log:[2021-12-08 22:18:28,167] {manager.py:1092} ERROR - Processor for some_dag.py with PID 1114 started at 2021-12-08T14:17:37.449065+00:00 has timed out, killing it.
```
While dag callbacks are executed by the processor as the source code indicated(in the process_file function), this timeout error could lead to callbacks missed.
I set up this [config](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#dag-file-processor-timeout) (to 100 seconds or more) and this problem seems to be fixed.
I think it's a not a good idea to execute callback while file procesing. Maybe to have a specialized process to handle callback is better.
@jpradass
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jpradass commented on issue #19767: dag callbacks are missed sometimes
Posted by GitBox <gi...@apache.org>.
jpradass commented on issue #19767:
URL: https://github.com/apache/airflow/issues/19767#issuecomment-989749279
> I had a look at the airflow sheduler logs and found this:
>
> ```
> ./logs/dag_processor_manager/dag_processor_manager.log:[2021-12-08 22:15:52,278] {manager.py:1092} ERROR - Processor for some_dag.py with PID 250 started at 2021-12-08T14:15:01.525929+00:00 has timed out, killing it.
> ./logs/dag_processor_manager/dag_processor_manager.log:[2021-12-08 22:16:43,632] {manager.py:1092} ERROR - Processor for some_dag.py with PID 581 started at 2021-12-08T14:15:53.491981+00:00 has timed out, killing it.
> ./logs/dag_processor_manager/dag_processor_manager.log:[2021-12-08 22:17:35,850] {manager.py:1092} ERROR - Processor for some_dag.py with PID 898 started at 2021-12-08T14:16:45.152570+00:00 has timed out, killing it.
> ./logs/dag_processor_manager/dag_processor_manager.log:[2021-12-08 22:18:28,167] {manager.py:1092} ERROR - Processor for some_dag.py with PID 1114 started at 2021-12-08T14:17:37.449065+00:00 has timed out, killing it.
> ```
>
> While dag callbacks are executed by the processor as the source code indicated(in the process_file function), this timeout error could lead to callbacks missed.
>
> I set up this [config](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#dag-file-processor-timeout) (to 100 seconds or more) and this problem seems to be fixed.
>
> I think it's a not a good idea to execute callback while file procesing. Maybe, use a specialized process or the executor to handle callback is better. (for airflowers :) )
>
> @jpradass
I'll give it a try and check if that works for me too.
Thank you @ghostbody
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ghostbody edited a comment on issue #19767: dag callbacks are missed sometimes
Posted by GitBox <gi...@apache.org>.
ghostbody edited a comment on issue #19767:
URL: https://github.com/apache/airflow/issues/19767#issuecomment-989709988
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on issue #19767: dag callbacks are missed sometimes
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on issue #19767:
URL: https://github.com/apache/airflow/issues/19767#issuecomment-976476614
Can you test this in 2.2.2?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jpradass commented on issue #19767: dag callbacks are missed sometimes
Posted by GitBox <gi...@apache.org>.
jpradass commented on issue #19767:
URL: https://github.com/apache/airflow/issues/19767#issuecomment-993266512
@ghostbody I've tried setting up the config you told about at 200 seconds but I'm getting the same issue so it doesn't seem to fix the problem.
Do we have any news from Airflow?
Thank you both!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on issue #19767: dag callbacks are missed sometimes
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on issue #19767:
URL: https://github.com/apache/airflow/issues/19767#issuecomment-989735855
> Also, to make more logs for callbacks is rather necessary. 😢
Callback logs are in logs/scheduler/{date}/{dagfilename}.log
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ghostbody commented on issue #19767: dag callbacks are missed sometimes
Posted by GitBox <gi...@apache.org>.
ghostbody commented on issue #19767:
URL: https://github.com/apache/airflow/issues/19767#issuecomment-1019507616
We rewrite our dag logic and replace the callback by a task. (as for `on_failure_hook`, we use a task with `one_failed`). More Information: https://airflow.apache.org/docs/apache-airflow/stable/concepts/dags.html#trigger-rules
Callbacks are unreliable as we tought. There are three reasons:
1. Callback executions are not logged to dag or dag_run which is hard to locate.
2. Callbacks can not be retried even when they fail.
3. Sometimes, they are missed.
Hope this help you~ @jpradass
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #19767: dag callbacks are missed sometimes
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #19767:
URL: https://github.com/apache/airflow/issues/19767#issuecomment-1018983218
This issue has been closed because it has not received response from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ghostbody commented on issue #19767: dag callbacks are missed sometimes
Posted by GitBox <gi...@apache.org>.
ghostbody commented on issue #19767:
URL: https://github.com/apache/airflow/issues/19767#issuecomment-988795792
+1.
I upgrade airflow to 2.2.2 and the problem still exists.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ghostbody commented on issue #19767: dag callbacks are missed sometimes
Posted by GitBox <gi...@apache.org>.
ghostbody commented on issue #19767:
URL: https://github.com/apache/airflow/issues/19767#issuecomment-989713793
Also, to make more logs for callbacks is rather necessary. 😢
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #19767: dag callbacks are missed sometimes
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #19767:
URL: https://github.com/apache/airflow/issues/19767#issuecomment-976406079
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ghostbody edited a comment on issue #19767: dag callbacks are missed sometimes
Posted by GitBox <gi...@apache.org>.
ghostbody edited a comment on issue #19767:
URL: https://github.com/apache/airflow/issues/19767#issuecomment-989709988
I had a look at the airflow sheduler logs and found this:
```
./logs/dag_processor_manager/dag_processor_manager.log:[2021-12-08 22:15:52,278] {manager.py:1092} ERROR - Processor for some_dag.py with PID 250 started at 2021-12-08T14:15:01.525929+00:00 has timed out, killing it.
./logs/dag_processor_manager/dag_processor_manager.log:[2021-12-08 22:16:43,632] {manager.py:1092} ERROR - Processor for some_dag.py with PID 581 started at 2021-12-08T14:15:53.491981+00:00 has timed out, killing it.
./logs/dag_processor_manager/dag_processor_manager.log:[2021-12-08 22:17:35,850] {manager.py:1092} ERROR - Processor for some_dag.py with PID 898 started at 2021-12-08T14:16:45.152570+00:00 has timed out, killing it.
./logs/dag_processor_manager/dag_processor_manager.log:[2021-12-08 22:18:28,167] {manager.py:1092} ERROR - Processor for some_dag.py with PID 1114 started at 2021-12-08T14:17:37.449065+00:00 has timed out, killing it.
```
While dag callbacks are executed by the processor as the source code indicated(in the process_file function), this timeout error could lead to callbacks missed.
I set up this [config](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#dag-file-processor-timeout) (to 100 seconds or more) and this problem seems to be fixed.
I think it's a not a good idea to execute callback while file procesing. Maybe to have a specialized process to handle callback is better. (for airflowers :) )
@jpradass
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ghostbody commented on issue #19767: dag callbacks are missed sometimes
Posted by GitBox <gi...@apache.org>.
ghostbody commented on issue #19767:
URL: https://github.com/apache/airflow/issues/19767#issuecomment-989709988
I had a look at the airflow sheduler logs and found this:
```
./logs/dag_processor_manager/dag_processor_manager.log:[2021-12-08 22:15:52,278] {manager.py:1092} ERROR - Processor for some_dag.py with PID 250 started at 2021-12-08T14:15:01.525929+00:00 has timed out, killing it.
./logs/dag_processor_manager/dag_processor_manager.log:[2021-12-08 22:16:43,632] {manager.py:1092} ERROR - Processor for some_dag.py with PID 581 started at 2021-12-08T14:15:53.491981+00:00 has timed out, killing it.
./logs/dag_processor_manager/dag_processor_manager.log:[2021-12-08 22:17:35,850] {manager.py:1092} ERROR - Processor for some_dag.py with PID 898 started at 2021-12-08T14:16:45.152570+00:00 has timed out, killing it.
./logs/dag_processor_manager/dag_processor_manager.log:[2021-12-08 22:18:28,167] {manager.py:1092} ERROR - Processor for some_dag.py with PID 1114 started at 2021-12-08T14:17:37.449065+00:00 has timed out, killing it.
```
While dag callbacks are executed by the processor as the source code indicated(in the process_file function), this timeout error could lead to callbacks missed.
I set up this [config](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#dag-file-processor-timeout) (to 100 seconds or more) and this problem seems to be fixed.
I think it's a not a good idea to execute callback while file procesing. Maybe to have a specialized process to handle callback is better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #19767: dag callbacks are missed sometimes
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #19767:
URL: https://github.com/apache/airflow/issues/19767#issuecomment-1012629919
This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ghostbody commented on issue #19767: dag callbacks are missed sometimes
Posted by GitBox <gi...@apache.org>.
ghostbody commented on issue #19767:
URL: https://github.com/apache/airflow/issues/19767#issuecomment-989814563
>
There is no log when callback is missed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jpradass commented on issue #19767: dag callbacks are missed sometimes
Posted by GitBox <gi...@apache.org>.
jpradass commented on issue #19767:
URL: https://github.com/apache/airflow/issues/19767#issuecomment-976499702
Sure. I'll try with the new version and if it happens again I'll let you know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org