You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "NBardelot (via GitHub)" <gi...@apache.org> on 2024/08/01 11:55:46 UTC
[I] Scheduler unable to process large number of orphan Datasets [airflow]
NBardelot opened a new issue, #41185:
URL: https://github.com/apache/airflow/issues/41185
### Apache Airflow version
2.9.3
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
When it starts, the scheduler starts marking Datasets as orphans in `airflow/jobs/scheduler_job_runner.py` in the method `_orphan_unreferenced_datasets`.
A large number of Datasets is queried from the Metadata DB (~200K). It seems the query does not filter Datasets that are already orphans (it queries with HAVING both no DAG nor Task reference; thus selecting also Datasets already flagged as orphans).
As we use Kubernetes, we have a livenessProbe failing for the Scheduler since it helplessly tries to orphan everything in a single run (Note: AIRFLOW__SCHEDULER__PARSING_CLEANUP_INTERVAL is 60s, but changing the value would not change this behaviour).
A lot of `INFO - Orphaning unreferenced dataset` are logged in the Scheduler output.
Kubernetes kills the unhealthy pod. The pod starts anew, and again fails to orphan everything at once. Etc. Etc.
### What you think should happen instead?
Several thing could be done to improve the situation:
1. Limit the number of Datasets selected in the query in `_orphan_unreferenced_datasets`, so that the `_set_orphaned` works by batches of Datasets to orphan (new config `AIRFLOW__SCHEDULER__UNREFERENCED_DATASETS_BATCH_SIZE` for example)
2. Do not select already orphaned Datasets, by improving the query in `_orphan_unreferenced_datasets`
3. Add an optional TTL to Datasets in the Metadata DB so that old Datasets can be ignored by the query and/or cleaned up without the need to orphan them beforehand (if the TTL has passed, then orphan or not is not relevant and the computation can be skipped)
### How to reproduce
Create a very large number of Datasets, that the Scheduler cannot process in one round, faster than its livenessProbe, so that the livenessProbe is not processed and the pod is considered unhealthy.
### Operating System
Kubernetes
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon
apache-airflow-providers-common-sql
apache-airflow-providers-elasticsearch
apache-airflow-providers-hashicorp
apache-airflow-providers-http
apache-airflow-providers-microsoft-winrm
apache-airflow-providers-microsoft-azure
apache-airflow-providers-opsgenie
apache-airflow-providers-postgres
apache-airflow-providers-redis
apache-airflow-providers-sftp
apache-airflow-providers-smtp
apache-airflow-providers-ssh
with versions as of Airflow 2.9.3 constraints.
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
AIRFLOW__SCHEDULER__PARSING_CLEANUP_INTERVAL = 60s (default value)
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Scheduler unable to process large number of orphan Datasets [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on issue #41185:
URL: https://github.com/apache/airflow/issues/41185#issuecomment-3489736705
Should this be closed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Scheduler unable to process large number of orphan Datasets [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr closed issue #41185: Scheduler unable to process large number of orphan Datasets
URL: https://github.com/apache/airflow/issues/41185
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Scheduler unable to process large number of orphan Datasets [airflow]
Posted by "NBardelot (via GitHub)" <gi...@apache.org>.
NBardelot commented on issue #41185:
URL: https://github.com/apache/airflow/issues/41185#issuecomment-2263378178
We found the root cause of the 200k Datasets.
We have a guidline for developers so that they do not put code in the DAG global scope, but here a developer was computing a date which was used in the Jinja template of the Dataset (so 2 bad practices colliding).
That being said, the proposals in the issue could still be useful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Scheduler unable to process large number of orphan Datasets [airflow]
Posted by "NBardelot (via GitHub)" <gi...@apache.org>.
NBardelot commented on issue #41185:
URL: https://github.com/apache/airflow/issues/41185#issuecomment-2295994492
See https://github.com/apache/airflow/pull/40806.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org