You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "trikker (via GitHub)" <gi...@apache.org> on 2024/11/18 13:51:48 UTC
[D] Why three schedulers are all sending UPDATE dag SQL to backend meta db? [airflow]
GitHub user trikker created a discussion: Why three schedulers are all sending UPDATE dag SQL to backend meta db?
HI, I am deploying a production airflow (2.10.3) environment on CentOS7 and I want to make sure every component has HA(highly available), including:
- webserver
- scheduler
- celery worker
The webserver and celery worker are OK because they are not stateful, but after I started three schedulers on different machines, I found that three schedulers were all sending UPDATE dag SQL to backend meta db(which is MySQL 8.0):
I am sure all of the three schedulers sent the update SQL to db:
```
UPDATE dag SET next_dagrun='2024-11-18 13:35:00', next_dagrun_data_interval_start='2024-11-18 13:35:00', next_dagrun_data_interval_end='2024-11-18 13:36:00', next_dagrun_create_after='2024-11-18 13:36:00' WHERE dag.dag_id = 'mytest_dag'
```
Does it means that the three schedulers are working at the same time and will possibly make data in meta db wrong?
I was following this part of document in airflow, which says we can run more than one scheduler at the same time to get HA of scheduler by using the feature `SELECT * FROM ... FOR UPDATE NOWAIT` of DB as long as the meta DB supports the SQL.
https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/scheduler.html#running-more-than-one-scheduler
Thanks for help!
GitHub link: https://github.com/apache/airflow/discussions/44141
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Why three schedulers are all sending UPDATE dag SQL to backend meta db? [airflow]
Posted by "trikker (via GitHub)" <gi...@apache.org>.
GitHub user trikker added a comment to the discussion: Why three schedulers are all sending UPDATE dag SQL to backend meta db?
Thanks, I think parsing the files for multiple times by multiple schedulers is ok because it has nothing to do with scheduling. I just want to avoid that one dag is scheduled by multiple schedulers by multiple times at the same time. So how can I confirm it?
GitHub link: https://github.com/apache/airflow/discussions/44141#discussioncomment-11299478
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Why three schedulers are all sending UPDATE dag SQL to backend meta db? [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk added a comment to the discussion: Why three schedulers are all sending UPDATE dag SQL to backend meta db?
No. they might do the same "work" - i.e. parse the same files and update their serialized and code. This is built-in in airflow scheduler design - DagFileProcessor is not supposed to "avoid" parsing the same file by multiple processors. There are various ways you can control it:
* you can have standalone dag file processor
* those processors can work only on a specified folders (but two dag file processors will still be able to parse and update info about the same DAG files)
* https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#file-parsing-sort-mode "random_seeded_by_host" sorting order is implemented so that your multiple schedulers (or multiple dag file processors) are not processing the DAG files in the same order - but randomly - to avoid the possibility that your dag file processors constantly process the same files at the same time. That allows you to decrease the parsing frequency as effectively your multiple processors will be processing the same files more frequently - but in different order.
GitHub link: https://github.com/apache/airflow/discussions/44141#discussioncomment-11296503
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Why three schedulers are all sending UPDATE dag SQL to backend meta db? [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk added a comment to the discussion: Why three schedulers are all sending UPDATE dag SQL to backend meta db?
What confirmation do you need? You can watch this talk that explains how scheduler works for example https://www.youtube.com/watch?v=DYC4-xElccE
GitHub link: https://github.com/apache/airflow/discussions/44141#discussioncomment-11299970
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org