You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "avkirilishin (via GitHub)" <gi...@apache.org> on 2023/12/20 23:01:44 UTC
[PR] feat: Add queues for triggerers. [airflow]
avkirilishin opened a new pull request, #36338:
URL: https://github.com/apache/airflow/pull/36338
Closes: #33818
Pull Request for discussing the approach. If this approach is acceptable, I will add queue support for more triggers, add tests, update the documentation, and assess the database load under a large number of triggers.
You can check how triggered's queues work with the sample DAG:
```python
from datetime import time
import pendulum
from airflow import DAG
from airflow.operators.empty import EmptyOperator
from airflow.sensors.time_sensor import TimeSensorAsync
with DAG(
'trigger-dag',
schedule='@daily',
start_date=pendulum.DateTime(2023, 12, 19),
) as dag:
sensor = TimeSensorAsync(task_id="sync_sensor", target_time=time(10, 59))
sensor_default = TimeSensorAsync(
task_id="sync_sensor_default", target_time=time(23, 59), trigger_queue="default")
sensor_q1 = TimeSensorAsync(task_id="sync_sensor_q1", target_time=time(10, 59), trigger_queue="queue1")
dummy_op = EmptyOperator(task_id="dummy_op")
sensor >> dummy_op
sensor_default >> dummy_op
sensor_q1 >> dummy_op
```
And you can run the second triggerer with the following command:
```bash
airflow triggerer -q queue1
```
<img width="1344" alt="image" src="https://github.com/avkirilishin/airflow/assets/54231417/51506483-869b-43a4-8c73-fecd1655e978">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add queues for triggerers. [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36338:
URL: https://github.com/apache/airflow/pull/36338#issuecomment-1937925306
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add queues for triggerers. [airflow]
Posted by "avkirilishin (via GitHub)" <gi...@apache.org>.
avkirilishin closed pull request #36338: feat: Add queues for triggerers.
URL: https://github.com/apache/airflow/pull/36338
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org