You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "pawansaha555 (via GitHub)" <gi...@apache.org> on 2025/07/16 16:43:47 UTC
[D] How to Speed Up Kafka Message Processing in Airflow Listener DAG Using AwaitMessageTriggerFunctionSensor? [airflow]
GitHub user pawansaha555 created a discussion: How to Speed Up Kafka Message Processing in Airflow Listener DAG Using AwaitMessageTriggerFunctionSensor?
**Hi Airflow Community,**
I’m working on a Kafka-based event ingestion pipeline using Airflow 2.7+ and the [AwaitMessageTriggerFunctionSensor](https://www.astronomer.io/docs/learn/airflow-kafka/) with the LocalExecutor.
**Setup:**
Kafka Topic is shared across multiple applications that publish large volumes of events.
A Listener DAG listens to this Kafka topic using AwaitMessageTriggerFunctionSensor, filters relevant events based on app name and selection state, and then uses TriggerDagRunOperator to launch a downstream DAG for document ingestion.
I’ve set NUM_PARALLEL_SENSORS = 50 with poll_interval=0.1 and poll_timeout=0.5.
**Problem:**
Even with parallel sensors, my specific application's event takes a long time to get picked up, due to the high volume of irrelevant events in the queue.
Listener DAG uses AwaitMessageTriggerFunctionSensor and TriggerDagRunOperator.
**What I Want to Know:**
1. Is there a way to prioritize or speed up filtering for relevant messages?
2. Are there any Kafka configuration tweaks or Airflow best practices (e.g., partitioning topics, optimizing sensors) that can help in such scenarios?
**Please find the codes for Listener DAG and Downstream DAG Attached for your reference.**
Any help or tuning suggestions to reduce latency in message detection and downstream DAG triggering would be appreciated. Thanks in advance!
[downstreamDAG.txt](https://github.com/user-attachments/files/21262359/downstreamDAG.txt)
[listenerDAG.txt](https://github.com/user-attachments/files/21262360/listenerDAG.txt)
`
GitHub link: https://github.com/apache/airflow/discussions/53426
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org