[D] Event listener for dag processing events [airflow]

["numice (via GitHub)" <gi...@apache.org>]

GitHub user numice created a discussion: Event listener for dag processing events

**Description**
I'd like to see if it's a good idea to have an event listener for Dag Processing events, for example, when a new or modified dag is loaded in the dag bag. Right now, I can see only events when a dag state changes or a task instance state changes but I don't see an event when a dag is loaded. 


**Use case/motivation**
In my use case, I'm planning to add some metadata to a dag file and read it when the dag is loaded and then run a small function like monkeypatching the dag depending on the metadata. 

There's a PEP [https://peps.python.org/pep-0723/](link) to one suggestion to embed a script inside a python file and I'd like to read this when a dag is processed and parsed.

So right now what I can think of is scanning the whole repo and parse the files, which could maybe be done by a running dag, but since Airflow is already processing new dags so I think it might be a better idea to have these event listeners and implementing event handlers.

GitHub link: https://github.com/apache/airflow/discussions/49372

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Event listeners for DAG processing events [airflow]

["numice (via GitHub)" <gi...@apache.org>]

GitHub user numice edited a discussion: Event listeners for DAG processing events

**Description**
I'd like to see if it's a good idea to have an event listener for Dag Processing events, for example, when a new or modified dag is loaded in the dag bag. Right now, I can see only events when a dag state changes or a task instance state changes but I don't see an event when a dag is loaded. 


**Use case/motivation**
In my use case, I'm planning to add some metadata to a dag file and read it when the dag is loaded and then run a small function like monkeypatching the dag depending on the metadata. 

There's a PEP [https://peps.python.org/pep-0723/](link) to one suggestion to embed a script inside a python file and I'd like to read this when a dag is processed and parsed.

So right now what I can think of is scanning the whole repo and parse the files, which could maybe be done by a running dag, but since Airflow is already processing new dags so I think it might be a better idea to have these event listeners and implementing event handlers.

GitHub link: https://github.com/apache/airflow/discussions/49372

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org