You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "jscheffl (via GitHub)" <gi...@apache.org> on 2025/06/29 17:05:15 UTC
[I] Load example Dags from standard provider [airflow]
jscheffl opened a new issue, #52469:
URL: https://github.com/apache/airflow/issues/52469
### Description
As described in https://cwiki.apache.org/confluence/display/AIRFLOW/Examples+Refurbish
In main (currently) a hacky mode is implemented do load example dags from standard provider as well as airflow-core. A proper way of loading from standard provider is to be implemented as feature
### Use case/motivation
Airflow Examples have been grown in number and focus over the past years. They purpose multiple things:
Serve as tutorials to learn Airflow DAG implementation
Serve with code snippets for documentation
Serve for testing the setup
(some) service for CI integration testing
Some example DAGs are in a good quality, some are not following best practices. Current examples do not follow a structure.
There are example DAGs contained in the Airflow core (currently pushed to standard provider/example_dags) as well as there are more examples in other providers. But examples from other providers are lot loaded automatically.
So in the [Airflow 3 Dev Calls](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-22May2025) there was a demand named to clean-up and optimize example DAGs.
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Load example Dags from standard provider [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on issue #52469:
URL: https://github.com/apache/airflow/issues/52469#issuecomment-3219412173
Per [Slack thread](https://apache-airflow.slack.com/archives/C07813CNKA8/p1755500006993699), targeting “example DAGs” won’t be possible in 3.1, so we’re moving this to Airflow 3.1+
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Load example Dags from standard provider [airflow]
Posted by "bhavaniravi (via GitHub)" <gi...@apache.org>.
bhavaniravi commented on issue #52469:
URL: https://github.com/apache/airflow/issues/52469#issuecomment-3448250625
@jscheffl When you say `is to be implemented as feature,` do you mean something like
1. `--load-provider-example standard,aws,slack`
or
2. When we set `--load-examples` to look through all providers available and load them. So Breeze will load all examples, whereas a user environment will only load what's installed
I started implementing (2) DagBundle in `_add_example_dag_bundle`. One thing I've noticed is that the majority of the test functions still use `Dagbag` to load examples.
Now I'm again at the crossroads: Should we load both dagbag `collect_dags` and Dagbundle, and which is used when?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Load example Dags from standard provider [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on issue #52469:
URL: https://github.com/apache/airflow/issues/52469#issuecomment-3448517589
> When you say is to be implemented as feature, do you mean something like...
The current breeze CLI with `--load-example-dags` just globally loads examples. I would keep it simple at first and with this flag load all examples from all providers (and assuming core examples are not existing in core in the future but all will be hosted in standard provider). Then the examples automatically vary depending on which providers are installed. If then in future people think this is overwhelming, then a filter switch can be added later as increment as well.
> Now I'm again at the crossroads: Should we load both dagbag collect_dags and Dagbundle, and which is used when?
That is a very good question. I have no strong opinion but using the Bundle mechansim looks like a superior way compared to loading examples from a hard-wired folder. And if we want to load from multiple folders the way of defining a Bundle per provider looks like a good approach. I'd also favor an opinion from @jedcunningham but he is on leave atm.
Is Dagbag noch indirectly loading the bundles as well? I am not too much into this detail but might be a good chance to clean-this up on a general level as tests should use the "normal" mechanisms. Maybe a fixture is needed to load the corresponsing test Dags into the testing environment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Load example Dags from standard provider [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #52469: Load example Dags from standard provider
URL: https://github.com/apache/airflow/issues/52469
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org