You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by John Thomas <jo...@astronomer.io.INVALID> on 2023/04/06 22:24:26 UTC
Apache Airflow Newsletter | March 2023
View this email in your browser
<https://mailchi.mp/bbe67aeb7e95/apache-airflow-newsletter-march-2023?e=ece21dcc4d>
Welcome to the latest edition of the Airflow newsletter! March was another
action-packed month in the Airflow Community. As a reminder, last month, we:
-
Announced that the Airflow Summit is back and will take place in Toronto
this coming September. The call for papers is still open, and we’re looking
for speakers, so get your sessions submitted by April 10th!
-
Elected two new PMC members, Pierre Jeambrun and Brent Bovenzi! We
appreciate their contributions to the Airflow community and look forward to
their continued support. Congratulations to you both! 🎉
-
Hit issue number 30,000
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=47d5949ded&e=ece21dcc4d>
in
the Airflow repository! It’s been a long journey, and I’m sure there are
many, many more to come.
As always, check the Airflow Meetups
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=b03ae52dc6&e=ece21dcc4d>
page
to find your local community, and feel free to reach out to me if you have
anything to include in the next month :)
Thanks for reading!
*Airflow Summit is back! *And we are looking for speakers who want to share
the exciting ways they are using, changing, and advancing Apache Airflow
Submit your session here! 👉 http://bit.ly/3y22fJM
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=1b4f420079&e=ece21dcc4d>
Get your tickets here! 👉 https://airflowsummit.org/tickets/
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=b6dfa50645&e=ece21dcc4d>
*Airflow*
On March 14, we released Airflow 2.5.2. This version closes a number of
longstanding bugs, includes some minor quality-of-life changes, such as
properly aligning our API datetime handling with RFC3339, and makes exposed
web server hostnames opt-in.
You can find the build artifacts using these links:
-
PyPI
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=5699ed2e7d&e=ece21dcc4d>
| Release Notes
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=f6dc8ba9b2&e=ece21dcc4d>
| Docs
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=f4c3f522a8&e=ece21dcc4d>
| Sources
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=591d49951a&e=ece21dcc4d>
| Docker
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=6d31c41ccd&e=ece21dcc4d>
*Providers*
This month, we updated 15 providers, including Amazon, Kubernetes, Flink,
SQL, Google, Github, and dbt.
*Ecosystem*
The Astronomer Registry
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=ecb7e83965&e=ece21dcc4d>,
a collection of Airflow providers, modules, and sample DAGs from public
Github repos, was recently overhauled and redesigned
The Astronomer providers versions 1.15.1 and 1.15.2 were released,
including the first versions of Astronomer’s Databricks provider
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=1d41a48e00&e=ece21dcc4d>
and async Fivetran provider
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=c27ccf11de&e=ece21dcc4d>,
and a beta version of the Universal Transfer provider.
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=92097143ad&e=ece21dcc4d>
-
April 4: How to test and debug your Airflow connections
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=211199abf5&e=ece21dcc4d>,
Webinar
-
April 4: Paris Apache Airflow Meetup
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=ab02eaeb56&e=ece21dcc4d>,
Paris Event
-
April 11: How to pass data between your Airflow tasks
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=d4d12a684e&e=ece21dcc4d>,
Webinar
-
April 19-27: Pycon US
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=d158237bab&e=ece21dcc4d>,
Salt Lake City Conference
-
April 23: TLV Apache Airflow Meetup, Spring ‘23
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=0ad6ec5501&e=ece21dcc4d>,
Tel Aviv Event
-
April 24: NYC Apache Airflow Meetup @ Etsy
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=13fa1b3137&e=ece21dcc4d>,
New York City Event
-
April 25: DC Apache Airflow Meetup @ Slalom
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=642715cce9&e=ece21dcc4d>,
Washington DC Event
- March 7: How to test your Airflow DAGs with dag.test()
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=083d89b160&e=ece21dcc4d>,
Webinar
- March 14: How to Automate and Scale Data Ingestion with Fivetran and
Airflow
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=ad1662a206&e=ece21dcc4d>,
Webinar
- March 21: How to Orchestrate Databricks Jobs Using Airflow
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=3ecdc95bca&e=ece21dcc4d>,
Webinar
- March 28: How to Orchestrate Machine Learning Workflows with Airflow
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=dec45d640d&e=ece21dcc4d>,
Webinar
- March 30: 4 Things To Consider When Deciding How to Run Airflow
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=b4d5a53722&e=ece21dcc4d>,
Webinar
@bbovenzi
#29413 Add a new graph inside of the grid view
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=9eb2c3c645&e=ece21dcc4d>
UI improvements are always exciting, and this PR, adding the graph to a tab
inside of the grid view, is more exciting than most! In addition to the
changes to the user flow, this PR also moves the graph off dagre-d3,
resolving a few outstanding issues.
-
“ELT with Airflow: a no-install click-and-play blueprint project running
in codespaces
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=35f6569431&e=ece21dcc4d>”
Tamara Fingerlin, Medium
-
“Passing Data Between Tasks with the KubernetesPodOperator in Apache
Airflow
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=c270356b6a&e=ece21dcc4d>”
Jeff Fletcher, Medium
- “Using Fivetran with Apache Airflow
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=b9826423dd&e=ece21dcc4d>”,
Astronomer, Documentation and Guides
-
Votes:
-
Release Airflow 2.5.3 from 2.5.3rc2 [accepted]
-
Release Airflow 2.5.3 from 2.5.3rc1 [canceled]
-
Release Airflow 2.5.2 from 2.5.2rc2 [accepted]
-
Release Airflow 2.5.2 from 2.5.2rc1 [canceled]
-
Airflow Providers - March 07, 2023 [accepted]
-
Airflow Providers - March 03, 2023 [accepted]
-
Discussed:
-
Describing (new) provider approval policies
-
formally describe process of what is needed to approve new
providers
-
AIP-55
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=d511288359&e=ece21dcc4d>
Rule-based
timetable with logical composition
-
Exclude some providers that hold us back from releasing
-
AIP-56
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=7e92362085&e=ece21dcc4d>
Extensible
user management
-
AIP-52
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=8d0238c600&e=ece21dcc4d>
updates
- setup / teardown tasks
-
Handling of external file processing and variable caching
-
Request for feedback on proposal for new OpenLineage provider in
Airflow [cont]
-
New SLA AIP
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=4bf1964bf9&e=ece21dcc4d>
-
request for feedback
-
New provider - Huawei Cloud
-
move of provider packages to isolated "providers" sub-folders
-
Allow shorter voting periods for subsequent RCs
-
Lazy Consensus
-
Introduce suspension process for providers due to dependencies
holding us back
-
Allow shorter voting periods for subsequent RCs
-
Unsubscribe links in users@ mailing list footers
-
Announced:
-
New PMC member: Pierre Jeambrun 🎉
-
New PMC member: Brent Bovenzi 🎉
-
Apache Airflow 2.5.2 Released
-
The issue / PR no. 30000 🎉
-
Google Summer of Code 2023 Mentor Registration
-
CVEs:
-
CVE-2023-25695
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=d2b063ce28&e=ece21dcc4d>
-
Generation of Error Message Containing Sensitive Information
-
Impacts Airflow versions before 2.5.2
-
*Airflow Helm Chart 1.9.0
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=ff938829a9&e=ece21dcc4d>*
-
*Airflow 2.6.0
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=5800e6e86d&e=ece21dcc4d>*
Please subscribe to this newsletter to ensure you always receive the latest
issue: http://eepurl.com/hXUA3r
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=f0ea2501fc&e=ece21dcc4d>
.
Know of an upcoming event or publication the community should know about?
Let me know at john.thomas@astronomer.io.