You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@airflow.apache.org by John Thomas via users <us...@airflow.apache.org> on 2022/11/02 19:29:46 UTC
Apache Airflow Newsletter | October 2022
October Monthly Digest
View this email in your browser
<https://mailchi.mp/79d21270332b/apache-airflow-newsletter-october-9295402?e=ece21dcc4d>
Another month passed, and Airflow continues to grow and improve! We had two
core releases 2.4.1 and 2.4.2 covering a wide swath of bug fixes and some
code cleanup.
Going into this winter, events are speeding up as well, with 3 conferences,
a meetup, and more. Find the details in the “Upcoming Events” section
below.
Also coming up, we’ll be looking to give this newsletter a facelift, with a
new name and some cleaner design. Look for a vote on the dev list soon!
Thanks for reading! See you next month
On October 24, we released Apache Airflow 2.4.2! 🎉
📦 PyPI: https://pypi.org/project/apache-airflow/2.4.2/
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=b5c5191150&e=ece21dcc4d>
📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.4.2
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=7db20ba21a&e=ece21dcc4d>
🛠️ Release Notes:
https://airflow.apache.org/docs/apache-airflow/2.4.2/release_notes.html
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=b92f356a01&e=ece21dcc4d>
🪶 Sources:
https://airflow.apache.org/docs/apache-airflow/2.4.2/installation/installing-from-sources.html
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=415957ed4a&e=ece21dcc4d>
On October 14, we also released Apache Airflow Helm chart 1.7.0.
📦 ArtifactHub: https://artifacthub.io/packages/helm/apache-airflow/airflow
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=7063ca7a45&e=ece21dcc4d>
📚 Docs: https://airflow.apache.org/docs/helm-chart/1.7.0/
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=2d3a60150b&e=ece21dcc4d>
🛠️ Release Notes:
https://airflow.apache.org/docs/helm-chart/1.7.0/release_notes.html
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=232a69c866&e=ece21dcc4d>
🪶 Sources:
https://airflow.apache.org/docs/helm-chart/1.7.0/installing-helm-chart-from-sources.html
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=06165130ac&e=ece21dcc4d>
Become the last #ApacheAirflow
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=324acfa286&e=ece21dcc4d>
speaker of 2022! Register here to present your awesome Airflow use case or
demo: http://bit.ly/3lhp1aR
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=338db7c499&e=ece21dcc4d>
& win some amazing swag!
Here are some upcoming Airflow-related events:
-
November 1: Batch Inference with Airflow and SageMaker webinar
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=853430f2f0&e=ece21dcc4d>,
Ganapathi Krishnamoorthi - Senior ML Solutions Architect at Amazon and
Faisal Hoda, Enterprise Architect at Astronomer
-
November 2, 9, and 16 or December 1: Data Engineering From Notebook to
Production
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=0471c3fa87&e=ece21dcc4d>
course
-
November 7: London Apache Airflow Meetup
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=fd7242d514&e=ece21dcc4d>
-
November 10: Trino Summit
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=cc9e5eb1ce&e=ece21dcc4d>
including Using Trino with Apache Airflow for (almost) all your data
problems with Philippe Gagnon
-
November 14-15, 16. International Conference on Semantic Technologies
for Big Data Integration and Analysis
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=5a772bf2c1&e=ece21dcc4d>,
Paris event
-
November 22-23, Sustainability Reporting & Communications Europe 2022
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=848b749a8a&e=ece21dcc4d>,
London conference
-
December 6, Astro Days
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=8b2bbf18c9&e=ece21dcc4d>,
Chicago event
If you missed any of these, you can catch a recap, watch on-demand, or dig
into the slides:
-
October 3-6: ApacheCon North America
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=3e1ea63e12&e=ece21dcc4d>,
New Orleans, LA - available slides
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=4b260ece02&e=ece21dcc4d>
-
October 4: What’s New in Airflow 2.4
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=818705f825&e=ece21dcc4d>
with Kenten Danas and Viraj Parekh
-
October 11: Data Driven Scheduling
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=2dcb065c70&e=ece21dcc4d>
with Kenten Danas and Viraj Parekhmediu
-
October 19: Data Lake Community Meetup- Apache Iceberg: An architectural
look under the covers
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=05d3e407f7&e=ece21dcc4d>
with Dipankar Mazumdar
This month we had a tie between two wholly diserving PRs!
*@kazanzhy
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=c809eedf01&e=ece21dcc4d>*
#25717 Add SQLExecuteQueryOperator
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=e3ba829e2e&e=ece21dcc4d>
Adds a catch-all operator to execute SQL queries against diverse databases
*@ephraimbuddy
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=96aedd80e1&e=ece21dcc4d>*
#27297 Fix IntegrityError during webserver startup
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=aba735b819&e=ece21dcc4d>
A fix to preload the webserver app before forking workers, fixing the
workers throwing IntegrityError
As Always, tag a PR with #protm to nominate it
-
How to keep Data Quality in check with Airflow
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=677be4c1be&e=ece21dcc4d>,
Tamara Fingerlin, Astronomer
-
Writing a good Airflow DAG - Part 1
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=3a2870a2b8&e=ece21dcc4d>
and Part 2
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=d077441a1b&e=ece21dcc4d>,
Najma Bader, Medium
-
Web scraping automation using Airflow
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=2aa4fa6437&e=ece21dcc4d>,
Muhammad Fadlil Ismail, Medium
-
Introducing Airflow SLA Miss Report
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=007d8c81a5&e=ece21dcc4d>,
Nikhil Manjunatha, Clairvoyant Blog
-
Tracking the War in Ukraine: A Data Engineering Project
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=db313747eb&e=ece21dcc4d>,
Nate DiRenzo, Python in Plain English
-
Airflow is on the Cloud | ELT Pipeline Orchestration With Airflow & AWS
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=0635893849&e=ece21dcc4d>,
Kaan Boke, Towards AI
-
Orchestration, or how to become a data-driven company
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=4df10f1d16&e=ece21dcc4d>,
Steven Hillion, Astronomer
-
Using Google Trends with Airflow
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=5444710fa5&e=ece21dcc4d>,
Davide Gazzè, Medium
-
AWS Managed Workflows for Apache Airflow vs. Glue
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=902adcab66&e=ece21dcc4d>,
Minseok Song, Towards Data Science
-
Using XCOMs with Airflow - scenario based examples with code
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=19e8f519f6&e=ece21dcc4d>,
Suman Kumar Gangopadhyay, Medium
-
Micropipelines: A Microservice Approach for DAG Authoring in Apache
Airflow
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=7730fe29ef&e=ece21dcc4d>,
Vikram Koka, Astronomer
-
Votes:
-
Release Apache Airflow Providers prepared on October 4 [approved]
-
Release Apache Airflow Helm Chart 1.7.0 based on 1.7.0rc1 [approved]
-
Release Airflow 2.4.2 from 2.4.2rc1 [approved]
-
Discussed:
-
Creating system to handle webserver_config.py in non-webserver pods
-
Proposed:
-
Clarifications of triage team role including strengthening importance
of active triaging
-
Split tracking for DAG SLAs off from tracking of queue to ensure
timely SLA pushes
-
Announced:
-
Apache Airflow 2.4.2 Released
-
Apache Airflow Helm Chart version 1.7.0 Released
-
Breeze rewrite to Python completed!
-
Apache Airflow 2.4.1 Released
-
Airflow 2.4.3
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=8bf5eeb1b4&e=ece21dcc4d>
-
Focused on Bugfixes
-
Airflow 2.5.0
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=b61e1a6fc2&e=ece21dcc4d>
-
Focused on new features
-
Airflow Helm Chart 1.8.0
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=e3b88fd3f5&e=ece21dcc4d>
-
For more info about upcoming releases and in-development projects, visit
the current projects’ issue (#10176
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=361e04fd3b&e=ece21dcc4d>
).
Please subscribe to this newsletter to ensure you always receive the latest
issue: http://eepurl.com/hXUA3r
<https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=455ce4a1fb&e=ece21dcc4d>
.
Know of an upcoming event or publication the community should know about?
Let us know at john.thomas@astronomer.io or julia@astronomer.io.