You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Briana Okyere <br...@astronomer.io.INVALID> on 2024/01/08 16:57:37 UTC
Apache Airflow Newsletter | December 2023
IF YOU WOULD LIKE TO SEE THE FORMATTED VERSION OF THE NEWSLETTER, PLEASE GO
HERE <https://www.astronomer.io/community-newsletter/december-2023/>
Hey Team,
Happy New Year! This community accomplished so much in the month of
December, and I’m excited to be sharing it all with you now! From a new
release to record breaking survey results, we closed out 2023 with the same
vivacity we entered it with. Highlights from the month of December include:
- The release of Airflow 2.8 <https://airflow.apache.org/blog/airflow-2.8.0/
>
- The launch of the 2023 Survey Results <https://airflow.apache.org/survey/>
- Announcing Airflow Summit in San Francisco Sept. 10-12th, 2024! <
https://airflowsummit.org/>
As always, please reach out with items for the next issue to: <
briana.okyere@astronomer.io>
Thanks for reading!
AIRFLOW SUMMIT 2024
SEPTEMBER 10TH-12TH, SAN FRANCISCO
It’s official folks! We’re hosting Airflow Summit 2024 on September 10-12th
in San Francisco, CA! Sign up for updates on the conference, including
access to early bird tickets here: <https://airflowsummit.org/>
THE AIRFLOW 2023 USER SURVEY RESULTS ARE IN!
The results are IN for the Airflow 2023 User Survey! This year, we had a
record breaking number of respondents, and are thankful to each and every
person that took the time to fill out the survey. Please view the full
Infographic here! <https://airflow.apache.org/survey/>
AIRFLOW 2.8.0
Airflow 2.8 was released on December 18th and includes a host of
significant enhancements and new features that will greatly benefit our
community.
- PyPI <https://pypi.org/project/apache-airflow/>
- Release Notes <https://pypi.org/project/apache-airflow/>
- Docs <https://airflow.apache.org/blog/airflow-2.8.0/>
- Sources <
https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-sources.html
>
- Docker <https://hub.docker.com/r/apache/airflow/tags/?page=1&name=2.8.0>
- Constraints <https://github.com/apache/airflow/tree/constraints-2.8.0>
PROVIDERS
New versions of 59 Airflow Providers packages were released.
For all source releases, PyPI packages and docs, see:
- Sources <
https://airflow.apache.org/docs/apache-airflow-providers/installing-from-sources
>
- PyPI <
https://airflow.apache.org/docs/apache-airflow-providers/installing-from-pypi
>
- Docs <https://airflow.apache.org/docs/>
Ecosystem
Initial releases:
Astronomer Cosmos 1.3.0 <https://pypi.org/project/astronomer-cosmos/>
OpenLineage 1.7.0 <
https://github.com/OpenLineage/OpenLineage/releases/tag/1.7.0>
UPCOMING EVENTS
1/11 Webinar: Airflow Town Hall <
https://astronomer.zoom.us/meeting/register/tZUpdOusrTItG90cw69WtvO2tOKfZbxxbQqH#/registration
>
1/11 Webinar: What’s new in Airflow 2.8 <
https://www.astronomer.io/events/webinars/whats-new-in-airflow-2-8/>
1/18 Webinar: How to manage connections in Airflow <
https://www.astronomer.io/events/webinars/how-to-manage-connections-in-airflow/
>
1/31/24 Webinar: Run MLOps production pipelines with Astronomer and
Snowpark <
https://www.snowflake.com/webinar/virtual-hands-on-labs/snowpark-day-where-python-developers-can-learn-the-latest-innovations-for-ai-ml-workflows-in-snowflake-2024-01-31/?utm_source=astronomer&utm_medium=partner&utm_campaign=na--en-customers&utm_content=-wb-snowpark-day-2024-01-31
>
2/3-4 Conference in Brussels: Data workflows: translating dbt to Apache
Airflow <
https://fosdem.org/2024/schedule/event/fosdem-2024-1651-data-workflows-translating-dbt-to-apache-airflow/>
& `New` Workflow Orchestrator in town: "Apache Airflow 2.x" <
https://fosdem.org/2024/schedule/event/fosdem-2024-1652--new-workflow-orchestrator-in-town-apache-airflow-2-x-/
>
2/9 Meetup in São Paulo: First São Paulo Airflow Meetup at iFood! <
https://www.meetup.com/sao-paulo-apache-airflow-meetup-group/events/298283822/
>
2/13 Meetup in Bangalore: Bangalore Airflow Meetup @ Cloudera <
https://www.meetup.com/bangalore-apache-airflow-meetup/events/298355258/>
RECENT EVENTS
12/7 Webinar: What’s New in the Astro Platform Release, Q4 2023 <
https://www.astronomer.io/events/webinars/whats-new-in-the-astro-platform-release-q4-2023/
>
12/7 Meetup in Prague: First Airflow Prague Meetup with Kiwi.com &
Astronomer <
https://www.meetup.com/prague-airflow-meetup-group/events/296837456/>
12/12 Meetup in Hyderabad: Hyderabad Airflow Meetup @ Microsoft HQ <
https://www.meetup.com/hyderabad-apache-airflow-meetup-group/events/297490529/
>
12/14 Webinar: Orchestrate next-generation AI tools: Explore six new
Airflow providers <
https://www.astronomer.io/events/webinars/orchestrate-next-generation-ai-tools-explore-six-new-airflow-providers/
>
PR OF THE MONTH
PR #35719
Add XCom tab to Grid <https://github.com/apache/airflow/pull/35719>
By @huyduong <https://github.com/hduong-mwam>
The December 2023 PR of the Month comes in amidst a medley of relevant PR’s
for the Airflow community. However, in the words of Committer Amogh Desai,
“Having a nice UI fix come in, that too from a first time contributor,
makes it a winner for me!”
COMMUNICATIONS DIGEST
Airflow Updates and Releases
Apache Airflow 2.8.0 is here by Ephraim Anierobi on the Airflow Blog <
https://airflow.apache.org/blog/airflow-2.8.0/>
Configuring the KubernetesExecutor to Hum at Etsy by Victor Chiapaikeo on
Airflow Medium <
https://medium.com/apache-airflow/configuring-the-kubernetesexecutor-to-hum-at-etsy-94c2924e9526
>
Airflow 2.8 Release: Meet the Contributors on Youtube <
https://youtu.be/Ubh-4i1pxX0?feature=shared>
Unraveling the Code: Navigating a CI/Release Security Vulnerability in
Apache Airflow by Jarek Potiuk on Airflow Medium <
https://medium.com/apache-airflow/unraveling-the-code-navigating-a-ci-release-security-vulnerability-in-apache-airflow-620214a96297
>
Introducing Airflow 2.8 by Kenten Danas on the Astronomer Blog <
https://www.astronomer.io/blog/introducing-airflow-2-8/>
Airflow in Practice and Use Cases
Apache Airflow at Adyen: Our journey and challenges to achieve reliability
at scale by Jorrick Sleijster and Natasha Shroff on Data Engineering Weekly
<
https://www.dataengineeringweekly.com/i/139860695/adyen-apache-airflow-at-adyen-our-journey-and-challenges-to-achieve-reliability-at-scale
>
How to share code across Airflow projects on Astronomer Learn <
https://docs.astronomer.io/learn/sharing-code-multiple-projects>
Airflow in action: providing environmental forecast services by Eloi
Codina-Torras on Airflow Medium <
https://medium.com/apache-airflow/airflow-in-action-providing-environmental-forecast-services-1a5a86110864
>
Use Airflow object storage to interact with cloud storage in an ML pipeline
on Astronomer Learn <
https://docs.astronomer.io/learn/airflow-object-storage-tutorial>
Leveraging Customization in Airflow with Databricks by Hanish Bansal on
Airflow Medium <
https://medium.com/apache-airflow/leveraging-customization-in-airflow-with-databricks-d263fde9a24d
>
LIVE with Astronomer: How Chartboost Uses Airflow Plugins with George Yates
on Youtube <https://www.youtube.com/watch?v=10-XCFKnmM4>
Deploy Rollbacks: Upgrade Airflow and Deploy DAGs with Confidence by David
Koenitzer on the Astronomer Blog <
https://www.astronomer.io/blog/deploy-rollbacks-upgrade-airflow-and-deploy-dags-with-confidence/
>
Use a listener to send a Slack notification when a dataset is updated on
Astronomer Learn <https://docs.astronomer.io/learn/airflow-listeners>
LIVE with Astronomer: DAG Serialization Deep Dive with Utkarsh! by George
Yates on Youtube <https://www.youtube.com/watch?v=yiGclH8RMzs>
Airflow Innovations
Introducing Apache Airflow™ on Astro – an Azure Native ISV Service by Conor
Beverland on the Astronomer Blog <
https://www.astronomer.io/blog/introducing-apache-airflow-on-astro-an-azure-native-isv-service/
>
Accelerating ML Application Development: Production-Ready Airflow
Integrations with Critical AI Tools by Steven Hillion on the Astronomer
Blog <
https://www.astronomer.io/blog/accelerating-ml-app-development-with-ai-airflow-integrations/
>
Enhanced Authentication Security to your Data Services on Azure with Astro
by Piotr Chomiak on the Astronomer Blog <
https://www.astronomer.io/blog/enhanced-authentication-security-to-your-data-services-on-azure-with-astro/
>
Exploring Airflow’s XComs: Unveiling the Power of Shared State by Amitabh
Roy Choudhary on Airflow Medium <
https://medium.com/apache-airflow/exploring-airflows-xcoms-unveiling-the-power-of-shared-state-a71270d824ba
>
Introducing the Astro Platform Release, Q4 2023 by Juliana O’Donohue on the
Astronomer Blog <
https://www.astronomer.io/blog/introducing-the-astro-platform-release-q4-2023/
>
DEVLIST DIGEST
Votes:
Airflow Providers prepared on December 08, 2023 <
https://lists.apache.org/thread/coy1t0dhtmpq1xyr7zr06ccmdfr70pkk>
[RESULT][VOTE] Airflow Providers - release of December 08, 2023 <
https://lists.apache.org/thread/cpbmng8z9wbrxgxgnyyr669xsosrtcsh>
Remove 'apache.scoop' provider <
https://lists.apache.org/thread/wnov3n2m2v0blnxl3q413mj8vphdrz4b>
Remove 'plexus' provider <
https://lists.apache.org/thread/6so1qh8kf088spd2y4nl0o5tpbyn0x3b>
Release Airflow 2.8.0 from 2.8.0rc1 <
https://lists.apache.org/thread/x5wt2z3ddvxpo1lkzotfbo94phncbs4n>
Release Airflow 2.8.0 from 2.8.0rc2 <
https://lists.apache.org/thread/3c8st8kyq2wbtbmsrh38sv1b34sy3w4y>
Release Airflow 2.8.0 from 2.8.0rc3 <
https://lists.apache.org/thread/7mxlh0m3v7xq717yobog0023djtpvbk2>
Airflow Providers prepared on December 12, 2023 <
https://lists.apache.org/thread/1r9w8mxw7165x29d118brgvpz3161zqw>
[RESULT][VOTE] Airflow Providers - release of 12 December 2023 <
https://lists.apache.org/thread/ll0pf7lmr0sfk4zgoj9y47ot5gy35hcm>
Release Airflow 2.8.0 from 2.8.0rc4 <
https://lists.apache.org/thread/x8tzt1k3rk63wdb71dqnbxvnkrhoc3nw>
[RESULT][VOTE] Release Airflow 2.8.0 from 2.8.0rc4 <
https://lists.apache.org/thread/ll0pf7lmr0sfk4zgoj9y47ot5gy35hcm>
Switch 2.8.0 images to use MariaDB client <
https://lists.apache.org/thread/rxbyxg11jg7y35k8om0f8wgb2l9h459l>
Making Listeners API non-experimental <
https://lists.apache.org/thread/8hr9bnfscgtf0ng3g44z13tthgk33oz4>
Release Airflow Python Client 2.8.0 from 2.8.0rc1 <
https://lists.apache.org/thread/pg9xod30s3prvk0ctp3gd220hmtl4r1o>
[RESULT][VOTE] Release Airflow Python Client 2.8.0 from 2.8.0rc1 <
https://lists.apache.org/thread/bv69z5qpjf2j3dz7fj4cdj2c3bfb04sl>
Airflow Providers prepared on December 23, 2023 <
https://lists.apache.org/thread/mrkqrsl7d4l482dr935wlozbgcfw56s1>
[RESULT][VOTE] Airflow Providers - release of 23rd of December 2023 <
https://lists.apache.org/thread/pctjv209k2gpro53dthytsh9xjqxbqmw>
Airflow Providers prepared on December 28, 2023 <
https://lists.apache.org/thread/cs6mcvpn2lk9w2p4oz43t20z3fg5nl7l>
[RESULT][VOTE] Airflow Providers - release of 28th December 2023 <
https://lists.apache.org/thread/brvbl5ndjcmfzfzcv6llrx0dvn186qo5>
Airflow Providers prepared on December 31, 2023 <
https://lists.apache.org/thread/gzfy6ccwr5hj2sp4bnzm03xqto9gk4y4>
Discussed:
Serialization in Apache Airflow: Clarifications and Future Directions <
https://lists.apache.org/thread/28r9x52cc51tcfxc5c6n3vh9frgp287x>
Helm Chart Version 1.9.0 Unavailable <
https://lists.apache.org/thread/kgp308ws0vo324o8070mtlc9m7o9qsq0>
Common.sql interface / API <
https://lists.apache.org/thread/pxt1tq6cmb69sncz1q4dp4z2o9zms7of>
Allowlist in serialization [was: Serialization in Apache Airflow] <
https://lists.apache.org/thread/1s6xpqcbdb3wrgtjv8op5zzkbsrsoc9f>
Capturing Architectural decisions (ADRS?) <
https://lists.apache.org/thread/kzz6w8gy9b1mxrqrbw1r5fkhk39qqbgc>
Using serde for PyODBC case/other cases [was Serialization in Apache
Airflow] <https://lists.apache.org/thread/z4sh6v9bnxhpgggxos8d0rc705tpz1k5>
Add BigQuery partition sensor for sequential (or just list of) partition <
https://lists.apache.org/thread/x3y79b2jq5g1tgm60wkf89r9zdly315l>
MappedArgument Object is not Subscriptable <
https://lists.apache.org/thread/g1jf7h04r31pphx3z48r5c6rz7s2mmz0>
Do the resources on the Community landing page need
reordering/reprioritizing? <
https://lists.apache.org/thread/22vnlrc039hk7bmflv7ym665y5os5x5w>
New Airflow Community Provider: Teradata <
https://lists.apache.org/thread/xj5v9gkmlr3gr20wtozh34l85n5g92fc>
"Require conversation resolution" in our PRs before merge? <
https://lists.apache.org/thread/hxrp5th37mn6g4q78grc1vrg1lx1srtb>
Looking for co-mentor in Major League Hacking Spring 2024 Fellows <
https://lists.apache.org/thread/0d0xt9fo9dt1m3rhfn1gm4oh3cfjq5ky>
Interested in joining/leaving the triage team? <
https://lists.apache.org/thread/37b2r9vjhs1ssrb4g2hm6d009c8jzp5r>
Enabling `pre-commit.ci` application for Airflow <
https://lists.apache.org/thread/fgoqj23hpl9mlo1n6gfkgvzjxvhfchzm>
Airflow vs Temporal <
https://lists.apache.org/thread/gq547w4ff5h1r8o0bq7cbfrf13116z7t>
Announced:
New committer: Utkarsh Sharma <
https://lists.apache.org/thread/p4b0jwsxofr2xsrnp81c45hcyp3wfg2t>
Apache Airflow Providers prepared on December 08, 2023 are released <
https://lists.apache.org/thread/lcooxhk1rhyysg1j9svx5hn9fb03xthw>
Apache Airflow Providers prepared on 12 December 2023 are released <
https://lists.apache.org/thread/41g5po1y6y7m6f9g7y1wsqdtlbofhw9d>
Apache Airflow 2.8.0 Released <
https://lists.apache.org/thread/pj6nolktscoltwd8wxn496wgc61opxyh>
Airflow Virtual Town Hall- Jan. 11th <
https://lists.apache.org/thread/c5vjk6ry9qqkvj60q089oc304vsjm6w5>
Save the date for Airflow Summit 2024! <
https://lists.apache.org/thread/jddytk5xvqlo9x5185rmtyswtpyfwzvn>
Airflow Survey is Closed! <
https://lists.apache.org/thread/33nsdboj9oxoyq6jb4ofzblq25v1rhot>
Meet our keynote speakers and register to Community Over Code EU! <
https://lists.apache.org/thread/zjktxfdtb97j48wypd8xtlm5x5rm5jjv>
Apache Airflow Python Client 2.8.0 Released <
https://lists.apache.org/thread/2s9glcwgsbn00ggmckdn5ynjkvrhh1jd>
Apache Airflow Providers prepared on 23rd December 2023 are released <
https://lists.apache.org/thread/b372nwq6q2z2bxjbsjjrry43ghcvc9zo>
Starting experimenting with "Require conversation resolution" setting <
https://lists.apache.org/thread/bzg00p8yolboqgwlwfnlkdfvnn996601>
Apache Airflow Providers prepared on 28th December 2023 are released <
https://lists.apache.org/thread/y3xgtwjq0g8qckchf9kotnt1jfs6kco9>
Proposed:
Security team rotation introduction to our process <
https://lists.apache.org/thread/mo52q5cv0shzqdcwzobxg8ffosldpzsg>
NEED URGENT FEEDBACK! MySQL Badly signed repo breaks Airflow images <
https://lists.apache.org/thread/dy3fgwhoc5m2f3tpqkwqcytrs6nhjlc4>
Making Listeners API non-experimental <
https://lists.apache.org/thread/cm89z2btvmw292dwlpw42f59zhgm2pqd>
IN-DEVELOPMENT
AIP-44: Airflow Internal API <
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-44+Airflow+Internal+API
>
AIP-49: Open Telemetry Support <
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-49+OpenTelemetry+Support+for+Apache+Airflow
>
AIP-51 Removing Executor Coupling from Core Airflow <
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-51+Removing+Executor+Coupling+from+Core+Airflow
>
AIP-56: Extensible user management <
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
>
AIP-57: Refactor SLA feature <
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-57+Refactor+SLA+Feature
>
AIP-58 Airflow ObjectStore (AS) <
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263430565>
AIP-59: Performance test framework <
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-59+Performance+tests+framework
>
AIP-60: Standard URI representation <
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-60+Standard+URI+representation+for+Airflow+Datasets
>
See the Wiki for more of the project’s approved AIPs. <
https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals
>
AIRFLOW FACT OF THE MONTH
Did you know that dolphins give names to each other? Research has revealed
that the marine mammals use a unique whistle to identify each other. A team
from the University of St Andrews in Scotland found that when the animals
hear their own call played back to them, they respond.
A humble reminder that humans are just another mammal ;).
Source: BBC News <https://www.bbc.com/news/science-environment-23410137>
STAY TUNED
Please subscribe to this newsletter to ensure you always receive the latest
issue: <http://eepurl.com/hXUA3r>
Know of an upcoming event or publication the community should know about?
Let me know at briana.okyere@astronomer.io.