Fwd: Apache Airflow Newsletter | August 2022

[Michael Robinson <mi...@astronomer.io.INVALID>]

> View this email in your browser <https://mailchi.mp/c750d1fc299c/apache-airflow-newsletter-august-9279594?e=a2e8531eaf>
> 
> 
> This month in the Airflow community, we:
> 
> released Airflow Client 2.3.0 <https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=22d09d7702&e=a2e8531eaf> and Airflow 2.3.4 <https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=ded85b2e1b&e=a2e8531eaf>
> held events on using plugins for Webserver <https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=4c01142009&e=a2e8531eaf> and building pipelines with dbt and Great Expectations <https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=1b94cf2f12&e=a2e8531eaf>
> discussed consolidating DAG scheduling params and other topics on the dev list <https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=c2b2315030&e=a2e8531eaf>
> and much more! 
> 
> Please subscribe to this newsletter to ensure you always receive the latest issue: http://eepurl.com/hXUA3r <https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=7a50486f2a&e=a2e8531eaf>.
> 
> 
> Airflow 2.4.0 <https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=29c2fc2ac7&e=a2e8531eaf>, expected in mid-September, will significantly upgrade Airflow’s data handling capabilities thanks to AIP-48 <https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=8dca048e93&e=a2e8531eaf>: data-aware scheduling.
> 
> From the proposal: 
> 
> "Airflow currently depends on XComs to handle data, but several common use cases would benefit from a more "data aware" approach. For example, in many organizations, different teams need to access and modify the same datasets. Currently, a data dependency can only be expressed as a task dependency within a single DAG, so organizations end up writing composite DAGs with many tasks. Even so, underlying data dependencies remain invisible.
> 
> AIP-48 will:
> 
> enable a data dependency mechanism which can be used across DAGs 
> 
> enable the triggering of DAGs based on these dataset updates
> 
> enable explicit, visible relationships between DAGs based on data dependencies."
> 
> 
> On August 23rd, we published Airflow 2.3.4. You can find the package and documentation using these links:
> PyPI <https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=6b4ee2800e&e=a2e8531eaf> | Docs <https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=539e5dde63&e=a2e8531eaf> | Notes <https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=c8323fcee2&e=a2e8531eaf> | Constraints <https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=610860c224&e=a2e8531eaf>
> Docker image: docker pull apache/airflow:2.3.4
> On August 3rd, we released Airflow Client 2.3.0. You can find it here:
> PyPI <https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=30878aa4c4&e=a2e8531eaf> | Docs <https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=85569d3c08&e=a2e8531eaf> 
> 
> This month we had a tie!
> 
> @rossturk <https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=a64b7ae4e9&e=a2e8531eaf>
> 
> #25888 Prefer the local Quick Start in docs <https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=6afbf67621&e=a2e8531eaf>
> Makes the Docker Compose Quick Start procedure into a How To, and puts the standalone procedure in its place
> 
> @pierrejeambrun <https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=6459dbed71&e=a2e8531eaf>
> 
> #25610 Grid logs for mapped instances <https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=7ec024e8d6&e=a2e8531eaf>
> Adds support for displaying mapped instance task logs in the Grid details side panel
> To nominate a PR <https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=81d56e4b76&e=a2e8531eaf> for a future newsletter, flag it by adding #protm to a comment.
> 
> Become the last #ApacheAirflow <https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=15053a04dd&e=a2e8531eaf> speaker of 2022! Sign up to host a meetup at http://bit.ly/3lhp1aR <https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=4229815273&e=a2e8531eaf> & win some amazing swag! 
> 
> September 27: Astronomer's Astro Days: NYC <https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=bd7e87cc48&e=a2e8531eaf>
> 
> If you missed any of these, you can catch a recap, watch on-demand, or dig into the slides:
> 
> August 9: "Airflow 101: Essential Tips for Beginners" <https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=cdf4b37a60&e=a2e8531eaf> with Tamara Fingerlin & Kenten Danas
> 
> August 10: “Build a Robust Data Pipeline with Airflow, dbt, and Great Expectations – with Interactivity” <https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=afe825ec84&e=a2e8531eaf> with Sam Bail
> 
> August 16: "Live with Astronomer: The SQL Table Check Operator" <https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=ff7ef9955e&e=a2e8531eaf> with Kenten Danas & Viraj Parekh
> 
> August 17: “Using Plugins for Webserver” <https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=a07eca048c&e=a2e8531eaf> with Faizan Qazi
> 
> August 23: "The Astro Python SDK" <https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=6a569d5131&e=a2e8531eaf> with Daniel Imberman & Kenten Danas
> 
> 
> Pierre Jeambrun @pierrejeambrun <https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=1b057ec1fe&e=a2e8531eaf>
> Where are you based?
> I live in Paris.
> 
> What is your current role (& industry if not tech)?
> I am a senior full stack developer. In January 2022 I started working freelance, since then I have been working for Engie doing back-end development.
> 
> Why did you start contributing to Airflow?
> After a few years in the industry using tons of amazing open source tools, I felt like it was time for me to contribute to the community. I was looking for a very cool project, not too 'low level' and most importantly with great people to work with. Apache Airflow checked all the boxes.
> 
> What do you use it for?
> I have used Airflow in a couple of side projects at small scale, but never in a production/commercial environment. Currently the only DAGs I write are for Airflow development purposes.
> 
> When was your first PR, and what was it?
> My first PR was in December 2021, the goal was to rewrite the Opsgenie hook to use the official Python SDK.
> 
> What was your latest PR?
> Lately I have been working on the Grid page of the UI, mostly around logs. My last contribution adds support for displaying logs of mapped task instances.
> 
> What do you like or enjoy about working on OSS projects?
> For me this is a unique opportunity to work with talented people from all over the world, to build an amazing product that can be used by everyone.
> 
> 
> “Everything You Should Know About Airflow 2.3.0’s New Grid View” <https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=1f2fd9872c&e=a2e8531eaf>, Brent Bovenzi, Astronomer Blog
> 
> “We’re All Using Airflow Wrong and How to Fix It” <https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=15df105edb&e=a2e8531eaf>, Jessica Laughlin, Medium
> 
> “Astro Is Now Available on All Major Cloud Providers” <https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=054005d742&e=a2e8531eaf>, Ryan Fox, Astronomer Blog 
> 
> “Reimagining Airflow for Data Engineers and Data Scientists with the Astro Python SDK” <https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=e4dafdbdf2&e=a2e8531eaf>, Kenten Danas & Mike Shwe, Astronomer Blog
> 
> Votes
> 
> AIP-44 - Airflow internal API [approved]
> 
> Airflow Providers release [approved]
> 
> Release Airflow 2.3.4 from 2.3.4rc1 [approved]
> 
> Lazy consensus
> 
> Consolidate DAG scheduling params
> 
> Discussed
> 
> Move “contrib” and all old classes to a separate package
> 
> AIP naming correction
> 
> AIP-44 Airflow database API
> 
> “Use existing venv support” for PythonVirtualenvOperator as counterpart to AIP-46
> 
> Vending AWS system test results back to the community
> 
> User Kinceo1 posting spam review approvals
> 
> Should value passed via XComArg be templated?
> 
> Proposed
> 
> A fix for the SLA alerts
> 
> Creating DAGs through the REST API
> 
> Airflow pluggable scheduler
> 
> Announced
> 
> Airflow Python client version 2.3.0 released
> 
> Airflow Providers released
> 
> Apache Airflow 2.3.4 released
> 
> 
> Airflow Helm Chart 1.7.0 <https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=dd6094e979&e=a2e8531eaf>
> The next release for the Airflow Helm Chart, fixing bugs and adding more features.
> 
> For more info about upcoming releases and in-development projects, visit the current projects’ issue (#10176 <https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=98b7fd9355&e=a2e8531eaf>).
> 
> Please subscribe to this newsletter to ensure you always receive the latest issue: http://eepurl.com/hXUA3r <https://apache.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=afc83edf5b&e=a2e8531eaf>.
> 
> Is there an upcoming event or publication the community should know about? Let us know at michael.robinson@astronomer.io <ma...@astronomer.io> or julia@astronomer.io <ma...@astronomer.io>.
> Want to change how you receive these emails?
> You can update your preferences <https://apache.us14.list-manage.com/profile?u=fe7ef7a8dbb32933f30a10466&id=65cb5665fa&e=a2e8531eaf&c=8dffcacaf4> or unsubscribe from this list <https://apache.us14.list-manage.com/unsubscribe?u=fe7ef7a8dbb32933f30a10466&id=65cb5665fa&e=a2e8531eaf&c=8dffcacaf4>.
>