You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Jarek Potiuk <ja...@potiuk.com> on 2023/05/18 09:09:32 UTC
Huge fependency bump in main (thanks to Google efforts)
Hello everyone,
Due to yesterday's dependency changes I recommend everyone to rebase your
PRs to the latest main and run `breeze ci-image build` if you are using
breeze (and likely upgrading/recreating your local venvs).
We have quite a huge dependency bump in main - thanks to the finally merged
dependency upgrade for Google provider:
https://github.com/apache/airflow/pull/30067. This took quite some time
(several months, countless rebases, some 25+ google libraries to upgrade in
a single PR and a lot of patience was needed as there were a lot of changes
we had to wait for. I also removed vendored-in `google-ads` as the upgrade
allows us to use the regular google-ads library, we have also switched to
the latest v13 API for ads - ahead of July's sunsetting of v12 API. That
should end the "drama" we had with ads-api being sunset and some of our
users being cut-off.
Note that the new Google provider is Airflow 2.4+ only, so if you want to
make use of it, users will have to upgrade Airflow to at least Airflow 2.4.
Elad is going to make an ad-hoc release for google provider right after we
release the current wave. I expect a bit more heavy testing (Google team is
going to take it to a spin) and some "teething" problems so it might take a
few RCs to get it released. But generally speaking it looks good.
This also opens up 3.11 migration (PR in progress here:
https://github.com/apache/airflow/pull/27264 and I expect it to land this
week so that Airflow 2.7 will have Python 3.11 support finally
Thanks to Łukasz Wyszomirski who has done a lot of coordinating work for a
long time and Augusto Hidalgo who led it to completion - but also some
unnamed :) people in the Google team who contributed to it.
That was a big effort that Google team undertool and we had to wait for
others to complete (for example apache.beam) but it finally landed -
freeing some long awaited dependency upgrades (protobuf 3 -> 4 being the
most problematic)
We also had a few "stabilizing" updates after (things got a bit slower for
tests and we will optimize it in the future) but generally speaking all
looks good
For the record - here is what changed (if you see strange errors - they
might come from one of those):
43a44
> Shapely==1.8.5.post1
56c57
< alembic==1.11.0
---
> alembic==1.11.1
63c64
< apache-beam==2.46.0
---
> apache-beam==2.47.0
108c109
< black==23.1a1
---
> black==23.3.0
163c164
< elasticsearch-dbapi==0.2.10
---
> elasticsearch-dbapi==0.2.2
165c166
< elasticsearch==7.13.4
---
> elasticsearch==7.14.2
190,191c191,193
< google-api-core==2.8.2
< google-api-python-client==1.12.11
---
> google-ads==21.1.0
> google-api-core==2.11.0
> google-api-python-client==2.86.0
193,205c195,207
< google-auth-oauthlib==0.8.0
< google-auth==2.18.0
< google-cloud-aiplatform==1.16.1
< google-cloud-appengine-logging==1.1.3
< google-cloud-audit-log==0.2.4
< google-cloud-automl==2.8.0
< google-cloud-bigquery-datatransfer==3.7.0
< google-cloud-bigquery-storage==2.14.1
< google-cloud-bigquery==2.34.4
> google-cloud-memcache==1.7.1
> google-cloud-monitoring==2.14.2
> google-cloud-orchestration-airflow==1.7.1
> google-cloud-os-login==2.9.1
> google-cloud-pubsub==2.17.0
> google-cloud-redis==2.12.1
> google-cloud-resource-manager==1.10.1
> google-cloud-secret-manager==2.16.1
> google-cloud-spanner==3.34.0
> google-cloud-speech==2.19.0
228,233c230,235
< google-cloud-tasks==2.10.1
< google-cloud-texttospeech==1.0.3
< google-cloud-translate==1.7.2
< google-cloud-videointelligence==1.16.3
< google-cloud-vision==1.0.2
< google-cloud-workflows==1.7.1
---
> google-cloud-tasks==2.13.1
> google-cloud-texttospeech==2.14.1
> google-cloud-translate==3.11.1
> google-cloud-videointelligence==2.11.1
> google-cloud-vision==3.4.1
> google-cloud-workflows==1.10.1
236c238
< googleapis-common-protos==1.56.4
---
> googleapis-common-protos==1.59.0
240c242
< grpc-google-iam-v1==0.12.4
---
> grpc-google-iam-v1==0.12.6
242c244
< grpcio-status==1.48.2
---
> grpcio-status==1.54.2
358,359c360,361
< packaging==21.3
< pandas-gbq==0.17.9
---
> packaging==23.1
> pandas-gbq==0.19.2
368c370
< pdpyras==5.0.1
---
> pdpyras==5.0.3
387,388c389,390
< proto-plus==1.19.6
< protobuf==3.20.0
---
> proto-plus==1.22.2
> protobuf==4.22.5
396c398
< pyarrow==9.0.0
---
> pyarrow==11.0.0
412c414
< pymongo==3.13.0
---
> pymongo==4.3.3
513c515
< tableauserverclient==0.24
---
> tableauserverclient==0.25
557c559
< uritemplate==3.0.1
---
> uritemplate==4.1.1
J.
Re: Huge fependency bump in main (thanks to Google efforts)
Posted by Daniel Standish <da...@astronomer.io.INVALID>.
nice
Re: Huge fependency bump in main (thanks to Google efforts)
Posted by Jarek Potiuk <ja...@potiuk.com>.
Huge DEPENDENCY dump of course :)
On Thu, May 18, 2023 at 11:09 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> Hello everyone,
>
> Due to yesterday's dependency changes I recommend everyone to rebase your
> PRs to the latest main and run `breeze ci-image build` if you are using
> breeze (and likely upgrading/recreating your local venvs).
>
> We have quite a huge dependency bump in main - thanks to the finally
> merged dependency upgrade for Google provider:
> https://github.com/apache/airflow/pull/30067. This took quite some time
> (several months, countless rebases, some 25+ google libraries to upgrade in
> a single PR and a lot of patience was needed as there were a lot of changes
> we had to wait for. I also removed vendored-in `google-ads` as the upgrade
> allows us to use the regular google-ads library, we have also switched to
> the latest v13 API for ads - ahead of July's sunsetting of v12 API. That
> should end the "drama" we had with ads-api being sunset and some of our
> users being cut-off.
>
> Note that the new Google provider is Airflow 2.4+ only, so if you want to
> make use of it, users will have to upgrade Airflow to at least Airflow 2.4.
>
> Elad is going to make an ad-hoc release for google provider right after we
> release the current wave. I expect a bit more heavy testing (Google team is
> going to take it to a spin) and some "teething" problems so it might take a
> few RCs to get it released. But generally speaking it looks good.
>
> This also opens up 3.11 migration (PR in progress here:
> https://github.com/apache/airflow/pull/27264 and I expect it to land this
> week so that Airflow 2.7 will have Python 3.11 support finally
>
> Thanks to Łukasz Wyszomirski who has done a lot of coordinating work for a
> long time and Augusto Hidalgo who led it to completion - but also some
> unnamed :) people in the Google team who contributed to it.
>
> That was a big effort that Google team undertool and we had to wait for
> others to complete (for example apache.beam) but it finally landed -
> freeing some long awaited dependency upgrades (protobuf 3 -> 4 being the
> most problematic)
>
> We also had a few "stabilizing" updates after (things got a bit slower for
> tests and we will optimize it in the future) but generally speaking all
> looks good
>
> For the record - here is what changed (if you see strange errors - they
> might come from one of those):
>
> 43a44
> > Shapely==1.8.5.post1
> 56c57
> < alembic==1.11.0
> ---
> > alembic==1.11.1
> 63c64
> < apache-beam==2.46.0
> ---
> > apache-beam==2.47.0
> 108c109
> < black==23.1a1
> ---
> > black==23.3.0
> 163c164
> < elasticsearch-dbapi==0.2.10
> ---
> > elasticsearch-dbapi==0.2.2
> 165c166
> < elasticsearch==7.13.4
> ---
> > elasticsearch==7.14.2
> 190,191c191,193
> < google-api-core==2.8.2
> < google-api-python-client==1.12.11
> ---
> > google-ads==21.1.0
> > google-api-core==2.11.0
> > google-api-python-client==2.86.0
> 193,205c195,207
> < google-auth-oauthlib==0.8.0
> < google-auth==2.18.0
> < google-cloud-aiplatform==1.16.1
> < google-cloud-appengine-logging==1.1.3
> < google-cloud-audit-log==0.2.4
> < google-cloud-automl==2.8.0
> < google-cloud-bigquery-datatransfer==3.7.0
> < google-cloud-bigquery-storage==2.14.1
> < google-cloud-bigquery==2.34.4
> > google-cloud-memcache==1.7.1
> > google-cloud-monitoring==2.14.2
> > google-cloud-orchestration-airflow==1.7.1
> > google-cloud-os-login==2.9.1
> > google-cloud-pubsub==2.17.0
> > google-cloud-redis==2.12.1
> > google-cloud-resource-manager==1.10.1
> > google-cloud-secret-manager==2.16.1
> > google-cloud-spanner==3.34.0
> > google-cloud-speech==2.19.0
> 228,233c230,235
> < google-cloud-tasks==2.10.1
> < google-cloud-texttospeech==1.0.3
> < google-cloud-translate==1.7.2
> < google-cloud-videointelligence==1.16.3
> < google-cloud-vision==1.0.2
> < google-cloud-workflows==1.7.1
> ---
> > google-cloud-tasks==2.13.1
> > google-cloud-texttospeech==2.14.1
> > google-cloud-translate==3.11.1
> > google-cloud-videointelligence==2.11.1
> > google-cloud-vision==3.4.1
> > google-cloud-workflows==1.10.1
> 236c238
> < googleapis-common-protos==1.56.4
> ---
> > googleapis-common-protos==1.59.0
> 240c242
> < grpc-google-iam-v1==0.12.4
> ---
> > grpc-google-iam-v1==0.12.6
> 242c244
> < grpcio-status==1.48.2
> ---
> > grpcio-status==1.54.2
> 358,359c360,361
> < packaging==21.3
> < pandas-gbq==0.17.9
> ---
> > packaging==23.1
> > pandas-gbq==0.19.2
> 368c370
> < pdpyras==5.0.1
> ---
> > pdpyras==5.0.3
> 387,388c389,390
> < proto-plus==1.19.6
> < protobuf==3.20.0
> ---
> > proto-plus==1.22.2
> > protobuf==4.22.5
> 396c398
> < pyarrow==9.0.0
> ---
> > pyarrow==11.0.0
> 412c414
> < pymongo==3.13.0
> ---
> > pymongo==4.3.3
> 513c515
> < tableauserverclient==0.24
> ---
> > tableauserverclient==0.25
> 557c559
> < uritemplate==3.0.1
> ---
> > uritemplate==4.1.1
>
> J.
>
>