You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/11/05 20:36:40 UTC
[GitHub] [airflow] mik-laj opened a new issue #12116: Update google-cloud deps to lv2
mik-laj opened a new issue #12116:
URL: https://github.com/apache/airflow/issues/12116
Hello,
Google Cloud API Libraries introduced breaking changes in 2.0.0 (https://github.com/googleapis/python-container/blob/master/UPGRADING.md) and they already caused a number of changes. For this reason, we limit all our google-cloud deps to <2.0.0. This limit means our updates will not receive updates, but everything will still work.
We should try to code all operators to use newer versions, so that our users can use the newest API versions.
At the moment (5 November 2020), this is the status of our packages:
| package_name | current_version | latest_version | diff_part |
|------------------------------------|-------------------|------------------|-------------|
| google-cloud-automl | 1.0.1 | 2.1.0 | 1-major |
| google-cloud-bigquery | 1.26.1 | 2.3.1 | 1-major |
| google-cloud-bigquery-datatransfer | 1.1.0 | 2.1.0 | 1-major |
| google-cloud-container | 1.0.1 | 2.1.0 | 1-major |
| google-cloud-datacatalog | 0.7.0 | 2.0.0 | 1-major |
| google-cloud-dataproc | 1.1.1 | 2.0.2 | 1-major |
| google-cloud-dlp | 1.0.0 | 2.0.0 | 1-major |
| google-cloud-kms | 1.4.0 | 2.2.0 | 1-major |
| google-cloud-language | 1.3.0 | 2.0.0 | 1-major |
| google-cloud-monitoring | 1.0.0 | 2.0.0 | 1-major |
| google-cloud-pubsub | 1.7.0 | 2.1.0 | 1-major |
| google-cloud-redis | 1.0.0 | 2.0.0 | 1-major |
| google-cloud-secret-manager | 1.0.0 | 2.0.0 | 1-major |
| google-cloud-speech | 1.3.2 | 2.0.0 | 1-major |
| google-cloud-tasks | 1.5.0 | 2.0.0 | 1-major |
| google-cloud-texttospeech | 1.0.1 | 2.2.0 | 1-major |
| google-cloud-translate | 1.7.0 | 3.0.1 | 1-major |
| google-cloud-vision | 1.0.0 | 2.0.0 | 1-major |
| google-cloud-bigtable | 1.4.0 | 1.5.1 | 2-minor |
| google-cloud-spanner | 1.17.1 | 1.19.1 | 2-minor |
| google-cloud-storage | 1.30.0 | 1.32.0 | 2-minor |
| google-cloud-videointelligence | 1.15.0 | 1.16.0 | 2-minor |
| google-cloud-core | 1.4.1 | 1.4.3 | 3-patch |
I generated these tables with the script:
https://gist.github.com/mik-laj/880b07bfbdbd5c65b4b2260f6c0fee72
Related issue: https://github.com/apache/airflow/issues/10316
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #12116: Update google-cloud deps to v2
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #12116:
URL: https://github.com/apache/airflow/issues/12116#issuecomment-722686840
Thanks @mik-laj . You are a gem!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #12116: Update google-cloud deps to v2
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #12116:
URL: https://github.com/apache/airflow/issues/12116#issuecomment-742502324
If we want users to be able to install backport packages, we should also remember about the limitations that exist in Airflow 1.10 We don't support them anymore, but users can still use them and it would be great if they could use new and old operators at the same time.
```
'google-cloud-bigtable>=1.0.0,<2.0.0',
'google-cloud-container>=0.1.1,<2.0.0',
'google-cloud-dlp>=0.11.0,<2.0.0',
'google-cloud-language>=1.1.1,<2.0.0',
'google-cloud-secret-manager>=0.2.0,<2.0.0',
'google-cloud-spanner>=1.10.0,<2.0.0',
'google-cloud-speech>=0.36.3,<2.0.0',
'google-cloud-storage>=1.16,<2.0.0',
'google-cloud-texttospeech>=0.4.0,<2',
'google-cloud-translate>=1.3.3,<2.0.0',
'google-cloud-videointelligence>=1.7.0,<2.0.0',
'google-cloud-vision>=0.35.2,<2.0.0',
```
We can only update these operators when we stop releasing backport packages.
> In addition, Backport providers within 1.10.x, will be supported for critical fixes for three months after Airflow 2.0.0 is released.
http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/upgrading-to-2.html
CC: @rafalbiegacz @potiuk
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #12116: Update google-cloud deps to v2
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #12116:
URL: https://github.com/apache/airflow/issues/12116#issuecomment-742502583
@michalslowikowski00 @turbaszek Fortunately, Dataproc does not conflict, but other libraries will cause problems.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj edited a comment on issue #12116: Update google-cloud deps to v2
Posted by GitBox <gi...@apache.org>.
mik-laj edited a comment on issue #12116:
URL: https://github.com/apache/airflow/issues/12116#issuecomment-748636445
Part of: https://github.com/apache/airflow/issues/12120
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] michalslowikowski00 commented on issue #12116: Update google-cloud deps to v2
Posted by GitBox <gi...@apache.org>.
michalslowikowski00 commented on issue #12116:
URL: https://github.com/apache/airflow/issues/12116#issuecomment-734209824
I am working on Dataproc package.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #12116: Update google-cloud deps to v2
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #12116:
URL: https://github.com/apache/airflow/issues/12116#issuecomment-744660908
I added a new column "To update?", which specifies whether we can update this library or rather we should wait until we drop Airflow 1.10 support.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #12116: Update google-cloud deps to v2
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #12116:
URL: https://github.com/apache/airflow/issues/12116#issuecomment-748636445
Depends: https://github.com/apache/airflow/issues/12120
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #12116: Update google-cloud deps to v2
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #12116:
URL: https://github.com/apache/airflow/issues/12116
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org