You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/04/28 12:24:51 UTC
[GitHub] [airflow] eladkal opened a new pull request, #23326: Google provider: Remove `bigquery_conn_id`, `google_cloud_storage_conn_id`
eladkal opened a new pull request, #23326:
URL: https://github.com/apache/airflow/pull/23326
The next Google provider release is major (breaking changes) so I want to take the opportunity to clean up some deprecations
* `bigquery_conn_id` is removed. Please use `gcp_conn_id`.
affected classes:
`BigQueryCheckOperator`
`BigQueryCreateEmptyDatasetOperator`
`BigQueryDeleteDatasetOperator`
`BigQueryDeleteTableOperator`
`BigQueryExecuteQueryOperator`
`BigQueryGetDataOperator`
`BigQueryHook`
`BigQueryIntervalCheckOperator`
`BigQueryTableExistenceSensor`
`BigQueryTablePartitionExistenceSensor`
`BigQueryToBigQueryOperator`
`BigQueryToGCSOperator`
`BigQueryUpdateTableSchemaOperator`
`BigQueryUpsertTableOperator`
`BigQueryValueCheckOperator`
`GCSToBigQueryOperator`
* `google_cloud_storage_conn_id` is removed. Please use `gcp_conn_id`.
affected classes:
`ADLSToGCSOperator`
`BaseSQLToGCSOperator`
`CassandraToGCSOperator`
`GCSBucketCreateAclEntryOperator`
`GCSCreateBucketOperator`
`GCSDeleteObjectsOperator`
`GCSHook`
`GCSListObjectsOperator`
`GCSObjectCreateAclEntryOperator`
`GCSToBigQueryOperator`
`GCSToGCSOperator`
`GCSToLocalFilesystemOperator`
`LocalFilesystemToGCSOperator`
Suppress: https://github.com/apache/airflow/pull/23244
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragement file, named `{pr_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal merged pull request #23326: Google provider: Remove deprecations of `bigquery_conn_id`, `google_cloud_storage_conn_id`
Posted by GitBox <gi...@apache.org>.
eladkal merged PR #23326:
URL: https://github.com/apache/airflow/pull/23326
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on pull request #23326: Google provider: Remove `bigquery_conn_id`, `google_cloud_storage_conn_id`
Posted by GitBox <gi...@apache.org>.
eladkal commented on PR #23326:
URL: https://github.com/apache/airflow/pull/23326#issuecomment-1112681159
@mik-laj
**For the first point:**
I didn't remove all project references to `bigquery_conn_id`, `google_cloud_storage_conn_id`.
I removed only those we set deprecation warning for.
There are operators which should use them - all are in cases where we have two references to gcp conn so we can't just name it `gcp_conn_id` as it will be confusing.
The [BigQueryCreateExternalTableOperator](https://github.com/apache/airflow/blob/d673c9dc86491a8edd88bc60b3ae9a6be2c3187d/airflow/providers/google/cloud/operators/bigquery.py#L889)
This operator has `bigquery_conn_id` and `google_cloud_storage_conn_id`. You don't have to use the same conn for BigQuery and for GCS so it make sense to use different names to distinguish the two conns you can set. I guess that was also the mind set of previous contributors as they decided not to deprecated these con_ids in this specific operator/
I guess the PR title is a bit misleading I'll change it to `Remove deprecations of ...`
Did I miss something else?
**For the second point:**
The gcs_to_s3 is in Aws not Google. The next release of Aws provider is not a major release so we can't remove it now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on pull request #23326: Google provider: Remove `bigquery_conn_id`, `google_cloud_storage_conn_id`
Posted by GitBox <gi...@apache.org>.
mik-laj commented on PR #23326:
URL: https://github.com/apache/airflow/pull/23326#issuecomment-1112558883
```
$ git ls-files -z | xargs -0 grep bigquery_conn_id
grep: .github/actions/checks-action: Is a directory
grep: .github/actions/codecov-action: Is a directory
grep: .github/actions/configure-aws-credentials: Is a directory
grep: .github/actions/get-workflow-origin: Is a directory
grep: .github/actions/github-push-action: Is a directory
grep: .github/actions/label-when-approved-action: Is a directory
RELEASE_NOTES.rst:which apply to most services. Parameters like ``datastore_conn_id``\ , ``bigquery_conn_id``\ ,
airflow/providers/google/CHANGELOG.rst:* ``bigquery_conn_id`` is removed. Please use ``gcp_conn_id``.
airflow/providers/google/cloud/operators/bigquery.py: :param bigquery_conn_id: [Optional] The connection ID used to connect to Google Cloud and
airflow/providers/google/cloud/operators/bigquery.py: bigquery_conn_id='airflow-conn-id',
airflow/providers/google/cloud/operators/bigquery.py: bigquery_conn_id='airflow-conn-id-account',
airflow/providers/google/cloud/operators/bigquery.py: bigquery_conn_id: str = 'google_cloud_default',
airflow/providers/google/cloud/operators/bigquery.py: self.bigquery_conn_id = bigquery_conn_id
airflow/providers/google/cloud/operators/bigquery.py: gcp_conn_id=self.bigquery_conn_id,
airflow/providers/google/cloud/operators/bigquery.py: :param bigquery_conn_id: (Optional) The connection ID used to connect to Google Cloud and
airflow/providers/google/cloud/operators/bigquery.py: bigquery_conn_id: str = 'google_cloud_default',
airflow/providers/google/cloud/operators/bigquery.py: self.bigquery_conn_id = bigquery_conn_id
airflow/providers/google/cloud/operators/bigquery.py: gcp_conn_id=self.bigquery_conn_id,
airflow/providers/google/cloud/operators/bigquery.py: :param bigquery_conn_id: (Deprecated) The connection ID used to connect to Google Cloud.
grep: provider_packages/dist: Is a directory
$ git ls-files -z | xargs -0 grep google_cloud_storage_conn_id
grep: .github/actions/checks-action: Is a directory
grep: .github/actions/codecov-action: Is a directory
grep: .github/actions/configure-aws-credentials: Is a directory
grep: .github/actions/get-workflow-origin: Is a directory
grep: .github/actions/github-push-action: Is a directory
grep: .github/actions/label-when-approved-action: Is a directory
RELEASE_NOTES.rst:``google_cloud_storage_conn_id`` and similar have been deprecated. Operators that require two connections are not changed.
airflow/providers/amazon/aws/transfers/gcs_to_s3.py: :param google_cloud_storage_conn_id: (Deprecated) The connection ID used to connect to Google Cloud.
airflow/providers/amazon/aws/transfers/gcs_to_s3.py: google_cloud_storage_conn_id: Optional[str] = None,
airflow/providers/amazon/aws/transfers/gcs_to_s3.py: if google_cloud_storage_conn_id:
airflow/providers/amazon/aws/transfers/gcs_to_s3.py: "The google_cloud_storage_conn_id parameter has been deprecated. You should pass "
airflow/providers/amazon/aws/transfers/gcs_to_s3.py: gcp_conn_id = google_cloud_storage_conn_id
airflow/providers/google/CHANGELOG.rst:* ``google_cloud_storage_conn_id`` is removed. Please use ``gcp_conn_id``.
airflow/providers/google/CHANGELOG.rst:* ``Replace 'google_cloud_storage_conn_id' by 'gcp_conn_id' when using 'GCSHook' (#13851)``
airflow/providers/google/cloud/operators/bigquery.py: :param google_cloud_storage_conn_id: [Optional] The connection ID used to connect to Google Cloud.
airflow/providers/google/cloud/operators/bigquery.py: google_cloud_storage_conn_id='airflow-conn-id'
airflow/providers/google/cloud/operators/bigquery.py: google_cloud_storage_conn_id='airflow-conn-id'
airflow/providers/google/cloud/operators/bigquery.py: google_cloud_storage_conn_id: str = 'google_cloud_default',
airflow/providers/google/cloud/operators/bigquery.py: self.google_cloud_storage_conn_id = google_cloud_storage_conn_id
airflow/providers/google/cloud/operators/bigquery.py: gcp_conn_id=self.google_cloud_storage_conn_id,
airflow/providers/google/cloud/operators/bigquery.py: :param google_cloud_storage_conn_id: (Optional) The connection ID used to connect to Google Cloud
airflow/providers/google/cloud/operators/bigquery.py: google_cloud_storage_conn_id: str = 'google_cloud_default',
airflow/providers/google/cloud/operators/bigquery.py: self.google_cloud_storage_conn_id = google_cloud_storage_conn_id
airflow/providers/google/cloud/operators/bigquery.py: gcp_conn_id=self.google_cloud_storage_conn_id,
docs/apache-airflow-providers-amazon/commits.rst:`f473ca7130 <https://github.com/apache/airflow/commit/f473ca7130f844bc59477674e641b42b80698bb7>`_ 2021-01-24 ``Replace 'google_cloud_storage_conn_id' by 'gcp_conn_id' when using 'GCSHook' (#13851)``
docs/apache-airflow-providers-google/commits.rst:`f473ca7130 <https://github.com/apache/airflow/commit/f473ca7130f844bc59477674e641b42b80698bb7>`_ 2021-01-24 ``Replace 'google_cloud_storage_conn_id' by 'gcp_conn_id' when using 'GCSHook' (#13851)``
grep: provider_packages/dist: Is a directory
```
Can you check `airflow/providers/google/cloud/operators/bigquery.py`, `airflow/providers/amazon/aws/transfers/gcs_to_s3.py` files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #23326: Google provider: Remove deprecations of `bigquery_conn_id`, `google_cloud_storage_conn_id`
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #23326:
URL: https://github.com/apache/airflow/pull/23326#issuecomment-1112720137
The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org