You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "BjornPrime (via GitHub)" <gi...@apache.org> on 2023/03/01 16:15:59 UTC
[GitHub] [beam] BjornPrime opened a new issue, #25676: Replace storage_v1_client with GCS client
BjornPrime opened a new issue, #25676:
URL: https://github.com/apache/beam/issues/25676
### What would you like to happen?
Replace references to the apitools-generated storage client (sdks/python/apache_beam/io/gcp/internal/clients/storage) with analogous calls to the [Google Cloud Storage](https://github.com/googleapis/python-storage) client, supplementing with other Google Cloud Python clients if necessary.
Please ensure that all client references are being thoroughly tested before swapping the GCS client in for the old client. The job is done when the old client can be safely deleted from Beam without impacting functionality.
Background:
apitools is deprecated and continued reliance on it is complicating Beam's dependency management. In order to remove it we need to replace the clients that it currently generates with [Google Cloud Python](https://github.com/googleapis/google-cloud-python) clients, which are being actively maintained (and should be reasonably up-to-date in google3, though please confirm for any particular libraries you end up using.)
### Issue Priority
Priority: 2 (default / most feature requests should be filed as P2)
### Issue Components
- [X] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Replace storage_v1_client with GCS client [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm closed issue #25676: Replace storage_v1_client with GCS client
URL: https://github.com/apache/beam/issues/25676
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on issue #25676: Replace storage_v1_client with GCS client
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on issue #25676:
URL: https://github.com/apache/beam/issues/25676#issuecomment-1630938080
Reopening since this got reverted
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] BjornPrime commented on issue #25676: Replace storage_v1_client with GCS client
Posted by "BjornPrime (via GitHub)" <gi...@apache.org>.
BjornPrime commented on issue #25676:
URL: https://github.com/apache/beam/issues/25676#issuecomment-1729875937
#28079 should resolve this. We're currently running some internal tests to ensure we don't break any internal users before merging again. If all goes well, that should be done by the end of this week, possibly even today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Replace storage_v1_client with GCS client [beam]
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on issue #25676:
URL: https://github.com/apache/beam/issues/25676#issuecomment-1843324121
Do we have any updates on this ahead of the 2.53.0 branch cut?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] BjornPrime commented on issue #25676: Replace storage_v1_client with GCS client
Posted by "BjornPrime (via GitHub)" <gi...@apache.org>.
BjornPrime commented on issue #25676:
URL: https://github.com/apache/beam/issues/25676#issuecomment-1450443452
.take-issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] BjornPrime commented on issue #25676: Replace storage_v1_client with GCS client
Posted by "BjornPrime (via GitHub)" <gi...@apache.org>.
BjornPrime commented on issue #25676:
URL: https://github.com/apache/beam/issues/25676#issuecomment-1658651175
Running into an issue with tar.gz objects (such as staged workflow files) being implicitly decompressed when retrieved from GCS (see #12619).
The way this was handled before seems to have possibly been apitools-specific.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Replace storage_v1_client with GCS client [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on issue #25676:
URL: https://github.com/apache/beam/issues/25676#issuecomment-1843328006
@shunping was this completed in https://github.com/apache/beam/pull/29360 or is there still remaining work?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on issue #25676: Replace storage_v1_client with GCS client
Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on issue #25676:
URL: https://github.com/apache/beam/issues/25676#issuecomment-1729694123
What is the status of this now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on issue #25676: Replace storage_v1_client with GCS client
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #25676:
URL: https://github.com/apache/beam/issues/25676#issuecomment-1730078657
It sounds like this will go to next release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] BjornPrime commented on issue #25676: Replace storage_v1_client with GCS client
Posted by "BjornPrime (via GitHub)" <gi...@apache.org>.
BjornPrime commented on issue #25676:
URL: https://github.com/apache/beam/issues/25676#issuecomment-1622262432
Closing now that PR #25965 has been merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] BjornPrime closed issue #25676: Replace storage_v1_client with GCS client
Posted by "BjornPrime (via GitHub)" <gi...@apache.org>.
BjornPrime closed issue #25676: Replace storage_v1_client with GCS client
URL: https://github.com/apache/beam/issues/25676
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Replace storage_v1_client with GCS client [beam]
Posted by "shunping (via GitHub)" <gi...@apache.org>.
shunping commented on issue #25676:
URL: https://github.com/apache/beam/issues/25676#issuecomment-1843738839
It was completed. We should just close it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org