You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/04 20:56:05 UTC
[GitHub] [beam] damccorm opened a new issue, #21019: Update Python libraries for GCP services
damccorm opened a new issue, #21019:
URL: https://github.com/apache/beam/issues/21019
Some libraries from the [GCP requirements](https://github.com/apache/beam/blob/cbb363f2f01d44dd3f7c063c6cd9d529b5fa9104/sdks/python/setup.py#L188) of the Apache Beam Python package reference old versions, e.g. Bigtable and Spanner (that had several major version bumps since).
The setup file describes those dependencies as only being required by tests, but it looks like [some transforms](https://github.com/apache/beam/blob/cbb363f2f01d44dd3f7c063c6cd9d529b5fa9104/sdks/python/apache_beam/io/gcp/experimental/spannerio.py) are referencing them. Therefore I'm not sure of the real cost behind upgrading them.
I'm currently using the Spanner client in a custom transform, but I have to stick to the `google-cloud-spanner` version used by Apache Beam. Would it be possible to upgrade those dependencies?
Cheers,
Flo
Imported from Jira [BEAM-12817](https://issues.apache.org/jira/browse/BEAM-12817). Original Jira may contain additional context.
Reported by: flovouin.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on issue #21019: Update Python libraries for GCP services
Posted by "AnandInguva (via GitHub)" <gi...@apache.org>.
AnandInguva commented on issue #21019:
URL: https://github.com/apache/beam/issues/21019#issuecomment-1454072653
Yes, PR addressing this issue https://github.com/apache/beam/pull/24599
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] amardeep commented on issue #21019: Update Python libraries for GCP services
Posted by "amardeep (via GitHub)" <gi...@apache.org>.
amardeep commented on issue #21019:
URL: https://github.com/apache/beam/issues/21019#issuecomment-1453933021
Any update on this?
Currently, when I install beam, I get `google-cloud-bigquery=1.28.2` which is really old.
Here are the steps to reproduce
```shell
conda create -p ./env -c conda-forge python=3.10 poetry
conda activate ./env
poetry init -q
poetry add 'apache-beam[gcp]=2.45.0'
```
This results in following versions of google libraries:
```shell
$ pip list | grep google
google-api-core 2.8.2
google-apitools 0.5.31
google-auth 2.16.2
google-auth-httplib2 0.1.0
google-cloud-bigquery 1.28.2
google-cloud-bigquery-storage 2.16.0
google-cloud-bigtable 1.7.3
google-cloud-core 2.3.2
google-cloud-datastore 1.15.5
google-cloud-dlp 3.9.0
google-cloud-language 1.3.2
google-cloud-pubsub 2.13.7
google-cloud-pubsublite 1.7.0
google-cloud-recommendations-ai 0.7.1
google-cloud-spanner 3.22.0
google-cloud-storage 2.1.0
google-cloud-videointelligence 1.16.3
google-cloud-vision 3.1.2
google-crc32c 1.5.0
google-resumable-media 1.3.3
googleapis-common-protos 1.56.4
grpc-google-iam-v1 0.12.4
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on issue #21019: Update Python libraries for GCP services
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on issue #21019:
URL: https://github.com/apache/beam/issues/21019#issuecomment-1204518360
We are also working on a better process of managing Python dependencies to avoid having outdated libraries in our chain.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on issue #21019: Update Python libraries for GCP services
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #21019:
URL: https://github.com/apache/beam/issues/21019#issuecomment-1453986783
cc: @AnandInguva
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] thclark commented on issue #21019: Update Python libraries for GCP services
Posted by GitBox <gi...@apache.org>.
thclark commented on issue #21019:
URL: https://github.com/apache/beam/issues/21019#issuecomment-1183069783
Yeah, it's really necessary that dependencies are updated; it makes it nearly impossible to do anything with current versions of google libraries, being blocked by installation of dependencies that aren't even used in the actual install, only in test.
As a simple step to mitigation, the test dependencies could be split into a separate section (e.g. `gcp-test`) to only get installed when actually testing beam, and not throw a spanner in the works when installing it.
My current workaround is to not install `apache-beam[gcp]` and just to install `apache-beam`, then manually install the gcp dependencies myself, thereby avoiding the infinite install [in this closely related issue](https://github.com/apache/beam/issues/22218) by pinning dependencies. It's a real pain to do that though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ElTav commented on issue #21019: Update Python libraries for GCP services
Posted by GitBox <gi...@apache.org>.
ElTav commented on issue #21019:
URL: https://github.com/apache/beam/issues/21019#issuecomment-1251168878
@tvalentyn Are there any plans to update the PubSub Python dependency?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on issue #21019: Update Python libraries for GCP services
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on issue #21019:
URL: https://github.com/apache/beam/issues/21019#issuecomment-1204517996
> I'm currently using the Spanner client in a custom transform, but I have to stick to the google-cloud-spanner version used by Apache Beam. Would it be possible to upgrade those dependencies?
We are evaluating current usage of Spanner dependency. It's currently an experimental IO, uses a very old client, and we may have to exclude it from the dependency chain since we plan to switch to X-Lang spanner IO.
I know this is not convenient, but (assuming you don't use the experimental spanner IO), you can have a custom build of Apache Beam that does not require the google-cloud-spanner, or just force-install the version you need even i there is a dependency conflict.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on issue #21019: Update Python libraries for GCP services
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on issue #21019:
URL: https://github.com/apache/beam/issues/21019#issuecomment-1261664088
which one?
https://pypi.org/project/google-cloud-pubsub/#history is within range:
https://github.com/apache/beam/blob/master/sdks/python/setup.py#L286
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on issue #21019: Update Python libraries for GCP services
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #21019:
URL: https://github.com/apache/beam/issues/21019#issuecomment-1453986548
we will have an update before next release for many dependencies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn closed issue #21019: Update Python libraries for GCP services
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn closed issue #21019: Update Python libraries for GCP services
URL: https://github.com/apache/beam/issues/21019
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org