You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/06/12 00:31:42 UTC
[GitHub] [beam] davidyan74 opened a new pull request #11996: [BEAM-10247] Pin google-api-core to 1.17.0
davidyan74 opened a new pull request #11996:
URL: https://github.com/apache/beam/pull/11996
Pin google-api-core to 1.17.0, because otherwise the pulled version 1.20.0 is incompatible with the pinned grpc.
R: @aaltay
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Samza | Spark
--- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on pull request #11996: [BEAM-10247] Pin google-api-core to 1.17.0
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on pull request #11996:
URL: https://github.com/apache/beam/pull/11996#issuecomment-644301960
David, can you add rsa<4.1; python_version < "3" to test dependencies to unblock this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on pull request #11996: [BEAM-10247] Pin google-api-core to 1.17.0
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on pull request #11996:
URL: https://github.com/apache/beam/pull/11996#issuecomment-644320783
Can't get Jenkins to run.. Also https://github.com/googleapis/google-auth-library-python/pull/535 is another attempt to fix this in google-auth.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on pull request #11996: [BEAM-10247] Pin google-api-core to 1.17.0
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on pull request #11996:
URL: https://github.com/apache/beam/pull/11996#issuecomment-644311666
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11996: [BEAM-10247] Pin google-api-core to 1.17.0
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #11996:
URL: https://github.com/apache/beam/pull/11996#issuecomment-644369082
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] davidyan74 commented on pull request #11996: [BEAM-10247] Pin google-api-core to 1.17.0
Posted by GitBox <gi...@apache.org>.
davidyan74 commented on pull request #11996:
URL: https://github.com/apache/beam/pull/11996#issuecomment-644295364
Looks like dependency check broke for python 2. Since python 2 is deprecated, I'm going to restrict the changes for just python 3.
```
10:09:10 google-auth 1.17.2 has requirement rsa<4.1; python_version < "3", but you have rsa 4.5.
10:09:10 ERROR: InvocationError for command /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py2/build/srcs/sdks/python/target/.tox-py27-cloud/py27-cloud/bin/pip check (exited with code 1)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn edited a comment on pull request #11996: [BEAM-10247] Pin google-api-core to 1.17.0
Posted by GitBox <gi...@apache.org>.
tvalentyn edited a comment on pull request #11996:
URL: https://github.com/apache/beam/pull/11996#issuecomment-643021914
I thinks setup.py should be as flexible as possible and as reasonable, it already does not include an upper bound on grpc until next major version. So in containers we can either upgrade grpc or downgrade the google-api-core.
Given the broad requirement in setup.py, I would prefer if we upgrade grpc instead, see: https://github.com/apache/beam/blob/63cc13ae0d135c9966fca3b87f34e06d142e412d/sdks/python/setup.py#L154
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #11996: [BEAM-10247] Pin google-api-core to 1.17.0
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #11996:
URL: https://github.com/apache/beam/pull/11996#issuecomment-643024901
> I thinks setup.py should be as flexible as possible and as reasonable, it already does not include an upper bound on grpc until next major version. So in containers we can either upgrade grpc or downgrade the google-api-core.
>
> Given the broad requirement in setup.py, I would prefer if we upgrade grpc instead, see:
>
> https://github.com/apache/beam/blob/63cc13ae0d135c9966fca3b87f34e06d142e412d/sdks/python/setup.py#L154
SGTM. Let's update and see if there will be any test issues.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on pull request #11996: [BEAM-10247] Pin google-api-core to 1.17.0
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on pull request #11996:
URL: https://github.com/apache/beam/pull/11996#issuecomment-643382060
I don't know why Jenkins lags. Tests are not triggered: https://builds.apache.org/job/beam_PostCommit_Python37_PR/.
PR LGTM but we need to test before merge.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] davidyan74 commented on pull request #11996: [BEAM-10247] Pin google-api-core to 1.17.0
Posted by GitBox <gi...@apache.org>.
davidyan74 commented on pull request #11996:
URL: https://github.com/apache/beam/pull/11996#issuecomment-643009569
FYI, this fixes the same issue described here: https://github.com/googleapis/python-api-core/issues/40
If we have grpcio < 1.29.0 with google-api-core > 1.17.0, I'd imagine a lot of GCP IOs would break.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] davidyan74 commented on pull request #11996: [BEAM-10247] Pin google-api-core to 1.17.0
Posted by GitBox <gi...@apache.org>.
davidyan74 commented on pull request #11996:
URL: https://github.com/apache/beam/pull/11996#issuecomment-643014103
To be safe, I think we should also pin google-api-core to 1.17.0 in setup.py, unless we are sure we are ready to require grpcio >= 1.29.0.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn merged pull request #11996: [BEAM-10247] Pin google-api-core to 1.17.0
Posted by GitBox <gi...@apache.org>.
tvalentyn merged pull request #11996:
URL: https://github.com/apache/beam/pull/11996
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn edited a comment on pull request #11996: [BEAM-10247] Pin google-api-core to 1.17.0
Posted by GitBox <gi...@apache.org>.
tvalentyn edited a comment on pull request #11996:
URL: https://github.com/apache/beam/pull/11996#issuecomment-643021914
I thinks setup.py should be as flexible as possible and as reasonable, it already does not include an upper bound on grpc until next major version. So we can either upgrade grpc in containers or downgrade the google-api-core.
Given the broad requirement in setup.py, I would prefer if we upgrade grpc instead, see: https://github.com/apache/beam/blob/63cc13ae0d135c9966fca3b87f34e06d142e412d/sdks/python/setup.py#L154
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on pull request #11996: [BEAM-10247] Pin google-api-core to 1.17.0
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on pull request #11996:
URL: https://github.com/apache/beam/pull/11996#issuecomment-643353234
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on pull request #11996: [BEAM-10247] Pin google-api-core to 1.17.0
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on pull request #11996:
URL: https://github.com/apache/beam/pull/11996#issuecomment-643022576
cc: @ihji FYI in case Dataflow containers also need an update.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on pull request #11996: [BEAM-10247] Pin google-api-core to 1.17.0
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on pull request #11996:
URL: https://github.com/apache/beam/pull/11996#issuecomment-644402456
I think tests passed, but Jenkins hook is lagging to propagate the green signal. https://builds.apache.org/job/beam_PreCommit_Python_Commit/13221/consoleText
PostCommits also passed on previous commit, merging.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #11996: [BEAM-10247] Pin google-api-core to 1.17.0
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #11996:
URL: https://github.com/apache/beam/pull/11996#issuecomment-643008565
This looks good to me.
@tvalentyn - do we need to pin anything else related to this? (setup.py?)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on pull request #11996: [BEAM-10247] Pin google-api-core to 1.17.0
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on pull request #11996:
URL: https://github.com/apache/beam/pull/11996#issuecomment-643021914
I thinks setup.py should be as flexible as possible and as reasonable, it already does not include an upper bound on grpc. So we can either upgrade grpc in containers or downgrade the google-api-core.
Given the broad requirement in setup.py, I would prefer if we upgrade grpc instead, see: https://github.com/apache/beam/blob/63cc13ae0d135c9966fca3b87f34e06d142e412d/sdks/python/setup.py#L154
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on pull request #11996: [BEAM-10247] Pin google-api-core to 1.17.0
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on pull request #11996:
URL: https://github.com/apache/beam/pull/11996#issuecomment-643381750
Run Python 3.7 PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #11996: [BEAM-10247] Pin google-api-core to 1.17.0
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #11996:
URL: https://github.com/apache/beam/pull/11996#issuecomment-643018640
I think we can consider updating grpcio as well. We will need to to do it eventually anyway.
Casting a wide net about grpcio. Any concerns about upgrading grpcio to 1.29.0 or later in python? (@chamikaramj @lukecwik @robertwb)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] davidyan74 commented on pull request #11996: [BEAM-10247] Pin google-api-core to 1.17.0
Posted by GitBox <gi...@apache.org>.
davidyan74 commented on pull request #11996:
URL: https://github.com/apache/beam/pull/11996#issuecomment-643340622
I'll add the grpc version requirement to both setup.py and base_image_requirements.txt.
BTW, the tests failed because of this:
18:13:16 ERROR: Failure: SyntaxError (Non-ASCII character '\xc3' in file /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/build/gradleenv/-194514014/local/lib/python2.7/site-packages/rsa/key.py on line 1, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details (key.py, line 1))
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on pull request #11996: [BEAM-10247] Pin google-api-core to 1.17.0
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on pull request #11996:
URL: https://github.com/apache/beam/pull/11996#issuecomment-644307698
Thank you. Test deps are declared here: https://github.com/apache/beam/blob/c328d2ac0f805cdfd9de8ffa513c611051549f65/sdks/python/setup.py#L181.
Sample syntax: https://github.com/apache/beam/blob/c328d2ac0f805cdfd9de8ffa513c611051549f65/sdks/python/setup.py#L151
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on pull request #11996: [BEAM-10247] Pin google-api-core to 1.17.0
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on pull request #11996:
URL: https://github.com/apache/beam/pull/11996#issuecomment-644256690
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #11996: [BEAM-10247] Pin google-api-core to 1.17.0
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #11996:
URL: https://github.com/apache/beam/pull/11996#issuecomment-643493248
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on pull request #11996: [BEAM-10247] Pin google-api-core to 1.17.0
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on pull request #11996:
URL: https://github.com/apache/beam/pull/11996#issuecomment-644311365
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] davidyan74 commented on pull request #11996: [BEAM-10247] Pin google-api-core to 1.17.0
Posted by GitBox <gi...@apache.org>.
davidyan74 commented on pull request #11996:
URL: https://github.com/apache/beam/pull/11996#issuecomment-644303753
Sure, will do so shortly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org