You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/04/04 21:25:51 UTC
[GitHub] [beam] veblush opened a new pull request #11315: [BEAM-8889]
Upgrades gcsio to 2.1.2
veblush opened a new pull request #11315: [BEAM-8889] Upgrades gcsio to 2.1.2
URL: https://github.com/apache/beam/pull/11315
This is part of [BEAM-8889](https://issues.apache.org/jira/projects/BEAM/issues/BEAM-8889) to get new GCS connector 2.1.2 after #10769. This doesn't add any feature except the new API `setGrpcEnabled` that does no-op currently, but enables #10786 to get merged. Once this and #10786 are merged, developers (mostly us) can try the unstable gRPC/GCS feature by changing the gcsio dependency to 2.2.x of their Beam project.
After gcsio releases 2.2.x, the the first version supporting gRPC/GCS protocol, that version will be added to Beam SDK so that users can try the feature by using `use_grpc_for_gcs` experiment flag (#11183).
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on issue #11315: [BEAM-8889] Upgrades
gcsio to 2.1.2
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #11315: [BEAM-8889] Upgrades gcsio to 2.1.2
URL: https://github.com/apache/beam/pull/11315#issuecomment-610644603
Run Java PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on issue #11315: [BEAM-8889] Upgrades
gcsio to 2.1.2
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #11315: [BEAM-8889] Upgrades gcsio to 2.1.2
URL: https://github.com/apache/beam/pull/11315#issuecomment-610644564
Retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on issue #11315: [BEAM-8889] Upgrades
gcsio to 2.1.2
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #11315: [BEAM-8889] Upgrades gcsio to 2.1.2
URL: https://github.com/apache/beam/pull/11315#issuecomment-611095743
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on issue #11315: [BEAM-8889] Upgrades
gcsio to 2.1.2
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #11315: [BEAM-8889] Upgrades gcsio to 2.1.2
URL: https://github.com/apache/beam/pull/11315#issuecomment-611064288
Can you check if failure here is due to the library update:
https://builds.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Commit/7467/testReport/junit/org.apache.beam.examples/WordCountIT/testE2EWordCount/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] veblush commented on issue #11315: [BEAM-8889] Upgrades
gcsio to 2.1.2
Posted by GitBox <gi...@apache.org>.
veblush commented on issue #11315: [BEAM-8889] Upgrades gcsio to 2.1.2
URL: https://github.com/apache/beam/pull/11315#issuecomment-611092137
Fortunately the failure in the example was gone now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj edited a comment on issue #11315: [BEAM-8889]
Upgrades gcsio to 2.1.2
Posted by GitBox <gi...@apache.org>.
chamikaramj edited a comment on issue #11315: [BEAM-8889] Upgrades gcsio to 2.1.2
URL: https://github.com/apache/beam/pull/11315#issuecomment-611064288
Can you check if failure here is due to the library update:
https://builds.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Commit/7467/testReport/junit/org.apache.beam.examples/WordCountIT/testE2EWordCount_2/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on issue #11315: [BEAM-8889] Upgrades
gcsio to 2.1.2
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #11315: [BEAM-8889] Upgrades gcsio to 2.1.2
URL: https://github.com/apache/beam/pull/11315#issuecomment-611066261
Run Java_Examples_Dataflow PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on issue #11315: [BEAM-8889] Upgrades
gcsio to 2.1.2
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #11315: [BEAM-8889] Upgrades gcsio to 2.1.2
URL: https://github.com/apache/beam/pull/11315#issuecomment-611256051
Don't think failure is related but test suite seems to be pretty flake. So will try re-run when it's more stable.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on issue #11315: [BEAM-8889] Upgrades
gcsio to 2.1.2
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #11315: [BEAM-8889] Upgrades gcsio to 2.1.2
URL: https://github.com/apache/beam/pull/11315#issuecomment-611063833
Run Python2_PVR_Flink PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on issue #11315: [BEAM-8889] Upgrades
gcsio to 2.1.2
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #11315: [BEAM-8889] Upgrades gcsio to 2.1.2
URL: https://github.com/apache/beam/pull/11315#issuecomment-611063709
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on issue #11315: [BEAM-8889] Upgrades
gcsio to 2.1.2
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #11315: [BEAM-8889] Upgrades gcsio to 2.1.2
URL: https://github.com/apache/beam/pull/11315#issuecomment-611066317
Run CommunityMetrics PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj merged pull request #11315: [BEAM-8889] Upgrades
gcsio to 2.1.2
Posted by GitBox <gi...@apache.org>.
chamikaramj merged pull request #11315: [BEAM-8889] Upgrades gcsio to 2.1.2
URL: https://github.com/apache/beam/pull/11315
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on issue #11315: [BEAM-8889] Upgrades
gcsio to 2.1.2
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #11315: [BEAM-8889] Upgrades gcsio to 2.1.2
URL: https://github.com/apache/beam/pull/11315#issuecomment-610644670
Run Dataflow ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] veblush commented on issue #11315: [BEAM-8889] Upgrades
gcsio to 2.1.2
Posted by GitBox <gi...@apache.org>.
veblush commented on issue #11315: [BEAM-8889] Upgrades gcsio to 2.1.2
URL: https://github.com/apache/beam/pull/11315#issuecomment-611048289
All failures other than `Java_Examples_Dataflow` don't seem related with the change. Can you run the test again?
```
# CommunityMetrics ("Run CommunityMetrics PreCommit")
Task :beam-test-infra-metrics:composeUp FAILED
-> dockerpycreds.errors.InitializationError: docker-credential-gcloud not installed or not availabl
# Java ("Run Java PreCommit")
org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactoryTest.loadBalancesBundles
-> Assert.assertTrue(System.currentTimeMillis() - tms >= 100 && closed.get());
# Java_Examples_Dataflow ("Run Java_Examples_Dataflow PreCommit")
org.apache.beam.examples.WordCountIT.testE2EWordCount
-> java.lang.RuntimeException: Failed to read from: gs://temp-storage-for-end-to-end-tests/WordCountIT-2020-04-07-22-11-56-654/output/results*-of-* with shard template '(?x) \S* (?<shardnum> \d+) -of- (?<numshards> \d+)'
# Python2_PVR_Flink ("Run Python2_PVR_Flink PreCommit")
Execution failed for task ':sdks:python:test-suites:portable:py2:flinkCompatibilityMatrixBatchLOOPBACK'.
-> ERROR:apache_beam.runners.worker.data_plane:Failed to read inputs in the data plane.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on issue #11315: [BEAM-8889] Upgrades
gcsio to 2.1.2
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #11315: [BEAM-8889] Upgrades gcsio to 2.1.2
URL: https://github.com/apache/beam/pull/11315#issuecomment-612074500
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on issue #11315: [BEAM-8889] Upgrades
gcsio to 2.1.2
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #11315: [BEAM-8889] Upgrades gcsio to 2.1.2
URL: https://github.com/apache/beam/pull/11315#issuecomment-612147550
'CommunityMetrics' failure is unrelated. Merging.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] veblush commented on issue #11315: [BEAM-8889] Upgrades
gcsio to 2.1.2
Posted by GitBox <gi...@apache.org>.
veblush commented on issue #11315: [BEAM-8889] Upgrades gcsio to 2.1.2
URL: https://github.com/apache/beam/pull/11315#issuecomment-611169834
Both `CommunityMetrics PreCommit` and `Java PreCommit` seem to fail because of environmental flakiness since those are observed by other PRs, too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on issue #11315: [BEAM-8889] Upgrades
gcsio to 2.1.2
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #11315: [BEAM-8889] Upgrades gcsio to 2.1.2
URL: https://github.com/apache/beam/pull/11315#issuecomment-611288279
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] veblush commented on issue #11315: [BEAM-8889] Upgrades
gcsio to 2.1.2
Posted by GitBox <gi...@apache.org>.
veblush commented on issue #11315: [BEAM-8889] Upgrades gcsio to 2.1.2
URL: https://github.com/apache/beam/pull/11315#issuecomment-609093332
R: @chamikaramj
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj edited a comment on issue #11315: [BEAM-8889]
Upgrades gcsio to 2.1.2
Posted by GitBox <gi...@apache.org>.
chamikaramj edited a comment on issue #11315: [BEAM-8889] Upgrades gcsio to 2.1.2
URL: https://github.com/apache/beam/pull/11315#issuecomment-611064288
Can you check if failure here is due to the library update:
https://builds.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Commit/7467/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services