You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/11/03 22:18:40 UTC
[GitHub] [beam] tysonjh opened a new pull request #13251: Add Dataflow Runner V2 ValidatesRunner streaming test configuration.
tysonjh opened a new pull request #13251:
URL: https://github.com/apache/beam/pull/13251
WIP - add Dataflow Runner V2 VR streaming tests and refactor build.gradle for reuse.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website | Whitespace | Typescript
--- | --- | --- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) <br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | --- | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on pull request #13251: Add Dataflow Runner V2 Java ValidatesRunner streaming test configuration.
Posted by GitBox <gi...@apache.org>.
kennknowles commented on pull request #13251:
URL: https://github.com/apache/beam/pull/13251#issuecomment-721419136
Seems like a change we can merge and check, since tweaking Jenkins.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tysonjh commented on pull request #13251: Add Dataflow Runner V2 ValidatesRunner streaming test configuration.
Posted by GitBox <gi...@apache.org>.
tysonjh commented on pull request #13251:
URL: https://github.com/apache/beam/pull/13251#issuecomment-721406075
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tysonjh commented on pull request #13251: Add Dataflow Runner V2 ValidatesRunner streaming test configuration.
Posted by GitBox <gi...@apache.org>.
tysonjh commented on pull request #13251:
URL: https://github.com/apache/beam/pull/13251#issuecomment-721414265
R: @kennknowles
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] udim commented on pull request #13251: Add Dataflow Runner V2 Java ValidatesRunner streaming test configuration.
Posted by GitBox <gi...@apache.org>.
udim commented on pull request #13251:
URL: https://github.com/apache/beam/pull/13251#issuecomment-726305100
I believe this PR has caused beam_PostCommit_Java_VR_Dataflow_V2 to start timing out (by adding more tests?)
https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/
I assume this PR belongs to https://issues.apache.org/jira/browse/BEAM-10967 ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles merged pull request #13251: Add Dataflow Runner V2 Java ValidatesRunner streaming test configuration.
Posted by GitBox <gi...@apache.org>.
kennknowles merged pull request #13251:
URL: https://github.com/apache/beam/pull/13251
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tysonjh commented on pull request #13251: Add Dataflow Runner V2 Java ValidatesRunner streaming test configuration.
Posted by GitBox <gi...@apache.org>.
tysonjh commented on pull request #13251:
URL: https://github.com/apache/beam/pull/13251#issuecomment-721847927
> Thanks! I'll monitor it.
This broke things but I'm investigating. I think it is because the containers weren't built, the `dependsOn buildAndPushDockerContainer` was put in the wrong spot.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tysonjh commented on pull request #13251: Add Dataflow Runner V2 ValidatesRunner streaming test configuration.
Posted by GitBox <gi...@apache.org>.
tysonjh commented on pull request #13251:
URL: https://github.com/apache/beam/pull/13251#issuecomment-721412377
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tysonjh commented on pull request #13251: Add Dataflow Runner V2 Java ValidatesRunner streaming test configuration.
Posted by GitBox <gi...@apache.org>.
tysonjh commented on pull request #13251:
URL: https://github.com/apache/beam/pull/13251#issuecomment-726310705
> I believe this PR has caused beam_PostCommit_Java_VR_Dataflow_V2 to start timing out (by adding more tests?)
> https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/
>
> I assume this PR belongs to https://issues.apache.org/jira/browse/BEAM-10967 ?
I'll take a look. The tests were consistently failing before and I've been trying to fix them (this PR) but it is slow since VR takes ~4-6hrs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tysonjh commented on pull request #13251: Add Dataflow Runner V2 Java ValidatesRunner streaming test configuration.
Posted by GitBox <gi...@apache.org>.
tysonjh commented on pull request #13251:
URL: https://github.com/apache/beam/pull/13251#issuecomment-726949436
> > I believe this PR has caused beam_PostCommit_Java_VR_Dataflow_V2 to start timing out (by adding more tests?)
> > https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/
> > I assume this PR belongs to https://issues.apache.org/jira/browse/BEAM-10967 ?
>
> I'll take a look. The tests were consistently failing before and I've been trying to fix them (this PR) but it is slow since VR takes ~4-6hrs.
I've updated the exclude lists. This was failing because there is a UsesTestStream test in the ValueState tests. I've added the exclude category to the streaming VR tests in the other PR I have open to fix other failures: https://github.com/apache/beam/pull/13259
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tysonjh commented on pull request #13251: Add Dataflow Runner V2 Java ValidatesRunner streaming test configuration.
Posted by GitBox <gi...@apache.org>.
tysonjh commented on pull request #13251:
URL: https://github.com/apache/beam/pull/13251#issuecomment-721419419
Thanks! I'll monitor it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tysonjh commented on pull request #13251: Add Dataflow Runner V2 ValidatesRunner streaming test configuration.
Posted by GitBox <gi...@apache.org>.
tysonjh commented on pull request #13251:
URL: https://github.com/apache/beam/pull/13251#issuecomment-721413900
Run Java Dataflow V2 ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org