You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/11/24 21:57:05 UTC
[GitHub] [beam] ibzib opened a new pull request #13419: [BEAM-10278] Move Dataflow v2 cross-language tests out of runner module.
ibzib opened a new pull request #13419:
URL: https://github.com/apache/beam/pull/13419
R: @kennknowles @ihji
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam
.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website | Whitespace | Typescript
--- | --- | --- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) <br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | --- | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib closed pull request #13419: [BEAM-10278] Move Dataflow v2 cross-language tests out of runner module.
Posted by GitBox <gi...@apache.org>.
ibzib closed pull request #13419:
URL: https://github.com/apache/beam/pull/13419
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ihji edited a comment on pull request #13419: [BEAM-10278] Move Dataflow v2 cross-language tests out of runner module.
Posted by GitBox <gi...@apache.org>.
ihji edited a comment on pull request #13419:
URL: https://github.com/apache/beam/pull/13419#issuecomment-733366568
Please don't merge this yet. I wanted try the other solution before merging this. We may also enable JavaUsingPython tests later and it will be little strange if we move this whole task under python test-suite.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib commented on pull request #13419: [BEAM-10278] Move Dataflow v2 cross-language tests out of runner module.
Posted by GitBox <gi...@apache.org>.
ibzib commented on pull request #13419:
URL: https://github.com/apache/beam/pull/13419#issuecomment-733273771
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib commented on pull request #13419: [BEAM-10278] Move Dataflow v2 cross-language tests out of runner module.
Posted by GitBox <gi...@apache.org>.
ibzib commented on pull request #13419:
URL: https://github.com/apache/beam/pull/13419#issuecomment-733263629
R: @amaliujia
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib commented on pull request #13419: [BEAM-10278] Move Dataflow v2 cross-language tests out of runner module.
Posted by GitBox <gi...@apache.org>.
ibzib commented on pull request #13419:
URL: https://github.com/apache/beam/pull/13419#issuecomment-733281861
BEAM-10921 strikes again!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib commented on pull request #13419: [BEAM-10278] Move Dataflow v2 cross-language tests out of runner module.
Posted by GitBox <gi...@apache.org>.
ibzib commented on pull request #13419:
URL: https://github.com/apache/beam/pull/13419#issuecomment-733386643
> It looks like the root cause is that we don't have python build.gradle in google3. I'm trying to skip createCrossLanguageValidatesRunnerTask if :sdks:python subproject doesn't exist. It's okay since we don't use that task in google3.
That would probably work, but I'm not sure it's the best solution. I prefer moving the tests to `dataflow/common.gradle` because that way all of the tests that depend on both the Dataflow runner and Python are in the same place.
Also note that `dataflow/common.gradle` already transitively depends on the Java SDK, so this PR is only removing dependencies through consolidation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib commented on pull request #13419: [BEAM-10278] Move Dataflow v2 cross-language tests out of runner module.
Posted by GitBox <gi...@apache.org>.
ibzib commented on pull request #13419:
URL: https://github.com/apache/beam/pull/13419#issuecomment-733256365
run seed job
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ihji commented on pull request #13419: [BEAM-10278] Move Dataflow v2 cross-language tests out of runner module.
Posted by GitBox <gi...@apache.org>.
ihji commented on pull request #13419:
URL: https://github.com/apache/beam/pull/13419#issuecomment-733378821
> > I wanted try the other solution before merging this.
>
> What other solution do you have in mind?
>
> I agree that it is kind of awkward to have runner tests in the Python module, but I don't know of a better place to put them currently, since these tests inherently need dependencies from across SDKs and runners.
It looks like the root cause is that we don't have python build.gradle in google3. I'm trying to skip createCrossLanguageValidatesRunnerTask if :sdks:python subproject doesn't exist. It's okay since we don't use that task in google3.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib commented on pull request #13419: [BEAM-10278] Move Dataflow v2 cross-language tests out of runner module.
Posted by GitBox <gi...@apache.org>.
ibzib commented on pull request #13419:
URL: https://github.com/apache/beam/pull/13419#issuecomment-733373967
> I wanted try the other solution before merging this.
What other solution do you have in mind?
I agree that it is kind of awkward to have runner tests in the Python module, but I don't know of a better place to put them currently, since these tests inherently need dependencies from across SDKs and runners.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib commented on pull request #13419: [BEAM-10278] Move Dataflow v2 cross-language tests out of runner module.
Posted by GitBox <gi...@apache.org>.
ibzib commented on pull request #13419:
URL: https://github.com/apache/beam/pull/13419#issuecomment-733840068
Okay, let's go with #13425 instead. We can always revisit this or a similar solution later if we need to for some reason.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ihji commented on pull request #13419: [BEAM-10278] Move Dataflow v2 cross-language tests out of runner module.
Posted by GitBox <gi...@apache.org>.
ihji commented on pull request #13419:
URL: https://github.com/apache/beam/pull/13419#issuecomment-733455263
> > It looks like the root cause is that we don't have python build.gradle in google3. I'm trying to skip createCrossLanguageValidatesRunnerTask if :sdks:python subproject doesn't exist. It's okay since we don't use that task in google3.
>
> That would probably work, but I'm not sure it's the best solution. I prefer moving the tests to `dataflow/common.gradle` because that way all of the tests that depend on both the Dataflow runner and Python are in the same place.
I see. Please note that `createCrossLanguageValidatesRunnerTask` creates not only the python tests but also java tests. When we put `createCrossLanguageValidatesRunnerTask` in the build file, it creates four variants of xlang tests:
- Java pipeline using Python external transforms
- Java pipeline using Java external transforms
- Python pipeline using Python external transforms
- Python pipeline using Java external transforms
It's not totally impossible but it may look awkward to launch Java pipelines using Java external transforms from python:test-suites module.
>
> Also note that `dataflow/common.gradle` already transitively depends on the Java SDK, so this PR is only removing dependencies through consolidation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ihji commented on pull request #13419: [BEAM-10278] Move Dataflow v2 cross-language tests out of runner module.
Posted by GitBox <gi...@apache.org>.
ihji commented on pull request #13419:
URL: https://github.com/apache/beam/pull/13419#issuecomment-733457541
> > > I wanted try the other solution before merging this.
> >
> >
> > What other solution do you have in mind?
> > I agree that it is kind of awkward to have runner tests in the Python module, but I don't know of a better place to put them currently, since these tests inherently need dependencies from across SDKs and runners.
>
> It looks like the root cause is that we don't have python build.gradle in google3. I'm trying to skip createCrossLanguageValidatesRunnerTask if :sdks:python subproject doesn't exist. It's okay since we don't use that task in google3.
Alternative fix: https://github.com/apache/beam/pull/13425
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib commented on pull request #13419: [BEAM-10278] Move Dataflow v2 cross-language tests out of runner module.
Posted by GitBox <gi...@apache.org>.
ibzib commented on pull request #13419:
URL: https://github.com/apache/beam/pull/13419#issuecomment-733261237
Run XVR_Dataflow PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ihji commented on pull request #13419: [BEAM-10278] Move Dataflow v2 cross-language tests out of runner module.
Posted by GitBox <gi...@apache.org>.
ihji commented on pull request #13419:
URL: https://github.com/apache/beam/pull/13419#issuecomment-733366568
Please wait for merging this yet. I wanted try the other solution before merging this. We may also enable JavaUsingPython tests later and it will be little strange if we move this whole task under python test-suite.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org