You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/04/28 18:21:33 UTC
[GitHub] [beam] rehmanmuradali opened a new pull request #11559: [BEAM-9836] Exculding spark runner for KeyTests
rehmanmuradali opened a new pull request #11559:
URL: https://github.com/apache/beam/pull/11559
**Please** add a meaningful description for your change here
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] iemejia commented on pull request #11559: [BEAM-9836] Excluding spark runner for KeyTests
Posted by GitBox <gi...@apache.org>.
iemejia commented on pull request #11559:
URL: https://github.com/apache/beam/pull/11559#issuecomment-624917207
Run Spark StructuredStreaming ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] iemejia removed a comment on pull request #11559: [BEAM-9836] Excluding spark runner for KeyTests
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on pull request #11559:
URL: https://github.com/apache/beam/pull/11559#issuecomment-624916990
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] reuvenlax commented on pull request #11559: [BEAM-9836] Exculding spark runner for KeyTests
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on pull request #11559:
URL: https://github.com/apache/beam/pull/11559#issuecomment-620779552
Run Java Flink PortableValidatesRunner Streaming
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] reuvenlax commented on pull request #11559: [BEAM-9836] Exculding spark runner for KeyTests
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on pull request #11559:
URL: https://github.com/apache/beam/pull/11559#issuecomment-620778779
run spark validatesrunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] iemejia commented on pull request #11559: [BEAM-9836] Exclude Spark runner from UsesKeyInParDo tests
Posted by GitBox <gi...@apache.org>.
iemejia commented on pull request #11559:
URL: https://github.com/apache/beam/pull/11559#issuecomment-624934968
Merged manually, closing now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] reuvenlax commented on pull request #11559: [BEAM-9836] Exculding spark runner for KeyTests
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on pull request #11559:
URL: https://github.com/apache/beam/pull/11559#issuecomment-620779173
run flink portable validatesrunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rehmanmuradali commented on pull request #11559: [BEAM-9836] Excluding spark runner for KeyTests
Posted by GitBox <gi...@apache.org>.
rehmanmuradali commented on pull request #11559:
URL: https://github.com/apache/beam/pull/11559#issuecomment-622972673
> Is this only used for Timers? If so what about `UsesKeyInTimers` or `UsesKeyTimers`.
@iemejia I think `Key` also makes sense in `OnWindowExpiration`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] iemejia commented on pull request #11559: [BEAM-9836] Excluding spark runner for KeyTests
Posted by GitBox <gi...@apache.org>.
iemejia commented on pull request #11559:
URL: https://github.com/apache/beam/pull/11559#issuecomment-624915813
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] reuvenlax commented on pull request #11559: [BEAM-9836] Excluding spark runner for KeyTests
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on pull request #11559:
URL: https://github.com/apache/beam/pull/11559#issuecomment-623580541
Key also makes sense in OnWindowExpiration, and it might make sense in processElements as well. However for test exclusion categories we can create overly-specific category names.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] iemejia commented on pull request #11559: [BEAM-9836] Excluding spark runner for KeyTests
Posted by GitBox <gi...@apache.org>.
iemejia commented on pull request #11559:
URL: https://github.com/apache/beam/pull/11559#issuecomment-624917065
Run Spark ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rehmanmuradali commented on pull request #11559: [BEAM-9836] Excluding spark runner for KeyTests
Posted by GitBox <gi...@apache.org>.
rehmanmuradali commented on pull request #11559:
URL: https://github.com/apache/beam/pull/11559#issuecomment-622916178
> Can we please rename that test Category to something more clear than `UsesKey`, that name is just too general.
>
> Can you also please create a Jira issue to tackle the implementation of this functionality for the Spark runner, and eventually self assign it if you intend to work on it.
@iemejia I suggest `UsesKeyParameterInParDo`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] iemejia removed a comment on pull request #11559: [BEAM-9836] Excluding spark runner for KeyTests
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on pull request #11559:
URL: https://github.com/apache/beam/pull/11559#issuecomment-624916303
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] iemejia commented on pull request #11559: [BEAM-9836] Excluding spark runner for KeyTests
Posted by GitBox <gi...@apache.org>.
iemejia commented on pull request #11559:
URL: https://github.com/apache/beam/pull/11559#issuecomment-624916303
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rehmanmuradali edited a comment on pull request #11559: [BEAM-9836] Exculding spark runner for KeyTests
Posted by GitBox <gi...@apache.org>.
rehmanmuradali edited a comment on pull request #11559:
URL: https://github.com/apache/beam/pull/11559#issuecomment-620776856
R: @reuvenlax
CC: @iemejia, @shehzaadn-vd
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] iemejia commented on pull request #11559: [BEAM-9836] Excluding spark runner for KeyTests
Posted by GitBox <gi...@apache.org>.
iemejia commented on pull request #11559:
URL: https://github.com/apache/beam/pull/11559#issuecomment-624916990
Run Spark ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] reuvenlax commented on pull request #11559: [BEAM-9836] Excluding spark runner for KeyTests
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on pull request #11559:
URL: https://github.com/apache/beam/pull/11559#issuecomment-622649935
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] iemejia commented on pull request #11559: [BEAM-9836] Excluding spark runner for KeyTests
Posted by GitBox <gi...@apache.org>.
iemejia commented on pull request #11559:
URL: https://github.com/apache/beam/pull/11559#issuecomment-622953813
Is this only used for Timers? If so what about `UsesKeyInTimers` or `UsesKeyTimers`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] iemejia commented on pull request #11559: [BEAM-9836] Excluding spark runner for KeyTests
Posted by GitBox <gi...@apache.org>.
iemejia commented on pull request #11559:
URL: https://github.com/apache/beam/pull/11559#issuecomment-623940748
I see, thanks for the info @rehmanmuradali and @reuvenlax.
It seems this is only for `ParDo`, my point here is not to be annoying it is that I can imagine in the future someone annotating their tests with `UsesKey` just because the tests use a Key (like in KV) and I want to prevent this. What about naming it `UsesKeyInParDo` ?
Also it seems then that the current documentation for the class is not ok too, can you please also mention of the `OnWindowExpiration` and `ProcessElement` case.
```java
/**
* Category tag for validation tests which use key. Tests tagged with {@link UsesKey} should be run
* for runners which support key parameter in {@link OnTimer}.
*/
```
Thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] iemejia commented on pull request #11559: [BEAM-9836] Excluding spark runner for KeyTests
Posted by GitBox <gi...@apache.org>.
iemejia commented on pull request #11559:
URL: https://github.com/apache/beam/pull/11559#issuecomment-624917133
Run Spark StructuredStreaming ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] reuvenlax commented on pull request #11559: [BEAM-9836] Exculding spark runner for KeyTests
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on pull request #11559:
URL: https://github.com/apache/beam/pull/11559#issuecomment-620778895
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] iemejia commented on pull request #11559: [BEAM-9836] Excluding spark runner for KeyTests
Posted by GitBox <gi...@apache.org>.
iemejia commented on pull request #11559:
URL: https://github.com/apache/beam/pull/11559#issuecomment-624917455
Run Java Spark PortableValidatesRunner Batch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rehmanmuradali commented on pull request #11559: [BEAM-9836] Exculding spark runner for KeyTests
Posted by GitBox <gi...@apache.org>.
rehmanmuradali commented on pull request #11559:
URL: https://github.com/apache/beam/pull/11559#issuecomment-620776856
R: @reuvenlax
CC: @iemejia
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] iemejia commented on pull request #11559: [BEAM-9836] Excluding spark runner for KeyTests
Posted by GitBox <gi...@apache.org>.
iemejia commented on pull request #11559:
URL: https://github.com/apache/beam/pull/11559#issuecomment-624917269
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org