You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2021/03/24 09:57:11 UTC
[GitHub] [beam] Amar3tto opened a new pull request #14322: [BEAM-449] Support PCollectionList in PAssert
Amar3tto opened a new pull request #14322:
URL: https://github.com/apache/beam/pull/14322
* Added `PAssert.thatFlattened()` method that runs an assertion on the flattened contents of the `PCollectionList`.
* Added `PAssert.thatList()` method that could run an assertion on each `PCollection` in the `PCollectionList` using one or multiple matchers.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
<table>
<thead>
<tr>
<th>Lang</th>
<th>SDK</th>
<th>ULR</th>
<th>Dataflow</th>
<th>Flink</th>
<th>Samza</th>
<th>Spark</th>
<th>Twister2</th>
</tr>
</thead>
<tbody>
<tr>
<td>Go</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/">
<img src="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
<tr>
<td>Java</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_ULR/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_ULR/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2_Streaming/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/badge/icon">
</a>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Java11_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Java11_Cron/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Dataflow_V2/lastCompletedBuild/badge/icon">
</a><br>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/badge/icon">
</a>
</td>
</tr>
<tr>
<td>Python</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
<tr>
<td>XLang</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
</tbody>
</table>
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
<table>
<thead>
<tr>
<th>---</th>
<th>Java</th>
<th>Python</th>
<th>Go</th>
<th>Website</th>
<th>Whitespace</th>
<th>Typescript</th>
</tr>
</thead>
<tbody>
<tr>
<td>Non-portable</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/badge/icon">
</a><br>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/">
<img src="https://camo.githubusercontent.com/4565d7b7e907114e6c1d12323408bd903aa252fefed5eeab93701b05c9628a84/68747470733a2f2f63692d6265616d2e6170616368652e6f72672f6a6f622f6265616d5f507265436f6d6d69745f507974686f6e446f636b65725f43726f6e2f62616467652f69636f6e" alt="Build Status" data-canonical-src="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/">
<img src="https://camo.githubusercontent.com/21afb097a5745440598bee5c59a027b140585eec871c3f3b883200610fabf722/68747470733a2f2f63692d6265616d2e6170616368652e6f72672f6a6f622f6265616d5f507265436f6d6d69745f507974686f6e446f63735f43726f6e2f62616467652f69636f6e" alt="Build Status" data-canonical-src="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
</tr>
<tr>
<td>Portable</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
</tbody>
</table>
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on a change in pull request #14322: [BEAM-449] Support PCollectionList in PAssert
Posted by GitBox <gi...@apache.org>.
kennknowles commented on a change in pull request #14322:
URL: https://github.com/apache/beam/pull/14322#discussion_r602433042
##########
File path: sdks/java/core/src/test/java/org/apache/beam/sdk/testing/PAssertTest.java
##########
@@ -596,4 +598,138 @@ public void countAssertsMultipleCallsIndependent() {
assertThat(PAssert.countAsserts(pipeline), equalTo(3));
}
+
+ @Test
+ public void testPAssertThatFlattened() {
Review comment:
I think this needs `@Category(NeedsRunner.class)` which will cause it to be executed on the direct runner in the `:runners:direct-java:needsRunnerTests` task. See here: https://scans.gradle.com/s/rlhjzeofa5ytc/tests?toggled=WyI6cnVubmVyczpkaXJlY3QtamF2YS1uZWVkc3J1bm5lcnRlc3RzLW9yZy5hcGFjaGUuYmVhbS5zZGsuaW8iLCI6cnVubmVyczpkaXJlY3QtamF2YS1uZWVkc3J1bm5lcnRlc3RzLW9yZy5hcGFjaGUuYmVhbS5zZGsudHJhbnNmb3JtcyIsIjpydW5uZXJzOmRpcmVjdC1qYXZhLW5lZWRzcnVubmVydGVzdHMtb3JnLmFwYWNoZS5iZWFtLnNkay5zY2hlbWFzLnRyYW5zZm9ybXMiXQ#:runners:direct-java-needsrunnertests-org.apache.beam.sdk.testing-passerttest
This test was executed in 66ms according to Jenkins but in fact the `:sdks:java:core:test` task was pulled FROM-CACHE as I guess a prior run on this PR populated the cache: https://scans.gradle.com/s/rlhjzeofa5ytc/timeline#cblz6an6jai7a
I am actually surprised it did not fail since it should have abandoned node enforcement, which is a feature of the `@Rule TestPipeline` to protect users from writing tests but forgetting to run the pipeline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Amar3tto commented on a change in pull request #14322: [BEAM-449] Support PCollectionList in PAssert
Posted by GitBox <gi...@apache.org>.
Amar3tto commented on a change in pull request #14322:
URL: https://github.com/apache/beam/pull/14322#discussion_r603251873
##########
File path: sdks/java/core/src/test/java/org/apache/beam/sdk/testing/PAssertTest.java
##########
@@ -596,4 +598,138 @@ public void countAssertsMultipleCallsIndependent() {
assertThat(PAssert.countAsserts(pipeline), equalTo(3));
}
+
+ @Test
+ public void testPAssertThatFlattened() {
Review comment:
Thank you for the explanation!
I fixed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Amar3tto commented on a change in pull request #14322: [BEAM-449] Support PCollectionList in PAssert
Posted by GitBox <gi...@apache.org>.
Amar3tto commented on a change in pull request #14322:
URL: https://github.com/apache/beam/pull/14322#discussion_r600748615
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java
##########
@@ -512,8 +513,72 @@ public int hashCode() {
PAssertionSite.capture(reason));
}
+ /**
+ * Constructs an {@link PCollectionListContentsAssert} for the provided {@link PCollectionList}.
+ */
+ public static <T> PCollectionListContentsAssert<T> thatList(PCollectionList<T> actual) {
+ return new PCollectionListContentsAssert<>(actual);
+ }
+
+ /**
+ * Constructs an {@link IterableAssert} for the elements of the flattened {@link PCollectionList}.
+ */
+ public static <T> IterableAssert<T> thatFlattened(PCollectionList<T> actual) {
+ PCollection<T> flatten = actual.apply(Flatten.pCollections());
+ return that(flatten.getName(), flatten);
+ }
+
+ /**
+ * Constructs an {@link IterableAssert} for the elements of the flattened {@link PCollectionList}
+ * with the specified reason.
+ */
+ public static <T> IterableAssert<T> thatFlattened(String reason, PCollectionList<T> actual) {
+ return new PCollectionContentsAssert<>(
+ actual.apply(Flatten.pCollections()), PAssertionSite.capture(reason));
+ }
+
////////////////////////////////////////////////////////////
+ /**
+ * An assert about the contents of each {@link PCollection} in the given {@link PCollectionList}.
+ */
+ protected static class PCollectionListContentsAssert<T> {
+ private final PCollectionList<T> pCollectionList;
+
+ public PCollectionListContentsAssert(PCollectionList<T> actual) {
+ this.pCollectionList = actual;
+ }
+
+ /**
+ * Applies one {@link SerializableFunction} to check the elements of each {@code PCollection} in
+ * the {@link #pCollectionList}.
+ *
+ * <p>Returns this {@code PCollectionListContentsAssert}.
+ */
+ public PCollectionListContentsAssert<T> satisfies(
+ SerializableFunction<Iterable<T>, Void> checkerFn) {
+ for (int i = 0; i < pCollectionList.size(); i++) {
+ PAssert.that(pCollectionList.get(i)).satisfies(checkerFn);
+ }
+ return this;
+ }
+
+ /**
+ * Takes list of {@link SerializableFunction}s of the same size as {@link #pCollectionList}, and
+ * applies each matcher to the {@code PCollection} with the identical index in the {@link
+ * #pCollectionList}.
+ *
+ * <p>Returns this {@code PCollectionListContentsAssert}.
+ */
+ public PCollectionListContentsAssert<T> satisfies(
+ List<SerializableFunction<Iterable<T>, Void>> checkerFnList) {
+ for (int i = 0; i < pCollectionList.size(); i++) {
+ PAssert.that(pCollectionList.get(i)).satisfies(checkerFnList.get(i));
Review comment:
Done
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java
##########
@@ -512,8 +513,72 @@ public int hashCode() {
PAssertionSite.capture(reason));
}
+ /**
+ * Constructs an {@link PCollectionListContentsAssert} for the provided {@link PCollectionList}.
+ */
+ public static <T> PCollectionListContentsAssert<T> thatList(PCollectionList<T> actual) {
+ return new PCollectionListContentsAssert<>(actual);
+ }
+
+ /**
+ * Constructs an {@link IterableAssert} for the elements of the flattened {@link PCollectionList}.
+ */
+ public static <T> IterableAssert<T> thatFlattened(PCollectionList<T> actual) {
+ PCollection<T> flatten = actual.apply(Flatten.pCollections());
+ return that(flatten.getName(), flatten);
+ }
+
+ /**
+ * Constructs an {@link IterableAssert} for the elements of the flattened {@link PCollectionList}
+ * with the specified reason.
+ */
+ public static <T> IterableAssert<T> thatFlattened(String reason, PCollectionList<T> actual) {
+ return new PCollectionContentsAssert<>(
+ actual.apply(Flatten.pCollections()), PAssertionSite.capture(reason));
+ }
+
////////////////////////////////////////////////////////////
+ /**
+ * An assert about the contents of each {@link PCollection} in the given {@link PCollectionList}.
+ */
+ protected static class PCollectionListContentsAssert<T> {
+ private final PCollectionList<T> pCollectionList;
+
+ public PCollectionListContentsAssert(PCollectionList<T> actual) {
+ this.pCollectionList = actual;
+ }
+
+ /**
+ * Applies one {@link SerializableFunction} to check the elements of each {@code PCollection} in
+ * the {@link #pCollectionList}.
Review comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] KhaninArtur commented on a change in pull request #14322: [BEAM-449] Support PCollectionList in PAssert
Posted by GitBox <gi...@apache.org>.
KhaninArtur commented on a change in pull request #14322:
URL: https://github.com/apache/beam/pull/14322#discussion_r600517005
##########
File path: sdks/java/core/src/test/java/org/apache/beam/sdk/testing/PAssertTest.java
##########
@@ -596,4 +598,59 @@ public void countAssertsMultipleCallsIndependent() {
assertThat(PAssert.countAsserts(pipeline), equalTo(3));
}
+
+ @Test
+ public void testPAssertThatFlattened() {
+ PCollection<Integer> firstCollection = pipeline.apply("FirstCreate", Create.of(1, 2, 3));
+ PCollection<Integer> secondCollection = pipeline.apply("SecondCreate", Create.of(4, 5, 6));
+
+ PCollectionList<Integer> collectionList =
+ PCollectionList.of(firstCollection).and(secondCollection);
+
+ PAssert.thatFlattened(collectionList).containsInAnyOrder(1, 2, 3, 4, 5, 6);
+ PAssert.thatFlattened("Reason", collectionList).containsInAnyOrder(1, 2, 3, 4, 5, 6);
+ }
+
+ @Test
+ public void testPAssertThatListSatisfiesOneMatcher() {
+ PCollection<Integer> firstCollection = pipeline.apply("FirstCreate", Create.of(1, 2, 3));
+ PCollection<Integer> secondCollection = pipeline.apply("SecondCreate", Create.of(4, 5, 6));
+
+ PCollectionList<Integer> collectionList =
+ PCollectionList.of(firstCollection).and(secondCollection);
+
+ PAssert.thatList(collectionList)
+ .satisfies(
+ input -> {
+ for (Integer element : input) {
+ assertTrue(element > 0);
+ }
+ return null;
+ });
+ }
+
+ @Test
+ public void testPAssertThatListSatisfiesMultipleMatchers() {
+ PCollection<Integer> firstCollection = pipeline.apply("FirstCreate", Create.of(1, 2, 3));
+ PCollection<Integer> secondCollection = pipeline.apply("SecondCreate", Create.of(4, 5, 6));
+
+ PCollectionList<Integer> collectionList =
+ PCollectionList.of(firstCollection).and(secondCollection);
+
+ PAssert.thatList(collectionList)
+ .satisfies(
+ ImmutableList.of(
+ input -> {
+ for (Integer element : input) {
+ assertTrue(element < 4);
+ }
+ return null;
+ },
+ input -> {
+ for (Integer element : input) {
+ assertTrue(element < 7);
+ }
+ return null;
+ }));
+ }
Review comment:
Should we also check that PAssert fails correctly? I.e. add negative test cases?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles merged pull request #14322: [BEAM-449] Support PCollectionList in PAssert
Posted by GitBox <gi...@apache.org>.
kennknowles merged pull request #14322:
URL: https://github.com/apache/beam/pull/14322
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] KhaninArtur commented on a change in pull request #14322: [BEAM-449] Support PCollectionList in PAssert
Posted by GitBox <gi...@apache.org>.
KhaninArtur commented on a change in pull request #14322:
URL: https://github.com/apache/beam/pull/14322#discussion_r600470178
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java
##########
@@ -512,8 +513,72 @@ public int hashCode() {
PAssertionSite.capture(reason));
}
+ /**
+ * Constructs an {@link PCollectionListContentsAssert} for the provided {@link PCollectionList}.
+ */
+ public static <T> PCollectionListContentsAssert<T> thatList(PCollectionList<T> actual) {
+ return new PCollectionListContentsAssert<>(actual);
+ }
+
+ /**
+ * Constructs an {@link IterableAssert} for the elements of the flattened {@link PCollectionList}.
+ */
+ public static <T> IterableAssert<T> thatFlattened(PCollectionList<T> actual) {
+ PCollection<T> flatten = actual.apply(Flatten.pCollections());
+ return that(flatten.getName(), flatten);
+ }
+
+ /**
+ * Constructs an {@link IterableAssert} for the elements of the flattened {@link PCollectionList}
+ * with the specified reason.
+ */
+ public static <T> IterableAssert<T> thatFlattened(String reason, PCollectionList<T> actual) {
+ return new PCollectionContentsAssert<>(
+ actual.apply(Flatten.pCollections()), PAssertionSite.capture(reason));
+ }
+
////////////////////////////////////////////////////////////
+ /**
+ * An assert about the contents of each {@link PCollection} in the given {@link PCollectionList}.
+ */
+ protected static class PCollectionListContentsAssert<T> {
+ private final PCollectionList<T> pCollectionList;
+
+ public PCollectionListContentsAssert(PCollectionList<T> actual) {
+ this.pCollectionList = actual;
+ }
+
+ /**
+ * Applies one {@link SerializableFunction} to check the elements of each {@code PCollection} in
+ * the {@link #pCollectionList}.
Review comment:
Shouldn't it be PCollectionList?
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java
##########
@@ -512,8 +513,72 @@ public int hashCode() {
PAssertionSite.capture(reason));
}
+ /**
+ * Constructs an {@link PCollectionListContentsAssert} for the provided {@link PCollectionList}.
+ */
+ public static <T> PCollectionListContentsAssert<T> thatList(PCollectionList<T> actual) {
+ return new PCollectionListContentsAssert<>(actual);
+ }
+
+ /**
+ * Constructs an {@link IterableAssert} for the elements of the flattened {@link PCollectionList}.
+ */
+ public static <T> IterableAssert<T> thatFlattened(PCollectionList<T> actual) {
+ PCollection<T> flatten = actual.apply(Flatten.pCollections());
+ return that(flatten.getName(), flatten);
+ }
+
+ /**
+ * Constructs an {@link IterableAssert} for the elements of the flattened {@link PCollectionList}
+ * with the specified reason.
+ */
+ public static <T> IterableAssert<T> thatFlattened(String reason, PCollectionList<T> actual) {
+ return new PCollectionContentsAssert<>(
+ actual.apply(Flatten.pCollections()), PAssertionSite.capture(reason));
+ }
+
////////////////////////////////////////////////////////////
+ /**
+ * An assert about the contents of each {@link PCollection} in the given {@link PCollectionList}.
+ */
+ protected static class PCollectionListContentsAssert<T> {
+ private final PCollectionList<T> pCollectionList;
+
+ public PCollectionListContentsAssert(PCollectionList<T> actual) {
+ this.pCollectionList = actual;
+ }
+
+ /**
+ * Applies one {@link SerializableFunction} to check the elements of each {@code PCollection} in
+ * the {@link #pCollectionList}.
+ *
+ * <p>Returns this {@code PCollectionListContentsAssert}.
+ */
+ public PCollectionListContentsAssert<T> satisfies(
+ SerializableFunction<Iterable<T>, Void> checkerFn) {
+ for (int i = 0; i < pCollectionList.size(); i++) {
+ PAssert.that(pCollectionList.get(i)).satisfies(checkerFn);
+ }
+ return this;
+ }
+
+ /**
+ * Takes list of {@link SerializableFunction}s of the same size as {@link #pCollectionList}, and
+ * applies each matcher to the {@code PCollection} with the identical index in the {@link
+ * #pCollectionList}.
+ *
+ * <p>Returns this {@code PCollectionListContentsAssert}.
+ */
+ public PCollectionListContentsAssert<T> satisfies(
+ List<SerializableFunction<Iterable<T>, Void>> checkerFnList) {
+ for (int i = 0; i < pCollectionList.size(); i++) {
+ PAssert.that(pCollectionList.get(i)).satisfies(checkerFnList.get(i));
Review comment:
Is there somewhere any validation that the size of the collection list and checker list are equal?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] daria-malkova commented on pull request #14322: [BEAM-449] Support PCollectionList in PAssert
Posted by GitBox <gi...@apache.org>.
daria-malkova commented on pull request #14322:
URL: https://github.com/apache/beam/pull/14322#issuecomment-811767505
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] daria-malkova removed a comment on pull request #14322: [BEAM-449] Support PCollectionList in PAssert
Posted by GitBox <gi...@apache.org>.
daria-malkova removed a comment on pull request #14322:
URL: https://github.com/apache/beam/pull/14322#issuecomment-811767505
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on pull request #14322: [BEAM-449] Support PCollectionList in PAssert
Posted by GitBox <gi...@apache.org>.
kennknowles commented on pull request #14322:
URL: https://github.com/apache/beam/pull/14322#issuecomment-815291827
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Amar3tto commented on pull request #14322: [BEAM-449] Support PCollectionList in PAssert
Posted by GitBox <gi...@apache.org>.
Amar3tto commented on pull request #14322:
URL: https://github.com/apache/beam/pull/14322#issuecomment-807353234
R: @lukecwik
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Amar3tto commented on pull request #14322: [BEAM-449] Support PCollectionList in PAssert
Posted by GitBox <gi...@apache.org>.
Amar3tto commented on pull request #14322:
URL: https://github.com/apache/beam/pull/14322#issuecomment-814730267
Hi, @kennknowles! Is there anything else I should update for this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on pull request #14322: [BEAM-449] Support PCollectionList in PAssert
Posted by GitBox <gi...@apache.org>.
kennknowles commented on pull request #14322:
URL: https://github.com/apache/beam/pull/14322#issuecomment-815154464
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on a change in pull request #14322: [BEAM-449] Support PCollectionList in PAssert
Posted by GitBox <gi...@apache.org>.
kennknowles commented on a change in pull request #14322:
URL: https://github.com/apache/beam/pull/14322#discussion_r608936771
##########
File path: sdks/java/core/src/test/java/org/apache/beam/sdk/testing/PAssertTest.java
##########
@@ -596,4 +598,138 @@ public void countAssertsMultipleCallsIndependent() {
assertThat(PAssert.countAsserts(pipeline), equalTo(3));
}
+
+ @Test
+ public void testPAssertThatFlattened() {
Review comment:
I see you added `ValidatesRunner` not `NeedsRunner`. This is actually OK. PAssert is not a primitive transform, so it does not fit the category. But a few runners do treat it special so it is good to include.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on pull request #14322: [BEAM-449] Support PCollectionList in PAssert
Posted by GitBox <gi...@apache.org>.
kennknowles commented on pull request #14322:
URL: https://github.com/apache/beam/pull/14322#issuecomment-815154765
run twister2 validatesrunner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Amar3tto commented on a change in pull request #14322: [BEAM-449] Support PCollectionList in PAssert
Posted by GitBox <gi...@apache.org>.
Amar3tto commented on a change in pull request #14322:
URL: https://github.com/apache/beam/pull/14322#discussion_r601415096
##########
File path: sdks/java/core/src/test/java/org/apache/beam/sdk/testing/PAssertTest.java
##########
@@ -596,4 +598,59 @@ public void countAssertsMultipleCallsIndependent() {
assertThat(PAssert.countAsserts(pipeline), equalTo(3));
}
+
+ @Test
+ public void testPAssertThatFlattened() {
+ PCollection<Integer> firstCollection = pipeline.apply("FirstCreate", Create.of(1, 2, 3));
+ PCollection<Integer> secondCollection = pipeline.apply("SecondCreate", Create.of(4, 5, 6));
+
+ PCollectionList<Integer> collectionList =
+ PCollectionList.of(firstCollection).and(secondCollection);
+
+ PAssert.thatFlattened(collectionList).containsInAnyOrder(1, 2, 3, 4, 5, 6);
+ PAssert.thatFlattened("Reason", collectionList).containsInAnyOrder(1, 2, 3, 4, 5, 6);
+ }
+
+ @Test
+ public void testPAssertThatListSatisfiesOneMatcher() {
+ PCollection<Integer> firstCollection = pipeline.apply("FirstCreate", Create.of(1, 2, 3));
+ PCollection<Integer> secondCollection = pipeline.apply("SecondCreate", Create.of(4, 5, 6));
+
+ PCollectionList<Integer> collectionList =
+ PCollectionList.of(firstCollection).and(secondCollection);
+
+ PAssert.thatList(collectionList)
+ .satisfies(
+ input -> {
+ for (Integer element : input) {
+ assertTrue(element > 0);
+ }
+ return null;
+ });
+ }
+
+ @Test
+ public void testPAssertThatListSatisfiesMultipleMatchers() {
+ PCollection<Integer> firstCollection = pipeline.apply("FirstCreate", Create.of(1, 2, 3));
+ PCollection<Integer> secondCollection = pipeline.apply("SecondCreate", Create.of(4, 5, 6));
+
+ PCollectionList<Integer> collectionList =
+ PCollectionList.of(firstCollection).and(secondCollection);
+
+ PAssert.thatList(collectionList)
+ .satisfies(
+ ImmutableList.of(
+ input -> {
+ for (Integer element : input) {
+ assertTrue(element < 4);
+ }
+ return null;
+ },
+ input -> {
+ for (Integer element : input) {
+ assertTrue(element < 7);
+ }
+ return null;
+ }));
+ }
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Amar3tto commented on pull request #14322: [BEAM-449] Support PCollectionList in PAssert
Posted by GitBox <gi...@apache.org>.
Amar3tto commented on pull request #14322:
URL: https://github.com/apache/beam/pull/14322#issuecomment-806764782
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on pull request #14322: [BEAM-449] Support PCollectionList in PAssert
Posted by GitBox <gi...@apache.org>.
kennknowles commented on pull request #14322:
URL: https://github.com/apache/beam/pull/14322#issuecomment-815155095
Run ULR Loopback ValidatesRunner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Amar3tto commented on pull request #14322: [BEAM-449] Support PCollectionList in PAssert
Posted by GitBox <gi...@apache.org>.
Amar3tto commented on pull request #14322:
URL: https://github.com/apache/beam/pull/14322#issuecomment-809310949
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org