You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2021/07/15 15:54:08 UTC
[GitHub] [beam] jrmccluskey opened a new pull request #15175: [BEAM-12548] Implement EqualsFloat test helper
jrmccluskey opened a new pull request #15175:
URL: https://github.com/apache/beam/pull/15175
Adds the EqualsFloat function, which ensures that all elements of a PCollection are within a given threshold of all of the elements in another PCollection. Also adds unit tests.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
`ValidatesRunner` compliance status (on master branch)
--------------------------------------------------------
<table>
<thead>
<tr>
<th>Lang</th>
<th>ULR</th>
<th>Dataflow</th>
<th>Flink</th>
<th>Samza</th>
<th>Spark</th>
<th>Twister2</th>
</tr>
</thead>
<tbody>
<tr>
<td>Go</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon">
</a>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Samza/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
<tr>
<td>Java</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_ULR/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_ULR/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon?subject=V1">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming/lastCompletedBuild/badge/icon?subject=V1+Streaming">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/badge/icon?subject=V1+Java+11">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2_Streaming/lastCompletedBuild/badge/icon?subject=V2+Streaming">
</a><br>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon?subject=Java+8">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/badge/icon?subject=Java+11">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon?subject=Portable">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon?subject=Portable+Streaming">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Samza/lastCompletedBuild/badge/icon?subject=Portable">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon?subject=Portable">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon?subject=Structured+Streaming">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/badge/icon">
</a>
</td>
</tr>
<tr>
<td>Python</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon?subject=V1">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon?subject=ValCont">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/badge/icon?subject=Portable">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
<tr>
<td>XLang</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
</tbody>
</table>
Examples testing status on various runners
--------------------------------------------------------
<table>
<thead>
<tr>
<th>Lang</th>
<th>ULR</th>
<th>Dataflow</th>
<th>Flink</th>
<th>Samza</th>
<th>Spark</th>
<th>Twister2</th>
</tr>
</thead>
<tbody>
<tr>
<td>Go</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
<tr>
<td>Java</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron/lastCompletedBuild/badge/icon?subject=V1">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Java11_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Java11_Cron/lastCompletedBuild/badge/icon?subject=V1+Java11">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
</a><br>
</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
<tr>
<td>Python</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
<tr>
<td>XLang</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
</tbody>
</table>
Post-Commit SDK/Transform Integration Tests Status (on master branch)
------------------------------------------------------------------------------------------------
<table>
<thead>
<tr>
<th>Go</th>
<th>Java</th>
<th>Python</th>
</tr>
</thead>
<tbody>
<tr>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon?subject=3.6">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon?subject=3.7">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/badge/icon?subject=3.8">
</a>
</td>
</tr>
</tbody>
</table>
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
<table>
<thead>
<tr>
<th>---</th>
<th>Java</th>
<th>Python</th>
<th>Go</th>
<th>Website</th>
<th>Whitespace</th>
<th>Typescript</th>
</tr>
</thead>
<tbody>
<tr>
<td>Non-portable</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/badge/icon">
</a><br>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/badge/icon?subject=Tests">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/badge/icon?subject=Lint">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/badge/icon?subject=Docker">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/badge/icon?subject=Docs">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
</tr>
<tr>
<td>Portable</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_GoPortable_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_GoPortable_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
</tbody>
</table>
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on a change in pull request #15175: [BEAM-12548] Implement EqualsFloat test helper
Posted by GitBox <gi...@apache.org>.
lostluck commented on a change in pull request #15175:
URL: https://github.com/apache/beam/pull/15175#discussion_r671343729
##########
File path: sdks/go/pkg/beam/testing/passert/floats.go
##########
@@ -26,6 +26,56 @@ import (
"github.com/apache/beam/sdks/go/pkg/beam/internal/errors"
)
+// EqualsFloat checks that two PCollections of floats are equal, with each element
+// being within a specified threshold of its corresponding element. Both PCollections
+// are loaded into memory, sorted, and compared element by element.
+func EqualsFloat(s beam.Scope, observed, expected beam.PCollection, threshold float64) {
Review comment:
Oh good point there. I would not try and capture errors from ParDo0. It would largely fail if the thresholdFn had any output PCollections, which we can see statically it doesn't, and the additional validation is sufficient to cover other errors it might pull since they are more specific than what is checked by ParDo0 at construction.
The examples are largely the beam.ParDo* functions themselves. There's a single general "TryParDo" that does all the work for the ParDo* functions. See https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/pardo.go#L86
I don't recommend duplicating all of that code or using the TryParDo there, just showing the example. Another example is TryExternal: https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/external.go#L38
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] jrmccluskey commented on pull request #15175: [BEAM-12548] Implement EqualsFloat test helper
Posted by GitBox <gi...@apache.org>.
jrmccluskey commented on pull request #15175:
URL: https://github.com/apache/beam/pull/15175#issuecomment-880820961
R: @lostluck
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] jrmccluskey commented on a change in pull request #15175: [BEAM-12548] Implement EqualsFloat test helper
Posted by GitBox <gi...@apache.org>.
jrmccluskey commented on a change in pull request #15175:
URL: https://github.com/apache/beam/pull/15175#discussion_r671334420
##########
File path: sdks/go/pkg/beam/testing/passert/floats.go
##########
@@ -26,6 +26,56 @@ import (
"github.com/apache/beam/sdks/go/pkg/beam/internal/errors"
)
+// EqualsFloat checks that two PCollections of floats are equal, with each element
+// being within a specified threshold of its corresponding element. Both PCollections
+// are loaded into memory, sorted, and compared element by element.
+func EqualsFloat(s beam.Scope, observed, expected beam.PCollection, threshold float64) {
Review comment:
Added the validation (and split out into a helper since AllWithinBounds did the same thing anyway.)
Is there a reference for splitting a DoFn into that Try function? I haven't been able to find another example of that design pattern in the SDK and I'm not seeing any examples of pulling the errors out of the ParDo0 call
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on a change in pull request #15175: [BEAM-12548] Implement EqualsFloat test helper
Posted by GitBox <gi...@apache.org>.
lostluck commented on a change in pull request #15175:
URL: https://github.com/apache/beam/pull/15175#discussion_r670813578
##########
File path: sdks/go/pkg/beam/testing/passert/floats.go
##########
@@ -26,6 +26,56 @@ import (
"github.com/apache/beam/sdks/go/pkg/beam/internal/errors"
)
+// EqualsFloat checks that two PCollections of floats are equal, with each element
+// being within a specified threshold of its corresponding element. Both PCollections
+// are loaded into memory, sorted, and compared element by element.
+func EqualsFloat(s beam.Scope, observed, expected beam.PCollection, threshold float64) {
+ s = s.Scope(fmt.Sprintf("passert.EqualsFloat[%v]", threshold))
+ beam.ParDo0(s, &thresholdFn{threshold: threshold}, beam.Impulse(s), beam.SideInput{Input: observed}, beam.SideInput{Input: expected})
+}
+
+type thresholdFn struct {
+ threshold float64
+}
+
+func (f *thresholdFn) ProcessElement(_ []byte, observed, expected func(*beam.T) bool) error {
+ var observedValues, expectedValues []float64
+ var observedInput, expectedInput beam.T
+ for observed(&observedInput) {
+ val := reflect.ValueOf(observedInput.(interface{})).Convert(reflectx.Float64).Interface().(float64)
Review comment:
Since we now have 4 places where this reflective snippet is used, consider writing a "toFloat" function to abstract it, making the calling code easier to read.
##########
File path: sdks/go/pkg/beam/testing/passert/floats.go
##########
@@ -26,6 +26,56 @@ import (
"github.com/apache/beam/sdks/go/pkg/beam/internal/errors"
)
+// EqualsFloat checks that two PCollections of floats are equal, with each element
+// being within a specified threshold of its corresponding element. Both PCollections
+// are loaded into memory, sorted, and compared element by element.
+func EqualsFloat(s beam.Scope, observed, expected beam.PCollection, threshold float64) {
+ s = s.Scope(fmt.Sprintf("passert.EqualsFloat[%v]", threshold))
+ beam.ParDo0(s, &thresholdFn{threshold: threshold}, beam.Impulse(s), beam.SideInput{Input: observed}, beam.SideInput{Input: expected})
+}
+
+type thresholdFn struct {
+ threshold float64
Review comment:
Export your Field names please.
While passert functions are intended to be run on single machines for testing, unexported Fields are never serialized as part of the DoFn, which makes them not function as intended on Portable runners.
Basically, all the rules the compiler enforces about Exported and unexported names, are respected by the underlying "reflect" package as well for access. This means that unexported fields can't be read and then subsequently written to reflectively, effectively dropping their data entirely.
It's legal to have DoFns that only have unexported fields (though it probably shouldn't be), and we likely should add a validation that there's at least one exported field for types used as PCollection elements. It's my number one comment to users with issues...
(unexported type and function names are fine though)
##########
File path: sdks/go/pkg/beam/testing/passert/floats.go
##########
@@ -26,6 +26,56 @@ import (
"github.com/apache/beam/sdks/go/pkg/beam/internal/errors"
)
+// EqualsFloat checks that two PCollections of floats are equal, with each element
+// being within a specified threshold of its corresponding element. Both PCollections
+// are loaded into memory, sorted, and compared element by element.
+func EqualsFloat(s beam.Scope, observed, expected beam.PCollection, threshold float64) {
Review comment:
As a second note to make this easier: I recommend putting the main implementation code into a "TryEqualsFloats" function that returns an error, with the "EqualsFloats" function calling that, panicking if the returned error isn't nil.
This lets you test your validation error without complex "defer recover" code, while letting users who accept the runtime panic use the shorter function call, with users who want to handle the error themselves use the "Try" call.
##########
File path: sdks/go/pkg/beam/testing/passert/floats.go
##########
@@ -26,6 +26,56 @@ import (
"github.com/apache/beam/sdks/go/pkg/beam/internal/errors"
)
+// EqualsFloat checks that two PCollections of floats are equal, with each element
+// being within a specified threshold of its corresponding element. Both PCollections
+// are loaded into memory, sorted, and compared element by element.
+func EqualsFloat(s beam.Scope, observed, expected beam.PCollection, threshold float64) {
Review comment:
We should likely check that both the observed and expected PCollection types are convertable to floats at pipeline construction time.
This is the tricky bit with leaning on "beam.T" in the DoFn's Process Element function. It doesn't validate all the types. As the transform and the code is written, the framework can only validate that the two side input types are identical, but this will pass construction if you provide two PCollections of strings, but will fail at pipeline runtime, which is undesirable.
So we should check it when the pipeline is constructed to fail as soon as possible. This is a guiding principle: If you can turn a Pipeline runtime failure into a Pipeline construction failure, Do it.
eg, [`beam.ValidateNonCompositeType` ](https://github.com/apache/beam/blob/c9f2321d70cac02aa59bd7ea4877cf27209ae4bc/sdks/go/pkg/beam/validate.go#L38) gets the reflect.Type out of the PCollection, which can then be checked to have a number underneath it or similar. (ala [validateNonComplexNumber](https://github.com/apache/beam/blob/c9f2321d70cac02aa59bd7ea4877cf27209ae4bc/sdks/go/pkg/beam/transforms/stats/util.go#L54) from the stats transforms package.
(and I just noticed that there's an extra "not" in the comment for that *sigh* )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck merged pull request #15175: [BEAM-12548] Implement EqualsFloat test helper
Posted by GitBox <gi...@apache.org>.
lostluck merged pull request #15175:
URL: https://github.com/apache/beam/pull/15175
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck merged pull request #15175: [BEAM-12548] Implement EqualsFloat test helper
Posted by GitBox <gi...@apache.org>.
lostluck merged pull request #15175:
URL: https://github.com/apache/beam/pull/15175
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] jrmccluskey commented on a change in pull request #15175: [BEAM-12548] Implement EqualsFloat test helper
Posted by GitBox <gi...@apache.org>.
jrmccluskey commented on a change in pull request #15175:
URL: https://github.com/apache/beam/pull/15175#discussion_r671281436
##########
File path: sdks/go/pkg/beam/testing/passert/floats.go
##########
@@ -26,6 +26,56 @@ import (
"github.com/apache/beam/sdks/go/pkg/beam/internal/errors"
)
+// EqualsFloat checks that two PCollections of floats are equal, with each element
+// being within a specified threshold of its corresponding element. Both PCollections
+// are loaded into memory, sorted, and compared element by element.
+func EqualsFloat(s beam.Scope, observed, expected beam.PCollection, threshold float64) {
+ s = s.Scope(fmt.Sprintf("passert.EqualsFloat[%v]", threshold))
+ beam.ParDo0(s, &thresholdFn{threshold: threshold}, beam.Impulse(s), beam.SideInput{Input: observed}, beam.SideInput{Input: expected})
+}
+
+type thresholdFn struct {
+ threshold float64
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck merged pull request #15175: [BEAM-12548] Implement EqualsFloat test helper
Posted by GitBox <gi...@apache.org>.
lostluck merged pull request #15175:
URL: https://github.com/apache/beam/pull/15175
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on a change in pull request #15175: [BEAM-12548] Implement EqualsFloat test helper
Posted by GitBox <gi...@apache.org>.
lostluck commented on a change in pull request #15175:
URL: https://github.com/apache/beam/pull/15175#discussion_r671346363
##########
File path: sdks/go/pkg/beam/testing/passert/floats.go
##########
@@ -70,3 +122,13 @@ func (f *boundsFn) ProcessElement(_ []byte, col func(*beam.T) bool) error {
}
return errors.New(strings.Join(errorStrings, "\n"))
}
+
+func toFloat(input beam.T) float64 {
+ return reflect.ValueOf(input.(interface{})).Convert(reflectx.Float64).Interface().(float64)
+}
+
+func validateNonComplexNumber(t reflect.Type) {
+ if !reflectx.IsNumber(t) || reflectx.IsComplex(t) {
+ panic(fmt.Sprintf("type must be a non-complex number: %v", t))
Review comment:
In this case, return an error, so the caller can add additional information to the error. "the observed PCollection elements have an incompatible type" or "the expected PCollection elements have an incompatible type" .
Good error messages make good frameworks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] jrmccluskey commented on a change in pull request #15175: [BEAM-12548] Implement EqualsFloat test helper
Posted by GitBox <gi...@apache.org>.
jrmccluskey commented on a change in pull request #15175:
URL: https://github.com/apache/beam/pull/15175#discussion_r671383900
##########
File path: sdks/go/pkg/beam/testing/passert/floats.go
##########
@@ -70,3 +122,13 @@ func (f *boundsFn) ProcessElement(_ []byte, col func(*beam.T) bool) error {
}
return errors.New(strings.Join(errorStrings, "\n"))
}
+
+func toFloat(input beam.T) float64 {
+ return reflect.ValueOf(input.(interface{})).Convert(reflectx.Float64).Interface().(float64)
+}
+
+func validateNonComplexNumber(t reflect.Type) {
+ if !reflectx.IsNumber(t) || reflectx.IsComplex(t) {
+ panic(fmt.Sprintf("type must be a non-complex number: %v", t))
Review comment:
Changed
##########
File path: sdks/go/pkg/beam/testing/passert/floats.go
##########
@@ -26,6 +26,56 @@ import (
"github.com/apache/beam/sdks/go/pkg/beam/internal/errors"
)
+// EqualsFloat checks that two PCollections of floats are equal, with each element
+// being within a specified threshold of its corresponding element. Both PCollections
+// are loaded into memory, sorted, and compared element by element.
+func EqualsFloat(s beam.Scope, observed, expected beam.PCollection, threshold float64) {
Review comment:
Done and unit tested
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] jrmccluskey commented on a change in pull request #15175: [BEAM-12548] Implement EqualsFloat test helper
Posted by GitBox <gi...@apache.org>.
jrmccluskey commented on a change in pull request #15175:
URL: https://github.com/apache/beam/pull/15175#discussion_r671281580
##########
File path: sdks/go/pkg/beam/testing/passert/floats.go
##########
@@ -26,6 +26,56 @@ import (
"github.com/apache/beam/sdks/go/pkg/beam/internal/errors"
)
+// EqualsFloat checks that two PCollections of floats are equal, with each element
+// being within a specified threshold of its corresponding element. Both PCollections
+// are loaded into memory, sorted, and compared element by element.
+func EqualsFloat(s beam.Scope, observed, expected beam.PCollection, threshold float64) {
+ s = s.Scope(fmt.Sprintf("passert.EqualsFloat[%v]", threshold))
+ beam.ParDo0(s, &thresholdFn{threshold: threshold}, beam.Impulse(s), beam.SideInput{Input: observed}, beam.SideInput{Input: expected})
+}
+
+type thresholdFn struct {
+ threshold float64
+}
+
+func (f *thresholdFn) ProcessElement(_ []byte, observed, expected func(*beam.T) bool) error {
+ var observedValues, expectedValues []float64
+ var observedInput, expectedInput beam.T
+ for observed(&observedInput) {
+ val := reflect.ValueOf(observedInput.(interface{})).Convert(reflectx.Float64).Interface().(float64)
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org