You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/04/28 05:36:00 UTC
[GitHub] [beam] youngoli opened a new pull request #11553: [BEAM-9799] Automated validation for RTrackers.
youngoli opened a new pull request #11553:
URL: https://github.com/apache/beam/pull/11553
Adds automated validation for RTrackers to the DoFn invokers. I ran the
ParDo benchmark with and without my change and got similar results, so
this doesn't seem to have an obvious performance impact.
Note that the currently documented benchmark number doesn't seem accurate before (or it's just my desktop that's different).
Benchmark before change:
BenchmarkParDo_EmitSumFn-6 948087 1918 ns/op 484 B/op 4 allocs/op
Benchmark after change:
`BenchmarkParDo_EmitSumFn-6 681343 1919 ns/op 521 B/op 4 allocs/op`
Note that exact numbers wavered a bunch. It seemed to average around 1900 ns/op, going as low as 1850 and up to 1950. I chose two examples that had similar runtimes.
The error message emitted looks like this to the end-user:
```
...
Caused by: java.lang.RuntimeException: Error received from SDK harness for instruction 2: process bundle failed for instruction 2 using plan 5-6 : <emitted error>
...
```
I was toying with making it a bit more descriptive, but it matches the structure you get when the user emits an error from ProcessElement, so I kept it as is for consistency.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [x] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] youngoli commented on pull request #11553: [BEAM-9799] Automated validation for RTrackers.
Posted by GitBox <gi...@apache.org>.
youngoli commented on pull request #11553:
URL: https://github.com/apache/beam/pull/11553#issuecomment-621568657
Completely forgot to actually send this for review.
R: @lostluck
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on a change in pull request #11553: [BEAM-9799] Automated validation for RTrackers.
Posted by GitBox <gi...@apache.org>.
lostluck commented on a change in pull request #11553:
URL: https://github.com/apache/beam/pull/11553#discussion_r419623182
##########
File path: sdks/go/pkg/beam/core/runtime/exec/pardo.go
##########
@@ -162,6 +168,14 @@ func (n *ParDo) processMainInput(mainIn *MainInput) error {
return nil
}
+func rtErrHelper(err error) error {
+ if err != nil {
+ return err
+ } else {
+ return errors.New("RTracker IsDone failed for unspecified reason")
+ }
Review comment:
```suggestion
}
return errors.New("RTracker.IsDone() failed for unspecified reason")
```
##########
File path: sdks/go/pkg/beam/core/runtime/exec/pardo.go
##########
@@ -162,6 +168,14 @@ func (n *ParDo) processMainInput(mainIn *MainInput) error {
return nil
}
+func rtErrHelper(err error) error {
+ if err != nil {
+ return err
+ } else {
+ return errors.New("RTracker IsDone failed for unspecified reason")
+ }
Review comment:
Also, in both places this is called, error or no, we don't know IsDone failed(), we know that the DoFn terminated without fully processing the restriction (If I understand IsDone's semantics correctly). Perhaps the message should indicate that?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] youngoli commented on a change in pull request #11553: [BEAM-9799] Automated validation for RTrackers.
Posted by GitBox <gi...@apache.org>.
youngoli commented on a change in pull request #11553:
URL: https://github.com/apache/beam/pull/11553#discussion_r419743023
##########
File path: sdks/go/pkg/beam/core/runtime/exec/pardo.go
##########
@@ -162,6 +168,14 @@ func (n *ParDo) processMainInput(mainIn *MainInput) error {
return nil
}
+func rtErrHelper(err error) error {
+ if err != nil {
+ return err
+ } else {
+ return errors.New("RTracker IsDone failed for unspecified reason")
+ }
Review comment:
Yeah good point. I'll rephrase it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org