You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2021/07/22 18:15:26 UTC
[GitHub] [beam] riteshghorse opened a new pull request #15206: [Beam-10166] Improve execution time errors
riteshghorse opened a new pull request #15206:
URL: https://github.com/apache/beam/pull/15206
Added a base skeleton for error message returned out of DoFns. Currently only for ParDo.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
`ValidatesRunner` compliance status (on master branch)
--------------------------------------------------------
<table>
<thead>
<tr>
<th>Lang</th>
<th>ULR</th>
<th>Dataflow</th>
<th>Flink</th>
<th>Samza</th>
<th>Spark</th>
<th>Twister2</th>
</tr>
</thead>
<tbody>
<tr>
<td>Go</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon">
</a>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Samza/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
<tr>
<td>Java</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_ULR/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_ULR/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon?subject=V1">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming/lastCompletedBuild/badge/icon?subject=V1+Streaming">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/badge/icon?subject=V1+Java+11">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2_Streaming/lastCompletedBuild/badge/icon?subject=V2+Streaming">
</a><br>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon?subject=Java+8">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/badge/icon?subject=Java+11">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon?subject=Portable">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon?subject=Portable+Streaming">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Samza/lastCompletedBuild/badge/icon?subject=Portable">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon?subject=Portable">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon?subject=Structured+Streaming">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/badge/icon">
</a>
</td>
</tr>
<tr>
<td>Python</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon?subject=V1">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon?subject=ValCont">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/badge/icon?subject=Portable">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Samza/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
<tr>
<td>XLang</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Samza/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
</tbody>
</table>
Examples testing status on various runners
--------------------------------------------------------
<table>
<thead>
<tr>
<th>Lang</th>
<th>ULR</th>
<th>Dataflow</th>
<th>Flink</th>
<th>Samza</th>
<th>Spark</th>
<th>Twister2</th>
</tr>
</thead>
<tbody>
<tr>
<td>Go</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
<tr>
<td>Java</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron/lastCompletedBuild/badge/icon?subject=V1">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Java11_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Java11_Cron/lastCompletedBuild/badge/icon?subject=V1+Java11">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
</a><br>
</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
<tr>
<td>Python</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
<tr>
<td>XLang</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
</tbody>
</table>
Post-Commit SDK/Transform Integration Tests Status (on master branch)
------------------------------------------------------------------------------------------------
<table>
<thead>
<tr>
<th>Go</th>
<th>Java</th>
<th>Python</th>
</tr>
</thead>
<tbody>
<tr>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon?subject=3.6">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon?subject=3.7">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/badge/icon?subject=3.8">
</a>
</td>
</tr>
</tbody>
</table>
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
<table>
<thead>
<tr>
<th>---</th>
<th>Java</th>
<th>Python</th>
<th>Go</th>
<th>Website</th>
<th>Whitespace</th>
<th>Typescript</th>
</tr>
</thead>
<tbody>
<tr>
<td>Non-portable</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/badge/icon">
</a><br>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/badge/icon?subject=Tests">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/badge/icon?subject=Lint">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/badge/icon?subject=Docker">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/badge/icon?subject=Docs">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
</tr>
<tr>
<td>Portable</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_GoPortable_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_GoPortable_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
</tbody>
</table>
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on a change in pull request #15206: [Beam-10166] Improve execution time errors
Posted by GitBox <gi...@apache.org>.
lostluck commented on a change in pull request #15206:
URL: https://github.com/apache/beam/pull/15206#discussion_r675788964
##########
File path: sdks/go/pkg/beam/core/runtime/exec/util.go
##########
@@ -33,13 +34,29 @@ func (g *GenID) New() UnitID {
return UnitID(g.last)
}
+type doFnError struct {
+ doFn string
+ err error
+ uid UnitID
+ pid string
+}
+
+func (e *doFnError) Error() string {
+ return fmt.Sprintf("DoFn[<%d>;<%s>]<%s> returned error:<%s>", e.uid, e.pid, e.doFn, e.err)
+}
+
// callNoPanic calls the given function and catches any panic.
func callNoPanic(ctx context.Context, fn func(context.Context) error) (err error) {
defer func() {
if r := recover(); r != nil {
- // Top level error is the panic itself, but also include the stack trace as the original error.
- // Higher levels can then add appropriate context without getting pushed down by the stack trace.
- err = errors.SetTopLevelMsgf(errors.Errorf("panic: %v %s", r, debug.Stack()), "panic: %v", r)
+ // Check if the panic value is from a failed DoFn, and return it without a pancic trace.
Review comment:
Typo
```suggestion
// Check if the panic value is from a failed DoFn, and return it without a panic trace.
```
##########
File path: sdks/go/pkg/beam/core/runtime/exec/util_test.go
##########
@@ -0,0 +1,69 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package exec
+
+import (
+ "context"
+ "strings"
+ "testing"
+
+ "github.com/apache/beam/sdks/go/pkg/beam/internal/errors"
+ "github.com/apache/beam/sdks/go/pkg/beam/util/errorx"
+)
+
+// testSimpleError tests for a simple case that
+// doesn't panic
+func TestCallNoPanic_simple(t *testing.T) {
+ ctx := context.Background()
+ want := errors.New("Simple error.")
+ got := callNoPanic(ctx, func(c context.Context) error { return errors.New("Simple error.") })
+
+ if got.Error() != want.Error() {
+ t.Errorf("callNoPanic(<func that returns error>) = %v, want %v", got, want)
+ }
+}
+
+// testPanicError tests for the case in which a normal
+// error is passed to panic, resulting in panic trace.
+func TestCallNoPanic_panic(t *testing.T) {
+ ctx := context.Background()
+ got := callNoPanic(ctx, func(c context.Context) error { panic("Panic error") })
+ if !strings.Contains(got.Error(), "panic:") {
+ t.Errorf("callNoPanic(<func that panics with a string>) didn't panic, got = %v", got)
+ }
+}
+
+// testWrapPanicError tests for the case in which error
+// is passed to panic from DoFn, resulting in
+// formatted error message for DoFn.
+func TestCallNoPanic_wrappedPanic(t *testing.T) {
+ ctx := context.Background()
+ parDoError := &doFnError{
+ doFn: "sumFn",
+ err: errors.New("SumFn error"),
+ uid: 1,
+ pid: "Plan ID",
+ }
+ want := "DoFn[<1>;<Plan ID>]<sumFn> returned error:<SumFn error>"
+ var err errorx.GuardedError
+ err.TrySetError(parDoError)
+
+ got := callNoPanic(ctx, func(c context.Context) error { panic(parDoError) })
+
+ if strings.Contains(got.Error(), "panic:") {
+ t.Errorf("callNoPanic(<func that panics with a wrapped know error>) did not filter panic, want %v, got %v", want, got)
Review comment:
Typo
```suggestion
t.Errorf("callNoPanic(<func that panics with a wrapped known error>) did not filter panic, want %v, got %v", want, got)
```
##########
File path: sdks/go/pkg/beam/core/runtime/exec/util.go
##########
@@ -33,13 +34,29 @@ func (g *GenID) New() UnitID {
return UnitID(g.last)
}
+type doFnError struct {
+ doFn string
+ err error
+ uid UnitID
+ pid string
+}
+
+func (e *doFnError) Error() string {
+ return fmt.Sprintf("DoFn[<%d>;<%s>]<%s> returned error:<%s>", e.uid, e.pid, e.doFn, e.err)
Review comment:
Consider something like this instead `fmt.Sprintf("DoFn[UID:%v, PID:%v, Name: %v] failed:\n%v", e.uid, e.pid, e.doFn, e.err)`
* It cuts out extra words and structure (all the <> make it harder to read the error)
* It doesn't assume that an error was returned (it might have been caught & converted earlier)
* It puts the error on it's own line which helps readability, since DoFn names will have the full package path.
* It also allows the more specific custom printing of these types (like error) rather than making type assumptions (eg. %d and %s which have input requirements...).
* Specifically Go's fmt package has an additional verb `%v` which can print most variables.
##########
File path: sdks/go/pkg/beam/core/runtime/exec/pardo.go
##########
@@ -332,8 +332,14 @@ func (n *ParDo) postInvoke() error {
func (n *ParDo) fail(err error) error {
n.status = Broken
- n.err.TrySetError(err)
- return err
+ parDoError := &doFnError{
Review comment:
You'll want to add something like the following to the tops of these methods (or add a wrapDoFnError function for wrapping the error, consistently)
```
if err2, ok := err.(*doFnError); ok {
return err2
}
```
This filters out the known error type and prevents it from being wrapped over and over again as it's returned up the stack of DoFns.
##########
File path: sdks/go/pkg/beam/core/runtime/exec/util_test.go
##########
@@ -0,0 +1,69 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package exec
+
+import (
+ "context"
+ "strings"
+ "testing"
+
+ "github.com/apache/beam/sdks/go/pkg/beam/internal/errors"
+ "github.com/apache/beam/sdks/go/pkg/beam/util/errorx"
+)
+
+// testSimpleError tests for a simple case that
+// doesn't panic
Review comment:
Go comments don't have line length restrictions, but tend to aim for around ~80 characters. That's not a hard limit.
In this case it should be a single line.
See also https://golang.org/doc/effective_go#commentary
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] riteshghorse commented on a change in pull request #15206: [Beam-10166] Improve execution time errors
Posted by GitBox <gi...@apache.org>.
riteshghorse commented on a change in pull request #15206:
URL: https://github.com/apache/beam/pull/15206#discussion_r675630522
##########
File path: sdks/go/pkg/beam/core/runtime/exec/util_test.go
##########
@@ -0,0 +1,71 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package exec
+
+import (
+ "context"
+ "errors"
+ "strings"
+ "testing"
+
+ "github.com/apache/beam/sdks/go/pkg/beam/util/errorx"
+)
+
+// testSimpleError tests for a simple case that
+// doesn't panic
+func testSimpleError(ctx context.Context, t *testing.T) {
+ expected := errors.New("Sample error.")
+ actual := callNoPanic(ctx, func(c context.Context) error { return errors.New("Simple error.") })
+
+ if errors.Unwrap(actual) != errors.Unwrap(expected) {
+ t.Errorf("Simple error reporting failed.")
+ }
+}
+
+// testPanicError tests for the case in which a normal
+// error is passed to panic, resulting in panic trace.
+func testPanicError(ctx context.Context, t *testing.T) {
+ actual := callNoPanic(ctx, func(c context.Context) error { panic("Panic error") })
+ if !strings.Contains(actual.Error(), "panic:") {
+ t.Errorf("Caught in panic.")
+ }
+}
+
+// testWrapPanicError tests for the case in which error
+// is passed to panic from DoFn, resulting in
+// formatted error message for DoFn.
+func testWrapPanicError(ctx context.Context, t *testing.T) {
+ parDoError := doFnError{
+ doFn: "sumFn",
+ err: errors.New("SumFn error"),
+ uid: 1,
+ pid: "Plan ID",
+ }
+ var err errorx.GuardedError
+ err.TrySetError(&parDoError)
+ actual := callNoPanic(ctx, func(c context.Context) error { panic(parDoError) })
+
+ if strings.Contains(actual.Error(), "panic:") {
+ t.Errorf("Error not wrapped! Caught in panic")
+ }
+}
+
+func TestCallNoPanic(t *testing.T) {
Review comment:
Got it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on a change in pull request #15206: [Beam-10166] Improve execution time errors
Posted by GitBox <gi...@apache.org>.
lostluck commented on a change in pull request #15206:
URL: https://github.com/apache/beam/pull/15206#discussion_r675222275
##########
File path: sdks/go/pkg/beam/core/runtime/exec/util.go
##########
@@ -33,13 +34,29 @@ func (g *GenID) New() UnitID {
return UnitID(g.last)
}
+type doFnError struct {
+ doFn string
+ err error
+ uid UnitID
+ pid string
+}
+
+func (e *doFnError) Error() string {
+ return fmt.Sprintf("DoFn[<%d>;<%s>]<%s> returned error:<%s>", e.uid, e.pid, e.doFn, e.err)
+}
+
// callNoPanic calls the given function and catches any panic.
func callNoPanic(ctx context.Context, fn func(context.Context) error) (err error) {
defer func() {
if r := recover(); r != nil {
- // Top level error is the panic itself, but also include the stack trace as the original error.
- // Higher levels can then add appropriate context without getting pushed down by the stack trace.
- err = errors.SetTopLevelMsgf(errors.Errorf("panic: %v %s", r, debug.Stack()), "panic: %v", r)
+ // check if error is of type DoFn then handle it appropriately
Review comment:
Please change this comment to
```suggestion
// Check if the panic value is from a failed DoFn, and return it without a panic trace.
```
A comment saying 'handle it appropriately' doesn't communicate anything to the reader. What does that mean?
##########
File path: sdks/go/pkg/beam/core/runtime/exec/util_test.go
##########
@@ -0,0 +1,71 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package exec
+
+import (
+ "context"
+ "errors"
+ "strings"
+ "testing"
+
+ "github.com/apache/beam/sdks/go/pkg/beam/util/errorx"
+)
+
+// testSimpleError tests for a simple case that
+// doesn't panic
+func testSimpleError(ctx context.Context, t *testing.T) {
+ expected := errors.New("Sample error.")
+ actual := callNoPanic(ctx, func(c context.Context) error { return errors.New("Simple error.") })
+
+ if errors.Unwrap(actual) != errors.Unwrap(expected) {
+ t.Errorf("Simple error reporting failed.")
+ }
+}
+
+// testPanicError tests for the case in which a normal
+// error is passed to panic, resulting in panic trace.
+func testPanicError(ctx context.Context, t *testing.T) {
+ actual := callNoPanic(ctx, func(c context.Context) error { panic("Panic error") })
+ if !strings.Contains(actual.Error(), "panic:") {
+ t.Errorf("Caught in panic.")
+ }
+}
+
+// testWrapPanicError tests for the case in which error
+// is passed to panic from DoFn, resulting in
+// formatted error message for DoFn.
+func testWrapPanicError(ctx context.Context, t *testing.T) {
+ parDoError := doFnError{
+ doFn: "sumFn",
+ err: errors.New("SumFn error"),
+ uid: 1,
+ pid: "Plan ID",
+ }
+ var err errorx.GuardedError
+ err.TrySetError(&parDoError)
+ actual := callNoPanic(ctx, func(c context.Context) error { panic(parDoError) })
+
+ if strings.Contains(actual.Error(), "panic:") {
+ t.Errorf("Error not wrapped! Caught in panic")
+ }
+}
+
+func TestCallNoPanic(t *testing.T) {
Review comment:
Split this into the 3 separate test functions instead of doing it like this. There's no reason to make this a "single" test.
In particular use an underscore to describe the cases
func TestCallNoPanic_simpleError(t *testing.T) { ...}
func TestCallNoPanic_panic(t *testing.T) { ...}
func TestCallNoPanic_wrappedPanic(t *testing.T) { ...}
Testing in Go is well described in the testing package overview.: https://pkg.go.dev/testing#pkg-overview
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] riteshghorse commented on pull request #15206: [Beam-10166] Improve execution time errors
Posted by GitBox <gi...@apache.org>.
riteshghorse commented on pull request #15206:
URL: https://github.com/apache/beam/pull/15206#issuecomment-885204790
​retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on a change in pull request #15206: [Beam-10166] Improve execution time errors
Posted by GitBox <gi...@apache.org>.
lostluck commented on a change in pull request #15206:
URL: https://github.com/apache/beam/pull/15206#discussion_r675716887
##########
File path: sdks/go/pkg/beam/core/runtime/exec/util_test.go
##########
@@ -0,0 +1,67 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package exec
+
+import (
+ "context"
+ "strings"
+ "testing"
+
+ "github.com/apache/beam/sdks/go/pkg/beam/internal/errors"
+ "github.com/apache/beam/sdks/go/pkg/beam/util/errorx"
+)
+
+// testSimpleError tests for a simple case that
+// doesn't panic
+func TestCallNoPanic_simple(t *testing.T) {
+ ctx := context.Background()
+ expected := errors.New("Simple error.")
+ actual := callNoPanic(ctx, func(c context.Context) error { return errors.New("Simple error.") })
Review comment:
Style nit: In go we use "got" and "want" rather than actual, expected. They're shorter, which makes them easier to read.
##########
File path: sdks/go/pkg/beam/core/runtime/exec/util.go
##########
@@ -33,13 +34,29 @@ func (g *GenID) New() UnitID {
return UnitID(g.last)
}
+type doFnError struct {
+ doFn string
+ err error
+ uid UnitID
+ pid string
+}
+
+func (e *doFnError) Error() string {
+ return fmt.Sprintf("DoFn[<%d>;<%s>]<%s> returned error:<%s>", e.uid, e.pid, e.doFn, e.err)
+}
+
// callNoPanic calls the given function and catches any panic.
func callNoPanic(ctx context.Context, fn func(context.Context) error) (err error) {
defer func() {
if r := recover(); r != nil {
- // Top level error is the panic itself, but also include the stack trace as the original error.
- // Higher levels can then add appropriate context without getting pushed down by the stack trace.
- err = errors.SetTopLevelMsgf(errors.Errorf("panic: %v %s", r, debug.Stack()), "panic: %v", r)
+ // Check if the panic value is from a failed DoFn, and return it without a pancic trace.
+ if e, ok := r.(doFnError); ok {
Review comment:
I don't think this line runs as expected: in the fail() call in pardo.go it's passed a *doFnError to the guarded error, but then it's type asserting the recovery value here as the non-pointer type. And then in the _wrappedPanic case, a non-pointer doFnError is used instead of the pointer, not replicating what's happening in fail.
##########
File path: sdks/go/pkg/beam/core/runtime/exec/util_test.go
##########
@@ -0,0 +1,67 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package exec
+
+import (
+ "context"
+ "strings"
+ "testing"
+
+ "github.com/apache/beam/sdks/go/pkg/beam/internal/errors"
+ "github.com/apache/beam/sdks/go/pkg/beam/util/errorx"
+)
+
+// testSimpleError tests for a simple case that
+// doesn't panic
+func TestCallNoPanic_simple(t *testing.T) {
+ ctx := context.Background()
+ expected := errors.New("Simple error.")
+ actual := callNoPanic(ctx, func(c context.Context) error { return errors.New("Simple error.") })
+
+ if actual.Error() != expected.Error() {
+ t.Errorf("Simple error reporting failed.")
Review comment:
Test error messages should identify the function under test, and the inputs to that function. In this case consider changing it to:
`t.Errorf("callNoPanic(<func that returns error>) = %v, want %v", got, want)`
The inputs here are more than we want to write (function pointers don't have a meaninful thing to print), but the specifics don't matter in this case, just that the function actually returns an error.
##########
File path: sdks/go/pkg/beam/core/runtime/exec/util_test.go
##########
@@ -0,0 +1,67 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package exec
+
+import (
+ "context"
+ "strings"
+ "testing"
+
+ "github.com/apache/beam/sdks/go/pkg/beam/internal/errors"
+ "github.com/apache/beam/sdks/go/pkg/beam/util/errorx"
+)
+
+// testSimpleError tests for a simple case that
+// doesn't panic
+func TestCallNoPanic_simple(t *testing.T) {
+ ctx := context.Background()
+ expected := errors.New("Simple error.")
+ actual := callNoPanic(ctx, func(c context.Context) error { return errors.New("Simple error.") })
+
+ if actual.Error() != expected.Error() {
+ t.Errorf("Simple error reporting failed.")
+ }
+}
+
+// testPanicError tests for the case in which a normal
+// error is passed to panic, resulting in panic trace.
+func TestCallNoPanic_panic(t *testing.T) {
+ ctx := context.Background()
+ actual := callNoPanic(ctx, func(c context.Context) error { panic("Panic error") })
+ if !strings.Contains(actual.Error(), "panic:") {
+ t.Errorf("Panic reporting failed.")
Review comment:
Similarly:
`t.Errorf("callNoPanic(<func that panics with a string>) didn't contain panic, got = %v", got)`
Note that in this case since we know the panic trace could be very long, or mis formatted, it's better to indicate the expectation before including what we actually got.
##########
File path: sdks/go/pkg/beam/core/runtime/exec/util_test.go
##########
@@ -0,0 +1,67 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package exec
+
+import (
+ "context"
+ "strings"
+ "testing"
+
+ "github.com/apache/beam/sdks/go/pkg/beam/internal/errors"
+ "github.com/apache/beam/sdks/go/pkg/beam/util/errorx"
+)
+
+// testSimpleError tests for a simple case that
+// doesn't panic
+func TestCallNoPanic_simple(t *testing.T) {
+ ctx := context.Background()
+ expected := errors.New("Simple error.")
+ actual := callNoPanic(ctx, func(c context.Context) error { return errors.New("Simple error.") })
+
+ if actual.Error() != expected.Error() {
+ t.Errorf("Simple error reporting failed.")
+ }
+}
+
+// testPanicError tests for the case in which a normal
+// error is passed to panic, resulting in panic trace.
+func TestCallNoPanic_panic(t *testing.T) {
+ ctx := context.Background()
+ actual := callNoPanic(ctx, func(c context.Context) error { panic("Panic error") })
+ if !strings.Contains(actual.Error(), "panic:") {
+ t.Errorf("Panic reporting failed.")
+ }
+}
+
+// testWrapPanicError tests for the case in which error
+// is passed to panic from DoFn, resulting in
+// formatted error message for DoFn.
+func TestCallNoPanic_wrappedPanic(t *testing.T) {
+ ctx := context.Background()
+ parDoError := doFnError{
+ doFn: "sumFn",
+ err: errors.New("SumFn error"),
+ uid: 1,
+ pid: "Plan ID",
+ }
+ var err errorx.GuardedError
+ err.TrySetError(&parDoError)
+ actual := callNoPanic(ctx, func(c context.Context) error { panic(parDoError) })
+
+ if strings.Contains(actual.Error(), "panic:") {
+ t.Errorf("Error not wrapped! Caught in panic.")
Review comment:
Similarly:
`t.Errorf("callNoPanic(<func that panics with a wrapped known error>) did not filter panic, want %v, got %", want, got)`
Usually test messages should match the "simple" case above, with the got coming before the want. However, in this case, since we know the got will be longer to print out when this test fails (since it presumably has a panic trace), it's better to print the want before the got.
##########
File path: sdks/go/pkg/beam/core/runtime/exec/util_test.go
##########
@@ -0,0 +1,67 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package exec
+
+import (
+ "context"
+ "strings"
+ "testing"
+
+ "github.com/apache/beam/sdks/go/pkg/beam/internal/errors"
+ "github.com/apache/beam/sdks/go/pkg/beam/util/errorx"
+)
+
+// testSimpleError tests for a simple case that
+// doesn't panic
+func TestCallNoPanic_simple(t *testing.T) {
+ ctx := context.Background()
+ expected := errors.New("Simple error.")
+ actual := callNoPanic(ctx, func(c context.Context) error { return errors.New("Simple error.") })
+
+ if actual.Error() != expected.Error() {
+ t.Errorf("Simple error reporting failed.")
+ }
+}
+
+// testPanicError tests for the case in which a normal
+// error is passed to panic, resulting in panic trace.
+func TestCallNoPanic_panic(t *testing.T) {
+ ctx := context.Background()
+ actual := callNoPanic(ctx, func(c context.Context) error { panic("Panic error") })
+ if !strings.Contains(actual.Error(), "panic:") {
+ t.Errorf("Panic reporting failed.")
+ }
+}
+
+// testWrapPanicError tests for the case in which error
+// is passed to panic from DoFn, resulting in
+// formatted error message for DoFn.
+func TestCallNoPanic_wrappedPanic(t *testing.T) {
+ ctx := context.Background()
+ parDoError := doFnError{
Review comment:
Make it a pointer type here, and remove the & passed to the TrySetError call.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] riteshghorse commented on a change in pull request #15206: [Beam-10166] Improve execution time errors
Posted by GitBox <gi...@apache.org>.
riteshghorse commented on a change in pull request #15206:
URL: https://github.com/apache/beam/pull/15206#discussion_r675734013
##########
File path: sdks/go/pkg/beam/core/runtime/exec/util.go
##########
@@ -33,13 +34,29 @@ func (g *GenID) New() UnitID {
return UnitID(g.last)
}
+type doFnError struct {
+ doFn string
+ err error
+ uid UnitID
+ pid string
+}
+
+func (e *doFnError) Error() string {
+ return fmt.Sprintf("DoFn[<%d>;<%s>]<%s> returned error:<%s>", e.uid, e.pid, e.doFn, e.err)
+}
+
// callNoPanic calls the given function and catches any panic.
func callNoPanic(ctx context.Context, fn func(context.Context) error) (err error) {
defer func() {
if r := recover(); r != nil {
- // Top level error is the panic itself, but also include the stack trace as the original error.
- // Higher levels can then add appropriate context without getting pushed down by the stack trace.
- err = errors.SetTopLevelMsgf(errors.Errorf("panic: %v %s", r, debug.Stack()), "panic: %v", r)
+ // Check if the panic value is from a failed DoFn, and return it without a pancic trace.
+ if e, ok := r.(doFnError); ok {
Review comment:
Yes, I missed this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on pull request #15206: [Beam-10166] Improve execution time errors
Posted by GitBox <gi...@apache.org>.
lostluck commented on pull request #15206:
URL: https://github.com/apache/beam/pull/15206#issuecomment-886880333
Run GoPortable PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] riteshghorse commented on pull request #15206: [Beam-10166] Improve execution time errors
Posted by GitBox <gi...@apache.org>.
riteshghorse commented on pull request #15206:
URL: https://github.com/apache/beam/pull/15206#issuecomment-885215482
R: @lostluck
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck merged pull request #15206: [Beam-10166] Improve execution time errors
Posted by GitBox <gi...@apache.org>.
lostluck merged pull request #15206:
URL: https://github.com/apache/beam/pull/15206
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] riteshghorse commented on a change in pull request #15206: [Beam-10166] Improve execution time errors
Posted by GitBox <gi...@apache.org>.
riteshghorse commented on a change in pull request #15206:
URL: https://github.com/apache/beam/pull/15206#discussion_r675630785
##########
File path: sdks/go/pkg/beam/core/runtime/exec/util.go
##########
@@ -33,13 +34,29 @@ func (g *GenID) New() UnitID {
return UnitID(g.last)
}
+type doFnError struct {
+ doFn string
+ err error
+ uid UnitID
+ pid string
+}
+
+func (e *doFnError) Error() string {
+ return fmt.Sprintf("DoFn[<%d>;<%s>]<%s> returned error:<%s>", e.uid, e.pid, e.doFn, e.err)
+}
+
// callNoPanic calls the given function and catches any panic.
func callNoPanic(ctx context.Context, fn func(context.Context) error) (err error) {
defer func() {
if r := recover(); r != nil {
- // Top level error is the panic itself, but also include the stack trace as the original error.
- // Higher levels can then add appropriate context without getting pushed down by the stack trace.
- err = errors.SetTopLevelMsgf(errors.Errorf("panic: %v %s", r, debug.Stack()), "panic: %v", r)
+ // check if error is of type DoFn then handle it appropriately
Review comment:
Right, should have explained it more clearly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on pull request #15206: [Beam-10166] Improve execution time errors
Posted by GitBox <gi...@apache.org>.
lostluck commented on pull request #15206:
URL: https://github.com/apache/beam/pull/15206#issuecomment-885935780
The licence failure is unrelated to this PR at least.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org