You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/12/04 10:34:46 UTC
[GitHub] [beam] kamilwu opened a new pull request #13484: [BEAM-11075] Return valid error on job failures
kamilwu opened a new pull request #13484:
URL: https://github.com/apache/beam/pull/13484
The 'err' object was being overridden by the output of newUniversalPipelineResult. As a result, the user was not able to catch any job errors.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam
.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website | Whitespace | Typescript
--- | --- | --- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) <br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | --- | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on a change in pull request #13484: [BEAM-11207] Return valid error on job failures
Posted by GitBox <gi...@apache.org>.
lostluck commented on a change in pull request #13484:
URL: https://github.com/apache/beam/pull/13484#discussion_r536493356
##########
File path: sdks/go/pkg/beam/runners/universal/runnerlib/execute.go
##########
@@ -94,11 +94,16 @@ func Execute(ctx context.Context, p *pipepb.Pipeline, endpoint string, opt *JobO
}
err = WaitForCompletion(ctx, client, jobID)
- res, err := newUniversalPipelineResult(ctx, jobID, client)
- if err != nil {
- return presult, err
+ res, presultErr := newUniversalPipelineResult(ctx, jobID, client)
+ if presultErr != nil {
+ if err != nil {
+ return presult, errors.Wrap(err, presultErr.Error())
+ } else {
+ return presult, presultErr
+ }
+ } else {
+ presult = res
}
Review comment:
In Go, there's no reason to try to have a single return, or maintain an else.
In Go, prefer having the happiest path have the least indentation, and strive to return to that lowest level. Simply return, and avoid the extra indentation of an else.
An if else block implies that either clause can be executed and those possibilities need to be kept in mind when reading later code. If the if returns (or continues, or breaks, as appropriate), there's no need for an else, as the code in the If never interacts with anything after it.
```suggestion
res, presultErr := newUniversalPipelineResult(ctx, jobID, client)
if presultErr != nil {
if err != nil {
return presult, errors.Wrap(err, presultErr.Error())
}
return presult, presultErr
}
return res, err
```
This in combination with my other suggestion below to remove the other line.
##########
File path: sdks/go/pkg/beam/runners/universal/runnerlib/execute.go
##########
@@ -94,11 +94,16 @@ func Execute(ctx context.Context, p *pipepb.Pipeline, endpoint string, opt *JobO
}
err = WaitForCompletion(ctx, client, jobID)
- res, err := newUniversalPipelineResult(ctx, jobID, client)
- if err != nil {
- return presult, err
+ res, presultErr := newUniversalPipelineResult(ctx, jobID, client)
+ if presultErr != nil {
+ if err != nil {
+ return presult, errors.Wrap(err, presultErr.Error())
+ } else {
+ return presult, presultErr
+ }
+ } else {
+ presult = res
}
- presult = res
return presult, err
}
Review comment:
```suggestion
}
```
cleanup of the old return that doesn't need to be preserved.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kamilwu commented on a change in pull request #13484: [BEAM-11207] Return valid error on job failures
Posted by GitBox <gi...@apache.org>.
kamilwu commented on a change in pull request #13484:
URL: https://github.com/apache/beam/pull/13484#discussion_r537412845
##########
File path: sdks/go/pkg/beam/runners/universal/runnerlib/execute.go
##########
@@ -94,11 +94,16 @@ func Execute(ctx context.Context, p *pipepb.Pipeline, endpoint string, opt *JobO
}
err = WaitForCompletion(ctx, client, jobID)
- res, err := newUniversalPipelineResult(ctx, jobID, client)
- if err != nil {
- return presult, err
+ res, presultErr := newUniversalPipelineResult(ctx, jobID, client)
+ if presultErr != nil {
+ if err != nil {
+ return presult, errors.Wrap(err, presultErr.Error())
+ } else {
+ return presult, presultErr
+ }
+ } else {
+ presult = res
}
Review comment:
Thanks, this definitely looks better. I pushed fixes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck merged pull request #13484: [BEAM-11207] Return valid error on job failures
Posted by GitBox <gi...@apache.org>.
lostluck merged pull request #13484:
URL: https://github.com/apache/beam/pull/13484
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kamilwu commented on pull request #13484: [BEAM-11075] Return valid error on job failures
Posted by GitBox <gi...@apache.org>.
kamilwu commented on pull request #13484:
URL: https://github.com/apache/beam/pull/13484#issuecomment-738709788
R: @lostluck
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kamilwu commented on pull request #13484: [BEAM-11075] Return valid error on job failures
Posted by GitBox <gi...@apache.org>.
kamilwu commented on pull request #13484:
URL: https://github.com/apache/beam/pull/13484#issuecomment-738709630
For example, [this](https://ci-beam.apache.org/job/beam_LoadTests_Go_CoGBK_Flink_Batch_PR/1/console) pipeline has failed, but the Jenkins job is a successful one, which is wrong.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org