You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/07/08 14:56:50 UTC
[GitHub] [beam] damccorm opened a new pull request, #22196: Collect heap dumps on Go worker panic for Dataflow
damccorm opened a new pull request, #22196:
URL: https://github.com/apache/beam/pull/22196
This collects a heap dump and uploads it to the temp_directory on Dataflow any time there is a worker panic before we exit. It departs from what #21797 recommends a little bit in that it doesn't use `GOTRACEBACK=crash` and instead relies on `debug.WriteHeapDump`. This is mostly because the GOTRACEBACK approach doesn't collect the dump until the top level Go routine exits with a panic - this currently isn't guaranteed to happen (and won't by default because we recover from panics and call os.exit), but if it did it would propogate up through boot.go. At that point, the VM is not guaranteed to be persisted (in fact, I think it will always be torn down), and this generally just breaks how we handle exiting. The approach of gathering the dump ourselves is also more in line with how Java handles this anyways - https://github.com/apache/beam/blob/4ffeae4d2b800f2df36d2ea2eab549f2204d5691/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/status/MemoryMonitor.java#L589
To test this, I inserted a panic into the harness code and verified that a heap dump was correctly uploaded to the temp directory:
- pipeline - https://pantheon.corp.google.com/dataflow/jobs/us-central1/2022-07-08_07_34_58-5641449712707134536;bottomTab=WORKER_LOGS;logsSeverity=INFO?project=google.com:clouddfe&pageState=(%22dfTime%22:(%22l%22:%22dfJobMaxTime%22))
- dumps from multiple pipeline runs - https://pantheon.corp.google.com/storage/browser/clouddfe-test/dannymccormick-tmp/heapDumps?pageState=(%22StorageObjectListTable%22:(%22f%22:%22%255B%255D%22))&project=google.com:clouddfe&prefix=&forceOnObjectsSortingFiltering=false)
Fixes #21797
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm closed pull request #22196: Collect heap dumps on Go worker panic for Dataflow
Posted by GitBox <gi...@apache.org>.
damccorm closed pull request #22196: Collect heap dumps on Go worker panic for Dataflow
URL: https://github.com/apache/beam/pull/22196
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on pull request #22196: Collect heap dumps on Go worker panic for Dataflow
Posted by GitBox <gi...@apache.org>.
damccorm commented on PR #22196:
URL: https://github.com/apache/beam/pull/22196#issuecomment-1179191154
> See https://stackoverflow.com/questions/57486620/are-all-runtime-errors-recoverable-in-go for some examples on how to trigger them.
Ah, that's helpful context. I'll need to play around with how this actually behaves, but I agree this isn't the right approach. I'll give this another pass
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #22196: Collect heap dumps on Go worker panic for Dataflow
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on PR #22196:
URL: https://github.com/apache/beam/pull/22196#issuecomment-1179085959
# [Codecov](https://codecov.io/gh/apache/beam/pull/22196?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#22196](https://codecov.io/gh/apache/beam/pull/22196?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e8ee08c) into [master](https://codecov.io/gh/apache/beam/commit/ec47b12cd54bef4632c8a8e0ebca6b88e597c327?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (ec47b12) will **decrease** coverage by `0.03%`.
> The diff coverage is `0.00%`.
```diff
@@ Coverage Diff @@
## master #22196 +/- ##
==========================================
- Coverage 74.21% 74.18% -0.04%
==========================================
Files 702 703 +1
Lines 92829 92870 +41
==========================================
Hits 68892 68892
- Misses 22670 22711 +41
Partials 1267 1267
```
| Flag | Coverage Δ | |
|---|---|---|
| go | `51.42% <0.00%> (-0.08%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/22196?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...ks/go/pkg/beam/core/runtime/harness/diagnostics.go](https://codecov.io/gh/apache/beam/pull/22196/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9kaWFnbm9zdGljcy5nbw==) | `0.00% <0.00%> (ø)` | |
| [sdks/go/pkg/beam/runners/dataflow/dataflow.go](https://codecov.io/gh/apache/beam/pull/22196/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RhdGFmbG93L2RhdGFmbG93Lmdv) | `59.77% <ø> (ø)` | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/22196?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/22196?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [ec47b12...e8ee08c](https://codecov.io/gh/apache/beam/pull/22196?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #22196: Collect heap dumps on Go worker panic for Dataflow
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #22196:
URL: https://github.com/apache/beam/pull/22196#issuecomment-1179125632
Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @lostluck for label go.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on a diff in pull request #22196: Collect heap dumps on Go worker panic for Dataflow
Posted by GitBox <gi...@apache.org>.
lostluck commented on code in PR #22196:
URL: https://github.com/apache/beam/pull/22196#discussion_r916984079
##########
sdks/go/pkg/beam/core/runtime/harness/init/init.go:
##########
@@ -88,10 +90,24 @@ func hook() {
runtime.GlobalOptions.Import(opt.Options)
}
+ // Since Init() is hijacking main, it's appropriate to do as main
+ // does, and establish the background context here.
+ // We produce a cancelFn here so runs in Loopback mode and similar can clean up
+ // any leftover goroutines.
+ ctx, cancelFn := context.WithCancel(context.Background())
+ defer cancelFn()
+ ctx = grpcx.WriteWorkerID(ctx, *id)
defer func() {
if r := recover(); r != nil {
fmt.Fprintf(os.Stderr, "Worker panic: %v\n", r)
debug.PrintStack()
+ log.Warnf(ctx, "Worker panic detected: %v", r)
+
+ if tempLocation := beam.PipelineOptions.Get("temp_location"); tempLocation != "" {
+ log.Warnf(ctx, "Capturing heap dump")
+ harness.UploadHeapDump(ctx, fmt.Sprintf("%v/heapDumps/dump-%d", strings.TrimSuffix(tempLocation, "/"), time.Now().Unix()))
Review Comment:
Please also use the worker-id `*id` and not just the time.
##########
sdks/go/pkg/beam/runners/dataflow/dataflow.go:
##########
@@ -117,18 +117,19 @@ var flagFilter = map[string]bool{
"network": true,
"subnetwork": true,
"no_use_public_ips": true,
- "temp_location": true,
- "template_location": true,
- "worker_machine_type": true,
- "min_cpu_platform": true,
- "dataflow_worker_jar": true,
- "worker_region": true,
- "worker_zone": true,
- "teardown_policy": true,
- "cpu_profiling": true,
- "session_recording": true,
- "update": true,
- "transform_name_mapping": true,
+ // Don't filter temp_location since we need this included in PipelineOptions to correctly upload heap dumps.
+ // "temp_location": true,
Review Comment:
I'd remove this line entirely, but still have a comment about it somewhere.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on pull request #22196: Collect heap dumps on Go worker panic for Dataflow
Posted by GitBox <gi...@apache.org>.
lostluck commented on PR #22196:
URL: https://github.com/apache/beam/pull/22196#issuecomment-1179182567
I was under the impression that there were panics we couldn't use "recover" from, such as
OOM and concurrent map writes+reads. Otherwise, this would be something else we need to bolster our "callNoPanic" path with since an OOM'd worker isn't likely to be able to recover.
See https://stackoverflow.com/questions/57486620/are-all-runtime-errors-recoverable-in-go for some examples on how to trigger them.
It does seem like we could probably have an actual integration test for this too that runs on Dataflow, pipeline that will start, and then OOM, and then we could observe the heap dump was written to the given GCS folder.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on pull request #22196: Collect heap dumps on Go worker panic for Dataflow
Posted by GitBox <gi...@apache.org>.
lostluck commented on PR #22196:
URL: https://github.com/apache/beam/pull/22196#issuecomment-1179188431
Also, it's very easy to subvert a recover:
```
go func(){ panic("escape recover") }
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org