You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "shunping (via GitHub)" <gi...@apache.org> on 2026/08/06 19:30:40 UTC
[PR] Fix GRPC channel leak [beam]
shunping opened a new pull request, #39663:
URL: https://github.com/apache/beam/pull/39663
### Description
#### Problem
When running Portable/YAML pipelines, gRPC outputs severe error logs indicating that ManagedChannel instances were garbage collected without being shut down:
```
SEVERE: *~*~*~ Previous channel ManagedChannelImpl{logId=..., target=...} was garbage collected without being shut down! ~*~*~*
Make sure to call shutdown()/shutdownNow()
java.lang.Throwable: java.lang.RuntimeException: ManagedChannel allocation site
at org.apache.beam.vendor.grpc.v1p69p0.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.<init>(...)
```
#### Cause
Three places in the Java Fn Harness allocated gRPC ManagedChannel instances without shutting them down upon task completion or harness shutdown:
- ExternalWorkerService: A ManagedChannel was created to query the provision service for worker options, but was left unclosed after obtaining provisionInfo.
- BeamFnDataGrpcClient & BeamFnDataGrpcMultiplexer: BeamFnDataGrpcClient created a ManagedChannel per BeamFnDataGrpcMultiplexer, but neither stored nor shut down the channel when close() was called.
- FnHarness: The control service ManagedChannel was created during harness startup but was never shut down when FnHarness.main() finished.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix GRPC channel leak [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #39663:
URL: https://github.com/apache/beam/pull/39663#discussion_r3731591386
##########
sdks/java/core/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataGrpcMultiplexer.java:
##########
@@ -194,6 +206,9 @@ public void close() throws Exception {
outboundObserver.onError(
Status.CANCELLED.withDescription("Multiplexer hanging up").asException());
inboundObserver.onCompleted();
+ if (channel != null) {
Review Comment:
Thanks. I was hopping the leak could have resolved in #39390, where BeamFnDataGrpcMultiplexer is closed there. Turns out there are still missing piece here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Java] Fix GRPC channel leak [beam]
Posted by "shunping (via GitHub)" <gi...@apache.org>.
shunping commented on code in PR #39663:
URL: https://github.com/apache/beam/pull/39663#discussion_r4290230293
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnHarness.java:
##########
@@ -437,6 +437,9 @@ private BeamFnApi.ProcessBundleDescriptor loadDescriptor(String id) {
beamFnStatusClient.close();
}
processBundleHandler.shutdown();
+ beamFnDataMultiplexer.close();
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Java] Fix GRPC channel leak [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #39663:
URL: https://github.com/apache/beam/pull/39663#issuecomment-6196078619
Assigning reviewers:
R: @Abacn for label java.
Backup reviewer: ahmedabu98 (if the main reviewer is unresponsive, you can escalate to them by tagging their username)
This pull request likely touches a core component ("core" label). Please review with scrutiny.
Note: If you would like to opt out of this review, comment `assign to next reviewer`.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix GRPC channel leak [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #39663:
URL: https://github.com/apache/beam/pull/39663#issuecomment-6016619846
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix GRPC channel leak [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #39663:
URL: https://github.com/apache/beam/pull/39663#issuecomment-6146099961
This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Java] Fix GRPC channel leak [beam]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #39663:
URL: https://github.com/apache/beam/pull/39663#issuecomment-6151619802
## [Codecov](https://app.codecov.io/gh/apache/beam/pull/39663?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `42.42424%` with `19 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 65.05%. Comparing base ([`76f6e6d`](https://app.codecov.io/gh/apache/beam/commit/76f6e6d7c0c1b2ce582c73d068ad9260cf15c5f3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`98687da`](https://app.codecov.io/gh/apache/beam/commit/98687da8f4d0e6f20b0da2543e0d8ee5fe84157b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 155 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/beam/pull/39663?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [.../apache/beam/fn/harness/ExternalWorkerService.java](https://app.codecov.io/gh/apache/beam/pull/39663?src=pr&el=tree&filepath=sdks%2Fjava%2Fharness%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fbeam%2Ffn%2Fharness%2FExternalWorkerService.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c2Rrcy9qYXZhL2hhcm5lc3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2JlYW0vZm4vaGFybmVzcy9FeHRlcm5hbFdvcmtlclNlcnZpY2UuamF2YQ==) | 0.00% | [12 Missing :warning: ](https://app.codecov.io/gh/apache/beam/pull/39663?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...m/fn/harness/state/BeamFnStateGrpcClientCache.java](https://app.codecov.io/gh/apache/beam/pull/39663?src=pr&el=tree&filepath=sdks%2Fjava%2Fharness%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fbeam%2Ffn%2Fharness%2Fstate%2FBeamFnStateGrpcClientCache.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c2Rrcy9qYXZhL2hhcm5lc3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2JlYW0vZm4vaGFybmVzcy9zdGF0ZS9CZWFtRm5TdGF0ZUdycGNDbGllbnRDYWNoZS5qYXZh) | 60.00% | [3 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/beam/pull/39663?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/beam/fn/harness/data/BeamFnDataGrpcClient.java](https://app.codecov.io/gh/apache/beam/pull/39663?src=pr&el=tree&filepath=sdks%2Fjava%2Fharness%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fbeam%2Ffn%2Fharness%2Fdata%2FBeamFnDataGrpcClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c2Rrcy9qYXZhL2hhcm5lc3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2JlYW0vZm4vaGFybmVzcy9kYXRhL0JlYW1GbkRhdGFHcnBjQ2xpZW50LmphdmE=) | 62.50% | [2 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/beam/pull/39663?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #39663 +/- ##
============================================
+ Coverage 56.16% 65.05% +8.88%
- Complexity 2288 13977 +11689
============================================
Files 1124 1729 +605
Lines 178336 108487 -69849
Branches 1489 11307 +9818
============================================
- Hits 100170 70571 -29599
+ Misses 75645 33707 -41938
- Partials 2521 4209 +1688
```
| [Flag](https://app.codecov.io/gh/apache/beam/pull/39663/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [go](https://app.codecov.io/gh/apache/beam/pull/39663/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java](https://app.codecov.io/gh/apache/beam/pull/39663/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `65.05% <42.42%> (-10.74%)` | :arrow_down: |
| [python](https://app.codecov.io/gh/apache/beam/pull/39663/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/beam/pull/39663?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix GRPC channel leak [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #39663: Fix GRPC channel leak
URL: https://github.com/apache/beam/pull/39663
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Java] Fix GRPC channel leak [beam]
Posted by "shunping (via GitHub)" <gi...@apache.org>.
shunping commented on PR #39663:
URL: https://github.com/apache/beam/pull/39663#issuecomment-6149219715
Converting it back to draft since some tests are failing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Java] Fix GRPC channel leak [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #39663:
URL: https://github.com/apache/beam/pull/39663#discussion_r4284914569
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnHarness.java:
##########
@@ -437,6 +437,9 @@ private BeamFnApi.ProcessBundleDescriptor loadDescriptor(String id) {
beamFnStatusClient.close();
}
processBundleHandler.shutdown();
+ beamFnDataMultiplexer.close();
Review Comment:
processBundleHandler.shutdown() already calls `beamFnStateGrpcClientCache.close();` and ` beamFnDataClient.close();`
This is redundant though probably harmless, however we should close client in the place where it owns it in general, to avoid potential pre-mature channel break
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Java] Fix GRPC channel leak [beam]
Posted by "shunping (via GitHub)" <gi...@apache.org>.
shunping merged PR #39663:
URL: https://github.com/apache/beam/pull/39663
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org