You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Claire McGinty (Jira)" <ji...@apache.org> on 2022/03/22 17:32:00 UTC
[jira] [Comment Edited] (BEAM-12356) BigQueryWriteClient in DatasetServiceImpl is not closed, which causes "ManagedChannel allocation site" exceptions
[ https://issues.apache.org/jira/browse/BEAM-12356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17510783#comment-17510783 ]
Claire McGinty edited comment on BEAM-12356 at 3/22/22, 5:31 PM:
-----------------------------------------------------------------
[~bhulette] Sure, here's an example! They're also all over our [integration test logs|https://github.com/spotify/scio/actions/workflows/it-tests.yml] if you want further context.
{{2022-03-22 11:17:11.415-0400 error [ManagedChannelOrphanWrapper] {*}{*}{*}~ Channel \{0} was not shutdown properly!!!{*} *~*}}{{ Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true. }}{{{}java.lang.RuntimeException: ManagedChannel allocation site{}}}{{{}at io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.<init>(ManagedChannelOrphanWrapper.java:93){}}}{{{}at io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:53){}}}{{{}at io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:44){}}}{{{}at io.grpc.internal.ManagedChannelImplBuilder.build(ManagedChannelImplBuilder.java:630){}}}{{{}at io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:297){}}}{{{}at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createSingleChannel(InstantiatingGrpcChannelProvider.java:395){}}}{{{}at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.access$1900(InstantiatingGrpcChannelProvider.java:82){}}}{{{}at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider$1.createSingleChannel(InstantiatingGrpcChannelProvider.java:240){}}}{{{}at com.google.api.gax.grpc.ChannelPool.create(ChannelPool.java:72){}}}{{{}at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createChannel(InstantiatingGrpcChannelProvider.java:250){}}}{{{}at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.getTransportChannel(InstantiatingGrpcChannelProvider.java:228){}}}{{{}at com.google.api.gax.rpc.ClientContext.create(ClientContext.java:205){}}}{{{}at com.google.cloud.bigquery.storage.v1.stub.GrpcBigQueryWriteStub.create(GrpcBigQueryWriteStub.java:132){}}}{{{}at com.google.cloud.bigquery.storage.v1.stub.BigQueryWriteStubSettings.createStub(BigQueryWriteStubSettings.java:144){}}}{{{}at com.google.cloud.bigquery.storage.v1.BigQueryWriteClient.<init>(BigQueryWriteClient.java:130){}}}{{{}at com.google.cloud.bigquery.storage.v1.BigQueryWriteClient.create(BigQueryWriteClient.java:111){}}}{{{}at org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.newBigQueryWriteClient(BigQueryServicesImpl.java:1353){}}}{{{}at org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.access$800(BigQueryServicesImpl.java:145){}}}{{{}at org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.<init>(BigQueryServicesImpl.java:518){}}}{{{}at org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.<init>(BigQueryServicesImpl.java:463){}}}{{{}at org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.getDatasetService(BigQueryServicesImpl.java:180){}}}{{{}at org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase.extractFiles(BigQuerySourceBase.java:117){}}}{{{}at org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase.split(BigQuerySourceBase.java:155){}}}{{{}at org.apache.beam.runners.direct.BoundedReadEvaluatorFactory$InputProvider.getInitialInputs(BoundedReadEvaluatorFactory.java:216){}}}{{{}at org.apache.beam.runners.direct.ReadEvaluatorFactory$InputProvider.getInitialInputs(ReadEvaluatorFactory.java:88){}}}{{{}at org.apache.beam.runners.direct.RootProviderRegistry.getInitialInputs(RootProviderRegistry.java:80){}}}{{{}at org.apache.beam.runners.direct.ExecutorServiceParallelExecutor.start(ExecutorServiceParallelExecutor.java:159){}}}{{{}at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213){}}}{{{}at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67){}}}{{{}at org.apache.beam.sdk.Pipeline.run(Pipeline.java:323){}}}{{{}at org.apache.beam.sdk.Pipeline.run(Pipeline.java:309){}}}
was (Author: clairemcginty):
[~bhulette] Sure, here's an example! They're also all over our [integration test logs|https://github.com/spotify/scio/actions/workflows/it-tests.yml] if you want further context.
2022-03-22 11:17:11.415-0400 error [ManagedChannelOrphanWrapper] *~*~*~ Channel \{0} was not shutdown properly!!! ~*~*~*
Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true.
java.lang.RuntimeException: ManagedChannel allocation site
at io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.<init>(ManagedChannelOrphanWrapper.java:93)
at io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:53)
at io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:44)
at io.grpc.internal.ManagedChannelImplBuilder.build(ManagedChannelImplBuilder.java:630)
at io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:297)
at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createSingleChannel(InstantiatingGrpcChannelProvider.java:395)
at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.access$1900(InstantiatingGrpcChannelProvider.java:82)
at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider$1.createSingleChannel(InstantiatingGrpcChannelProvider.java:240)
at com.google.api.gax.grpc.ChannelPool.create(ChannelPool.java:72)
at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createChannel(InstantiatingGrpcChannelProvider.java:250)
at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.getTransportChannel(InstantiatingGrpcChannelProvider.java:228)
at com.google.api.gax.rpc.ClientContext.create(ClientContext.java:205)
at com.google.cloud.bigquery.storage.v1.stub.GrpcBigQueryWriteStub.create(GrpcBigQueryWriteStub.java:132)
at com.google.cloud.bigquery.storage.v1.stub.BigQueryWriteStubSettings.createStub(BigQueryWriteStubSettings.java:144)
at com.google.cloud.bigquery.storage.v1.BigQueryWriteClient.<init>(BigQueryWriteClient.java:130)
at com.google.cloud.bigquery.storage.v1.BigQueryWriteClient.create(BigQueryWriteClient.java:111)
at org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.newBigQueryWriteClient(BigQueryServicesImpl.java:1353)
at org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.access$800(BigQueryServicesImpl.java:145)
at org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.<init>(BigQueryServicesImpl.java:518)
at org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.<init>(BigQueryServicesImpl.java:463)
at org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.getDatasetService(BigQueryServicesImpl.java:180)
at org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase.extractFiles(BigQuerySourceBase.java:117)
at org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase.split(BigQuerySourceBase.java:155)
at org.apache.beam.runners.direct.BoundedReadEvaluatorFactory$InputProvider.getInitialInputs(BoundedReadEvaluatorFactory.java:216)
at org.apache.beam.runners.direct.ReadEvaluatorFactory$InputProvider.getInitialInputs(ReadEvaluatorFactory.java:88)
at org.apache.beam.runners.direct.RootProviderRegistry.getInitialInputs(RootProviderRegistry.java:80)
at org.apache.beam.runners.direct.ExecutorServiceParallelExecutor.start(ExecutorServiceParallelExecutor.java:159)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:323)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:309)
> BigQueryWriteClient in DatasetServiceImpl is not closed, which causes "ManagedChannel allocation site" exceptions
> -----------------------------------------------------------------------------------------------------------------
>
> Key: BEAM-12356
> URL: https://issues.apache.org/jira/browse/BEAM-12356
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp
> Affects Versions: 2.29.0, 2.32.0, 2.33.0
> Reporter: Minbo Bae
> Assignee: Reuven Lax
> Priority: P2
> Fix For: 2.34.0
>
> Attachments: bigquery_grpc.log
>
> Time Spent: 5.5h
> Remaining Estimate: 0h
>
> [BigQueryWriteClient|https://github.com/apache/beam/blob/v2.29.0/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java#L461] in DatasetServiceImpl (added at [https://github.com/apache/beam/pull/14309)] is not closed. This causes the error logs in gRPC orphan channel clean up. See "bigquery_grpc.log" in attachments which is extracted from GCP Dataflow. I don't think this issue affect pipeline runs except the error logs, but could you take a look at that?
> A similar issue is reported for {{CloudBigtableIO}} at [https://github.com/googleapis/java-bigtable-hbase/issues/2658]
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)