You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by "Kaymak, Tobias" <to...@ricardo.ch> on 2021/09/22 16:10:41 UTC
Beam/Flink's netty versions seems to clash (2.32.0 / 1.13.1)
Hello,
while upgrading our cluster and pipelines to the new visions I noticed:
java.lang.NoClassDefFoundError:
io/grpc/netty/shaded/io/netty/channel/FailedChannelFuture
so I checked the maven dependency tree and found that the:
org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.32.0:compile
dependency is using netty 4.1.52 - while Flink 1.13.1 seems to use 4.1.49.
Is there a way to fix this?
Best,
Tobi
Re: Beam/Flink's netty versions seems to clash (2.32.0 / 1.13.1)
Posted by "Kaymak, Tobias" <to...@ricardo.ch>.
Ok, thank you!
On Thu, Sep 23, 2021 at 6:06 PM Reuven Lax <re...@google.com> wrote:
> The bug will cause these logs and might also cause some performance
> problems. It should no cause any data correctness through.
>
> The bug fix will be available in Beam 2.3.4, ETA probably this November.
>
> On Thu, Sep 23, 2021 at 6:22 AM Kaymak, Tobias <to...@ricardo.ch>
> wrote:
>
>> Hi +Reuven Lax <re...@google.com> -
>>
>> I saw your post when researching the error above:
>> https://stackoverflow.com/a/69111493 :)
>>
>> As we are on Flink 1.13.1 in the middle of a move, it would be tough to
>> have to downgrade Flink to downgrade Beam here.
>> Is the warning something of concern or can we expect the data to be
>> delivered (and ignore this warning for now, and wait for the next release
>> and then upgrade to it)?
>>
>> Best,
>> Tobi
>>
>> On Thu, Sep 23, 2021 at 2:51 PM Kaymak, Tobias <to...@ricardo.ch>
>> wrote:
>>
>>> Hello,
>>> I am using the GCP IOs to connect to BigQuery and Bigtable. I seem to
>>> have improved the situation by including Bigtable and BigQuery in my POM
>>> with the exact same version as the Beam 2.32.0 SDK. However I still get a
>>> lot of these errors:
>>>
>>> io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference
>>> cleanQueue
>>> SEVERE: *~*~*~ Channel ManagedChannelImpl{logId=3, target=
>>> bigquerystorage.googleapis.com:443} was not shutdown properly!!! ~*~*~*
>>> Make sure to call shutdown()/shutdownNow() and wait until
>>> awaitTermination() returns true.
>>> java.lang.RuntimeException: ManagedChannel allocation site
>>> at
>>> io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.<init>(ManagedChannelOrphanWrapper.java:93)
>>> at
>>> io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:53)
>>> at
>>> io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:44)
>>> at
>>> io.grpc.internal.ManagedChannelImplBuilder.build(ManagedChannelImplBuilder.java:615)
>>> at
>>> io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:261)
>>> at
>>> com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createSingleChannel(InstantiatingGrpcChannelProvider.java:327)
>>> at
>>> com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.access$1700(InstantiatingGrpcChannelProvider.java:74)
>>> at
>>> com.google.api.gax.grpc.InstantiatingGrpcChannelProvider$1.createSingleChannel(InstantiatingGrpcChannelProvider.java:220)
>>> at
>>> com.google.api.gax.grpc.ChannelPool.create(ChannelPool.java:72)
>>> at
>>> com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createChannel(InstantiatingGrpcChannelProvider.java:227)
>>> at
>>> com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.getTransportChannel(InstantiatingGrpcChannelProvider.java:210)
>>> at
>>> com.google.api.gax.rpc.ClientContext.create(ClientContext.java:169)
>>> at
>>> com.google.cloud.bigquery.storage.v1beta2.stub.GrpcBigQueryWriteStub.create(GrpcBigQueryWriteStub.java:138)
>>> at
>>> com.google.cloud.bigquery.storage.v1beta2.stub.BigQueryWriteStubSettings.createStub(BigQueryWriteStubSettings.java:145)
>>> at
>>> com.google.cloud.bigquery.storage.v1beta2.BigQueryWriteClient.<init>(BigQueryWriteClient.java:128)
>>> at
>>> com.google.cloud.bigquery.storage.v1beta2.BigQueryWriteClient.create(BigQueryWriteClient.java:109)
>>> at
>>> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.newBigQueryWriteClient(BigQueryServicesImpl.java:1279)
>>> at
>>> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.access$800(BigQueryServicesImpl.java:139)
>>> at
>>> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.<init>(BigQueryServicesImpl.java:510)
>>> at
>>> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.<init>(BigQueryServicesImpl.java:455)
>>> at
>>> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.getDatasetService(BigQueryServicesImpl.java:172)
>>> at
>>> org.apache.beam.sdk.io.gcp.bigquery.WriteTables$WriteTablesDoFn.finishBundle(WriteTables.java:284)
>>> at
>>> org.apache.beam.sdk.io.gcp.bigquery.WriteTables$WriteTablesDoFn$DoFnInvoker.invokeFinishBundle(Unknown
>>> Source)
>>> at
>>> org.apache.beam.runners.core.SimpleDoFnRunner.finishBundle(SimpleDoFnRunner.java:242)
>>> at
>>> org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate.finishBundle(DoFnRunnerWithMetricsUpdate.java:89)
>>> at
>>> org.apache.beam.runners.core.SimplePushbackSideInputDoFnRunner.finishBundle(SimplePushbackSideInputDoFnRunner.java:125)
>>> at
>>> org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.invokeFinishBundle(DoFnOperator.java:894)
>>> at
>>> org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.checkInvokeFinishBundleByTime(DoFnOperator.java:887)
>>> at
>>> org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.lambda$open$2(DoFnOperator.java:529)
>>> at
>>> org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1419)
>>> at
>>> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$null$16(StreamTask.java:1410)
>>> at
>>> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
>>> at
>>> org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90)
>>> at
>>> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsNonBlocking(MailboxProcessor.java:359)
>>> at
>>> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:323)
>>> at
>>> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:202)
>>> at
>>> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:681)
>>> at
>>> org.apache.flink.streaming.runtime.tasks.StreamTask.executeInvoke(StreamTask.java:636)
>>> at
>>> org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:647)
>>> at
>>> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:620)
>>> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:779)
>>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)
>>> at java.base/java.lang.Thread.run(Unknown Source)
>>>
>>> dependency-tree:
>>>
>>> ch.ricardo.di:di-beam:jar:2.14.0
>>> +- org.apache.beam:beam-sdks-java-core:jar:2.32.0:compile
>>> | +- org.apache.beam:beam-model-pipeline:jar:2.32.0:compile
>>> | +- org.apache.beam:beam-model-job-management:jar:2.32.0:compile
>>> | +- org.apache.beam:beam-vendor-bytebuddy-1_11_0:jar:0.1:compile
>>> | +- org.apache.beam:beam-vendor-grpc-1_36_0:jar:0.2:compile
>>> | +- org.apache.beam:beam-vendor-guava-26_0-jre:jar:0.1:compile
>>> | +- com.google.code.findbugs:jsr305:jar:3.0.2:compile
>>> | +- com.google.errorprone:error_prone_annotations:jar:2.3.4:compile
>>> | +- com.fasterxml.jackson.core:jackson-core:jar:2.12.1:compile
>>> | +- com.fasterxml.jackson.core:jackson-annotations:jar:2.12.1:compile
>>> | +- com.fasterxml.jackson.core:jackson-databind:jar:2.12.1:compile
>>> | +- org.apache.avro:avro:jar:1.8.2:compile
>>> | | +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
>>> | | +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
>>> | | +- com.thoughtworks.paranamer:paranamer:jar:2.7:compile
>>> | | +- org.apache.commons:commons-compress:jar:1.8.1:compile
>>> | | \- org.tukaani:xz:jar:1.5:compile
>>> | \- org.xerial.snappy:snappy-java:jar:1.1.8.4:compile
>>> +-
>>> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.32.0:compile
>>> | +- org.checkerframework:checker-qual:jar:3.10.0:compile
>>> | +- org.apache.beam:beam-runners-core-java:jar:2.32.0:compile
>>> | | \- org.apache.beam:beam-sdks-java-fn-execution:jar:2.32.0:compile
>>> | | \- org.apache.beam:beam-model-fn-execution:jar:2.32.0:compile
>>> | +-
>>> org.apache.beam:beam-sdks-java-expansion-service:jar:2.32.0:compile
>>> | | \-
>>> org.apache.beam:beam-runners-java-fn-execution:jar:2.32.0:compile
>>> | +-
>>> org.apache.beam:beam-sdks-java-extensions-google-cloud-platform-core:jar:2.32.0:compile
>>> | | +- com.google.cloud.bigdataoss:gcsio:jar:2.2.2:compile
>>> | | \-
>>> com.google.apis:google-api-services-cloudresourcemanager:jar:v1-rev20210331-1.31.0:compile
>>> | +-
>>> org.apache.beam:beam-runners-core-construction-java:jar:2.32.0:compile
>>> | | \- io.github.classgraph:classgraph:jar:4.8.104:compile
>>> | +- org.apache.beam:beam-sdks-java-extensions-arrow:jar:2.32.0:compile
>>> | +- com.google.cloud.bigdataoss:util:jar:2.2.2:compile
>>> | | +-
>>> com.google.api-client:google-api-client-jackson2:jar:1.31.3:compile
>>> | | +-
>>> com.google.apis:google-api-services-iamcredentials:jar:v1-rev20201022-1.31.0:compile
>>> | | \- com.google.flogger:google-extensions:jar:0.5.1:compile
>>> | +- com.google.flogger:flogger-system-backend:jar:0.6:compile
>>> | | \- com.google.flogger:flogger:jar:0.6:compile
>>> | +- com.google.api:gax:jar:1.63.0:compile
>>> | +- com.google.api:gax-grpc:jar:1.63.0:compile
>>> | +- com.google.api:gax-httpjson:jar:0.80.0:compile
>>> | +- com.google.api:api-common:jar:1.10.1:compile
>>> | +-
>>> com.google.apis:google-api-services-healthcare:jar:v1beta1-rev20210407-1.31.0:compile
>>> | +-
>>> com.google.apis:google-api-services-pubsub:jar:v1-rev20210322-1.31.0:compile
>>> | +-
>>> com.google.apis:google-api-services-storage:jar:v1-rev20210127-1.31.0:compile
>>> | +- com.google.auth:google-auth-library-credentials:jar:0.25.2:compile
>>> | +- com.google.auth:google-auth-library-oauth2-http:jar:0.25.2:compile
>>> | +- com.google.cloud:google-cloud-bigquerystorage:jar:1.17.0:compile
>>> | | +-
>>> com.google.api.grpc:proto-google-cloud-bigquerystorage-v1alpha2:jar:0.117.0:compile
>>> | | +-
>>> com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta1:jar:0.117.0:compile
>>> | | +- org.json:json:jar:20200518:compile
>>> | | +-
>>> com.google.api.grpc:grpc-google-cloud-bigquerystorage-v1beta1:jar:0.117.0:compile
>>> | | +-
>>> com.google.api.grpc:grpc-google-cloud-bigquerystorage-v1beta2:jar:0.117.0:compile
>>> | | \-
>>> com.google.api.grpc:grpc-google-cloud-bigquerystorage-v1:jar:1.17.0:compile
>>> | +- com.google.cloud.bigtable:bigtable-client-core:jar:1.19.1:compile
>>> | | +- com.google.api.grpc:grpc-google-common-protos:jar:2.1.0:compile
>>> | | +-
>>> com.google.api.grpc:grpc-google-cloud-bigtable-admin-v2:jar:1.22.0:compile
>>> | | +- io.opencensus:opencensus-contrib-grpc-util:jar:0.24.0:compile
>>> | | \- io.dropwizard.metrics:metrics-core:jar:3.2.6:compile
>>> | +- com.google.cloud:google-cloud-core:jar:1.94.6:compile
>>> | +- com.google.cloud:google-cloud-core-grpc:jar:1.94.6:compile
>>> | +-
>>> com.google.cloud.datastore:datastore-v1-proto-client:jar:1.6.3:compile
>>> | | \-
>>> com.google.http-client:google-http-client-protobuf:jar:1.39.2:compile
>>> | +- com.google.cloud:google-cloud-firestore:jar:2.2.5:compile
>>> | | \-
>>> com.google.cloud:proto-google-cloud-firestore-bundle-v1:jar:2.2.5:compile
>>> | +- com.google.cloud:google-cloud-pubsublite:jar:0.13.2:compile
>>> | +- com.google.cloud:google-cloud-pubsub:jar:1.112.0:compile
>>> | +- com.google.cloud:google-cloud-spanner:jar:6.2.0:compile
>>> | | +-
>>> com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:jar:6.2.0:compile
>>> | | +-
>>> com.google.api.grpc:grpc-google-cloud-spanner-admin-instance-v1:jar:6.2.0:compile
>>> | | +-
>>> com.google.api.grpc:grpc-google-cloud-spanner-v1:jar:6.2.0:compile
>>> | | \-
>>> com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:jar:6.2.0:compile
>>> | +- com.google.code.gson:gson:jar:2.8.6:compile
>>> | +-
>>> com.google.http-client:google-http-client-jackson2:jar:1.39.2:compile
>>> | +- io.grpc:grpc-alts:jar:1.37.0:compile
>>> | +- io.grpc:grpc-api:jar:1.37.0:compile
>>> | +- io.grpc:grpc-auth:jar:1.37.0:compile
>>> | +- io.grpc:grpc-core:jar:1.37.0:compile
>>> | +- io.grpc:grpc-context:jar:1.37.0:compile
>>> | +- io.grpc:grpc-grpclb:jar:1.37.0:compile
>>> | +- io.grpc:grpc-netty:jar:1.37.0:compile
>>> | | +- io.netty:netty-codec-http2:jar:4.1.52.Final:compile
>>> | | | \- io.netty:netty-codec-http:jar:4.1.52.Final:compile
>>> | | \- io.netty:netty-handler-proxy:jar:4.1.52.Final:runtime
>>> | | \- io.netty:netty-codec-socks:jar:4.1.52.Final:runtime
>>> | +- io.grpc:grpc-netty-shaded:jar:1.37.0:compile
>>> | +- io.grpc:grpc-stub:jar:1.37.0:compile
>>> | +- com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:1.94.0:compile
>>> | +-
>>> com.google.api.grpc:grpc-google-cloud-pubsublite-v1:jar:0.13.2:compile
>>> | +- com.google.guava:guava:jar:30.1-jre:compile
>>> | +- org.hamcrest:hamcrest:jar:2.1:compile
>>> | +- org.apache.httpcomponents:httpclient:jar:4.5.10:compile
>>> | +- org.apache.httpcomponents:httpcore:jar:4.4.12:compile
>>> | +- io.netty:netty-handler:jar:4.1.52.Final:compile
>>> | | +- io.netty:netty-common:jar:4.1.52.Final:compile
>>> | | +- io.netty:netty-resolver:jar:4.1.52.Final:compile
>>> | | +- io.netty:netty-buffer:jar:4.1.52.Final:compile
>>> | | +- io.netty:netty-transport:jar:4.1.52.Final:compile
>>> | | \- io.netty:netty-codec:jar:4.1.52.Final:compile
>>> | +- io.netty:netty-tcnative-boringssl-static:jar:2.0.33.Final:compile
>>> | +-
>>> com.google.api.grpc:proto-google-cloud-bigquerystorage-v1:jar:1.17.0:compile
>>> | +-
>>> com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta2:jar:0.117.0:compile
>>> | +-
>>> com.google.api.grpc:proto-google-cloud-bigtable-admin-v2:jar:1.22.0:compile
>>> | +-
>>> com.google.api.grpc:proto-google-cloud-bigtable-v2:jar:1.22.0:compile
>>> | +-
>>> com.google.api.grpc:proto-google-cloud-datastore-v1:jar:0.89.0:compile
>>> | +-
>>> com.google.api.grpc:proto-google-cloud-firestore-v1:jar:2.2.5:compile
>>> | +-
>>> com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:1.94.0:compile
>>> | +-
>>> com.google.api.grpc:proto-google-cloud-pubsublite-v1:jar:0.13.2:compile
>>> | +-
>>> com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:jar:6.2.0:compile
>>> | +-
>>> com.google.api.grpc:proto-google-cloud-spanner-v1:jar:6.2.0:compile
>>> | +- com.google.api.grpc:proto-google-common-protos:jar:2.1.0:compile
>>> | +- com.google.protobuf:protobuf-java:jar:3.15.8:compile
>>> | +- org.apache.arrow:arrow-memory-core:jar:4.0.0:compile
>>> | +- org.apache.arrow:arrow-vector:jar:4.0.0:compile
>>> | | +- org.apache.arrow:arrow-format:jar:4.0.0:compile
>>> | | \- com.google.flatbuffers:flatbuffers-java:jar:1.12.0:compile
>>> | \- org.threeten:threetenbp:jar:1.4.4:compile
>>> +- com.google.api-client:google-api-client:jar:1.31.1:compile
>>> | +- com.google.oauth-client:google-oauth-client:jar:1.31.2:compile
>>> | \-
>>> com.google.http-client:google-http-client-apache-v2:jar:1.39.2:compile
>>> +-
>>> com.google.apis:google-api-services-bigquery:jar:v2-rev20210410-1.31.0:compile
>>> +- com.google.http-client:google-http-client:jar:1.39.2:compile
>>> | +- com.google.j2objc:j2objc-annotations:jar:1.3:compile
>>> | +- io.opencensus:opencensus-api:jar:0.28.0:compile
>>> | \- io.opencensus:opencensus-contrib-http-util:jar:0.28.0:compile
>>> +- joda-time:joda-time:jar:2.10.10:compile
>>> +- org.slf4j:slf4j-api:jar:1.7.30:compile
>>> +- org.slf4j:slf4j-jdk14:jar:1.7.30:runtime
>>> +- org.hamcrest:hamcrest-core:jar:2.1:compile
>>> +- org.hamcrest:hamcrest-library:jar:2.1:compile
>>> +- junit:junit:jar:4.13.1:compile
>>> +- org.apache.beam:beam-runners-direct-java:jar:2.32.0:runtime
>>> +- org.apache.beam:beam-sdks-java-io-kafka:jar:2.32.0:compile
>>> | +- org.springframework:spring-expression:jar:4.3.18.RELEASE:compile
>>> | | \- org.springframework:spring-core:jar:4.3.18.RELEASE:compile
>>> | +- io.confluent:kafka-avro-serializer:jar:5.3.2:compile
>>> | | +- io.confluent:common-config:jar:5.3.2:compile
>>> | | \- io.confluent:common-utils:jar:5.3.2:compile
>>> | | \- com.101tec:zkclient:jar:0.10:compile
>>> | \- io.confluent:kafka-schema-registry-client:jar:5.3.2:compile
>>> +- org.apache.kafka:kafka-clients:jar:2.6.0:compile
>>> | +- com.github.luben:zstd-jni:jar:1.4.4-7:compile
>>> | \- org.lz4:lz4-java:jar:1.7.1:compile
>>> +- com.google.protobuf:protobuf-java-util:jar:3.6.1:compile
>>> +- ch.ricardo.schema:proto-schemas:jar:3.1.68:compile
>>> | \- javax.annotation:javax.annotation-api:jar:1.3.2:compile
>>> +- org.mockito:mockito-core:jar:3.7.7:test
>>> | +- net.bytebuddy:byte-buddy:jar:1.10.19:test
>>> | +- net.bytebuddy:byte-buddy-agent:jar:1.10.19:test
>>> | \- org.objenesis:objenesis:jar:3.1:test
>>> +- com.google.auto.value:auto-value-annotations:jar:1.6.2:compile
>>> +- com.google.auto.value:auto-value:jar:1.6.2:provided
>>> +- org.projectlombok:lombok:jar:1.16.22:compile
>>> +- com.jayway.jsonpath:json-path:jar:2.3.0:compile
>>> | \- net.minidev:json-smart:jar:2.3:compile
>>> | \- net.minidev:accessors-smart:jar:1.2:compile
>>> | \- org.ow2.asm:asm:jar:5.0.4:compile
>>> +- org.apache.commons:commons-lang3:jar:3.8.1:compile
>>> +- com.google.cloud:google-cloud-bigquery:jar:1.127.12:compile
>>> | +- com.google.api.grpc:proto-google-iam-v1:jar:1.0.12:compile
>>> | +- com.google.http-client:google-http-client-gson:jar:1.39.2:compile
>>> | +- com.google.cloud:google-cloud-core-http:jar:1.94.6:compile
>>> | +-
>>> com.google.http-client:google-http-client-appengine:jar:1.39.2:compile
>>> | +- commons-logging:commons-logging:jar:1.2:compile
>>> | +- commons-codec:commons-codec:jar:1.15:compile
>>> | +- org.checkerframework:checker-compat-qual:jar:2.5.5:compile
>>> | +- com.google.guava:failureaccess:jar:1.0.1:compile
>>> | \-
>>> com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:compile
>>> +- com.google.cloud:google-cloud-bigtable:jar:1.22.0:compile
>>> | +-
>>> com.google.api.grpc:grpc-google-cloud-bigtable-v2:jar:1.22.0:compile
>>> | +- org.conscrypt:conscrypt-openjdk-uber:jar:2.5.1:compile
>>> | +- org.codehaus.mojo:animal-sniffer-annotations:jar:1.20:runtime
>>> | +- com.google.android:annotations:jar:4.1.1.4:runtime
>>> | +- io.perfmark:perfmark-api:jar:0.23.0:runtime
>>> | +- io.grpc:grpc-protobuf:jar:1.37.0:compile
>>> | \- io.grpc:grpc-protobuf-lite:jar:1.37.0:compile
>>> +- org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.32.0:compile
>>> +- org.jetbrains:annotations:jar:20.1.0:compile
>>> +- com.google.cloud:google-cloud-bigtable-emulator:jar:0.128.2:test
>>> \- commons-io:commons-io:jar:2.6:compile
>>>
>>> Best,
>>> Tobi
>>>
>>> On Thu, Sep 23, 2021 at 6:15 AM Tomo Suzuki <su...@google.com> wrote:
>>>
>>>> From the package name, I think you want to focus on grpc-netty-shaded
>>>> artifact rather than netty.
>>>>
>>>> Stacktrace and the dependency tree would be helpful to understand why
>>>> the discrepancy occurred.
>>>>
>>>> Regards,
>>>> Tomo
>>>>
>>>> On Wed, Sep 22, 2021 at 12:21 Reuven Lax <re...@google.com> wrote:
>>>>
>>>>> Are you using any of the GCP IOs in your pipeline?
>>>>>
>>>>> On Wed, Sep 22, 2021 at 9:11 AM Kaymak, Tobias <
>>>>> tobias.kaymak@ricardo.ch> wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> while upgrading our cluster and pipelines to the new visions I
>>>>>> noticed:
>>>>>>
>>>>>> java.lang.NoClassDefFoundError:
>>>>>> io/grpc/netty/shaded/io/netty/channel/FailedChannelFuture
>>>>>>
>>>>>> so I checked the maven dependency tree and found that the:
>>>>>>
>>>>>>
>>>>>> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.32.0:compile
>>>>>>
>>>>>> dependency is using netty 4.1.52 - while Flink 1.13.1 seems to use
>>>>>> 4.1.49.
>>>>>>
>>>>>> Is there a way to fix this?
>>>>>>
>>>>>> Best,
>>>>>> Tobi
>>>>>>
>>>>> --
>>>> Regards,
>>>> Tomo
>>>>
>>>
Re: Beam/Flink's netty versions seems to clash (2.32.0 / 1.13.1)
Posted by Reuven Lax <re...@google.com>.
The bug will cause these logs and might also cause some performance
problems. It should no cause any data correctness through.
The bug fix will be available in Beam 2.3.4, ETA probably this November.
On Thu, Sep 23, 2021 at 6:22 AM Kaymak, Tobias <to...@ricardo.ch>
wrote:
> Hi +Reuven Lax <re...@google.com> -
>
> I saw your post when researching the error above:
> https://stackoverflow.com/a/69111493 :)
>
> As we are on Flink 1.13.1 in the middle of a move, it would be tough to
> have to downgrade Flink to downgrade Beam here.
> Is the warning something of concern or can we expect the data to be
> delivered (and ignore this warning for now, and wait for the next release
> and then upgrade to it)?
>
> Best,
> Tobi
>
> On Thu, Sep 23, 2021 at 2:51 PM Kaymak, Tobias <to...@ricardo.ch>
> wrote:
>
>> Hello,
>> I am using the GCP IOs to connect to BigQuery and Bigtable. I seem to
>> have improved the situation by including Bigtable and BigQuery in my POM
>> with the exact same version as the Beam 2.32.0 SDK. However I still get a
>> lot of these errors:
>>
>> io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference
>> cleanQueue
>> SEVERE: *~*~*~ Channel ManagedChannelImpl{logId=3, target=
>> bigquerystorage.googleapis.com:443} was not shutdown properly!!! ~*~*~*
>> Make sure to call shutdown()/shutdownNow() and wait until
>> awaitTermination() returns true.
>> java.lang.RuntimeException: ManagedChannel allocation site
>> at
>> io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.<init>(ManagedChannelOrphanWrapper.java:93)
>> at
>> io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:53)
>> at
>> io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:44)
>> at
>> io.grpc.internal.ManagedChannelImplBuilder.build(ManagedChannelImplBuilder.java:615)
>> at
>> io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:261)
>> at
>> com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createSingleChannel(InstantiatingGrpcChannelProvider.java:327)
>> at
>> com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.access$1700(InstantiatingGrpcChannelProvider.java:74)
>> at
>> com.google.api.gax.grpc.InstantiatingGrpcChannelProvider$1.createSingleChannel(InstantiatingGrpcChannelProvider.java:220)
>> at com.google.api.gax.grpc.ChannelPool.create(ChannelPool.java:72)
>> at
>> com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createChannel(InstantiatingGrpcChannelProvider.java:227)
>> at
>> com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.getTransportChannel(InstantiatingGrpcChannelProvider.java:210)
>> at
>> com.google.api.gax.rpc.ClientContext.create(ClientContext.java:169)
>> at
>> com.google.cloud.bigquery.storage.v1beta2.stub.GrpcBigQueryWriteStub.create(GrpcBigQueryWriteStub.java:138)
>> at
>> com.google.cloud.bigquery.storage.v1beta2.stub.BigQueryWriteStubSettings.createStub(BigQueryWriteStubSettings.java:145)
>> at
>> com.google.cloud.bigquery.storage.v1beta2.BigQueryWriteClient.<init>(BigQueryWriteClient.java:128)
>> at
>> com.google.cloud.bigquery.storage.v1beta2.BigQueryWriteClient.create(BigQueryWriteClient.java:109)
>> at
>> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.newBigQueryWriteClient(BigQueryServicesImpl.java:1279)
>> at
>> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.access$800(BigQueryServicesImpl.java:139)
>> at
>> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.<init>(BigQueryServicesImpl.java:510)
>> at
>> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.<init>(BigQueryServicesImpl.java:455)
>> at
>> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.getDatasetService(BigQueryServicesImpl.java:172)
>> at
>> org.apache.beam.sdk.io.gcp.bigquery.WriteTables$WriteTablesDoFn.finishBundle(WriteTables.java:284)
>> at
>> org.apache.beam.sdk.io.gcp.bigquery.WriteTables$WriteTablesDoFn$DoFnInvoker.invokeFinishBundle(Unknown
>> Source)
>> at
>> org.apache.beam.runners.core.SimpleDoFnRunner.finishBundle(SimpleDoFnRunner.java:242)
>> at
>> org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate.finishBundle(DoFnRunnerWithMetricsUpdate.java:89)
>> at
>> org.apache.beam.runners.core.SimplePushbackSideInputDoFnRunner.finishBundle(SimplePushbackSideInputDoFnRunner.java:125)
>> at
>> org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.invokeFinishBundle(DoFnOperator.java:894)
>> at
>> org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.checkInvokeFinishBundleByTime(DoFnOperator.java:887)
>> at
>> org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.lambda$open$2(DoFnOperator.java:529)
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1419)
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$null$16(StreamTask.java:1410)
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
>> at
>> org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90)
>> at
>> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsNonBlocking(MailboxProcessor.java:359)
>> at
>> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:323)
>> at
>> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:202)
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:681)
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.executeInvoke(StreamTask.java:636)
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:647)
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:620)
>> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:779)
>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)
>> at java.base/java.lang.Thread.run(Unknown Source)
>>
>> dependency-tree:
>>
>> ch.ricardo.di:di-beam:jar:2.14.0
>> +- org.apache.beam:beam-sdks-java-core:jar:2.32.0:compile
>> | +- org.apache.beam:beam-model-pipeline:jar:2.32.0:compile
>> | +- org.apache.beam:beam-model-job-management:jar:2.32.0:compile
>> | +- org.apache.beam:beam-vendor-bytebuddy-1_11_0:jar:0.1:compile
>> | +- org.apache.beam:beam-vendor-grpc-1_36_0:jar:0.2:compile
>> | +- org.apache.beam:beam-vendor-guava-26_0-jre:jar:0.1:compile
>> | +- com.google.code.findbugs:jsr305:jar:3.0.2:compile
>> | +- com.google.errorprone:error_prone_annotations:jar:2.3.4:compile
>> | +- com.fasterxml.jackson.core:jackson-core:jar:2.12.1:compile
>> | +- com.fasterxml.jackson.core:jackson-annotations:jar:2.12.1:compile
>> | +- com.fasterxml.jackson.core:jackson-databind:jar:2.12.1:compile
>> | +- org.apache.avro:avro:jar:1.8.2:compile
>> | | +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
>> | | +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
>> | | +- com.thoughtworks.paranamer:paranamer:jar:2.7:compile
>> | | +- org.apache.commons:commons-compress:jar:1.8.1:compile
>> | | \- org.tukaani:xz:jar:1.5:compile
>> | \- org.xerial.snappy:snappy-java:jar:1.1.8.4:compile
>> +-
>> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.32.0:compile
>> | +- org.checkerframework:checker-qual:jar:3.10.0:compile
>> | +- org.apache.beam:beam-runners-core-java:jar:2.32.0:compile
>> | | \- org.apache.beam:beam-sdks-java-fn-execution:jar:2.32.0:compile
>> | | \- org.apache.beam:beam-model-fn-execution:jar:2.32.0:compile
>> | +- org.apache.beam:beam-sdks-java-expansion-service:jar:2.32.0:compile
>> | | \-
>> org.apache.beam:beam-runners-java-fn-execution:jar:2.32.0:compile
>> | +-
>> org.apache.beam:beam-sdks-java-extensions-google-cloud-platform-core:jar:2.32.0:compile
>> | | +- com.google.cloud.bigdataoss:gcsio:jar:2.2.2:compile
>> | | \-
>> com.google.apis:google-api-services-cloudresourcemanager:jar:v1-rev20210331-1.31.0:compile
>> | +-
>> org.apache.beam:beam-runners-core-construction-java:jar:2.32.0:compile
>> | | \- io.github.classgraph:classgraph:jar:4.8.104:compile
>> | +- org.apache.beam:beam-sdks-java-extensions-arrow:jar:2.32.0:compile
>> | +- com.google.cloud.bigdataoss:util:jar:2.2.2:compile
>> | | +-
>> com.google.api-client:google-api-client-jackson2:jar:1.31.3:compile
>> | | +-
>> com.google.apis:google-api-services-iamcredentials:jar:v1-rev20201022-1.31.0:compile
>> | | \- com.google.flogger:google-extensions:jar:0.5.1:compile
>> | +- com.google.flogger:flogger-system-backend:jar:0.6:compile
>> | | \- com.google.flogger:flogger:jar:0.6:compile
>> | +- com.google.api:gax:jar:1.63.0:compile
>> | +- com.google.api:gax-grpc:jar:1.63.0:compile
>> | +- com.google.api:gax-httpjson:jar:0.80.0:compile
>> | +- com.google.api:api-common:jar:1.10.1:compile
>> | +-
>> com.google.apis:google-api-services-healthcare:jar:v1beta1-rev20210407-1.31.0:compile
>> | +-
>> com.google.apis:google-api-services-pubsub:jar:v1-rev20210322-1.31.0:compile
>> | +-
>> com.google.apis:google-api-services-storage:jar:v1-rev20210127-1.31.0:compile
>> | +- com.google.auth:google-auth-library-credentials:jar:0.25.2:compile
>> | +- com.google.auth:google-auth-library-oauth2-http:jar:0.25.2:compile
>> | +- com.google.cloud:google-cloud-bigquerystorage:jar:1.17.0:compile
>> | | +-
>> com.google.api.grpc:proto-google-cloud-bigquerystorage-v1alpha2:jar:0.117.0:compile
>> | | +-
>> com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta1:jar:0.117.0:compile
>> | | +- org.json:json:jar:20200518:compile
>> | | +-
>> com.google.api.grpc:grpc-google-cloud-bigquerystorage-v1beta1:jar:0.117.0:compile
>> | | +-
>> com.google.api.grpc:grpc-google-cloud-bigquerystorage-v1beta2:jar:0.117.0:compile
>> | | \-
>> com.google.api.grpc:grpc-google-cloud-bigquerystorage-v1:jar:1.17.0:compile
>> | +- com.google.cloud.bigtable:bigtable-client-core:jar:1.19.1:compile
>> | | +- com.google.api.grpc:grpc-google-common-protos:jar:2.1.0:compile
>> | | +-
>> com.google.api.grpc:grpc-google-cloud-bigtable-admin-v2:jar:1.22.0:compile
>> | | +- io.opencensus:opencensus-contrib-grpc-util:jar:0.24.0:compile
>> | | \- io.dropwizard.metrics:metrics-core:jar:3.2.6:compile
>> | +- com.google.cloud:google-cloud-core:jar:1.94.6:compile
>> | +- com.google.cloud:google-cloud-core-grpc:jar:1.94.6:compile
>> | +-
>> com.google.cloud.datastore:datastore-v1-proto-client:jar:1.6.3:compile
>> | | \-
>> com.google.http-client:google-http-client-protobuf:jar:1.39.2:compile
>> | +- com.google.cloud:google-cloud-firestore:jar:2.2.5:compile
>> | | \-
>> com.google.cloud:proto-google-cloud-firestore-bundle-v1:jar:2.2.5:compile
>> | +- com.google.cloud:google-cloud-pubsublite:jar:0.13.2:compile
>> | +- com.google.cloud:google-cloud-pubsub:jar:1.112.0:compile
>> | +- com.google.cloud:google-cloud-spanner:jar:6.2.0:compile
>> | | +-
>> com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:jar:6.2.0:compile
>> | | +-
>> com.google.api.grpc:grpc-google-cloud-spanner-admin-instance-v1:jar:6.2.0:compile
>> | | +-
>> com.google.api.grpc:grpc-google-cloud-spanner-v1:jar:6.2.0:compile
>> | | \-
>> com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:jar:6.2.0:compile
>> | +- com.google.code.gson:gson:jar:2.8.6:compile
>> | +-
>> com.google.http-client:google-http-client-jackson2:jar:1.39.2:compile
>> | +- io.grpc:grpc-alts:jar:1.37.0:compile
>> | +- io.grpc:grpc-api:jar:1.37.0:compile
>> | +- io.grpc:grpc-auth:jar:1.37.0:compile
>> | +- io.grpc:grpc-core:jar:1.37.0:compile
>> | +- io.grpc:grpc-context:jar:1.37.0:compile
>> | +- io.grpc:grpc-grpclb:jar:1.37.0:compile
>> | +- io.grpc:grpc-netty:jar:1.37.0:compile
>> | | +- io.netty:netty-codec-http2:jar:4.1.52.Final:compile
>> | | | \- io.netty:netty-codec-http:jar:4.1.52.Final:compile
>> | | \- io.netty:netty-handler-proxy:jar:4.1.52.Final:runtime
>> | | \- io.netty:netty-codec-socks:jar:4.1.52.Final:runtime
>> | +- io.grpc:grpc-netty-shaded:jar:1.37.0:compile
>> | +- io.grpc:grpc-stub:jar:1.37.0:compile
>> | +- com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:1.94.0:compile
>> | +-
>> com.google.api.grpc:grpc-google-cloud-pubsublite-v1:jar:0.13.2:compile
>> | +- com.google.guava:guava:jar:30.1-jre:compile
>> | +- org.hamcrest:hamcrest:jar:2.1:compile
>> | +- org.apache.httpcomponents:httpclient:jar:4.5.10:compile
>> | +- org.apache.httpcomponents:httpcore:jar:4.4.12:compile
>> | +- io.netty:netty-handler:jar:4.1.52.Final:compile
>> | | +- io.netty:netty-common:jar:4.1.52.Final:compile
>> | | +- io.netty:netty-resolver:jar:4.1.52.Final:compile
>> | | +- io.netty:netty-buffer:jar:4.1.52.Final:compile
>> | | +- io.netty:netty-transport:jar:4.1.52.Final:compile
>> | | \- io.netty:netty-codec:jar:4.1.52.Final:compile
>> | +- io.netty:netty-tcnative-boringssl-static:jar:2.0.33.Final:compile
>> | +-
>> com.google.api.grpc:proto-google-cloud-bigquerystorage-v1:jar:1.17.0:compile
>> | +-
>> com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta2:jar:0.117.0:compile
>> | +-
>> com.google.api.grpc:proto-google-cloud-bigtable-admin-v2:jar:1.22.0:compile
>> | +-
>> com.google.api.grpc:proto-google-cloud-bigtable-v2:jar:1.22.0:compile
>> | +-
>> com.google.api.grpc:proto-google-cloud-datastore-v1:jar:0.89.0:compile
>> | +-
>> com.google.api.grpc:proto-google-cloud-firestore-v1:jar:2.2.5:compile
>> | +- com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:1.94.0:compile
>> | +-
>> com.google.api.grpc:proto-google-cloud-pubsublite-v1:jar:0.13.2:compile
>> | +-
>> com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:jar:6.2.0:compile
>> | +- com.google.api.grpc:proto-google-cloud-spanner-v1:jar:6.2.0:compile
>> | +- com.google.api.grpc:proto-google-common-protos:jar:2.1.0:compile
>> | +- com.google.protobuf:protobuf-java:jar:3.15.8:compile
>> | +- org.apache.arrow:arrow-memory-core:jar:4.0.0:compile
>> | +- org.apache.arrow:arrow-vector:jar:4.0.0:compile
>> | | +- org.apache.arrow:arrow-format:jar:4.0.0:compile
>> | | \- com.google.flatbuffers:flatbuffers-java:jar:1.12.0:compile
>> | \- org.threeten:threetenbp:jar:1.4.4:compile
>> +- com.google.api-client:google-api-client:jar:1.31.1:compile
>> | +- com.google.oauth-client:google-oauth-client:jar:1.31.2:compile
>> | \-
>> com.google.http-client:google-http-client-apache-v2:jar:1.39.2:compile
>> +-
>> com.google.apis:google-api-services-bigquery:jar:v2-rev20210410-1.31.0:compile
>> +- com.google.http-client:google-http-client:jar:1.39.2:compile
>> | +- com.google.j2objc:j2objc-annotations:jar:1.3:compile
>> | +- io.opencensus:opencensus-api:jar:0.28.0:compile
>> | \- io.opencensus:opencensus-contrib-http-util:jar:0.28.0:compile
>> +- joda-time:joda-time:jar:2.10.10:compile
>> +- org.slf4j:slf4j-api:jar:1.7.30:compile
>> +- org.slf4j:slf4j-jdk14:jar:1.7.30:runtime
>> +- org.hamcrest:hamcrest-core:jar:2.1:compile
>> +- org.hamcrest:hamcrest-library:jar:2.1:compile
>> +- junit:junit:jar:4.13.1:compile
>> +- org.apache.beam:beam-runners-direct-java:jar:2.32.0:runtime
>> +- org.apache.beam:beam-sdks-java-io-kafka:jar:2.32.0:compile
>> | +- org.springframework:spring-expression:jar:4.3.18.RELEASE:compile
>> | | \- org.springframework:spring-core:jar:4.3.18.RELEASE:compile
>> | +- io.confluent:kafka-avro-serializer:jar:5.3.2:compile
>> | | +- io.confluent:common-config:jar:5.3.2:compile
>> | | \- io.confluent:common-utils:jar:5.3.2:compile
>> | | \- com.101tec:zkclient:jar:0.10:compile
>> | \- io.confluent:kafka-schema-registry-client:jar:5.3.2:compile
>> +- org.apache.kafka:kafka-clients:jar:2.6.0:compile
>> | +- com.github.luben:zstd-jni:jar:1.4.4-7:compile
>> | \- org.lz4:lz4-java:jar:1.7.1:compile
>> +- com.google.protobuf:protobuf-java-util:jar:3.6.1:compile
>> +- ch.ricardo.schema:proto-schemas:jar:3.1.68:compile
>> | \- javax.annotation:javax.annotation-api:jar:1.3.2:compile
>> +- org.mockito:mockito-core:jar:3.7.7:test
>> | +- net.bytebuddy:byte-buddy:jar:1.10.19:test
>> | +- net.bytebuddy:byte-buddy-agent:jar:1.10.19:test
>> | \- org.objenesis:objenesis:jar:3.1:test
>> +- com.google.auto.value:auto-value-annotations:jar:1.6.2:compile
>> +- com.google.auto.value:auto-value:jar:1.6.2:provided
>> +- org.projectlombok:lombok:jar:1.16.22:compile
>> +- com.jayway.jsonpath:json-path:jar:2.3.0:compile
>> | \- net.minidev:json-smart:jar:2.3:compile
>> | \- net.minidev:accessors-smart:jar:1.2:compile
>> | \- org.ow2.asm:asm:jar:5.0.4:compile
>> +- org.apache.commons:commons-lang3:jar:3.8.1:compile
>> +- com.google.cloud:google-cloud-bigquery:jar:1.127.12:compile
>> | +- com.google.api.grpc:proto-google-iam-v1:jar:1.0.12:compile
>> | +- com.google.http-client:google-http-client-gson:jar:1.39.2:compile
>> | +- com.google.cloud:google-cloud-core-http:jar:1.94.6:compile
>> | +-
>> com.google.http-client:google-http-client-appengine:jar:1.39.2:compile
>> | +- commons-logging:commons-logging:jar:1.2:compile
>> | +- commons-codec:commons-codec:jar:1.15:compile
>> | +- org.checkerframework:checker-compat-qual:jar:2.5.5:compile
>> | +- com.google.guava:failureaccess:jar:1.0.1:compile
>> | \-
>> com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:compile
>> +- com.google.cloud:google-cloud-bigtable:jar:1.22.0:compile
>> | +-
>> com.google.api.grpc:grpc-google-cloud-bigtable-v2:jar:1.22.0:compile
>> | +- org.conscrypt:conscrypt-openjdk-uber:jar:2.5.1:compile
>> | +- org.codehaus.mojo:animal-sniffer-annotations:jar:1.20:runtime
>> | +- com.google.android:annotations:jar:4.1.1.4:runtime
>> | +- io.perfmark:perfmark-api:jar:0.23.0:runtime
>> | +- io.grpc:grpc-protobuf:jar:1.37.0:compile
>> | \- io.grpc:grpc-protobuf-lite:jar:1.37.0:compile
>> +- org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.32.0:compile
>> +- org.jetbrains:annotations:jar:20.1.0:compile
>> +- com.google.cloud:google-cloud-bigtable-emulator:jar:0.128.2:test
>> \- commons-io:commons-io:jar:2.6:compile
>>
>> Best,
>> Tobi
>>
>> On Thu, Sep 23, 2021 at 6:15 AM Tomo Suzuki <su...@google.com> wrote:
>>
>>> From the package name, I think you want to focus on grpc-netty-shaded
>>> artifact rather than netty.
>>>
>>> Stacktrace and the dependency tree would be helpful to understand why
>>> the discrepancy occurred.
>>>
>>> Regards,
>>> Tomo
>>>
>>> On Wed, Sep 22, 2021 at 12:21 Reuven Lax <re...@google.com> wrote:
>>>
>>>> Are you using any of the GCP IOs in your pipeline?
>>>>
>>>> On Wed, Sep 22, 2021 at 9:11 AM Kaymak, Tobias <
>>>> tobias.kaymak@ricardo.ch> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> while upgrading our cluster and pipelines to the new visions I noticed:
>>>>>
>>>>> java.lang.NoClassDefFoundError:
>>>>> io/grpc/netty/shaded/io/netty/channel/FailedChannelFuture
>>>>>
>>>>> so I checked the maven dependency tree and found that the:
>>>>>
>>>>>
>>>>> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.32.0:compile
>>>>>
>>>>> dependency is using netty 4.1.52 - while Flink 1.13.1 seems to use
>>>>> 4.1.49.
>>>>>
>>>>> Is there a way to fix this?
>>>>>
>>>>> Best,
>>>>> Tobi
>>>>>
>>>> --
>>> Regards,
>>> Tomo
>>>
>>
Re: Beam/Flink's netty versions seems to clash (2.32.0 / 1.13.1)
Posted by "Kaymak, Tobias" <to...@ricardo.ch>.
Hi +Reuven Lax <re...@google.com> -
I saw your post when researching the error above:
https://stackoverflow.com/a/69111493 :)
As we are on Flink 1.13.1 in the middle of a move, it would be tough to
have to downgrade Flink to downgrade Beam here.
Is the warning something of concern or can we expect the data to be
delivered (and ignore this warning for now, and wait for the next release
and then upgrade to it)?
Best,
Tobi
On Thu, Sep 23, 2021 at 2:51 PM Kaymak, Tobias <to...@ricardo.ch>
wrote:
> Hello,
> I am using the GCP IOs to connect to BigQuery and Bigtable. I seem to have
> improved the situation by including Bigtable and BigQuery in my POM
> with the exact same version as the Beam 2.32.0 SDK. However I still get a
> lot of these errors:
>
> io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference
> cleanQueue
> SEVERE: *~*~*~ Channel ManagedChannelImpl{logId=3, target=
> bigquerystorage.googleapis.com:443} was not shutdown properly!!! ~*~*~*
> Make sure to call shutdown()/shutdownNow() and wait until
> awaitTermination() returns true.
> java.lang.RuntimeException: ManagedChannel allocation site
> at
> io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.<init>(ManagedChannelOrphanWrapper.java:93)
> at
> io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:53)
> at
> io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:44)
> at
> io.grpc.internal.ManagedChannelImplBuilder.build(ManagedChannelImplBuilder.java:615)
> at
> io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:261)
> at
> com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createSingleChannel(InstantiatingGrpcChannelProvider.java:327)
> at
> com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.access$1700(InstantiatingGrpcChannelProvider.java:74)
> at
> com.google.api.gax.grpc.InstantiatingGrpcChannelProvider$1.createSingleChannel(InstantiatingGrpcChannelProvider.java:220)
> at com.google.api.gax.grpc.ChannelPool.create(ChannelPool.java:72)
> at
> com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createChannel(InstantiatingGrpcChannelProvider.java:227)
> at
> com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.getTransportChannel(InstantiatingGrpcChannelProvider.java:210)
> at
> com.google.api.gax.rpc.ClientContext.create(ClientContext.java:169)
> at
> com.google.cloud.bigquery.storage.v1beta2.stub.GrpcBigQueryWriteStub.create(GrpcBigQueryWriteStub.java:138)
> at
> com.google.cloud.bigquery.storage.v1beta2.stub.BigQueryWriteStubSettings.createStub(BigQueryWriteStubSettings.java:145)
> at
> com.google.cloud.bigquery.storage.v1beta2.BigQueryWriteClient.<init>(BigQueryWriteClient.java:128)
> at
> com.google.cloud.bigquery.storage.v1beta2.BigQueryWriteClient.create(BigQueryWriteClient.java:109)
> at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.newBigQueryWriteClient(BigQueryServicesImpl.java:1279)
> at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.access$800(BigQueryServicesImpl.java:139)
> at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.<init>(BigQueryServicesImpl.java:510)
> at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.<init>(BigQueryServicesImpl.java:455)
> at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.getDatasetService(BigQueryServicesImpl.java:172)
> at
> org.apache.beam.sdk.io.gcp.bigquery.WriteTables$WriteTablesDoFn.finishBundle(WriteTables.java:284)
> at
> org.apache.beam.sdk.io.gcp.bigquery.WriteTables$WriteTablesDoFn$DoFnInvoker.invokeFinishBundle(Unknown
> Source)
> at
> org.apache.beam.runners.core.SimpleDoFnRunner.finishBundle(SimpleDoFnRunner.java:242)
> at
> org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate.finishBundle(DoFnRunnerWithMetricsUpdate.java:89)
> at
> org.apache.beam.runners.core.SimplePushbackSideInputDoFnRunner.finishBundle(SimplePushbackSideInputDoFnRunner.java:125)
> at
> org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.invokeFinishBundle(DoFnOperator.java:894)
> at
> org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.checkInvokeFinishBundleByTime(DoFnOperator.java:887)
> at
> org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.lambda$open$2(DoFnOperator.java:529)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1419)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$null$16(StreamTask.java:1410)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsNonBlocking(MailboxProcessor.java:359)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:323)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:202)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:681)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.executeInvoke(StreamTask.java:636)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:647)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:620)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:779)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)
> at java.base/java.lang.Thread.run(Unknown Source)
>
> dependency-tree:
>
> ch.ricardo.di:di-beam:jar:2.14.0
> +- org.apache.beam:beam-sdks-java-core:jar:2.32.0:compile
> | +- org.apache.beam:beam-model-pipeline:jar:2.32.0:compile
> | +- org.apache.beam:beam-model-job-management:jar:2.32.0:compile
> | +- org.apache.beam:beam-vendor-bytebuddy-1_11_0:jar:0.1:compile
> | +- org.apache.beam:beam-vendor-grpc-1_36_0:jar:0.2:compile
> | +- org.apache.beam:beam-vendor-guava-26_0-jre:jar:0.1:compile
> | +- com.google.code.findbugs:jsr305:jar:3.0.2:compile
> | +- com.google.errorprone:error_prone_annotations:jar:2.3.4:compile
> | +- com.fasterxml.jackson.core:jackson-core:jar:2.12.1:compile
> | +- com.fasterxml.jackson.core:jackson-annotations:jar:2.12.1:compile
> | +- com.fasterxml.jackson.core:jackson-databind:jar:2.12.1:compile
> | +- org.apache.avro:avro:jar:1.8.2:compile
> | | +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
> | | +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
> | | +- com.thoughtworks.paranamer:paranamer:jar:2.7:compile
> | | +- org.apache.commons:commons-compress:jar:1.8.1:compile
> | | \- org.tukaani:xz:jar:1.5:compile
> | \- org.xerial.snappy:snappy-java:jar:1.1.8.4:compile
> +-
> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.32.0:compile
> | +- org.checkerframework:checker-qual:jar:3.10.0:compile
> | +- org.apache.beam:beam-runners-core-java:jar:2.32.0:compile
> | | \- org.apache.beam:beam-sdks-java-fn-execution:jar:2.32.0:compile
> | | \- org.apache.beam:beam-model-fn-execution:jar:2.32.0:compile
> | +- org.apache.beam:beam-sdks-java-expansion-service:jar:2.32.0:compile
> | | \- org.apache.beam:beam-runners-java-fn-execution:jar:2.32.0:compile
> | +-
> org.apache.beam:beam-sdks-java-extensions-google-cloud-platform-core:jar:2.32.0:compile
> | | +- com.google.cloud.bigdataoss:gcsio:jar:2.2.2:compile
> | | \-
> com.google.apis:google-api-services-cloudresourcemanager:jar:v1-rev20210331-1.31.0:compile
> | +-
> org.apache.beam:beam-runners-core-construction-java:jar:2.32.0:compile
> | | \- io.github.classgraph:classgraph:jar:4.8.104:compile
> | +- org.apache.beam:beam-sdks-java-extensions-arrow:jar:2.32.0:compile
> | +- com.google.cloud.bigdataoss:util:jar:2.2.2:compile
> | | +-
> com.google.api-client:google-api-client-jackson2:jar:1.31.3:compile
> | | +-
> com.google.apis:google-api-services-iamcredentials:jar:v1-rev20201022-1.31.0:compile
> | | \- com.google.flogger:google-extensions:jar:0.5.1:compile
> | +- com.google.flogger:flogger-system-backend:jar:0.6:compile
> | | \- com.google.flogger:flogger:jar:0.6:compile
> | +- com.google.api:gax:jar:1.63.0:compile
> | +- com.google.api:gax-grpc:jar:1.63.0:compile
> | +- com.google.api:gax-httpjson:jar:0.80.0:compile
> | +- com.google.api:api-common:jar:1.10.1:compile
> | +-
> com.google.apis:google-api-services-healthcare:jar:v1beta1-rev20210407-1.31.0:compile
> | +-
> com.google.apis:google-api-services-pubsub:jar:v1-rev20210322-1.31.0:compile
> | +-
> com.google.apis:google-api-services-storage:jar:v1-rev20210127-1.31.0:compile
> | +- com.google.auth:google-auth-library-credentials:jar:0.25.2:compile
> | +- com.google.auth:google-auth-library-oauth2-http:jar:0.25.2:compile
> | +- com.google.cloud:google-cloud-bigquerystorage:jar:1.17.0:compile
> | | +-
> com.google.api.grpc:proto-google-cloud-bigquerystorage-v1alpha2:jar:0.117.0:compile
> | | +-
> com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta1:jar:0.117.0:compile
> | | +- org.json:json:jar:20200518:compile
> | | +-
> com.google.api.grpc:grpc-google-cloud-bigquerystorage-v1beta1:jar:0.117.0:compile
> | | +-
> com.google.api.grpc:grpc-google-cloud-bigquerystorage-v1beta2:jar:0.117.0:compile
> | | \-
> com.google.api.grpc:grpc-google-cloud-bigquerystorage-v1:jar:1.17.0:compile
> | +- com.google.cloud.bigtable:bigtable-client-core:jar:1.19.1:compile
> | | +- com.google.api.grpc:grpc-google-common-protos:jar:2.1.0:compile
> | | +-
> com.google.api.grpc:grpc-google-cloud-bigtable-admin-v2:jar:1.22.0:compile
> | | +- io.opencensus:opencensus-contrib-grpc-util:jar:0.24.0:compile
> | | \- io.dropwizard.metrics:metrics-core:jar:3.2.6:compile
> | +- com.google.cloud:google-cloud-core:jar:1.94.6:compile
> | +- com.google.cloud:google-cloud-core-grpc:jar:1.94.6:compile
> | +-
> com.google.cloud.datastore:datastore-v1-proto-client:jar:1.6.3:compile
> | | \-
> com.google.http-client:google-http-client-protobuf:jar:1.39.2:compile
> | +- com.google.cloud:google-cloud-firestore:jar:2.2.5:compile
> | | \-
> com.google.cloud:proto-google-cloud-firestore-bundle-v1:jar:2.2.5:compile
> | +- com.google.cloud:google-cloud-pubsublite:jar:0.13.2:compile
> | +- com.google.cloud:google-cloud-pubsub:jar:1.112.0:compile
> | +- com.google.cloud:google-cloud-spanner:jar:6.2.0:compile
> | | +-
> com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:jar:6.2.0:compile
> | | +-
> com.google.api.grpc:grpc-google-cloud-spanner-admin-instance-v1:jar:6.2.0:compile
> | | +-
> com.google.api.grpc:grpc-google-cloud-spanner-v1:jar:6.2.0:compile
> | | \-
> com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:jar:6.2.0:compile
> | +- com.google.code.gson:gson:jar:2.8.6:compile
> | +-
> com.google.http-client:google-http-client-jackson2:jar:1.39.2:compile
> | +- io.grpc:grpc-alts:jar:1.37.0:compile
> | +- io.grpc:grpc-api:jar:1.37.0:compile
> | +- io.grpc:grpc-auth:jar:1.37.0:compile
> | +- io.grpc:grpc-core:jar:1.37.0:compile
> | +- io.grpc:grpc-context:jar:1.37.0:compile
> | +- io.grpc:grpc-grpclb:jar:1.37.0:compile
> | +- io.grpc:grpc-netty:jar:1.37.0:compile
> | | +- io.netty:netty-codec-http2:jar:4.1.52.Final:compile
> | | | \- io.netty:netty-codec-http:jar:4.1.52.Final:compile
> | | \- io.netty:netty-handler-proxy:jar:4.1.52.Final:runtime
> | | \- io.netty:netty-codec-socks:jar:4.1.52.Final:runtime
> | +- io.grpc:grpc-netty-shaded:jar:1.37.0:compile
> | +- io.grpc:grpc-stub:jar:1.37.0:compile
> | +- com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:1.94.0:compile
> | +-
> com.google.api.grpc:grpc-google-cloud-pubsublite-v1:jar:0.13.2:compile
> | +- com.google.guava:guava:jar:30.1-jre:compile
> | +- org.hamcrest:hamcrest:jar:2.1:compile
> | +- org.apache.httpcomponents:httpclient:jar:4.5.10:compile
> | +- org.apache.httpcomponents:httpcore:jar:4.4.12:compile
> | +- io.netty:netty-handler:jar:4.1.52.Final:compile
> | | +- io.netty:netty-common:jar:4.1.52.Final:compile
> | | +- io.netty:netty-resolver:jar:4.1.52.Final:compile
> | | +- io.netty:netty-buffer:jar:4.1.52.Final:compile
> | | +- io.netty:netty-transport:jar:4.1.52.Final:compile
> | | \- io.netty:netty-codec:jar:4.1.52.Final:compile
> | +- io.netty:netty-tcnative-boringssl-static:jar:2.0.33.Final:compile
> | +-
> com.google.api.grpc:proto-google-cloud-bigquerystorage-v1:jar:1.17.0:compile
> | +-
> com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta2:jar:0.117.0:compile
> | +-
> com.google.api.grpc:proto-google-cloud-bigtable-admin-v2:jar:1.22.0:compile
> | +-
> com.google.api.grpc:proto-google-cloud-bigtable-v2:jar:1.22.0:compile
> | +-
> com.google.api.grpc:proto-google-cloud-datastore-v1:jar:0.89.0:compile
> | +-
> com.google.api.grpc:proto-google-cloud-firestore-v1:jar:2.2.5:compile
> | +- com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:1.94.0:compile
> | +-
> com.google.api.grpc:proto-google-cloud-pubsublite-v1:jar:0.13.2:compile
> | +-
> com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:jar:6.2.0:compile
> | +- com.google.api.grpc:proto-google-cloud-spanner-v1:jar:6.2.0:compile
> | +- com.google.api.grpc:proto-google-common-protos:jar:2.1.0:compile
> | +- com.google.protobuf:protobuf-java:jar:3.15.8:compile
> | +- org.apache.arrow:arrow-memory-core:jar:4.0.0:compile
> | +- org.apache.arrow:arrow-vector:jar:4.0.0:compile
> | | +- org.apache.arrow:arrow-format:jar:4.0.0:compile
> | | \- com.google.flatbuffers:flatbuffers-java:jar:1.12.0:compile
> | \- org.threeten:threetenbp:jar:1.4.4:compile
> +- com.google.api-client:google-api-client:jar:1.31.1:compile
> | +- com.google.oauth-client:google-oauth-client:jar:1.31.2:compile
> | \-
> com.google.http-client:google-http-client-apache-v2:jar:1.39.2:compile
> +-
> com.google.apis:google-api-services-bigquery:jar:v2-rev20210410-1.31.0:compile
> +- com.google.http-client:google-http-client:jar:1.39.2:compile
> | +- com.google.j2objc:j2objc-annotations:jar:1.3:compile
> | +- io.opencensus:opencensus-api:jar:0.28.0:compile
> | \- io.opencensus:opencensus-contrib-http-util:jar:0.28.0:compile
> +- joda-time:joda-time:jar:2.10.10:compile
> +- org.slf4j:slf4j-api:jar:1.7.30:compile
> +- org.slf4j:slf4j-jdk14:jar:1.7.30:runtime
> +- org.hamcrest:hamcrest-core:jar:2.1:compile
> +- org.hamcrest:hamcrest-library:jar:2.1:compile
> +- junit:junit:jar:4.13.1:compile
> +- org.apache.beam:beam-runners-direct-java:jar:2.32.0:runtime
> +- org.apache.beam:beam-sdks-java-io-kafka:jar:2.32.0:compile
> | +- org.springframework:spring-expression:jar:4.3.18.RELEASE:compile
> | | \- org.springframework:spring-core:jar:4.3.18.RELEASE:compile
> | +- io.confluent:kafka-avro-serializer:jar:5.3.2:compile
> | | +- io.confluent:common-config:jar:5.3.2:compile
> | | \- io.confluent:common-utils:jar:5.3.2:compile
> | | \- com.101tec:zkclient:jar:0.10:compile
> | \- io.confluent:kafka-schema-registry-client:jar:5.3.2:compile
> +- org.apache.kafka:kafka-clients:jar:2.6.0:compile
> | +- com.github.luben:zstd-jni:jar:1.4.4-7:compile
> | \- org.lz4:lz4-java:jar:1.7.1:compile
> +- com.google.protobuf:protobuf-java-util:jar:3.6.1:compile
> +- ch.ricardo.schema:proto-schemas:jar:3.1.68:compile
> | \- javax.annotation:javax.annotation-api:jar:1.3.2:compile
> +- org.mockito:mockito-core:jar:3.7.7:test
> | +- net.bytebuddy:byte-buddy:jar:1.10.19:test
> | +- net.bytebuddy:byte-buddy-agent:jar:1.10.19:test
> | \- org.objenesis:objenesis:jar:3.1:test
> +- com.google.auto.value:auto-value-annotations:jar:1.6.2:compile
> +- com.google.auto.value:auto-value:jar:1.6.2:provided
> +- org.projectlombok:lombok:jar:1.16.22:compile
> +- com.jayway.jsonpath:json-path:jar:2.3.0:compile
> | \- net.minidev:json-smart:jar:2.3:compile
> | \- net.minidev:accessors-smart:jar:1.2:compile
> | \- org.ow2.asm:asm:jar:5.0.4:compile
> +- org.apache.commons:commons-lang3:jar:3.8.1:compile
> +- com.google.cloud:google-cloud-bigquery:jar:1.127.12:compile
> | +- com.google.api.grpc:proto-google-iam-v1:jar:1.0.12:compile
> | +- com.google.http-client:google-http-client-gson:jar:1.39.2:compile
> | +- com.google.cloud:google-cloud-core-http:jar:1.94.6:compile
> | +-
> com.google.http-client:google-http-client-appengine:jar:1.39.2:compile
> | +- commons-logging:commons-logging:jar:1.2:compile
> | +- commons-codec:commons-codec:jar:1.15:compile
> | +- org.checkerframework:checker-compat-qual:jar:2.5.5:compile
> | +- com.google.guava:failureaccess:jar:1.0.1:compile
> | \-
> com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:compile
> +- com.google.cloud:google-cloud-bigtable:jar:1.22.0:compile
> | +- com.google.api.grpc:grpc-google-cloud-bigtable-v2:jar:1.22.0:compile
> | +- org.conscrypt:conscrypt-openjdk-uber:jar:2.5.1:compile
> | +- org.codehaus.mojo:animal-sniffer-annotations:jar:1.20:runtime
> | +- com.google.android:annotations:jar:4.1.1.4:runtime
> | +- io.perfmark:perfmark-api:jar:0.23.0:runtime
> | +- io.grpc:grpc-protobuf:jar:1.37.0:compile
> | \- io.grpc:grpc-protobuf-lite:jar:1.37.0:compile
> +- org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.32.0:compile
> +- org.jetbrains:annotations:jar:20.1.0:compile
> +- com.google.cloud:google-cloud-bigtable-emulator:jar:0.128.2:test
> \- commons-io:commons-io:jar:2.6:compile
>
> Best,
> Tobi
>
> On Thu, Sep 23, 2021 at 6:15 AM Tomo Suzuki <su...@google.com> wrote:
>
>> From the package name, I think you want to focus on grpc-netty-shaded
>> artifact rather than netty.
>>
>> Stacktrace and the dependency tree would be helpful to understand why the
>> discrepancy occurred.
>>
>> Regards,
>> Tomo
>>
>> On Wed, Sep 22, 2021 at 12:21 Reuven Lax <re...@google.com> wrote:
>>
>>> Are you using any of the GCP IOs in your pipeline?
>>>
>>> On Wed, Sep 22, 2021 at 9:11 AM Kaymak, Tobias <to...@ricardo.ch>
>>> wrote:
>>>
>>>> Hello,
>>>>
>>>> while upgrading our cluster and pipelines to the new visions I noticed:
>>>>
>>>> java.lang.NoClassDefFoundError:
>>>> io/grpc/netty/shaded/io/netty/channel/FailedChannelFuture
>>>>
>>>> so I checked the maven dependency tree and found that the:
>>>>
>>>>
>>>> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.32.0:compile
>>>>
>>>> dependency is using netty 4.1.52 - while Flink 1.13.1 seems to use
>>>> 4.1.49.
>>>>
>>>> Is there a way to fix this?
>>>>
>>>> Best,
>>>> Tobi
>>>>
>>> --
>> Regards,
>> Tomo
>>
>
Re: Beam/Flink's netty versions seems to clash (2.32.0 / 1.13.1)
Posted by "Kaymak, Tobias" <to...@ricardo.ch>.
Hello,
I am using the GCP IOs to connect to BigQuery and Bigtable. I seem to have
improved the situation by including Bigtable and BigQuery in my POM
with the exact same version as the Beam 2.32.0 SDK. However I still get a
lot of these errors:
io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference
cleanQueue
SEVERE: *~*~*~ Channel ManagedChannelImpl{logId=3, target=
bigquerystorage.googleapis.com:443} was not shutdown properly!!! ~*~*~*
Make sure to call shutdown()/shutdownNow() and wait until
awaitTermination() returns true.
java.lang.RuntimeException: ManagedChannel allocation site
at
io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.<init>(ManagedChannelOrphanWrapper.java:93)
at
io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:53)
at
io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:44)
at
io.grpc.internal.ManagedChannelImplBuilder.build(ManagedChannelImplBuilder.java:615)
at
io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:261)
at
com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createSingleChannel(InstantiatingGrpcChannelProvider.java:327)
at
com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.access$1700(InstantiatingGrpcChannelProvider.java:74)
at
com.google.api.gax.grpc.InstantiatingGrpcChannelProvider$1.createSingleChannel(InstantiatingGrpcChannelProvider.java:220)
at com.google.api.gax.grpc.ChannelPool.create(ChannelPool.java:72)
at
com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createChannel(InstantiatingGrpcChannelProvider.java:227)
at
com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.getTransportChannel(InstantiatingGrpcChannelProvider.java:210)
at
com.google.api.gax.rpc.ClientContext.create(ClientContext.java:169)
at
com.google.cloud.bigquery.storage.v1beta2.stub.GrpcBigQueryWriteStub.create(GrpcBigQueryWriteStub.java:138)
at
com.google.cloud.bigquery.storage.v1beta2.stub.BigQueryWriteStubSettings.createStub(BigQueryWriteStubSettings.java:145)
at
com.google.cloud.bigquery.storage.v1beta2.BigQueryWriteClient.<init>(BigQueryWriteClient.java:128)
at
com.google.cloud.bigquery.storage.v1beta2.BigQueryWriteClient.create(BigQueryWriteClient.java:109)
at
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.newBigQueryWriteClient(BigQueryServicesImpl.java:1279)
at
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.access$800(BigQueryServicesImpl.java:139)
at
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.<init>(BigQueryServicesImpl.java:510)
at
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.<init>(BigQueryServicesImpl.java:455)
at
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.getDatasetService(BigQueryServicesImpl.java:172)
at
org.apache.beam.sdk.io.gcp.bigquery.WriteTables$WriteTablesDoFn.finishBundle(WriteTables.java:284)
at
org.apache.beam.sdk.io.gcp.bigquery.WriteTables$WriteTablesDoFn$DoFnInvoker.invokeFinishBundle(Unknown
Source)
at
org.apache.beam.runners.core.SimpleDoFnRunner.finishBundle(SimpleDoFnRunner.java:242)
at
org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate.finishBundle(DoFnRunnerWithMetricsUpdate.java:89)
at
org.apache.beam.runners.core.SimplePushbackSideInputDoFnRunner.finishBundle(SimplePushbackSideInputDoFnRunner.java:125)
at
org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.invokeFinishBundle(DoFnOperator.java:894)
at
org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.checkInvokeFinishBundleByTime(DoFnOperator.java:887)
at
org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.lambda$open$2(DoFnOperator.java:529)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1419)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$null$16(StreamTask.java:1410)
at
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
at
org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90)
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsNonBlocking(MailboxProcessor.java:359)
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:323)
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:202)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:681)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.executeInvoke(StreamTask.java:636)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:647)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:620)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:779)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)
at java.base/java.lang.Thread.run(Unknown Source)
dependency-tree:
ch.ricardo.di:di-beam:jar:2.14.0
+- org.apache.beam:beam-sdks-java-core:jar:2.32.0:compile
| +- org.apache.beam:beam-model-pipeline:jar:2.32.0:compile
| +- org.apache.beam:beam-model-job-management:jar:2.32.0:compile
| +- org.apache.beam:beam-vendor-bytebuddy-1_11_0:jar:0.1:compile
| +- org.apache.beam:beam-vendor-grpc-1_36_0:jar:0.2:compile
| +- org.apache.beam:beam-vendor-guava-26_0-jre:jar:0.1:compile
| +- com.google.code.findbugs:jsr305:jar:3.0.2:compile
| +- com.google.errorprone:error_prone_annotations:jar:2.3.4:compile
| +- com.fasterxml.jackson.core:jackson-core:jar:2.12.1:compile
| +- com.fasterxml.jackson.core:jackson-annotations:jar:2.12.1:compile
| +- com.fasterxml.jackson.core:jackson-databind:jar:2.12.1:compile
| +- org.apache.avro:avro:jar:1.8.2:compile
| | +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
| | +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
| | +- com.thoughtworks.paranamer:paranamer:jar:2.7:compile
| | +- org.apache.commons:commons-compress:jar:1.8.1:compile
| | \- org.tukaani:xz:jar:1.5:compile
| \- org.xerial.snappy:snappy-java:jar:1.1.8.4:compile
+-
org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.32.0:compile
| +- org.checkerframework:checker-qual:jar:3.10.0:compile
| +- org.apache.beam:beam-runners-core-java:jar:2.32.0:compile
| | \- org.apache.beam:beam-sdks-java-fn-execution:jar:2.32.0:compile
| | \- org.apache.beam:beam-model-fn-execution:jar:2.32.0:compile
| +- org.apache.beam:beam-sdks-java-expansion-service:jar:2.32.0:compile
| | \- org.apache.beam:beam-runners-java-fn-execution:jar:2.32.0:compile
| +-
org.apache.beam:beam-sdks-java-extensions-google-cloud-platform-core:jar:2.32.0:compile
| | +- com.google.cloud.bigdataoss:gcsio:jar:2.2.2:compile
| | \-
com.google.apis:google-api-services-cloudresourcemanager:jar:v1-rev20210331-1.31.0:compile
| +-
org.apache.beam:beam-runners-core-construction-java:jar:2.32.0:compile
| | \- io.github.classgraph:classgraph:jar:4.8.104:compile
| +- org.apache.beam:beam-sdks-java-extensions-arrow:jar:2.32.0:compile
| +- com.google.cloud.bigdataoss:util:jar:2.2.2:compile
| | +-
com.google.api-client:google-api-client-jackson2:jar:1.31.3:compile
| | +-
com.google.apis:google-api-services-iamcredentials:jar:v1-rev20201022-1.31.0:compile
| | \- com.google.flogger:google-extensions:jar:0.5.1:compile
| +- com.google.flogger:flogger-system-backend:jar:0.6:compile
| | \- com.google.flogger:flogger:jar:0.6:compile
| +- com.google.api:gax:jar:1.63.0:compile
| +- com.google.api:gax-grpc:jar:1.63.0:compile
| +- com.google.api:gax-httpjson:jar:0.80.0:compile
| +- com.google.api:api-common:jar:1.10.1:compile
| +-
com.google.apis:google-api-services-healthcare:jar:v1beta1-rev20210407-1.31.0:compile
| +-
com.google.apis:google-api-services-pubsub:jar:v1-rev20210322-1.31.0:compile
| +-
com.google.apis:google-api-services-storage:jar:v1-rev20210127-1.31.0:compile
| +- com.google.auth:google-auth-library-credentials:jar:0.25.2:compile
| +- com.google.auth:google-auth-library-oauth2-http:jar:0.25.2:compile
| +- com.google.cloud:google-cloud-bigquerystorage:jar:1.17.0:compile
| | +-
com.google.api.grpc:proto-google-cloud-bigquerystorage-v1alpha2:jar:0.117.0:compile
| | +-
com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta1:jar:0.117.0:compile
| | +- org.json:json:jar:20200518:compile
| | +-
com.google.api.grpc:grpc-google-cloud-bigquerystorage-v1beta1:jar:0.117.0:compile
| | +-
com.google.api.grpc:grpc-google-cloud-bigquerystorage-v1beta2:jar:0.117.0:compile
| | \-
com.google.api.grpc:grpc-google-cloud-bigquerystorage-v1:jar:1.17.0:compile
| +- com.google.cloud.bigtable:bigtable-client-core:jar:1.19.1:compile
| | +- com.google.api.grpc:grpc-google-common-protos:jar:2.1.0:compile
| | +-
com.google.api.grpc:grpc-google-cloud-bigtable-admin-v2:jar:1.22.0:compile
| | +- io.opencensus:opencensus-contrib-grpc-util:jar:0.24.0:compile
| | \- io.dropwizard.metrics:metrics-core:jar:3.2.6:compile
| +- com.google.cloud:google-cloud-core:jar:1.94.6:compile
| +- com.google.cloud:google-cloud-core-grpc:jar:1.94.6:compile
| +-
com.google.cloud.datastore:datastore-v1-proto-client:jar:1.6.3:compile
| | \-
com.google.http-client:google-http-client-protobuf:jar:1.39.2:compile
| +- com.google.cloud:google-cloud-firestore:jar:2.2.5:compile
| | \-
com.google.cloud:proto-google-cloud-firestore-bundle-v1:jar:2.2.5:compile
| +- com.google.cloud:google-cloud-pubsublite:jar:0.13.2:compile
| +- com.google.cloud:google-cloud-pubsub:jar:1.112.0:compile
| +- com.google.cloud:google-cloud-spanner:jar:6.2.0:compile
| | +-
com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:jar:6.2.0:compile
| | +-
com.google.api.grpc:grpc-google-cloud-spanner-admin-instance-v1:jar:6.2.0:compile
| | +- com.google.api.grpc:grpc-google-cloud-spanner-v1:jar:6.2.0:compile
| | \-
com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:jar:6.2.0:compile
| +- com.google.code.gson:gson:jar:2.8.6:compile
| +- com.google.http-client:google-http-client-jackson2:jar:1.39.2:compile
| +- io.grpc:grpc-alts:jar:1.37.0:compile
| +- io.grpc:grpc-api:jar:1.37.0:compile
| +- io.grpc:grpc-auth:jar:1.37.0:compile
| +- io.grpc:grpc-core:jar:1.37.0:compile
| +- io.grpc:grpc-context:jar:1.37.0:compile
| +- io.grpc:grpc-grpclb:jar:1.37.0:compile
| +- io.grpc:grpc-netty:jar:1.37.0:compile
| | +- io.netty:netty-codec-http2:jar:4.1.52.Final:compile
| | | \- io.netty:netty-codec-http:jar:4.1.52.Final:compile
| | \- io.netty:netty-handler-proxy:jar:4.1.52.Final:runtime
| | \- io.netty:netty-codec-socks:jar:4.1.52.Final:runtime
| +- io.grpc:grpc-netty-shaded:jar:1.37.0:compile
| +- io.grpc:grpc-stub:jar:1.37.0:compile
| +- com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:1.94.0:compile
| +-
com.google.api.grpc:grpc-google-cloud-pubsublite-v1:jar:0.13.2:compile
| +- com.google.guava:guava:jar:30.1-jre:compile
| +- org.hamcrest:hamcrest:jar:2.1:compile
| +- org.apache.httpcomponents:httpclient:jar:4.5.10:compile
| +- org.apache.httpcomponents:httpcore:jar:4.4.12:compile
| +- io.netty:netty-handler:jar:4.1.52.Final:compile
| | +- io.netty:netty-common:jar:4.1.52.Final:compile
| | +- io.netty:netty-resolver:jar:4.1.52.Final:compile
| | +- io.netty:netty-buffer:jar:4.1.52.Final:compile
| | +- io.netty:netty-transport:jar:4.1.52.Final:compile
| | \- io.netty:netty-codec:jar:4.1.52.Final:compile
| +- io.netty:netty-tcnative-boringssl-static:jar:2.0.33.Final:compile
| +-
com.google.api.grpc:proto-google-cloud-bigquerystorage-v1:jar:1.17.0:compile
| +-
com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta2:jar:0.117.0:compile
| +-
com.google.api.grpc:proto-google-cloud-bigtable-admin-v2:jar:1.22.0:compile
| +- com.google.api.grpc:proto-google-cloud-bigtable-v2:jar:1.22.0:compile
| +-
com.google.api.grpc:proto-google-cloud-datastore-v1:jar:0.89.0:compile
| +- com.google.api.grpc:proto-google-cloud-firestore-v1:jar:2.2.5:compile
| +- com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:1.94.0:compile
| +-
com.google.api.grpc:proto-google-cloud-pubsublite-v1:jar:0.13.2:compile
| +-
com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:jar:6.2.0:compile
| +- com.google.api.grpc:proto-google-cloud-spanner-v1:jar:6.2.0:compile
| +- com.google.api.grpc:proto-google-common-protos:jar:2.1.0:compile
| +- com.google.protobuf:protobuf-java:jar:3.15.8:compile
| +- org.apache.arrow:arrow-memory-core:jar:4.0.0:compile
| +- org.apache.arrow:arrow-vector:jar:4.0.0:compile
| | +- org.apache.arrow:arrow-format:jar:4.0.0:compile
| | \- com.google.flatbuffers:flatbuffers-java:jar:1.12.0:compile
| \- org.threeten:threetenbp:jar:1.4.4:compile
+- com.google.api-client:google-api-client:jar:1.31.1:compile
| +- com.google.oauth-client:google-oauth-client:jar:1.31.2:compile
| \-
com.google.http-client:google-http-client-apache-v2:jar:1.39.2:compile
+-
com.google.apis:google-api-services-bigquery:jar:v2-rev20210410-1.31.0:compile
+- com.google.http-client:google-http-client:jar:1.39.2:compile
| +- com.google.j2objc:j2objc-annotations:jar:1.3:compile
| +- io.opencensus:opencensus-api:jar:0.28.0:compile
| \- io.opencensus:opencensus-contrib-http-util:jar:0.28.0:compile
+- joda-time:joda-time:jar:2.10.10:compile
+- org.slf4j:slf4j-api:jar:1.7.30:compile
+- org.slf4j:slf4j-jdk14:jar:1.7.30:runtime
+- org.hamcrest:hamcrest-core:jar:2.1:compile
+- org.hamcrest:hamcrest-library:jar:2.1:compile
+- junit:junit:jar:4.13.1:compile
+- org.apache.beam:beam-runners-direct-java:jar:2.32.0:runtime
+- org.apache.beam:beam-sdks-java-io-kafka:jar:2.32.0:compile
| +- org.springframework:spring-expression:jar:4.3.18.RELEASE:compile
| | \- org.springframework:spring-core:jar:4.3.18.RELEASE:compile
| +- io.confluent:kafka-avro-serializer:jar:5.3.2:compile
| | +- io.confluent:common-config:jar:5.3.2:compile
| | \- io.confluent:common-utils:jar:5.3.2:compile
| | \- com.101tec:zkclient:jar:0.10:compile
| \- io.confluent:kafka-schema-registry-client:jar:5.3.2:compile
+- org.apache.kafka:kafka-clients:jar:2.6.0:compile
| +- com.github.luben:zstd-jni:jar:1.4.4-7:compile
| \- org.lz4:lz4-java:jar:1.7.1:compile
+- com.google.protobuf:protobuf-java-util:jar:3.6.1:compile
+- ch.ricardo.schema:proto-schemas:jar:3.1.68:compile
| \- javax.annotation:javax.annotation-api:jar:1.3.2:compile
+- org.mockito:mockito-core:jar:3.7.7:test
| +- net.bytebuddy:byte-buddy:jar:1.10.19:test
| +- net.bytebuddy:byte-buddy-agent:jar:1.10.19:test
| \- org.objenesis:objenesis:jar:3.1:test
+- com.google.auto.value:auto-value-annotations:jar:1.6.2:compile
+- com.google.auto.value:auto-value:jar:1.6.2:provided
+- org.projectlombok:lombok:jar:1.16.22:compile
+- com.jayway.jsonpath:json-path:jar:2.3.0:compile
| \- net.minidev:json-smart:jar:2.3:compile
| \- net.minidev:accessors-smart:jar:1.2:compile
| \- org.ow2.asm:asm:jar:5.0.4:compile
+- org.apache.commons:commons-lang3:jar:3.8.1:compile
+- com.google.cloud:google-cloud-bigquery:jar:1.127.12:compile
| +- com.google.api.grpc:proto-google-iam-v1:jar:1.0.12:compile
| +- com.google.http-client:google-http-client-gson:jar:1.39.2:compile
| +- com.google.cloud:google-cloud-core-http:jar:1.94.6:compile
| +-
com.google.http-client:google-http-client-appengine:jar:1.39.2:compile
| +- commons-logging:commons-logging:jar:1.2:compile
| +- commons-codec:commons-codec:jar:1.15:compile
| +- org.checkerframework:checker-compat-qual:jar:2.5.5:compile
| +- com.google.guava:failureaccess:jar:1.0.1:compile
| \-
com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:compile
+- com.google.cloud:google-cloud-bigtable:jar:1.22.0:compile
| +- com.google.api.grpc:grpc-google-cloud-bigtable-v2:jar:1.22.0:compile
| +- org.conscrypt:conscrypt-openjdk-uber:jar:2.5.1:compile
| +- org.codehaus.mojo:animal-sniffer-annotations:jar:1.20:runtime
| +- com.google.android:annotations:jar:4.1.1.4:runtime
| +- io.perfmark:perfmark-api:jar:0.23.0:runtime
| +- io.grpc:grpc-protobuf:jar:1.37.0:compile
| \- io.grpc:grpc-protobuf-lite:jar:1.37.0:compile
+- org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.32.0:compile
+- org.jetbrains:annotations:jar:20.1.0:compile
+- com.google.cloud:google-cloud-bigtable-emulator:jar:0.128.2:test
\- commons-io:commons-io:jar:2.6:compile
Best,
Tobi
On Thu, Sep 23, 2021 at 6:15 AM Tomo Suzuki <su...@google.com> wrote:
> From the package name, I think you want to focus on grpc-netty-shaded
> artifact rather than netty.
>
> Stacktrace and the dependency tree would be helpful to understand why the
> discrepancy occurred.
>
> Regards,
> Tomo
>
> On Wed, Sep 22, 2021 at 12:21 Reuven Lax <re...@google.com> wrote:
>
>> Are you using any of the GCP IOs in your pipeline?
>>
>> On Wed, Sep 22, 2021 at 9:11 AM Kaymak, Tobias <to...@ricardo.ch>
>> wrote:
>>
>>> Hello,
>>>
>>> while upgrading our cluster and pipelines to the new visions I noticed:
>>>
>>> java.lang.NoClassDefFoundError:
>>> io/grpc/netty/shaded/io/netty/channel/FailedChannelFuture
>>>
>>> so I checked the maven dependency tree and found that the:
>>>
>>>
>>> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.32.0:compile
>>>
>>> dependency is using netty 4.1.52 - while Flink 1.13.1 seems to use
>>> 4.1.49.
>>>
>>> Is there a way to fix this?
>>>
>>> Best,
>>> Tobi
>>>
>> --
> Regards,
> Tomo
>
Re: Beam/Flink's netty versions seems to clash (2.32.0 / 1.13.1)
Posted by Tomo Suzuki <su...@google.com>.
From the package name, I think you want to focus on grpc-netty-shaded
artifact rather than netty.
Stacktrace and the dependency tree would be helpful to understand why the
discrepancy occurred.
Regards,
Tomo
On Wed, Sep 22, 2021 at 12:21 Reuven Lax <re...@google.com> wrote:
> Are you using any of the GCP IOs in your pipeline?
>
> On Wed, Sep 22, 2021 at 9:11 AM Kaymak, Tobias <to...@ricardo.ch>
> wrote:
>
>> Hello,
>>
>> while upgrading our cluster and pipelines to the new visions I noticed:
>>
>> java.lang.NoClassDefFoundError:
>> io/grpc/netty/shaded/io/netty/channel/FailedChannelFuture
>>
>> so I checked the maven dependency tree and found that the:
>>
>> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.32.0:compile
>>
>> dependency is using netty 4.1.52 - while Flink 1.13.1 seems to use 4.1.49.
>>
>> Is there a way to fix this?
>>
>> Best,
>> Tobi
>>
> --
Regards,
Tomo
Re: Beam/Flink's netty versions seems to clash (2.32.0 / 1.13.1)
Posted by Reuven Lax <re...@google.com>.
Are you using any of the GCP IOs in your pipeline?
On Wed, Sep 22, 2021 at 9:11 AM Kaymak, Tobias <to...@ricardo.ch>
wrote:
> Hello,
>
> while upgrading our cluster and pipelines to the new visions I noticed:
>
> java.lang.NoClassDefFoundError:
> io/grpc/netty/shaded/io/netty/channel/FailedChannelFuture
>
> so I checked the maven dependency tree and found that the:
>
> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.32.0:compile
>
> dependency is using netty 4.1.52 - while Flink 1.13.1 seems to use 4.1.49.
>
> Is there a way to fix this?
>
> Best,
> Tobi
>