You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Luke Cwik (JIRA)" <ji...@apache.org> on 2019/06/04 22:11:00 UTC
[jira] [Resolved] (BEAM-6249) Vendored gRPC doesn't seem to work
with dataflow
[ https://issues.apache.org/jira/browse/BEAM-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Luke Cwik resolved BEAM-6249.
-----------------------------
Resolution: Fixed
Assignee: Kenneth Knowles (was: Tyler Akidau)
Fix Version/s: 2.10.0
> Vendored gRPC doesn't seem to work with dataflow
> ------------------------------------------------
>
> Key: BEAM-6249
> URL: https://issues.apache.org/jira/browse/BEAM-6249
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow
> Affects Versions: 2.9.0
> Reporter: Steve Niemitz
> Assignee: Kenneth Knowles
> Priority: Major
> Fix For: 2.10.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> I attempted to migrate an existing pipeline (that worked in 2.8.0) to 2.9.0. This pipeline is using the experimental streaming engine (–experiments=enable_streaming_engine).
> The pipeline fails to start with these logs:
> {code:java}
> D Unable to load the library 'org_apache_beam_vendor_grpc_v1_13_1_netty_tcnative_linux_x86_64', trying other loading mechanism.
> D org_apache_beam_vendor_grpc_v1_13_1_netty_tcnative_linux_x86_64 cannot be loaded from java.libary.path, now trying export to -Dio.netty.native.workdir: /tmp
> D Unable to load the library '/tmp/liborg_apache_beam_vendor_grpc_v1_13_1_netty_tcnative_linux_x86_646918605450681921540.so', trying other loading mechanism.
> D Unable to load the library 'netty_tcnative_linux_x86_64', trying next name...
> D Unable to load the library 'org_apache_beam_vendor_grpc_v1_13_1_netty_tcnative_linux_x86_64_fedora', trying other loading mechanism.
> D org_apache_beam_vendor_grpc_v1_13_1_netty_tcnative_linux_x86_64_fedora cannot be loaded from java.libary.path, now trying export to -Dio.netty.native.workdir: /tmp
> D Unable to load the library 'netty_tcnative_linux_x86_64_fedora', trying next name...
> D Unable to load the library 'org_apache_beam_vendor_grpc_v1_13_1_netty_tcnative_x86_64', trying other loading mechanism.
> D org_apache_beam_vendor_grpc_v1_13_1_netty_tcnative_x86_64 cannot be loaded from java.libary.path, now trying export to -Dio.netty.native.workdir: /tmp
> D Unable to load the library 'netty_tcnative_x86_64', trying next name...
> D Unable to load the library 'org_apache_beam_vendor_grpc_v1_13_1_netty_tcnative', trying other loading mechanism.
> D org_apache_beam_vendor_grpc_v1_13_1_netty_tcnative cannot be loaded from java.libary.path, now trying export to -Dio.netty.native.workdir: /tmp
> D Unable to load the library 'netty_tcnative', trying next name...
> D Failed to load netty-tcnative; OpenSslEngine will be unavailable, unless the application has already loaded the symbols by some other means. See http://netty.io/wiki/forked-tomcat-native.html for more information.
> D Failed to initialize netty-tcnative; OpenSslEngine will be unavailable. See http://netty.io/wiki/forked-tomcat-native.html for more information.
> I netty-tcnative unavailable (this may be normal)
> I Conscrypt not found (this may be normal)
> I Jetty ALPN unavailable (this may be normal)
> E Uncaught exception in main thread. Exiting with status code 1.
> W Please use a logger instead of System.out or System.err.
> Please switch to using org.slf4j.Logger.
> See: https://cloud.google.com/dataflow/pipelines/logging
> E Uncaught exception in main thread. Exiting with status code 1.
> E java.lang.IllegalStateException: Could not find TLS ALPN provider; no working netty-tcnative, Conscrypt, or Jetty NPN/ALPN available
> E at org.apache.beam.vendor.grpc.v1_13_1.io.grpc.netty.GrpcSslContexts.defaultSslProvider(GrpcSslContexts.java:256)
> E at org.apache.beam.vendor.grpc.v1_13_1.io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:171)
> E at org.apache.beam.vendor.grpc.v1_13_1.io.grpc.netty.GrpcSslContexts.forClient(GrpcSslContexts.java:120)
> E at org.apache.beam.runners.dataflow.worker.windmill.GrpcWindmillServer.remoteChannel(GrpcWindmillServer.java:343)
> E at org.apache.beam.runners.dataflow.worker.windmill.GrpcWindmillServer.initializeWindmillService(GrpcWindmillServer.java:312)
> {code}
>
> The interesting part is in the netty load failure, the stack trace is:
> {code:java}
> exception: "java.lang.UnsatisfiedLinkError at org.apache.beam.vendor.grpc.v1_13_1.io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:276) at org.apache.beam.vendor.grpc.v1_13_1.io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:233) at org.apache.beam.vendor.grpc.v1_13_1.io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:187) at org.apache.beam.vendor.grpc.v1_13_1.io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:85) at org.apache.beam.vendor.grpc.v1_13_1.io.netty.handler.ssl.OpenSsl.loadTcNative(OpenSsl.java:430) at org.apache.beam.vendor.grpc.v1_13_1.io.netty.handler.ssl.OpenSsl.<clinit>(OpenSsl.java:97) at org.apache.beam.vendor.grpc.v1_13_1.io.grpc.netty.GrpcSslContexts.defaultSslProvider(GrpcSslContexts.java:242) at org.apache.beam.vendor.grpc.v1_13_1.io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:171) at org.apache.beam.vendor.grpc.v1_13_1.io.grpc.netty.GrpcSslContexts.forClient(GrpcSslContexts.java:120) at org.apache.beam.runners.dataflow.worker.windmill.GrpcWindmillServer.remoteChannel(GrpcWindmillServer.java:343) at org.apache.beam.runners.dataflow.worker.windmill.GrpcWindmillServer.initializeWindmillService(GrpcWindmillServer.java:312) at org.apache.beam.runners.dataflow.worker.windmill.GrpcWindmillServer.setWindmillServiceEndpoints(GrpcWindmillServer.java:192) at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.getConfigFromDataflowService(StreamingDataflowWorker.java:1528) at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.getConfig(StreamingDataflowWorker.java:1583) at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.getGlobalConfig(StreamingDataflowWorker.java:1568) at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.schedulePeriodicGlobalConfigRequests(StreamingDataflowWorker.java:1543) at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.start(StreamingDataflowWorker.java:704) at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.main(StreamingDataflowWorker.java:228) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.beam.vendor.grpc.v1_13_1.io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:263) at java.security.AccessController.doPrivileged(Native Method) at org.apache.beam.vendor.grpc.v1_13_1.io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:255) ... 17 more Caused by: java.lang.NoClassDefFoundError: org/apache/beam/vendor/grpc/v1/13/1/io/netty/internal/tcnative/Library at java.lang.ClassLoader$NativeLibrary.load(Native Method) at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941) at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824) at java.lang.Runtime.load0(Runtime.java:809) at java.lang.System.load(System.java:1086) at org.apache.beam.vendor.grpc.v1_13_1.io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:36) ... 24 more Caused by: java.lang.ClassNotFoundException: org.apache.beam.vendor.grpc.v1.13.1.io.netty.internal.tcnative.Library at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 30 more{code}
>
> Notice that the class attempting to be loaded is:
> org.apache.beam.vendor.grpc.v1.13.1.io.netty.internal.tcnative.Library, but it's actually defined in the jar as org.apache.beam.vendor.grpc.v1_13_1.io.netty.internal.tcnative.Library.
> I traced this back to the jni interop code in tcnative:
> [https://github.com/netty/netty-tcnative/blob/master/openssl-dynamic/src/main/c/jnilib.c#L266]
> Here it replaces all _ in the package prefix with /, which won't work here. The fix seems like it would be to repackage the vendored gRPC with a different prefix that doesn't contain underscores.
> I'm curious how this ever worked though? Maybe the streaming engine is the only thing using this vendored gRPC code?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)