You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@drill.apache.org by Ron Cecchini <ro...@comcast.net> on 2020/02/05 23:28:55 UTC
Runtime Kotlin + Drill JDBC netty problem
TL;DR: my Maven Fu is failing and I'm encountering a bad transitive dependency...
Sorry I couldn't keep this shorter. But I'll get right to the point:
The Drill server is up ... but I've been struggling mightily to connect to it via JDBC called from a Kotlin web app.
First of all, this Kotlin project has the following 2 "ktor" dependencies ("ktor" is a Kotlin web framework):
io.ktor:ktor-server-netty:1.0.1
io.ktor:ktor-server-core:1.0.1
(There is a newer version of both ktor deps, 1.3.0, but upgrading the version caused a bunch of other problems.)
I added the drill-jdbc-all:1.17.0 dependency:
<dependency>
<groupId>org.apache.drill.exec</groupId>
<artifactId>drill-jdbc-all</artifactId>
<version>1.17.0</version>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>log4j-over-slf4j</artifactId>
</exclusion>
</exclusions>
</dependency>
It took a while to discover the <exclusion> but the project won't compile without it.
Anyway, the project starts, but any attempts (when doing a query) to load the Drill JDBC driver throws a runtime exception.
I experimented with a number of things (versions, scopes, exclusions, adding dependency 'netty-all' while excluding io.netty:* from everything else) but I couldn't hit on the magic formula.
There is some dependency clash I can't figure out how to exclude/get around, or how/where to specify the correct version... I believe the NoClassDefFoundError is a red herring -- after all, the drill-jdbc-all should be all I need; I shouldn't need to add anything from io.netty. It's got to be some clash with that other ktor dependency that has a transitive dependency on something io.netty deep in its bowels.
At any rate, I'm waving the white flag.
Below is the Maven dependency tree and then the full stack trace.
Thank you so much for any help!
----------
% mvn dependency:tree -Dverbose
[INFO] +- io.ktor:ktor-server-netty:jar:1.0.1:compile
...
[INFO] | +- io.netty:netty-codec-http2:jar:4.1.24.Final:compile
[INFO] | | +- io.netty:netty-codec-http:jar:4.1.24.Final:compile
[INFO] | | | \- (io.netty:netty-codec:jar:4.1.24.Final:compile - omitted for conflict with 4.0.48.Final)
[INFO] | | \- (io.netty:netty-handler:jar:4.1.24.Final:compile - omitted for conflict with 4.0.48.Final)
[INFO] | \- org.eclipse.jetty.alpn:alpn-api:jar:1.1.3.v20160715:compile
+- org.apache.drill.exec:drill-jdbc-all:jar:1.17.0:compile
[INFO] | +- org.slf4j:jul-to-slf4j:jar:1.7.26:compile
[INFO] | | \- (org.slf4j:slf4j-api:jar:1.7.26:compile - omitted for conflict with 1.7.25)
[INFO] | +- commons-io:commons-io:jar:2.4:compile
[INFO] | +- org.xerial.snappy:snappy-java:jar:1.1.2.6:compile
[INFO] | +- commons-codec:commons-codec:jar:1.10:compile
[INFO] | +- io.netty:netty-handler:jar:4.0.48.Final:compile
[INFO] | | +- io.netty:netty-buffer:jar:4.0.48.Final:compile
[INFO] | | | \- (io.netty:netty-common:jar:4.0.48.Final:compile - omitted for duplicate)
[INFO] | | +- io.netty:netty-transport:jar:4.0.48.Final:compile
[INFO] | | | \- (io.netty:netty-buffer:jar:4.0.48.Final:compile - omitted for duplicate)
[INFO] | | \- io.netty:netty-codec:jar:4.0.48.Final:compile
[INFO] | | \- (io.netty:netty-transport:jar:4.0.48.Final:compile - omitted for duplicate)
[INFO] | +- io.netty:netty-common:jar:4.0.48.Final:compile
[INFO] | +- org.apache.drill:drill-shaded-guava:jar:23.0:compile
[INFO] | | +- com.google.code.findbugs:jsr305:jar:1.3.9:compile
[INFO] | | +- com.google.errorprone:error_prone_annotations:jar:2.0.18:compile
[INFO] | | +- com.google.j2objc:j2objc-annotations:jar:1.1:compile
[INFO] | | \- org.codehaus.mojo:animal-sniffer-annotations:jar:1.14:compile
[INFO] | +- (org.slf4j:slf4j-api:jar:1.7.26:compile - omitted for conflict with 1.7.25)
[INFO] | \- org.slf4j:jcl-over-slf4j:jar:1.7.26:compile
[INFO] | \- (org.slf4j:slf4j-api:jar:1.7.26:compile - omitted for conflict with 1.7.25)
----------
05 Feb 22:50:43 WARN nettyWorkerPool-3-1 io.netty.channel.ChannelInitializer - Failed to initialize a channel. Closing: [id: 0xcfe46a52, L:/10.128.3.74:8080 - R:/10.131.0.1:38430]
java.lang.NoClassDefFoundError: io/netty/handler/codec/DecoderResultProvider
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at io.netty.handler.codec.http.HttpServerCodec.<init>(HttpServerCodec.java:51)
at io.netty.handler.codec.http.HttpServerCodec.<init>(HttpServerCodec.java:44)
at io.ktor.server.netty.NettyApplicationEngine$Configuration$httpServerCodec$1.invoke(NettyApplicationEngine.kt:53)
at io.ktor.server.netty.NettyApplicationEngine$Configuration$httpServerCodec$1.invoke(NettyApplicationEngine.kt:24)
at io.ktor.server.netty.NettyChannelInitializer.configurePipeline(NettyChannelInitializer.kt:102)
at io.ktor.server.netty.NettyChannelInitializer.initChannel(NettyChannelInitializer.kt:78)
at io.ktor.server.netty.NettyChannelInitializer.initChannel(NettyChannelInitializer.kt:19)
at io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:113)
at io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:105)
at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:582)
at io.netty.channel.DefaultChannelPipeline.access$000(DefaultChannelPipeline.java:43)
at io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1347)
at io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1082)
at io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:632)
at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:452)
at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:374)
at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:424)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: io.netty.handler.codec.DecoderResultProvider
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 46 more
Re: Runtime Kotlin + Drill JDBC netty problem
Posted by Paul Rogers <pa...@yahoo.com.INVALID>.
Hi Ron,
Glad you figured it out! Yes, Drill dependencies are overly complex. One of the reasons I don't enthusiastically encourage the use of Drill as an emedded query library is that we don't want to put people through the hassles you are encountering. Besides Netty, Guava is an ongoing source of pain, even within Drill itself.
If there is interest in using Drill in embedded mode inside other apps, then someone should figure out how we a) run Drill in a dedicated class loader, and b) load plugins in their own class loaders. Java 9 was supposed to provide mechanisms to help with this, but I think there are limits.
There is also the challenge that the JDBC driver itself includes a large amount of Drill's dependencies (to run the RPC mechanism work with vectors, etc.) We should a) simplify the client, or b) load Drill's "guts" in a separate class loader, as in the above scenario.
Anyone looking for a challenging Drill project; this is a good candidate.
Sounds like you know the Maven tricks: use dependency tree to figure out who is including what, then try to whack the problems one by one with dependency management. (That is one of the most painful parts of Maven IMHO.)
Thanks,
- Paul
On Thursday, February 6, 2020, 8:27:31 PM PST, Ron Cecchini <ro...@comcast.net> wrote:
Quick apology for asking a boneheaded question... esp. when the answer was actually staring me in the face.
Apparently I just kept messing up the Maven exclusions on the 'drill-jdbc-all' dependency...
FWIW, I had to exclude both the 4.0.48 'netty-handler' and 'netty-common' from 'drill-jdbc-all' to allow the Kotlin stuff to pull in the newer 4.1.24 versions of the netty libraries. (using Maven's <dependencyManagement> to encorce 4.1.24 also worked.)
> On February 5, 2020 at 6:28 PM Ron Cecchini <ro...@comcast.net> wrote:
>
>
> TL;DR: my Maven Fu is failing and I'm encountering a bad transitive dependency...
Re: Runtime Kotlin + Drill JDBC netty problem
Posted by Ron Cecchini <ro...@comcast.net>.
Quick apology for asking a boneheaded question... esp. when the answer was actually staring me in the face.
Apparently I just kept messing up the Maven exclusions on the 'drill-jdbc-all' dependency...
FWIW, I had to exclude both the 4.0.48 'netty-handler' and 'netty-common' from 'drill-jdbc-all' to allow the Kotlin stuff to pull in the newer 4.1.24 versions of the netty libraries. (using Maven's <dependencyManagement> to encorce 4.1.24 also worked.)
> On February 5, 2020 at 6:28 PM Ron Cecchini <ro...@comcast.net> wrote:
>
>
> TL;DR: my Maven Fu is failing and I'm encountering a bad transitive dependency...