You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by do...@dbruhn.de on 2017/12/18 14:07:43 UTC
Flink 1.4 with cassandra-connector: Shading error
Hey everyone,
I'm trying to migrate one of my jobs to Flink 1.4 and I'm running into a
classpath/shading error.
What happens is that when Flink calls into Cluster.connect(), somewhere
down in the stream, the cassandra library tries to initialize Netty, and
I'm getting the following exception:
Caused by: java.lang.AssertionError: Cannot locate Netty classes in the
classpath:java.lang.ClassNotFoundException:
com.datastax.shaded.netty.channel.Channel.
Full exception here:
https://gist.github.com/anonymous/16a44eabb45ad7f20d551dd29b83d2fb
I think this can be explained very easily:
1. The flink-cassandra-connector-1.4 jar contains the Datastax Cassandra
driver inside
2. The flink-cassandra-connector-1.4 jar contains a shaded (to
org.apache.flink.cassandra.shaded.io.netty) netty
3. The DataStax driver executes the following code when it initializes
netty (in NettyUtils starting line 58):
----
try {
Class.forName(String.format("%s.%s.channel.Channel", "io",
"netty"));
shaded = false;
} catch (ClassNotFoundException var9) {
try {
Class.forName("com.datastax.shaded.netty.channel.Channel");
shaded = true;
} catch (ClassNotFoundException var8) {
throw new AssertionError("Cannot locate Netty classes in
the classpath:" + var8);
}
}
----
Neither of the two packages (io.netty.channel and
com.datastax.shaded.netty.channel.Channel) are available and the shaded
one from Flink is not in the list and is never used here.
My question: Did anyone ever use the cassandra connector on 1.4. As it
looks to me it is completely broken and can never work. But maybe I'm
failing to see something. I don't include netty in my dependencies, that
could of course fix it, but I'm suspecting I will run into more
dependency problems then.
Here are my dependencies:
https://gist.github.com/anonymous/565a0ad017976502a62b2919115b31fd
What additional information can I provide?
Thanks,
Dominik
Re: Flink 1.4 with cassandra-connector: Shading error
Posted by Timo Walther <tw...@apache.org>.
I opened an issue for it: https://issues.apache.org/jira/browse/FLINK-8295
Timo
Am 12/19/17 um 11:14 AM schrieb Nico Kruber:
> Hi Dominik,
> nice assessment of the issue: in the version of the cassandra-driver we
> use there is even a comment about why:
>
> ----
> try {
> // prevent this string from being shaded
> Class.forName(String.format("%s.%s.channel.Channel", "io", "netty"));
> shaded = false;
> } catch (ClassNotFoundException e) {
> try {
> Class.forName("com.datastax.shaded.netty.channel.Channel");
> shaded = true;
> } catch (ClassNotFoundException e1) {
> throw new AssertionError("Cannot locate Netty classes in the
> classpath:" + e1);
> }
> }
> ----
>
> @Chesnay: Should we instead shade into datastax' namespace as shown?
> This would also make sure to follow the shaded path in that class which,
> for example, deactivates epoll.
>
>
> Nico
>
>
> On 18/12/17 15:43, Timo Walther wrote:
>> Hi Dominik,
>>
>> thanks for reporting your issue. I will loop in Chesnay that might know
>> more about your problem.
>>
>> There were a lot of dependency changes in 1.4 to make the future more
>> dependency friendly. Maybe this has not been tested properly.
>>
>> Regards,
>> Timo
>>
>>
>> Am 12/18/17 um 3:07 PM schrieb dominik@dbruhn.de:
>>> Hey everyone,
>>> I'm trying to migrate one of my jobs to Flink 1.4 and I'm running into
>>> a classpath/shading error.
>>>
>>> What happens is that when Flink calls into Cluster.connect(),
>>> somewhere down in the stream, the cassandra library tries to
>>> initialize Netty, and I'm getting the following exception:
>>>
>>> Caused by: java.lang.AssertionError: Cannot locate Netty classes in
>>> the classpath:java.lang.ClassNotFoundException:
>>> com.datastax.shaded.netty.channel.Channel.
>>> Full exception here:
>>> https://gist.github.com/anonymous/16a44eabb45ad7f20d551dd29b83d2fb
>>>
>>> I think this can be explained very easily:
>>> 1. The flink-cassandra-connector-1.4 jar contains the Datastax
>>> Cassandra driver inside
>>> 2. The flink-cassandra-connector-1.4 jar contains a shaded (to
>>> org.apache.flink.cassandra.shaded.io.netty) netty
>>> 3. The DataStax driver executes the following code when it initializes
>>> netty (in NettyUtils starting line 58):
>>>
>>> ----
>>> try {
>>> Class.forName(String.format("%s.%s.channel.Channel", "io",
>>> "netty"));
>>> shaded = false;
>>> } catch (ClassNotFoundException var9) {
>>> try {
>>> Class.forName("com.datastax.shaded.netty.channel.Channel");
>>> shaded = true;
>>> } catch (ClassNotFoundException var8) {
>>> throw new AssertionError("Cannot locate Netty classes
>>> in the classpath:" + var8);
>>> }
>>> }
>>> ----
>>>
>>>
>>> Neither of the two packages (io.netty.channel and
>>> com.datastax.shaded.netty.channel.Channel) are available and the
>>> shaded one from Flink is not in the list and is never used here.
>>>
>>> My question: Did anyone ever use the cassandra connector on 1.4. As it
>>> looks to me it is completely broken and can never work. But maybe I'm
>>> failing to see something. I don't include netty in my dependencies,
>>> that could of course fix it, but I'm suspecting I will run into more
>>> dependency problems then.
>>>
>>> Here are my dependencies:
>>> https://gist.github.com/anonymous/565a0ad017976502a62b2919115b31fd
>>>
>>> What additional information can I provide?
>>>
>>> Thanks,
>>> Dominik
>>
Re: Flink 1.4 with cassandra-connector: Shading error
Posted by Nico Kruber <ni...@data-artisans.com>.
Hi Dominik,
nice assessment of the issue: in the version of the cassandra-driver we
use there is even a comment about why:
----
try {
// prevent this string from being shaded
Class.forName(String.format("%s.%s.channel.Channel", "io", "netty"));
shaded = false;
} catch (ClassNotFoundException e) {
try {
Class.forName("com.datastax.shaded.netty.channel.Channel");
shaded = true;
} catch (ClassNotFoundException e1) {
throw new AssertionError("Cannot locate Netty classes in the
classpath:" + e1);
}
}
----
@Chesnay: Should we instead shade into datastax' namespace as shown?
This would also make sure to follow the shaded path in that class which,
for example, deactivates epoll.
Nico
On 18/12/17 15:43, Timo Walther wrote:
> Hi Dominik,
>
> thanks for reporting your issue. I will loop in Chesnay that might know
> more about your problem.
>
> There were a lot of dependency changes in 1.4 to make the future more
> dependency friendly. Maybe this has not been tested properly.
>
> Regards,
> Timo
>
>
> Am 12/18/17 um 3:07 PM schrieb dominik@dbruhn.de:
>> Hey everyone,
>> I'm trying to migrate one of my jobs to Flink 1.4 and I'm running into
>> a classpath/shading error.
>>
>> What happens is that when Flink calls into Cluster.connect(),
>> somewhere down in the stream, the cassandra library tries to
>> initialize Netty, and I'm getting the following exception:
>>
>> Caused by: java.lang.AssertionError: Cannot locate Netty classes in
>> the classpath:java.lang.ClassNotFoundException:
>> com.datastax.shaded.netty.channel.Channel.
>> Full exception here:
>> https://gist.github.com/anonymous/16a44eabb45ad7f20d551dd29b83d2fb
>>
>> I think this can be explained very easily:
>> 1. The flink-cassandra-connector-1.4 jar contains the Datastax
>> Cassandra driver inside
>> 2. The flink-cassandra-connector-1.4 jar contains a shaded (to
>> org.apache.flink.cassandra.shaded.io.netty) netty
>> 3. The DataStax driver executes the following code when it initializes
>> netty (in NettyUtils starting line 58):
>>
>> ----
>> try {
>> Class.forName(String.format("%s.%s.channel.Channel", "io",
>> "netty"));
>> shaded = false;
>> } catch (ClassNotFoundException var9) {
>> try {
>> Class.forName("com.datastax.shaded.netty.channel.Channel");
>> shaded = true;
>> } catch (ClassNotFoundException var8) {
>> throw new AssertionError("Cannot locate Netty classes
>> in the classpath:" + var8);
>> }
>> }
>> ----
>>
>>
>> Neither of the two packages (io.netty.channel and
>> com.datastax.shaded.netty.channel.Channel) are available and the
>> shaded one from Flink is not in the list and is never used here.
>>
>> My question: Did anyone ever use the cassandra connector on 1.4. As it
>> looks to me it is completely broken and can never work. But maybe I'm
>> failing to see something. I don't include netty in my dependencies,
>> that could of course fix it, but I'm suspecting I will run into more
>> dependency problems then.
>>
>> Here are my dependencies:
>> https://gist.github.com/anonymous/565a0ad017976502a62b2919115b31fd
>>
>> What additional information can I provide?
>>
>> Thanks,
>> Dominik
>
>
Re: Flink 1.4 with cassandra-connector: Shading error
Posted by Timo Walther <tw...@apache.org>.
Hi Dominik,
thanks for reporting your issue. I will loop in Chesnay that might know
more about your problem.
There were a lot of dependency changes in 1.4 to make the future more
dependency friendly. Maybe this has not been tested properly.
Regards,
Timo
Am 12/18/17 um 3:07 PM schrieb dominik@dbruhn.de:
> Hey everyone,
> I'm trying to migrate one of my jobs to Flink 1.4 and I'm running into
> a classpath/shading error.
>
> What happens is that when Flink calls into Cluster.connect(),
> somewhere down in the stream, the cassandra library tries to
> initialize Netty, and I'm getting the following exception:
>
> Caused by: java.lang.AssertionError: Cannot locate Netty classes in
> the classpath:java.lang.ClassNotFoundException:
> com.datastax.shaded.netty.channel.Channel.
> Full exception here:
> https://gist.github.com/anonymous/16a44eabb45ad7f20d551dd29b83d2fb
>
> I think this can be explained very easily:
> 1. The flink-cassandra-connector-1.4 jar contains the Datastax
> Cassandra driver inside
> 2. The flink-cassandra-connector-1.4 jar contains a shaded (to
> org.apache.flink.cassandra.shaded.io.netty) netty
> 3. The DataStax driver executes the following code when it initializes
> netty (in NettyUtils starting line 58):
>
> ----
> try {
> Class.forName(String.format("%s.%s.channel.Channel", "io",
> "netty"));
> shaded = false;
> } catch (ClassNotFoundException var9) {
> try {
> Class.forName("com.datastax.shaded.netty.channel.Channel");
> shaded = true;
> } catch (ClassNotFoundException var8) {
> throw new AssertionError("Cannot locate Netty classes
> in the classpath:" + var8);
> }
> }
> ----
>
>
> Neither of the two packages (io.netty.channel and
> com.datastax.shaded.netty.channel.Channel) are available and the
> shaded one from Flink is not in the list and is never used here.
>
> My question: Did anyone ever use the cassandra connector on 1.4. As it
> looks to me it is completely broken and can never work. But maybe I'm
> failing to see something. I don't include netty in my dependencies,
> that could of course fix it, but I'm suspecting I will run into more
> dependency problems then.
>
> Here are my dependencies:
> https://gist.github.com/anonymous/565a0ad017976502a62b2919115b31fd
>
> What additional information can I provide?
>
> Thanks,
> Dominik