You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Federico D'Ambrosio <fe...@smartlab.ws> on 2017/08/25 15:36:51 UTC
Possible conflict between in Flink connectors
Hello everyone, I'm new to Flink and am encountering a nasty problem while
trying to submit a streaming Flink Job. I'll try to explain it as
thoroughly as possible.
Premise: I'm using an HDP 2.6 hadoop cluster, with hadoop version
2.7.3.2.6.1.0-129, Flink compiled from sources accordingly (maven 3.0.5) as
per documentation
<https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/building.html#vendor-specific-versions>
and I submit the job using yarn-session.sh and then flink run.
Getting into more details, the Flink Job is a fat jar built with sbt
assembly and the following dependencies:
libraryDependencies += "org.apache.flink" %% "flink-scala" %
flinkVersion % "provided"
libraryDependencies += "org.apache.flink" %% "flink-streaming-scala" %
flinkVersion % "provided"
libraryDependencies += "org.apache.flink" %% "flink-connector-kinesis"
% flinkVersion % "provided"
libraryDependencies += "org.apache.flink" %% "flink-hbase" %
flinkVersion % "provided"
libraryDependencies += "org.apache.flink" %%
"flink-connector-filesystem" % flinkVersion % "provided"
libraryDependencies += "org.apache.hbase" % "hbase-common" % "1.3.0"
libraryDependencies += "org.joda" % "joda-convert" % "1.8.3"
libraryDependencies += "com.typesafe.play" %% "play-json" % "2.6.2"
assemblyOption in assembly := (assemblyOption in
assembly).value.copy(includeScala = false)
inside the flink lib folder I have the following Jars:
flink-connector-filesystem_2.10-1.3.2.jar
flink-python_2.10-1.3.2.jar
flink-connector-kinesis_2.10-1.3.2.jar
flink-shaded-hadoop2-uber-1.3.2.jar
flink-dist_2.10-1.3.2.jar
log4j-1.2.17.jar
flink-hbase_2.10-1.3.2.jar
slf4j-log4j12-1.7.7.jar
Right now, the issue is that when I flink run the job, I get
NoClassDefFound for org/apache/hadoop/hbase/client/Put class, despite it
being inside the fat jar. So, I tried putting both hbase-common.jar and
hbase-client.jars inside the lib folder, getting as a result another
NoClassDefFound, only, this time, for com/google/common/collect/ListMultimap.
Now, I noticed that flink-connector-kinesis and all the hbase packages
(both hbase-common and flink-hbase deps) have a different version of guava
as dependency: the first one needs version 18.0, while the second one needs
12.0.1 version. I, then, tried to enable the shading of guava packages with
the following:
assemblyShadeRules in assembly := Seq(
ShadeRule.rename("com.google.**" -> "shade.com.google.@1").inAll
)
Only to no avail, getting the exact same error as before. And I really
don't know how to submit the job without pouring every single dependency
inside the lib folder. I thought it could be a compatibility issue with the
YARN cluster, but I tried launching the job locally only to get the same
error.
As an additional info, I get the following warnings if I don't specify
flink packages as provided:
[warn] * org.scalamacros:quasiquotes_2.10:2.1.0 is selected over 2.0.1
[warn] +- org.typelevel:macro-compat_2.10:1.1.1
(depends on 2.1.0)
[warn] +- org.apache.flink:flink-scala_2.10:1.3.2
(depends on 2.0.1)
[warn]
[warn] * org.mortbay.jetty:jetty-util:6.1.26 is selected over 6.1.26.hwx
[warn] +- org.apache.hbase:hbase-common:1.3.0
(depends on 6.1.26)
[warn] +- org.apache.flink:flink-shaded-hadoop2:1.3.2
(depends on 6.1.26.hwx)
[warn]
[warn] * io.netty:netty:3.8.0.Final is selected over 3.7.0.Final
[warn] +- com.data-artisans:flakka-remote_2.10:2.3-custom
(depends on 3.8.0.Final)
[warn] +- org.apache.zookeeper:zookeeper:3.4.6
(depends on 3.7.0.Final)
[warn]
[warn] * com.google.guava:guava:18.0 is selected over 12.0.1
[warn] +- org.apache.flink:flink-connector-kinesis_2.10:1.3.2
(depends on 18.0)
[warn] +- org.apache.hbase:hbase-prefix-tree:1.3.0
(depends on 12.0.1)
[warn] +- org.apache.hbase:hbase-procedure:1.3.0
(depends on 12.0.1)
[warn] +- org.apache.hbase:hbase-client:1.3.0
(depends on 12.0.1)
[warn] +- org.apache.hbase:hbase-common:1.3.0
(depends on 12.0.1)
[warn] +- org.apache.hbase:hbase-server:1.3.0
(depends on 12.0.1)
[warn]
[warn] * io.netty:netty-all:4.0.27.Final is selected over 4.0.23.Final
[warn] +- org.apache.flink:flink-runtime_2.10:1.3.2
(depends on 4.0.27.Final)
[warn] +- org.apache.flink:flink-shaded-hadoop2:1.3.2
(depends on 4.0.27.Final)
[warn] +- org.apache.hbase:hbase-prefix-tree:1.3.0
(depends on 4.0.23.Final)
[warn] +- org.apache.hbase:hbase-client:1.3.0
(depends on 4.0.23.Final)
[warn] +- org.apache.hbase:hbase-server:1.3.0
(depends on 4.0.23.Final)
[warn]
[warn] * junit:junit:4.12 is selected over 3.8.1
[warn] +- org.apache.hbase:hbase-protocol:1.3.0
(depends on 4.12)
[warn] +- org.apache.hbase:hbase-annotations:1.3.0
(depends on 4.12)
[warn] +- org.apache.hbase:hbase-prefix-tree:1.3.0
(depends on 4.12)
[warn] +- org.apache.hbase:hbase-procedure:1.3.0
(depends on 4.12)
[warn] +- org.apache.hbase:hbase-client:1.3.0
(depends on 4.12)
[warn] +- org.apache.hbase:hbase-common:1.3.0
(depends on 4.12)
[warn] +- org.apache.hbase:hbase-server:1.3.0
(depends on 4.12)
[warn] +- jline:jline:0.9.94
(depends on 3.8.1)
And here I am asking for help here.
Thank you very much for your attention,
Kind regards,
Federico D'Ambrosio
Re: Possible conflict between in Flink connectors
Posted by Ted Yu <yu...@gmail.com>.
Looking at dependencies for flink-hbase, we have :
[INFO] +- org.apache.hbase:hbase-server:jar:1.3.1:compile
[INFO] | +- org.apache.hbase:hbase-common:jar:1.3.1:compile
[INFO] | +- org.apache.hbase:hbase-protocol:jar:1.3.1:compile
[INFO] | +- org.apache.hbase:hbase-procedure:jar:1.3.1:compile
[INFO] | | \- org.apache.hbase:hbase-common:jar:tests:1.3.1:compile
[INFO] | +- org.apache.hbase:hbase-client:jar:1.3.1:compile
On Fri, Aug 25, 2017 at 2:03 PM, Robert Metzger <rm...@apache.org> wrote:
> Hi,
>
> why do you need to add hbase-common as a separate dependency? Doesn't the
> "flink-hbase" dependency transitively pull in hbase?
>
> On Fri, Aug 25, 2017 at 6:35 PM, Ted Yu <yu...@gmail.com> wrote:
>
>> If Guava 18.0 is used to build hbase 1.3, there would be compilation
>> errors such as the following:
>>
>> [ERROR] /mnt/disk2/a/1.3-h/hbase-server/src/main/java/org/apache/had
>> oop/hbase/replication/regionserver/ReplicationSource.java:[271,25]
>> error: cannot find symbol
>> [ERROR] symbol: method stopAndWait()
>> [ERROR] location: variable replicationEndpoint of type
>> ReplicationEndpoint
>> [ERROR] /mnt/disk2/a/1.3-h/hbase-server/src/main/java/org/apache/had
>> oop/hbase/replication/regionserver/ReplicationSource.java:[281,47]
>> error: cannot find symbol
>> [ERROR] symbol: method start()
>> [ERROR] location: variable replicationEndpoint of type
>> ReplicationEndpoint
>>
>> Maybe you can shade the guava dependency in hbase 1.3
>>
>> In the upcoming hbase 2.0 release, third party dependencies such as guava
>> and netty are shaded. Meaning there wouldn't be such conflict with other
>> components you may use.
>>
>> On Fri, Aug 25, 2017 at 8:36 AM, Federico D'Ambrosio <
>> federico.dambrosio@smartlab.ws> wrote:
>>
>>> Hello everyone, I'm new to Flink and am encountering a nasty problem
>>> while trying to submit a streaming Flink Job. I'll try to explain it as
>>> thoroughly as possible.
>>>
>>> Premise: I'm using an HDP 2.6 hadoop cluster, with hadoop version
>>> 2.7.3.2.6.1.0-129, Flink compiled from sources accordingly (maven 3.0.5) as
>>> per documentation
>>> <https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/building.html#vendor-specific-versions>
>>> and I submit the job using yarn-session.sh and then flink run.
>>>
>>> Getting into more details, the Flink Job is a fat jar built with sbt
>>> assembly and the following dependencies:
>>>
>>> libraryDependencies += "org.apache.flink" %% "flink-scala" % flinkVersion % "provided"
>>> libraryDependencies += "org.apache.flink" %% "flink-streaming-scala" % flinkVersion % "provided"
>>> libraryDependencies += "org.apache.flink" %% "flink-connector-kinesis" % flinkVersion % "provided"
>>> libraryDependencies += "org.apache.flink" %% "flink-hbase" % flinkVersion % "provided"
>>> libraryDependencies += "org.apache.flink" %% "flink-connector-filesystem" % flinkVersion % "provided"
>>> libraryDependencies += "org.apache.hbase" % "hbase-common" % "1.3.0"
>>> libraryDependencies += "org.joda" % "joda-convert" % "1.8.3"
>>> libraryDependencies += "com.typesafe.play" %% "play-json" % "2.6.2"
>>>
>>>
>>> assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = false)
>>>
>>>
>>> inside the flink lib folder I have the following Jars:
>>>
>>> flink-connector-filesystem_2.10-1.3.2.jar
>>> flink-python_2.10-1.3.2.jar
>>> flink-connector-kinesis_2.10-1.3.2.jar
>>> flink-shaded-hadoop2-uber-1.3.2.jar
>>> flink-dist_2.10-1.3.2.jar
>>> log4j-1.2.17.jar
>>> flink-hbase_2.10-1.3.2.jar
>>> slf4j-log4j12-1.7.7.jar
>>>
>>> Right now, the issue is that when I flink run the job, I get
>>> NoClassDefFound for org/apache/hadoop/hbase/client/Put class, despite
>>> it being inside the fat jar. So, I tried putting both hbase-common.jar and
>>> hbase-client.jars inside the lib folder, getting as a result another
>>> NoClassDefFound, only, this time, for com/google/common/collect/List
>>> Multimap. Now, I noticed that flink-connector-kinesis and all the hbase
>>> packages (both hbase-common and flink-hbase deps) have a different version
>>> of guava as dependency: the first one needs version 18.0, while the second
>>> one needs 12.0.1 version. I, then, tried to enable the shading of guava
>>> packages with the following:
>>>
>>> assemblyShadeRules in assembly := Seq(
>>> ShadeRule.rename("com.google.**" -> "shade.com.google.@1").inAll
>>> )
>>>
>>>
>>> Only to no avail, getting the exact same error as before. And I really
>>> don't know how to submit the job without pouring every single dependency
>>> inside the lib folder. I thought it could be a compatibility issue with the
>>> YARN cluster, but I tried launching the job locally only to get the same
>>> error.
>>>
>>> As an additional info, I get the following warnings if I don't specify
>>> flink packages as provided:
>>>
>>> [warn] * org.scalamacros:quasiquotes_2.10:2.1.0 is selected over
>>> 2.0.1
>>> [warn] +- org.typelevel:macro-compat_2.10:1.1.1
>>> (depends on 2.1.0)
>>> [warn] +- org.apache.flink:flink-scala_2.10:1.3.2
>>> (depends on 2.0.1)
>>> [warn]
>>> [warn] * org.mortbay.jetty:jetty-util:6.1.26 is selected over
>>> 6.1.26.hwx
>>> [warn] +- org.apache.hbase:hbase-common:1.3.0
>>> (depends on 6.1.26)
>>> [warn] +- org.apache.flink:flink-shaded-hadoop2:1.3.2
>>> (depends on 6.1.26.hwx)
>>> [warn]
>>> [warn] * io.netty:netty:3.8.0.Final is selected over 3.7.0.Final
>>> [warn] +- com.data-artisans:flakka-remote_2.10:2.3-custom
>>> (depends on 3.8.0.Final)
>>> [warn] +- org.apache.zookeeper:zookeeper:3.4.6
>>> (depends on 3.7.0.Final)
>>> [warn]
>>> [warn] * com.google.guava:guava:18.0 is selected over 12.0.1
>>> [warn] +- org.apache.flink:flink-connector-kinesis_2.10:1.3.2
>>> (depends on 18.0)
>>> [warn] +- org.apache.hbase:hbase-prefix-tree:1.3.0
>>> (depends on 12.0.1)
>>> [warn] +- org.apache.hbase:hbase-procedure:1.3.0
>>> (depends on 12.0.1)
>>> [warn] +- org.apache.hbase:hbase-client:1.3.0
>>> (depends on 12.0.1)
>>> [warn] +- org.apache.hbase:hbase-common:1.3.0
>>> (depends on 12.0.1)
>>> [warn] +- org.apache.hbase:hbase-server:1.3.0
>>> (depends on 12.0.1)
>>> [warn]
>>> [warn] * io.netty:netty-all:4.0.27.Final is selected over
>>> 4.0.23.Final
>>> [warn] +- org.apache.flink:flink-runtime_2.10:1.3.2
>>> (depends on 4.0.27.Final)
>>> [warn] +- org.apache.flink:flink-shaded-hadoop2:1.3.2
>>> (depends on 4.0.27.Final)
>>> [warn] +- org.apache.hbase:hbase-prefix-tree:1.3.0
>>> (depends on 4.0.23.Final)
>>> [warn] +- org.apache.hbase:hbase-client:1.3.0
>>> (depends on 4.0.23.Final)
>>> [warn] +- org.apache.hbase:hbase-server:1.3.0
>>> (depends on 4.0.23.Final)
>>> [warn]
>>> [warn] * junit:junit:4.12 is selected over 3.8.1
>>> [warn] +- org.apache.hbase:hbase-protocol:1.3.0
>>> (depends on 4.12)
>>> [warn] +- org.apache.hbase:hbase-annotations:1.3.0
>>> (depends on 4.12)
>>> [warn] +- org.apache.hbase:hbase-prefix-tree:1.3.0
>>> (depends on 4.12)
>>> [warn] +- org.apache.hbase:hbase-procedure:1.3.0
>>> (depends on 4.12)
>>> [warn] +- org.apache.hbase:hbase-client:1.3.0
>>> (depends on 4.12)
>>> [warn] +- org.apache.hbase:hbase-common:1.3.0
>>> (depends on 4.12)
>>> [warn] +- org.apache.hbase:hbase-server:1.3.0
>>> (depends on 4.12)
>>> [warn] +- jline:jline:0.9.94
>>> (depends on 3.8.1)
>>>
>>> And here I am asking for help here.
>>>
>>> Thank you very much for your attention,
>>> Kind regards,
>>>
>>> Federico D'Ambrosio
>>>
>>
>>
>
Re: Possible conflict between in Flink connectors
Posted by Federico D'Ambrosio <fe...@smartlab.ws>.
Hi,
At the beginning, I was wondering myself that too, and I don't know why
hbase-common wasn''t being downloaded and included, so I added it
explicitly.
I was in the process to write that maybe I've solved this weird issue:
apparently the shading worked and the ClassDefNotFound issue was caused by
the missing hbase-client jar inside the fat jar, even though there should
have been! So, I added again the explicit dependency for it in the
build.sbt and now it's working.
2017-08-25 23:03 GMT+02:00 Robert Metzger <rm...@apache.org>:
> Hi,
>
> why do you need to add hbase-common as a separate dependency? Doesn't the
> "flink-hbase" dependency transitively pull in hbase?
>
> On Fri, Aug 25, 2017 at 6:35 PM, Ted Yu <yu...@gmail.com> wrote:
>
>> If Guava 18.0 is used to build hbase 1.3, there would be compilation
>> errors such as the following:
>>
>> [ERROR] /mnt/disk2/a/1.3-h/hbase-server/src/main/java/org/apache/had
>> oop/hbase/replication/regionserver/ReplicationSource.java:[271,25]
>> error: cannot find symbol
>> [ERROR] symbol: method stopAndWait()
>> [ERROR] location: variable replicationEndpoint of type
>> ReplicationEndpoint
>> [ERROR] /mnt/disk2/a/1.3-h/hbase-server/src/main/java/org/apache/had
>> oop/hbase/replication/regionserver/ReplicationSource.java:[281,47]
>> error: cannot find symbol
>> [ERROR] symbol: method start()
>> [ERROR] location: variable replicationEndpoint of type
>> ReplicationEndpoint
>>
>> Maybe you can shade the guava dependency in hbase 1.3
>>
>> In the upcoming hbase 2.0 release, third party dependencies such as guava
>> and netty are shaded. Meaning there wouldn't be such conflict with other
>> components you may use.
>>
>> On Fri, Aug 25, 2017 at 8:36 AM, Federico D'Ambrosio <
>> federico.dambrosio@smartlab.ws> wrote:
>>
>>> Hello everyone, I'm new to Flink and am encountering a nasty problem
>>> while trying to submit a streaming Flink Job. I'll try to explain it as
>>> thoroughly as possible.
>>>
>>> Premise: I'm using an HDP 2.6 hadoop cluster, with hadoop version
>>> 2.7.3.2.6.1.0-129, Flink compiled from sources accordingly (maven 3.0.5) as
>>> per documentation
>>> <https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/building.html#vendor-specific-versions>
>>> and I submit the job using yarn-session.sh and then flink run.
>>>
>>> Getting into more details, the Flink Job is a fat jar built with sbt
>>> assembly and the following dependencies:
>>>
>>> libraryDependencies += "org.apache.flink" %% "flink-scala" % flinkVersion % "provided"
>>> libraryDependencies += "org.apache.flink" %% "flink-streaming-scala" % flinkVersion % "provided"
>>> libraryDependencies += "org.apache.flink" %% "flink-connector-kinesis" % flinkVersion % "provided"
>>> libraryDependencies += "org.apache.flink" %% "flink-hbase" % flinkVersion % "provided"
>>> libraryDependencies += "org.apache.flink" %% "flink-connector-filesystem" % flinkVersion % "provided"
>>> libraryDependencies += "org.apache.hbase" % "hbase-common" % "1.3.0"
>>> libraryDependencies += "org.joda" % "joda-convert" % "1.8.3"
>>> libraryDependencies += "com.typesafe.play" %% "play-json" % "2.6.2"
>>>
>>>
>>> assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = false)
>>>
>>>
>>> inside the flink lib folder I have the following Jars:
>>>
>>> flink-connector-filesystem_2.10-1.3.2.jar
>>> flink-python_2.10-1.3.2.jar
>>> flink-connector-kinesis_2.10-1.3.2.jar
>>> flink-shaded-hadoop2-uber-1.3.2.jar
>>> flink-dist_2.10-1.3.2.jar
>>> log4j-1.2.17.jar
>>> flink-hbase_2.10-1.3.2.jar
>>> slf4j-log4j12-1.7.7.jar
>>>
>>> Right now, the issue is that when I flink run the job, I get
>>> NoClassDefFound for org/apache/hadoop/hbase/client/Put class, despite
>>> it being inside the fat jar. So, I tried putting both hbase-common.jar and
>>> hbase-client.jars inside the lib folder, getting as a result another
>>> NoClassDefFound, only, this time, for com/google/common/collect/List
>>> Multimap. Now, I noticed that flink-connector-kinesis and all the hbase
>>> packages (both hbase-common and flink-hbase deps) have a different version
>>> of guava as dependency: the first one needs version 18.0, while the second
>>> one needs 12.0.1 version. I, then, tried to enable the shading of guava
>>> packages with the following:
>>>
>>> assemblyShadeRules in assembly := Seq(
>>> ShadeRule.rename("com.google.**" -> "shade.com.google.@1").inAll
>>> )
>>>
>>>
>>> Only to no avail, getting the exact same error as before. And I really
>>> don't know how to submit the job without pouring every single dependency
>>> inside the lib folder. I thought it could be a compatibility issue with the
>>> YARN cluster, but I tried launching the job locally only to get the same
>>> error.
>>>
>>> As an additional info, I get the following warnings if I don't specify
>>> flink packages as provided:
>>>
>>> [warn] * org.scalamacros:quasiquotes_2.10:2.1.0 is selected over
>>> 2.0.1
>>> [warn] +- org.typelevel:macro-compat_2.10:1.1.1
>>> (depends on 2.1.0)
>>> [warn] +- org.apache.flink:flink-scala_2.10:1.3.2
>>> (depends on 2.0.1)
>>> [warn]
>>> [warn] * org.mortbay.jetty:jetty-util:6.1.26 is selected over
>>> 6.1.26.hwx
>>> [warn] +- org.apache.hbase:hbase-common:1.3.0
>>> (depends on 6.1.26)
>>> [warn] +- org.apache.flink:flink-shaded-hadoop2:1.3.2
>>> (depends on 6.1.26.hwx)
>>> [warn]
>>> [warn] * io.netty:netty:3.8.0.Final is selected over 3.7.0.Final
>>> [warn] +- com.data-artisans:flakka-remote_2.10:2.3-custom
>>> (depends on 3.8.0.Final)
>>> [warn] +- org.apache.zookeeper:zookeeper:3.4.6
>>> (depends on 3.7.0.Final)
>>> [warn]
>>> [warn] * com.google.guava:guava:18.0 is selected over 12.0.1
>>> [warn] +- org.apache.flink:flink-connector-kinesis_2.10:1.3.2
>>> (depends on 18.0)
>>> [warn] +- org.apache.hbase:hbase-prefix-tree:1.3.0
>>> (depends on 12.0.1)
>>> [warn] +- org.apache.hbase:hbase-procedure:1.3.0
>>> (depends on 12.0.1)
>>> [warn] +- org.apache.hbase:hbase-client:1.3.0
>>> (depends on 12.0.1)
>>> [warn] +- org.apache.hbase:hbase-common:1.3.0
>>> (depends on 12.0.1)
>>> [warn] +- org.apache.hbase:hbase-server:1.3.0
>>> (depends on 12.0.1)
>>> [warn]
>>> [warn] * io.netty:netty-all:4.0.27.Final is selected over
>>> 4.0.23.Final
>>> [warn] +- org.apache.flink:flink-runtime_2.10:1.3.2
>>> (depends on 4.0.27.Final)
>>> [warn] +- org.apache.flink:flink-shaded-hadoop2:1.3.2
>>> (depends on 4.0.27.Final)
>>> [warn] +- org.apache.hbase:hbase-prefix-tree:1.3.0
>>> (depends on 4.0.23.Final)
>>> [warn] +- org.apache.hbase:hbase-client:1.3.0
>>> (depends on 4.0.23.Final)
>>> [warn] +- org.apache.hbase:hbase-server:1.3.0
>>> (depends on 4.0.23.Final)
>>> [warn]
>>> [warn] * junit:junit:4.12 is selected over 3.8.1
>>> [warn] +- org.apache.hbase:hbase-protocol:1.3.0
>>> (depends on 4.12)
>>> [warn] +- org.apache.hbase:hbase-annotations:1.3.0
>>> (depends on 4.12)
>>> [warn] +- org.apache.hbase:hbase-prefix-tree:1.3.0
>>> (depends on 4.12)
>>> [warn] +- org.apache.hbase:hbase-procedure:1.3.0
>>> (depends on 4.12)
>>> [warn] +- org.apache.hbase:hbase-client:1.3.0
>>> (depends on 4.12)
>>> [warn] +- org.apache.hbase:hbase-common:1.3.0
>>> (depends on 4.12)
>>> [warn] +- org.apache.hbase:hbase-server:1.3.0
>>> (depends on 4.12)
>>> [warn] +- jline:jline:0.9.94
>>> (depends on 3.8.1)
>>>
>>> And here I am asking for help here.
>>>
>>> Thank you very much for your attention,
>>> Kind regards,
>>>
>>> Federico D'Ambrosio
>>>
>>
>>
>
Re: Possible conflict between in Flink connectors
Posted by Robert Metzger <rm...@apache.org>.
Hi,
why do you need to add hbase-common as a separate dependency? Doesn't the
"flink-hbase" dependency transitively pull in hbase?
On Fri, Aug 25, 2017 at 6:35 PM, Ted Yu <yu...@gmail.com> wrote:
> If Guava 18.0 is used to build hbase 1.3, there would be compilation
> errors such as the following:
>
> [ERROR] /mnt/disk2/a/1.3-h/hbase-server/src/main/java/org/apache/
> hadoop/hbase/replication/regionserver/ReplicationSource.java:[271,25]
> error: cannot find symbol
> [ERROR] symbol: method stopAndWait()
> [ERROR] location: variable replicationEndpoint of type
> ReplicationEndpoint
> [ERROR] /mnt/disk2/a/1.3-h/hbase-server/src/main/java/org/apache/
> hadoop/hbase/replication/regionserver/ReplicationSource.java:[281,47]
> error: cannot find symbol
> [ERROR] symbol: method start()
> [ERROR] location: variable replicationEndpoint of type
> ReplicationEndpoint
>
> Maybe you can shade the guava dependency in hbase 1.3
>
> In the upcoming hbase 2.0 release, third party dependencies such as guava
> and netty are shaded. Meaning there wouldn't be such conflict with other
> components you may use.
>
> On Fri, Aug 25, 2017 at 8:36 AM, Federico D'Ambrosio <
> federico.dambrosio@smartlab.ws> wrote:
>
>> Hello everyone, I'm new to Flink and am encountering a nasty problem
>> while trying to submit a streaming Flink Job. I'll try to explain it as
>> thoroughly as possible.
>>
>> Premise: I'm using an HDP 2.6 hadoop cluster, with hadoop version
>> 2.7.3.2.6.1.0-129, Flink compiled from sources accordingly (maven 3.0.5) as
>> per documentation
>> <https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/building.html#vendor-specific-versions>
>> and I submit the job using yarn-session.sh and then flink run.
>>
>> Getting into more details, the Flink Job is a fat jar built with sbt
>> assembly and the following dependencies:
>>
>> libraryDependencies += "org.apache.flink" %% "flink-scala" % flinkVersion % "provided"
>> libraryDependencies += "org.apache.flink" %% "flink-streaming-scala" % flinkVersion % "provided"
>> libraryDependencies += "org.apache.flink" %% "flink-connector-kinesis" % flinkVersion % "provided"
>> libraryDependencies += "org.apache.flink" %% "flink-hbase" % flinkVersion % "provided"
>> libraryDependencies += "org.apache.flink" %% "flink-connector-filesystem" % flinkVersion % "provided"
>> libraryDependencies += "org.apache.hbase" % "hbase-common" % "1.3.0"
>> libraryDependencies += "org.joda" % "joda-convert" % "1.8.3"
>> libraryDependencies += "com.typesafe.play" %% "play-json" % "2.6.2"
>>
>>
>> assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = false)
>>
>>
>> inside the flink lib folder I have the following Jars:
>>
>> flink-connector-filesystem_2.10-1.3.2.jar
>> flink-python_2.10-1.3.2.jar
>> flink-connector-kinesis_2.10-1.3.2.jar
>> flink-shaded-hadoop2-uber-1.3.2.jar
>> flink-dist_2.10-1.3.2.jar
>> log4j-1.2.17.jar
>> flink-hbase_2.10-1.3.2.jar
>> slf4j-log4j12-1.7.7.jar
>>
>> Right now, the issue is that when I flink run the job, I get
>> NoClassDefFound for org/apache/hadoop/hbase/client/Put class, despite it
>> being inside the fat jar. So, I tried putting both hbase-common.jar and
>> hbase-client.jars inside the lib folder, getting as a result another
>> NoClassDefFound, only, this time, for com/google/common/collect/List
>> Multimap. Now, I noticed that flink-connector-kinesis and all the hbase
>> packages (both hbase-common and flink-hbase deps) have a different version
>> of guava as dependency: the first one needs version 18.0, while the second
>> one needs 12.0.1 version. I, then, tried to enable the shading of guava
>> packages with the following:
>>
>> assemblyShadeRules in assembly := Seq(
>> ShadeRule.rename("com.google.**" -> "shade.com.google.@1").inAll
>> )
>>
>>
>> Only to no avail, getting the exact same error as before. And I really
>> don't know how to submit the job without pouring every single dependency
>> inside the lib folder. I thought it could be a compatibility issue with the
>> YARN cluster, but I tried launching the job locally only to get the same
>> error.
>>
>> As an additional info, I get the following warnings if I don't specify
>> flink packages as provided:
>>
>> [warn] * org.scalamacros:quasiquotes_2.10:2.1.0 is selected over
>> 2.0.1
>> [warn] +- org.typelevel:macro-compat_2.10:1.1.1
>> (depends on 2.1.0)
>> [warn] +- org.apache.flink:flink-scala_2.10:1.3.2
>> (depends on 2.0.1)
>> [warn]
>> [warn] * org.mortbay.jetty:jetty-util:6.1.26 is selected over
>> 6.1.26.hwx
>> [warn] +- org.apache.hbase:hbase-common:1.3.0
>> (depends on 6.1.26)
>> [warn] +- org.apache.flink:flink-shaded-hadoop2:1.3.2
>> (depends on 6.1.26.hwx)
>> [warn]
>> [warn] * io.netty:netty:3.8.0.Final is selected over 3.7.0.Final
>> [warn] +- com.data-artisans:flakka-remote_2.10:2.3-custom
>> (depends on 3.8.0.Final)
>> [warn] +- org.apache.zookeeper:zookeeper:3.4.6
>> (depends on 3.7.0.Final)
>> [warn]
>> [warn] * com.google.guava:guava:18.0 is selected over 12.0.1
>> [warn] +- org.apache.flink:flink-connector-kinesis_2.10:1.3.2
>> (depends on 18.0)
>> [warn] +- org.apache.hbase:hbase-prefix-tree:1.3.0
>> (depends on 12.0.1)
>> [warn] +- org.apache.hbase:hbase-procedure:1.3.0
>> (depends on 12.0.1)
>> [warn] +- org.apache.hbase:hbase-client:1.3.0
>> (depends on 12.0.1)
>> [warn] +- org.apache.hbase:hbase-common:1.3.0
>> (depends on 12.0.1)
>> [warn] +- org.apache.hbase:hbase-server:1.3.0
>> (depends on 12.0.1)
>> [warn]
>> [warn] * io.netty:netty-all:4.0.27.Final is selected over
>> 4.0.23.Final
>> [warn] +- org.apache.flink:flink-runtime_2.10:1.3.2
>> (depends on 4.0.27.Final)
>> [warn] +- org.apache.flink:flink-shaded-hadoop2:1.3.2
>> (depends on 4.0.27.Final)
>> [warn] +- org.apache.hbase:hbase-prefix-tree:1.3.0
>> (depends on 4.0.23.Final)
>> [warn] +- org.apache.hbase:hbase-client:1.3.0
>> (depends on 4.0.23.Final)
>> [warn] +- org.apache.hbase:hbase-server:1.3.0
>> (depends on 4.0.23.Final)
>> [warn]
>> [warn] * junit:junit:4.12 is selected over 3.8.1
>> [warn] +- org.apache.hbase:hbase-protocol:1.3.0
>> (depends on 4.12)
>> [warn] +- org.apache.hbase:hbase-annotations:1.3.0
>> (depends on 4.12)
>> [warn] +- org.apache.hbase:hbase-prefix-tree:1.3.0
>> (depends on 4.12)
>> [warn] +- org.apache.hbase:hbase-procedure:1.3.0
>> (depends on 4.12)
>> [warn] +- org.apache.hbase:hbase-client:1.3.0
>> (depends on 4.12)
>> [warn] +- org.apache.hbase:hbase-common:1.3.0
>> (depends on 4.12)
>> [warn] +- org.apache.hbase:hbase-server:1.3.0
>> (depends on 4.12)
>> [warn] +- jline:jline:0.9.94
>> (depends on 3.8.1)
>>
>> And here I am asking for help here.
>>
>> Thank you very much for your attention,
>> Kind regards,
>>
>> Federico D'Ambrosio
>>
>
>
Re: Possible conflict between in Flink connectors
Posted by Ted Yu <yu...@gmail.com>.
If Guava 18.0 is used to build hbase 1.3, there would be compilation errors
such as the following:
[ERROR] /mnt/disk2/a/1.3-h/hbase-server/src/main/java/org/
apache/hadoop/hbase/replication/regionserver/ReplicationSource.java:[271,25]
error: cannot find symbol
[ERROR] symbol: method stopAndWait()
[ERROR] location: variable replicationEndpoint of type ReplicationEndpoint
[ERROR] /mnt/disk2/a/1.3-h/hbase-server/src/main/java/org/
apache/hadoop/hbase/replication/regionserver/ReplicationSource.java:[281,47]
error: cannot find symbol
[ERROR] symbol: method start()
[ERROR] location: variable replicationEndpoint of type ReplicationEndpoint
Maybe you can shade the guava dependency in hbase 1.3
In the upcoming hbase 2.0 release, third party dependencies such as guava
and netty are shaded. Meaning there wouldn't be such conflict with other
components you may use.
On Fri, Aug 25, 2017 at 8:36 AM, Federico D'Ambrosio <
federico.dambrosio@smartlab.ws> wrote:
> Hello everyone, I'm new to Flink and am encountering a nasty problem while
> trying to submit a streaming Flink Job. I'll try to explain it as
> thoroughly as possible.
>
> Premise: I'm using an HDP 2.6 hadoop cluster, with hadoop version
> 2.7.3.2.6.1.0-129, Flink compiled from sources accordingly (maven 3.0.5) as
> per documentation
> <https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/building.html#vendor-specific-versions>
> and I submit the job using yarn-session.sh and then flink run.
>
> Getting into more details, the Flink Job is a fat jar built with sbt
> assembly and the following dependencies:
>
> libraryDependencies += "org.apache.flink" %% "flink-scala" % flinkVersion % "provided"
> libraryDependencies += "org.apache.flink" %% "flink-streaming-scala" % flinkVersion % "provided"
> libraryDependencies += "org.apache.flink" %% "flink-connector-kinesis" % flinkVersion % "provided"
> libraryDependencies += "org.apache.flink" %% "flink-hbase" % flinkVersion % "provided"
> libraryDependencies += "org.apache.flink" %% "flink-connector-filesystem" % flinkVersion % "provided"
> libraryDependencies += "org.apache.hbase" % "hbase-common" % "1.3.0"
> libraryDependencies += "org.joda" % "joda-convert" % "1.8.3"
> libraryDependencies += "com.typesafe.play" %% "play-json" % "2.6.2"
>
>
> assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = false)
>
>
> inside the flink lib folder I have the following Jars:
>
> flink-connector-filesystem_2.10-1.3.2.jar
> flink-python_2.10-1.3.2.jar
> flink-connector-kinesis_2.10-1.3.2.jar
> flink-shaded-hadoop2-uber-1.3.2.jar
> flink-dist_2.10-1.3.2.jar
> log4j-1.2.17.jar
> flink-hbase_2.10-1.3.2.jar
> slf4j-log4j12-1.7.7.jar
>
> Right now, the issue is that when I flink run the job, I get
> NoClassDefFound for org/apache/hadoop/hbase/client/Put class, despite it
> being inside the fat jar. So, I tried putting both hbase-common.jar and
> hbase-client.jars inside the lib folder, getting as a result another
> NoClassDefFound, only, this time, for com/google/common/collect/
> ListMultimap. Now, I noticed that flink-connector-kinesis and all the
> hbase packages (both hbase-common and flink-hbase deps) have a different
> version of guava as dependency: the first one needs version 18.0, while the
> second one needs 12.0.1 version. I, then, tried to enable the shading of
> guava packages with the following:
>
> assemblyShadeRules in assembly := Seq(
> ShadeRule.rename("com.google.**" -> "shade.com.google.@1").inAll
> )
>
>
> Only to no avail, getting the exact same error as before. And I really
> don't know how to submit the job without pouring every single dependency
> inside the lib folder. I thought it could be a compatibility issue with the
> YARN cluster, but I tried launching the job locally only to get the same
> error.
>
> As an additional info, I get the following warnings if I don't specify
> flink packages as provided:
>
> [warn] * org.scalamacros:quasiquotes_2.10:2.1.0 is selected over 2.0.1
> [warn] +- org.typelevel:macro-compat_2.10:1.1.1
> (depends on 2.1.0)
> [warn] +- org.apache.flink:flink-scala_2.10:1.3.2
> (depends on 2.0.1)
> [warn]
> [warn] * org.mortbay.jetty:jetty-util:6.1.26 is selected over
> 6.1.26.hwx
> [warn] +- org.apache.hbase:hbase-common:1.3.0
> (depends on 6.1.26)
> [warn] +- org.apache.flink:flink-shaded-hadoop2:1.3.2
> (depends on 6.1.26.hwx)
> [warn]
> [warn] * io.netty:netty:3.8.0.Final is selected over 3.7.0.Final
> [warn] +- com.data-artisans:flakka-remote_2.10:2.3-custom
> (depends on 3.8.0.Final)
> [warn] +- org.apache.zookeeper:zookeeper:3.4.6
> (depends on 3.7.0.Final)
> [warn]
> [warn] * com.google.guava:guava:18.0 is selected over 12.0.1
> [warn] +- org.apache.flink:flink-connector-kinesis_2.10:1.3.2
> (depends on 18.0)
> [warn] +- org.apache.hbase:hbase-prefix-tree:1.3.0
> (depends on 12.0.1)
> [warn] +- org.apache.hbase:hbase-procedure:1.3.0
> (depends on 12.0.1)
> [warn] +- org.apache.hbase:hbase-client:1.3.0
> (depends on 12.0.1)
> [warn] +- org.apache.hbase:hbase-common:1.3.0
> (depends on 12.0.1)
> [warn] +- org.apache.hbase:hbase-server:1.3.0
> (depends on 12.0.1)
> [warn]
> [warn] * io.netty:netty-all:4.0.27.Final is selected over 4.0.23.Final
> [warn] +- org.apache.flink:flink-runtime_2.10:1.3.2
> (depends on 4.0.27.Final)
> [warn] +- org.apache.flink:flink-shaded-hadoop2:1.3.2
> (depends on 4.0.27.Final)
> [warn] +- org.apache.hbase:hbase-prefix-tree:1.3.0
> (depends on 4.0.23.Final)
> [warn] +- org.apache.hbase:hbase-client:1.3.0
> (depends on 4.0.23.Final)
> [warn] +- org.apache.hbase:hbase-server:1.3.0
> (depends on 4.0.23.Final)
> [warn]
> [warn] * junit:junit:4.12 is selected over 3.8.1
> [warn] +- org.apache.hbase:hbase-protocol:1.3.0
> (depends on 4.12)
> [warn] +- org.apache.hbase:hbase-annotations:1.3.0
> (depends on 4.12)
> [warn] +- org.apache.hbase:hbase-prefix-tree:1.3.0
> (depends on 4.12)
> [warn] +- org.apache.hbase:hbase-procedure:1.3.0
> (depends on 4.12)
> [warn] +- org.apache.hbase:hbase-client:1.3.0
> (depends on 4.12)
> [warn] +- org.apache.hbase:hbase-common:1.3.0
> (depends on 4.12)
> [warn] +- org.apache.hbase:hbase-server:1.3.0
> (depends on 4.12)
> [warn] +- jline:jline:0.9.94
> (depends on 3.8.1)
>
> And here I am asking for help here.
>
> Thank you very much for your attention,
> Kind regards,
>
> Federico D'Ambrosio
>