You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by Michael Williams <mw...@phemi.com> on 2018/10/12 23:38:27 UTC
Build failure on 0.8.0 when using CDH hadoop
Hey all,
I'm hitting some dependency issues when trying to build Zeppelin 0.8.0 with
CDH-5.14.4. Here's the maven command I'm using:
mvn clean install -DskipTests -Pbuild-distr -Pvendor-repo
> -Dhadoop.version=2.6.0-cdh5.14.4 -Dcheckstyle.skip=true -Pr -Pspark-2.2
> -Dspark.version=2.2.0 -Pscala-2.11
>
With that I run into some Jackson and Zookeeper issues, mainly with
hadoop-client and hadoop-azure. I see that this has been an issue before
and there was a PR to fix CDH compatibility issues for 0.8.0
<https://github.com/apache/zeppelin/pull/2723>, but it looks like those
changes have been overwritten at some point.
Wondering if this is a known issue or are CDH builds supposed to be
working? Was that commit to fix this overwritten by accident or not? In the
case that is should be working, is there something that anyone can see
wrong with what I am doing?
Thanks,
Michael
Unsubscribe
Posted by "Richard A. Bross" <rb...@oaktreepeak.com>.
Unsubscribe
----- Original Message -----
From: "Jeff Zhang" <zj...@gmail.com>
To: users@zeppelin.apache.org
Sent: Friday, October 12, 2018 8:33:35 PM
Subject: Re: Build failure on 0.8.0 when using CDH hadoop
You don't need to build with CDH to run zeppelin under CDH, you can just run the following command to run zeppelin under CDH
mvn clean package -DskipTests
Michael Williams < mwilliams@phemi.com >于2018年10月13日周六 上午7:38写道：
Hey all,
I'm hitting some dependency issues when trying to build Zeppelin 0.8.0 with CDH-5.14.4. Here's the maven command I'm using:
mvn clean install -DskipTests -Pbuild-distr -Pvendor-repo -Dhadoop.version=2.6.0-cdh5.14.4 -Dcheckstyle.skip=true -Pr -Pspark-2.2 -Dspark.version=2.2.0 -Pscala-2.11
With that I run into some Jackson and Zookeeper issues, mainly with hadoop-client and hadoop-azure. I see that this has been an issue before and there was a PR to fix CDH compatibility issues for 0.8.0 , but it looks like those changes have been overwritten at some point.
Wondering if this is a known issue or are CDH builds supposed to be working? Was that commit to fix this overwritten by accident or not? In the case that is should be working, is there something that anyone can see wrong with what I am doing?
Thanks,
Michael
Re: Build failure on 0.8.0 when using CDH hadoop
Posted by Michael Williams <mw...@phemi.com>.
Thanks for the response, Ruslan.
I've noticed that the maven profiles seem to have changed since 0.7.3, so
specifying the profile "-Phadoop-2.6" is ignored and still builds with
Hadoop 2.7.3, the default version. The actual profile is "-Phadoop2" and
I'm assuming that from there you are to specify the specific version with
"-Dhadoop.version=2.6.0....". Let me know if this is not the case for you.
I guess what Jeff was saying is that now Zeppelin shades all dependencies
I can see how shading will solve the dependency issues faced when trying to
build with CDH at the moment but there are still modules that are not being
shaded that will cause transitive deps issues. The biggest culprit being
"zeppelin-zengine", which runs into the same jackson/zookeeper dependency
issues.
On Mon, Oct 15, 2018 at 11:02 AM Ruslan Dautkhanov <da...@gmail.com>
wrote:
> Michael,
>
> This is my build command for Cloudera:
>
> mvn clean package -DskipTests -Pspark-2.2 -Phadoop-2.6 -Pvendor-repo
> -Pscala-2.10 -Psparkr -pl
> '!alluxio,!flink,!ignite,!lens,!cassandra,!bigquery,!scio' -e
>
> It works okay with CDH. Including CDH 5.14 you mentioned.
>
> We used to have -Dhadoop.version=2.6.0-cdh5.12.1 way back but it had some
> issues in more recent Zeppelin upstream.
>
> I guess what Jeff was saying is that now Zeppelin shades all dependencies
> See https://github.com/apache/zeppelin/pull/3170 that was committed last
> month.
> So I think you now don't even need to provide hadoop-2.6 profile at all,
> but I haven't tested that.
> You can test and let us know which way works for you.
>
> --
> Ruslan Dautkhanov
>
>
> On Mon, Oct 15, 2018 at 11:48 AM Michael Williams <mw...@phemi.com>
> wrote:
>
>> I understand it's possible to build and run Zeppelin using plain Hadoop,
>> but we are always running on Cloudera clusters so it makes sense for us to
>> build against Cloudera's Hadoop dist. Or would you recommend building using
>> plain Hadoop, not as a workaround but for some other reason?
>>
>> Since we are running on Cloudera, it would be nice to just have one
>> version of the Hadoop jars floating around.
>>
>> On Fri, Oct 12, 2018 at 5:33 PM Jeff Zhang <zj...@gmail.com> wrote:
>>
>>> You don't need to build with CDH to run zeppelin under CDH, you can just
>>> run the following command to run zeppelin under CDH
>>>
>>> mvn clean package -DskipTests
>>>
>>>
>>> Michael Williams <mw...@phemi.com>于2018年10月13日周六 上午7:38写道：
>>>
>>>> Hey all,
>>>>
>>>> I'm hitting some dependency issues when trying to build Zeppelin 0.8.0
>>>> with CDH-5.14.4. Here's the maven command I'm using:
>>>>
>>>> mvn clean install -DskipTests -Pbuild-distr -Pvendor-repo
>>>>> -Dhadoop.version=2.6.0-cdh5.14.4 -Dcheckstyle.skip=true -Pr -Pspark-2.2
>>>>> -Dspark.version=2.2.0 -Pscala-2.11
>>>>>
>>>>
>>>> With that I run into some Jackson and Zookeeper issues, mainly with
>>>> hadoop-client and hadoop-azure. I see that this has been an issue before
>>>> and there was a PR to fix CDH compatibility issues for 0.8.0
>>>> <https://github.com/apache/zeppelin/pull/2723>, but it looks like
>>>> those changes have been overwritten at some point.
>>>>
>>>> Wondering if this is a known issue or are CDH builds supposed to be
>>>> working? Was that commit to fix this overwritten by accident or not? In the
>>>> case that is should be working, is there something that anyone can see
>>>> wrong with what I am doing?
>>>>
>>>> Thanks,
>>>> Michael
>>>>
>>>
Re: Build failure on 0.8.0 when using CDH hadoop
Posted by Ruslan Dautkhanov <da...@gmail.com>.
Michael,
This is my build command for Cloudera:
mvn clean package -DskipTests -Pspark-2.2 -Phadoop-2.6 -Pvendor-repo
-Pscala-2.10 -Psparkr -pl
'!alluxio,!flink,!ignite,!lens,!cassandra,!bigquery,!scio' -e
It works okay with CDH. Including CDH 5.14 you mentioned.
We used to have -Dhadoop.version=2.6.0-cdh5.12.1 way back but it had some
issues in more recent Zeppelin upstream.
I guess what Jeff was saying is that now Zeppelin shades all dependencies
See https://github.com/apache/zeppelin/pull/3170 that was committed last
month.
So I think you now don't even need to provide hadoop-2.6 profile at all,
but I haven't tested that.
You can test and let us know which way works for you.
--
Ruslan Dautkhanov
On Mon, Oct 15, 2018 at 11:48 AM Michael Williams <mw...@phemi.com>
wrote:
> I understand it's possible to build and run Zeppelin using plain Hadoop,
> but we are always running on Cloudera clusters so it makes sense for us to
> build against Cloudera's Hadoop dist. Or would you recommend building using
> plain Hadoop, not as a workaround but for some other reason?
>
> Since we are running on Cloudera, it would be nice to just have one
> version of the Hadoop jars floating around.
>
> On Fri, Oct 12, 2018 at 5:33 PM Jeff Zhang <zj...@gmail.com> wrote:
>
>> You don't need to build with CDH to run zeppelin under CDH, you can just
>> run the following command to run zeppelin under CDH
>>
>> mvn clean package -DskipTests
>>
>>
>> Michael Williams <mw...@phemi.com>于2018年10月13日周六 上午7:38写道：
>>
>>> Hey all,
>>>
>>> I'm hitting some dependency issues when trying to build Zeppelin 0.8.0
>>> with CDH-5.14.4. Here's the maven command I'm using:
>>>
>>> mvn clean install -DskipTests -Pbuild-distr -Pvendor-repo
>>>> -Dhadoop.version=2.6.0-cdh5.14.4 -Dcheckstyle.skip=true -Pr -Pspark-2.2
>>>> -Dspark.version=2.2.0 -Pscala-2.11
>>>>
>>>
>>> With that I run into some Jackson and Zookeeper issues, mainly with
>>> hadoop-client and hadoop-azure. I see that this has been an issue before
>>> and there was a PR to fix CDH compatibility issues for 0.8.0
>>> <https://github.com/apache/zeppelin/pull/2723>, but it looks like those
>>> changes have been overwritten at some point.
>>>
>>> Wondering if this is a known issue or are CDH builds supposed to be
>>> working? Was that commit to fix this overwritten by accident or not? In the
>>> case that is should be working, is there something that anyone can see
>>> wrong with what I am doing?
>>>
>>> Thanks,
>>> Michael
>>>
>>
Re: Build failure on 0.8.0 when using CDH hadoop
Posted by Michael Williams <mw...@phemi.com>.
I understand it's possible to build and run Zeppelin using plain Hadoop,
but we are always running on Cloudera clusters so it makes sense for us to
build against Cloudera's Hadoop dist. Or would you recommend building using
plain Hadoop, not as a workaround but for some other reason?
Since we are running on Cloudera, it would be nice to just have one version
of the Hadoop jars floating around.
On Fri, Oct 12, 2018 at 5:33 PM Jeff Zhang <zj...@gmail.com> wrote:
> You don't need to build with CDH to run zeppelin under CDH, you can just
> run the following command to run zeppelin under CDH
>
> mvn clean package -DskipTests
>
>
> Michael Williams <mw...@phemi.com>于2018年10月13日周六 上午7:38写道：
>
>> Hey all,
>>
>> I'm hitting some dependency issues when trying to build Zeppelin 0.8.0
>> with CDH-5.14.4. Here's the maven command I'm using:
>>
>> mvn clean install -DskipTests -Pbuild-distr -Pvendor-repo
>>> -Dhadoop.version=2.6.0-cdh5.14.4 -Dcheckstyle.skip=true -Pr -Pspark-2.2
>>> -Dspark.version=2.2.0 -Pscala-2.11
>>>
>>
>> With that I run into some Jackson and Zookeeper issues, mainly with
>> hadoop-client and hadoop-azure. I see that this has been an issue before
>> and there was a PR to fix CDH compatibility issues for 0.8.0
>> <https://github.com/apache/zeppelin/pull/2723>, but it looks like those
>> changes have been overwritten at some point.
>>
>> Wondering if this is a known issue or are CDH builds supposed to be
>> working? Was that commit to fix this overwritten by accident or not? In the
>> case that is should be working, is there something that anyone can see
>> wrong with what I am doing?
>>
>> Thanks,
>> Michael
>>
>
Re: Build failure on 0.8.0 when using CDH hadoop
Posted by Jeff Zhang <zj...@gmail.com>.
You don't need to build with CDH to run zeppelin under CDH, you can just
run the following command to run zeppelin under CDH
mvn clean package -DskipTests
Michael Williams <mw...@phemi.com>于2018年10月13日周六 上午7:38写道：
> Hey all,
>
> I'm hitting some dependency issues when trying to build Zeppelin 0.8.0
> with CDH-5.14.4. Here's the maven command I'm using:
>
> mvn clean install -DskipTests -Pbuild-distr -Pvendor-repo
>> -Dhadoop.version=2.6.0-cdh5.14.4 -Dcheckstyle.skip=true -Pr -Pspark-2.2
>> -Dspark.version=2.2.0 -Pscala-2.11
>>
>
> With that I run into some Jackson and Zookeeper issues, mainly with
> hadoop-client and hadoop-azure. I see that this has been an issue before
> and there was a PR to fix CDH compatibility issues for 0.8.0
> <https://github.com/apache/zeppelin/pull/2723>, but it looks like those
> changes have been overwritten at some point.
>
> Wondering if this is a known issue or are CDH builds supposed to be
> working? Was that commit to fix this overwritten by accident or not? In the
> case that is should be working, is there something that anyone can see
> wrong with what I am doing?
>
> Thanks,
> Michael
>