You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Lars Francke <la...@gmail.com> on 2014/02/28 13:13:17 UTC
YARN Maven build questions
Hey,
I'm trying to dig into Spark's code but am running into a couple of problems.
1) The yarn-common directory is not included in the Maven build
causing things to fail because the dependency is missing. If I see the
history correct it used to be a Maven module but is not anymore.
2) When I try to include the yarn-common directory in the build things
start going bad. Compilation failures all over the place and I think
there are some dependency issues in there as well.
This leads me to believe that either the Maven build system isn't
maintained for YARN or the whole YARN branch isn't. What's the status
here?
Without YARN things build fine for me using Maven.
Thanks for your help.
Cheers,
Lars
Re: YARN Maven build questions
Posted by Usman Ghani <us...@platfora.com>.
Tom,
I tried the command line you used and it didn't work for me.
On Fri, Feb 28, 2014 at 6:05 AM, Tom Graves <tg...@yahoo.com> wrote:
> what build command are you using? What do you mean when you say YARN
> branch?
>
> The yarn builds have been working fine for me with maven. Build command
> I use against hadoop 2.2 or higher: mvn -Dyarn.version=2.2.0
> -Dhadoop.version=2.2.0 -Pyarn clean package -DskipTests
>
> Tom
>
>
>
> On Friday, February 28, 2014 6:14 AM, Lars Francke <la...@gmail.com>
> wrote:
>
> Hey,
>
> I'm trying to dig into Spark's code but am running into a couple of
> problems.
>
> 1) The yarn-common directory is not included in the Maven build
> causing things to fail because the dependency is missing. If I see the
> history correct it used to be a Maven module but is not anymore.
>
> 2) When I try to include the yarn-common directory in the build things
> start going bad. Compilation failures all over the place and I think
> there are some dependency issues in there as well.
>
> This leads me to believe that either the Maven build system isn't
> maintained for YARN or the whole YARN branch isn't. What's the status
> here?
>
> Without YARN things build fine for me using Maven.
>
> Thanks for your help.
>
> Cheers,
> Lars
>
Re: YARN Maven build questions
Posted by Sandy Ryza <sa...@cloudera.com>.
Hi Lars,
Unfortunately, due to some incompatible changes we pulled in to be closer
to YARN trunk, Spark-on-YARN does not work against CDH 4.4+ (but does work
against CDH5)
-Sandy
On Tue, Mar 4, 2014 at 6:33 AM, Tom Graves <tg...@yahoo.com> wrote:
> What is your question about Any hints?
> The maven build worked for me yesterday again fine.
>
> You should create a jira for any pull request like the documentation
> states. The jira thing is new so I think people are still getting used to
> it.
>
> Tom
>
>
>
> On Tuesday, March 4, 2014 2:51 AM, Lars Francke <la...@gmail.com>
> wrote:
>
> Hi,
>
> sorry to bother again.
>
> As a newbie to the project it's hard to judge whether I'm doing
> anything wrong, the documentation is outdated or the Maven/SBT files
> have diverged from the actual code by defining older/now incompatible
> versions or something else going wrong.
>
> Any hints?
>
> Also an unrelated note/question: I see tons of pull requests being
> accepted without a JIRA but the documentation says to create a JIRA
> issue first[1]. So I assume it's okay to just send pull requests?
>
> Thanks for your help.
>
> Cheers,
> Lars
>
> [1] <
> https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark>
>
>
> On Fri, Feb 28, 2014 at 6:41 PM, Lars Francke <la...@gmail.com>
> wrote:
> > Hey,
> >
> > so currently it doesn't work because of
> > <https://github.com/apache/spark/pull/6#issuecomment-36343187>
> >
> > IntelliJ reports a lot of warnings with default settings and I haven't
> > found a way to tell IntellJ to use different Hadoop versions yet.
> > mvn clean compile -Pyarn fails as well (compilation errror
> >
> > Your command works indeed. Default yarn version is 0.23.7 which
> > doesn't seem to work with the default 2.2.0 Hadoop version (anymore?)
> >
> > I was basically trying to follow the documentation:
> > <http://spark.incubator.apache.org/docs/latest/building-with-maven.html>
> >
> > mvn clean compile -Pyarn-alpha -Dhadoop.version=2.0.0-cdh4.5.0
> > -Dyarn.version=2.0.0-cdh4.5.0 fails as well as does mvn clean compile
> > -Pyarn-alpha
> >
> > Thanks for showing me a configuration that works. Unfortunately the
> > default ones and at least one of the documented ones fail.
> >
> > Cheers,
> > Lars
> >
> >
> > On Fri, Feb 28, 2014 at 3:05 PM, Tom Graves <tg...@yahoo.com>
> wrote:
> >> what build command are you using? What do you mean when you say YARN
> branch?
> >>
> >> The yarn builds have been working fine for me with maven. Build
> command I use against hadoop 2.2 or higher: mvn -Dyarn.version=2.2.0
> -Dhadoop.version=2.2.0 -Pyarn clean package -DskipTests
> >>
> >> Tom
> >>
> >>
> >>
> >> On Friday, February 28, 2014 6:14 AM, Lars Francke <
> lars.francke@gmail.com> wrote:
> >>
> >> Hey,
> >>
> >> I'm trying to dig into Spark's code but am running into a couple of
> problems.
> >>
> >> 1) The yarn-common directory is not included in the Maven build
> >> causing things to fail because the dependency is missing. If I see the
> >> history correct it used to be a Maven module but is not anymore.
> >>
> >> 2) When I try to include the yarn-common directory in the build things
> >> start going bad. Compilation failures all over the place and I think
> >> there are some dependency issues in there as well.
> >>
> >> This leads me to believe that either the Maven build system isn't
> >> maintained for YARN or the whole YARN branch isn't. What's the status
> >> here?
> >>
> >> Without YARN things build fine for me using Maven.
> >>
> >> Thanks for your help.
> >>
> >> Cheers,
> >> Lars
>
Re: YARN Maven build questions
Posted by Tom Graves <tg...@yahoo.com>.
What is your question about Any hints?
The maven build worked for me yesterday again fine.
You should create a jira for any pull request like the documentation states. The jira thing is new so I think people are still getting used to it.
Tom
On Tuesday, March 4, 2014 2:51 AM, Lars Francke <la...@gmail.com> wrote:
Hi,
sorry to bother again.
As a newbie to the project it's hard to judge whether I'm doing
anything wrong, the documentation is outdated or the Maven/SBT files
have diverged from the actual code by defining older/now incompatible
versions or something else going wrong.
Any hints?
Also an unrelated note/question: I see tons of pull requests being
accepted without a JIRA but the documentation says to create a JIRA
issue first[1]. So I assume it's okay to just send pull requests?
Thanks for your help.
Cheers,
Lars
[1] <https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark>
On Fri, Feb 28, 2014 at 6:41 PM, Lars Francke <la...@gmail.com> wrote:
> Hey,
>
> so currently it doesn't work because of
> <https://github.com/apache/spark/pull/6#issuecomment-36343187>
>
> IntelliJ reports a lot of warnings with default settings and I haven't
> found a way to tell IntellJ to use different Hadoop versions yet.
> mvn clean compile -Pyarn fails as well (compilation errror
>
> Your command works indeed. Default yarn version is 0.23.7 which
> doesn't seem to work with the default 2.2.0 Hadoop version (anymore?)
>
> I was basically trying to follow the documentation:
> <http://spark.incubator.apache.org/docs/latest/building-with-maven.html>
>
> mvn clean compile -Pyarn-alpha -Dhadoop.version=2.0.0-cdh4.5.0
> -Dyarn.version=2.0.0-cdh4.5.0 fails as well as does mvn clean compile
> -Pyarn-alpha
>
> Thanks for showing me a configuration that works. Unfortunately the
> default ones and at least one of the documented ones fail.
>
> Cheers,
> Lars
>
>
> On Fri, Feb 28, 2014 at 3:05 PM, Tom Graves <tg...@yahoo.com> wrote:
>> what build command are you using? What do you mean when you say YARN branch?
>>
>> The yarn builds have been working fine for me with maven. Build command I use against hadoop 2.2 or higher: mvn -Dyarn.version=2.2.0 -Dhadoop.version=2.2.0 -Pyarn clean package -DskipTests
>>
>> Tom
>>
>>
>>
>> On Friday, February 28, 2014 6:14 AM, Lars Francke <la...@gmail.com> wrote:
>>
>> Hey,
>>
>> I'm trying to dig into Spark's code but am running into a couple of problems.
>>
>> 1) The yarn-common directory is not included in the Maven build
>> causing things to fail because the dependency is missing. If I see the
>> history correct it used to be a Maven module but is not anymore.
>>
>> 2) When I try to include the yarn-common directory in the build things
>> start going bad. Compilation failures all over the place and I think
>> there are some dependency issues in there as well.
>>
>> This leads me to believe that either the Maven build system isn't
>> maintained for YARN or the whole YARN branch isn't. What's the status
>> here?
>>
>> Without YARN things build fine for me using Maven.
>>
>> Thanks for your help.
>>
>> Cheers,
>> Lars
Re: YARN Maven build questions
Posted by Lars Francke <la...@gmail.com>.
Hi,
sorry to bother again.
As a newbie to the project it's hard to judge whether I'm doing
anything wrong, the documentation is outdated or the Maven/SBT files
have diverged from the actual code by defining older/now incompatible
versions or something else going wrong.
Any hints?
Also an unrelated note/question: I see tons of pull requests being
accepted without a JIRA but the documentation says to create a JIRA
issue first[1]. So I assume it's okay to just send pull requests?
Thanks for your help.
Cheers,
Lars
[1] <https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark>
On Fri, Feb 28, 2014 at 6:41 PM, Lars Francke <la...@gmail.com> wrote:
> Hey,
>
> so currently it doesn't work because of
> <https://github.com/apache/spark/pull/6#issuecomment-36343187>
>
> IntelliJ reports a lot of warnings with default settings and I haven't
> found a way to tell IntellJ to use different Hadoop versions yet.
> mvn clean compile -Pyarn fails as well (compilation errror
>
> Your command works indeed. Default yarn version is 0.23.7 which
> doesn't seem to work with the default 2.2.0 Hadoop version (anymore?)
>
> I was basically trying to follow the documentation:
> <http://spark.incubator.apache.org/docs/latest/building-with-maven.html>
>
> mvn clean compile -Pyarn-alpha -Dhadoop.version=2.0.0-cdh4.5.0
> -Dyarn.version=2.0.0-cdh4.5.0 fails as well as does mvn clean compile
> -Pyarn-alpha
>
> Thanks for showing me a configuration that works. Unfortunately the
> default ones and at least one of the documented ones fail.
>
> Cheers,
> Lars
>
>
> On Fri, Feb 28, 2014 at 3:05 PM, Tom Graves <tg...@yahoo.com> wrote:
>> what build command are you using? What do you mean when you say YARN branch?
>>
>> The yarn builds have been working fine for me with maven. Build command I use against hadoop 2.2 or higher: mvn -Dyarn.version=2.2.0 -Dhadoop.version=2.2.0 -Pyarn clean package -DskipTests
>>
>> Tom
>>
>>
>>
>> On Friday, February 28, 2014 6:14 AM, Lars Francke <la...@gmail.com> wrote:
>>
>> Hey,
>>
>> I'm trying to dig into Spark's code but am running into a couple of problems.
>>
>> 1) The yarn-common directory is not included in the Maven build
>> causing things to fail because the dependency is missing. If I see the
>> history correct it used to be a Maven module but is not anymore.
>>
>> 2) When I try to include the yarn-common directory in the build things
>> start going bad. Compilation failures all over the place and I think
>> there are some dependency issues in there as well.
>>
>> This leads me to believe that either the Maven build system isn't
>> maintained for YARN or the whole YARN branch isn't. What's the status
>> here?
>>
>> Without YARN things build fine for me using Maven.
>>
>> Thanks for your help.
>>
>> Cheers,
>> Lars
Re: YARN Maven build questions
Posted by Lars Francke <la...@gmail.com>.
Hey,
so currently it doesn't work because of
<https://github.com/apache/spark/pull/6#issuecomment-36343187>
IntelliJ reports a lot of warnings with default settings and I haven't
found a way to tell IntellJ to use different Hadoop versions yet.
mvn clean compile -Pyarn fails as well (compilation errror
Your command works indeed. Default yarn version is 0.23.7 which
doesn't seem to work with the default 2.2.0 Hadoop version (anymore?)
I was basically trying to follow the documentation:
<http://spark.incubator.apache.org/docs/latest/building-with-maven.html>
mvn clean compile -Pyarn-alpha -Dhadoop.version=2.0.0-cdh4.5.0
-Dyarn.version=2.0.0-cdh4.5.0 fails as well as does mvn clean compile
-Pyarn-alpha
Thanks for showing me a configuration that works. Unfortunately the
default ones and at least one of the documented ones fail.
Cheers,
Lars
On Fri, Feb 28, 2014 at 3:05 PM, Tom Graves <tg...@yahoo.com> wrote:
> what build command are you using? What do you mean when you say YARN branch?
>
> The yarn builds have been working fine for me with maven. Build command I use against hadoop 2.2 or higher: mvn -Dyarn.version=2.2.0 -Dhadoop.version=2.2.0 -Pyarn clean package -DskipTests
>
> Tom
>
>
>
> On Friday, February 28, 2014 6:14 AM, Lars Francke <la...@gmail.com> wrote:
>
> Hey,
>
> I'm trying to dig into Spark's code but am running into a couple of problems.
>
> 1) The yarn-common directory is not included in the Maven build
> causing things to fail because the dependency is missing. If I see the
> history correct it used to be a Maven module but is not anymore.
>
> 2) When I try to include the yarn-common directory in the build things
> start going bad. Compilation failures all over the place and I think
> there are some dependency issues in there as well.
>
> This leads me to believe that either the Maven build system isn't
> maintained for YARN or the whole YARN branch isn't. What's the status
> here?
>
> Without YARN things build fine for me using Maven.
>
> Thanks for your help.
>
> Cheers,
> Lars
Re: YARN Maven build questions
Posted by Tom Graves <tg...@yahoo.com>.
what build command are you using? What do you mean when you say YARN branch?
The yarn builds have been working fine for me with maven. Build command I use against hadoop 2.2 or higher: mvn -Dyarn.version=2.2.0 -Dhadoop.version=2.2.0 -Pyarn clean package -DskipTests
Tom
On Friday, February 28, 2014 6:14 AM, Lars Francke <la...@gmail.com> wrote:
Hey,
I'm trying to dig into Spark's code but am running into a couple of problems.
1) The yarn-common directory is not included in the Maven build
causing things to fail because the dependency is missing. If I see the
history correct it used to be a Maven module but is not anymore.
2) When I try to include the yarn-common directory in the build things
start going bad. Compilation failures all over the place and I think
there are some dependency issues in there as well.
This leads me to believe that either the Maven build system isn't
maintained for YARN or the whole YARN branch isn't. What's the status
here?
Without YARN things build fine for me using Maven.
Thanks for your help.
Cheers,
Lars