You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Matei Zaharia <ma...@gmail.com> on 2013/10/29 19:34:35 UTC
Re: spark-0.8.0 and hadoop-2.1.0-beta
I’m curious, Viren, do you have a patch you could post to build this against YARN 2.1 / 2.2? It would be nice to see how big the changes are.
Matei
On Sep 30, 2013, at 10:14 AM, viren kumar <vi...@gmail.com> wrote:
> I was able to get Spark 0.8.0 to compile with Hadoop/Yarn 2.1.0-beta, by following some of the changes described here: http://hortonworks.com/blog/stabilizing-yarn-apis-for-apache-hadoop-2-beta-and-beyond/
>
> That should help you build most of it. One change not covered there is the change from ProtoUtils.convertFromProtoFormat(containerToken, cmAddress) to ConverterUtils.convertFromYarn(containerToken, cmAddress).
>
> Not 100% sure that my changes are correct.
>
> Hope that helps,
> Viren
>
>
> On Sun, Sep 29, 2013 at 8:59 AM, Matei Zaharia <ma...@gmail.com> wrote:
> Hi Terence,
>
> YARN's API changed in an incompatible way in Hadoop 2.1.0, so I'd suggest sticking with 2.0.x for now. We may create a different branch for this version. Unfortunately due to the API change it may not be possible to support this version while also supporting other widely-used versions like 0.23.x.
>
> Matei
>
> On Sep 29, 2013, at 11:00 AM, Terance Dias <te...@gmail.com> wrote:
>
> >
> > Hi, I'm trying to build spark-0.8.0 with hadoop-2.1.0-beta.
> > I have changed the following properties in SparkBuild.scala file.
> >
> > val DEFAULT_HADOOP_VERSION = "2.1.0-beta"
> > val DEFAULT_YARN = true
> >
> > when i do sbt clean compile, I get an error saying
> >
> > [error] /usr/local/spark-0.8.0-incubating/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:42: not found: type AMRMProtocol
> > [error] private var resourceManager: AMRMProtocol = null
> >
> > Thanks,
> > Terance.
> >
>
>
RE: spark-0.8.0 and hadoop-2.1.0-beta
Posted by "Liu, Raymond" <ra...@intel.com>.
After simply replace the API with new one and ignore those I could not find a replacement. I can make it pass compile, But just fail at AM registering
13/10/30 13:22:01 INFO yarn.ApplicationMaster: Registering the ApplicationMaster
13/10/30 13:22:02 ERROR security.UserGroupInformation: PriviledgedActionException as:root (auth:SIMPLE) cause:org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN]
Seems need a few more investigation...
Best Regards,
Raymond Liu
-----Original Message-----
From: Liu, Raymond [mailto:raymond.liu@intel.com]
Sent: Wednesday, October 30, 2013 8:45 AM
To: user@spark.incubator.apache.org
Subject: RE: spark-0.8.0 and hadoop-2.1.0-beta
I am also working on porting the trunk code onto 2.2.0. Seems quite many API changes but many of them are just a rename work.
While Yarn 2.1.0 beta also add some client API for easy interaction with YARN framework, but there are not many examples on how to use them ( API and wiki doc are both old and not reflecting the new API), some part of SPARK YARN code will need to be rewritten with the new client API And I am not quite familiar with the user certification part of code, it might take times for it seems to me this part of codes also change a little bit, some methods gone, and I don't find the replacement or they are not need anymore.
Best Regards,
Raymond Liu
From: Matei Zaharia [mailto:matei.zaharia@gmail.com]
Sent: Wednesday, October 30, 2013 2:35 AM
To: user@spark.incubator.apache.org
Subject: Re: spark-0.8.0 and hadoop-2.1.0-beta
I'm curious, Viren, do you have a patch you could post to build this against YARN 2.1 / 2.2? It would be nice to see how big the changes are.
Matei
On Sep 30, 2013, at 10:14 AM, viren kumar <vi...@gmail.com> wrote:
I was able to get Spark 0.8.0 to compile with Hadoop/Yarn 2.1.0-beta, by following some of the changes described here: http://hortonworks.com/blog/stabilizing-yarn-apis-for-apache-hadoop-2-beta-and-beyond/
That should help you build most of it. One change not covered there is the change from ProtoUtils.convertFromProtoFormat(containerToken, cmAddress) to ConverterUtils.convertFromYarn(containerToken, cmAddress).
Not 100% sure that my changes are correct.
Hope that helps,
Viren
On Sun, Sep 29, 2013 at 8:59 AM, Matei Zaharia <ma...@gmail.com> wrote:
Hi Terence,
YARN's API changed in an incompatible way in Hadoop 2.1.0, so I'd suggest sticking with 2.0.x for now. We may create a different branch for this version. Unfortunately due to the API change it may not be possible to support this version while also supporting other widely-used versions like 0.23.x.
Matei
On Sep 29, 2013, at 11:00 AM, Terance Dias <te...@gmail.com> wrote:
>
> Hi, I'm trying to build spark-0.8.0 with hadoop-2.1.0-beta.
> I have changed the following properties in SparkBuild.scala file.
>
> val DEFAULT_HADOOP_VERSION = "2.1.0-beta"
> val DEFAULT_YARN = true
>
> when i do sbt clean compile, I get an error saying
>
> [error]
> /usr/local/spark-0.8.0-incubating/yarn/src/main/scala/org/apache/spark
> /deploy/yarn/ApplicationMaster.scala:42: not found: type AMRMProtocol
> [error] private var resourceManager: AMRMProtocol = null
>
> Thanks,
> Terance.
>
RE: spark-0.8.0 and hadoop-2.1.0-beta
Posted by "Liu, Raymond" <ra...@intel.com>.
I am also working on porting the trunk code onto 2.2.0. Seems quite many API changes but many of them are just a rename work.
While Yarn 2.1.0 beta also add some client API for easy interaction with YARN framework, but there are not many examples on how to use them ( API and wiki doc are both old and not reflecting the new API), some part of SPARK YARN code will need to be rewritten with the new client API
And I am not quite familiar with the user certification part of code, it might take times for it seems to me this part of codes also change a little bit, some methods gone, and I don't find the replacement or they are not need anymore.
Best Regards,
Raymond Liu
From: Matei Zaharia [mailto:matei.zaharia@gmail.com]
Sent: Wednesday, October 30, 2013 2:35 AM
To: user@spark.incubator.apache.org
Subject: Re: spark-0.8.0 and hadoop-2.1.0-beta
I'm curious, Viren, do you have a patch you could post to build this against YARN 2.1 / 2.2? It would be nice to see how big the changes are.
Matei
On Sep 30, 2013, at 10:14 AM, viren kumar <vi...@gmail.com> wrote:
I was able to get Spark 0.8.0 to compile with Hadoop/Yarn 2.1.0-beta, by following some of the changes described here: http://hortonworks.com/blog/stabilizing-yarn-apis-for-apache-hadoop-2-beta-and-beyond/
That should help you build most of it. One change not covered there is the change from ProtoUtils.convertFromProtoFormat(containerToken, cmAddress) to ConverterUtils.convertFromYarn(containerToken, cmAddress).
Not 100% sure that my changes are correct.
Hope that helps,
Viren
On Sun, Sep 29, 2013 at 8:59 AM, Matei Zaharia <ma...@gmail.com> wrote:
Hi Terence,
YARN's API changed in an incompatible way in Hadoop 2.1.0, so I'd suggest sticking with 2.0.x for now. We may create a different branch for this version. Unfortunately due to the API change it may not be possible to support this version while also supporting other widely-used versions like 0.23.x.
Matei
On Sep 29, 2013, at 11:00 AM, Terance Dias <te...@gmail.com> wrote:
>
> Hi, I'm trying to build spark-0.8.0 with hadoop-2.1.0-beta.
> I have changed the following properties in SparkBuild.scala file.
>
> val DEFAULT_HADOOP_VERSION = "2.1.0-beta"
> val DEFAULT_YARN = true
>
> when i do sbt clean compile, I get an error saying
>
> [error] /usr/local/spark-0.8.0-incubating/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:42: not found: type AMRMProtocol
> [error] private var resourceManager: AMRMProtocol = null
>
> Thanks,
> Terance.
>