You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Mohit Jaggi <mo...@gmail.com> on 2015/01/30 00:33:31 UTC
spark with cdh 5.2.1
Hi All,
I noticed in pom.xml that there is no entry for Hadoop 2.5. Has anyone tried Spark with 2.5.0-cdh5.2.1? Will replicating the 2.4 entry be sufficient to make this work?
Mohit.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: spark with cdh 5.2.1
Posted by Mohit Jaggi <mo...@gmail.com>.
Thanks Nobuhiro. I have my own fork so I can’t use precompiled repos. I copy pasted the hadoop 2.4 profile in pom.xml and replaced 2.4 with 2.5. It seems to have worked, though I wouldn’t be sure until I run more tests.
> On Jan 29, 2015, at 5:41 PM, Nobuhiro Sue <ns...@e-mail.ne.jp> wrote:
>
> Mohit,
>
> I'm using spark modules provided by Cloudera repos, it works fine.
>
> Please add Cloudera maven repo, and specify dependencies with CDH
> version, like "spark-core_2.10-1.1.0-cdh5.2.1".
>
> To add Cloudera maven repo, see:
> http://www.cloudera.com/content/cloudera/en/documentation/core/v5-2-x/topics/cdh_vd_cdh5_maven_repo.html
>
> Nobuhiro Sue(PetaData inc.)
>
> 2015-01-30 8:33 GMT+09:00 Mohit Jaggi <mo...@gmail.com>:
>> Hi All,
>> I noticed in pom.xml that there is no entry for Hadoop 2.5. Has anyone tried Spark with 2.5.0-cdh5.2.1? Will replicating the 2.4 entry be sufficient to make this work?
>>
>> Mohit.
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: spark with cdh 5.2.1
Posted by Nobuhiro Sue <ns...@e-mail.ne.jp>.
Mohit,
I'm using spark modules provided by Cloudera repos, it works fine.
Please add Cloudera maven repo, and specify dependencies with CDH
version, like "spark-core_2.10-1.1.0-cdh5.2.1".
To add Cloudera maven repo, see:
http://www.cloudera.com/content/cloudera/en/documentation/core/v5-2-x/topics/cdh_vd_cdh5_maven_repo.html
Nobuhiro Sue(PetaData inc.)
2015-01-30 8:33 GMT+09:00 Mohit Jaggi <mo...@gmail.com>:
> Hi All,
> I noticed in pom.xml that there is no entry for Hadoop 2.5. Has anyone tried Spark with 2.5.0-cdh5.2.1? Will replicating the 2.4 entry be sufficient to make this work?
>
> Mohit.
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: spark with cdh 5.2.1
Posted by Mohit Jaggi <mo...@gmail.com>.
yep...it was unnecessary to create a 2.5 profile. i struggled a bit because
it wasn't clear that i *need* to select a profile using -P option. i didn't
have to do that for earlier hadoop versions.
On Fri, Jan 30, 2015 at 12:11 AM, Sean Owen <so...@cloudera.com> wrote:
> There is no need for a 2.5 profile. The hadoop-2.4 profile is for
> Hadoop 2.4 and beyond. You can set the particular version you want
> with -Dhadoop.version=
>
> You do not need to make any new profile to compile vs 2.5.0-cdh5.2.1.
> Again, the hadoop-2.4 profile is what you need.
>
> On Thu, Jan 29, 2015 at 11:33 PM, Mohit Jaggi <mo...@gmail.com>
> wrote:
> > Hi All,
> > I noticed in pom.xml that there is no entry for Hadoop 2.5. Has anyone
> tried Spark with 2.5.0-cdh5.2.1? Will replicating the 2.4 entry be
> sufficient to make this work?
> >
> > Mohit.
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> > For additional commands, e-mail: user-help@spark.apache.org
> >
>
Re: spark with cdh 5.2.1
Posted by Sean Owen <so...@cloudera.com>.
There is no need for a 2.5 profile. The hadoop-2.4 profile is for
Hadoop 2.4 and beyond. You can set the particular version you want
with -Dhadoop.version=
You do not need to make any new profile to compile vs 2.5.0-cdh5.2.1.
Again, the hadoop-2.4 profile is what you need.
On Thu, Jan 29, 2015 at 11:33 PM, Mohit Jaggi <mo...@gmail.com> wrote:
> Hi All,
> I noticed in pom.xml that there is no entry for Hadoop 2.5. Has anyone tried Spark with 2.5.0-cdh5.2.1? Will replicating the 2.4 entry be sufficient to make this work?
>
> Mohit.
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org