Building Spark 0.9.x for CDH5 with mrv1 installation (Protobuf 2.5 upgrade)

[Gary Malouf <ma...@gmail.com>]

Today, our cluster setup is as follows:

Mesos 0.15,
CDH 4.2.1-MRV1,
Spark 0.9-pre-scala-2.10 off master build targeted at appropriate CDH4
version


We are looking to upgrade all of these in order to get protobuf 2.5 working
properly.  The question is, which 'Hadoop version build' of Spark 0.9 is
compatible with the HDFS from Hadoop 2.2 and Cloudera's CDH5 MRV1
installation?  Is there one?

Re: Building Spark 0.9.x for CDH5 with mrv1 installation (Protobuf 2.5 upgrade)

[Patrick Wendell <pw...@gmail.com>]

I'm not sure exactly how your cluster is configured. But as far as I can
tell Cloudera's MR1 CDH5 dependencies are against Hadoop 2.3. I'd just find
the exact CDH version you have and link against the `mr1` version of their
published dependencies in that version.

So I think you wan't "2.3.0-mr1-cdh5.0.0"
https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/hadoop/hadoop-client/2.3.0-mr1-cdh5.0.0/

The full list of Cloudera versions is here:
https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/hadoop/hadoop-client/



On Tue, Mar 25, 2014 at 6:42 PM, Gary Malouf <ma...@gmail.com> wrote:

> Today, our cluster setup is as follows:
>
> Mesos 0.15,
> CDH 4.2.1-MRV1,
> Spark 0.9-pre-scala-2.10 off master build targeted at appropriate CDH4
> version
>
>
> We are looking to upgrade all of these in order to get protobuf 2.5
> working properly.  The question is, which 'Hadoop version build' of Spark
> 0.9 is compatible with the HDFS from Hadoop 2.2 and Cloudera's CDH5 MRV1
> installation?  Is there one?
>