You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mesos.apache.org by Gary Malouf <ma...@gmail.com> on 2014/03/26 00:57:43 UTC
Mesos + Hadoop 2.2 for protobuf 2.5 dependency
For various reasons, our team needs to keep all of our projects on the same
protobuf version. We've now hit a point where we need to upgrade protobuf
from 2.4.1 to 2.5.0 across the board in our projects and dependent
platforms.
*Current stack*: Mesos 0.15, Chronos, CDH 4.2.1-MRV1, Spark
0.9-pre-scala-2.10 build off master
*Ideal stack after upgrade*: Mesos 0.17, Chronos, CDH5 beta2, Spark 0.9.1
(hadoop 2.2 build)
>From what we understand, we need a dependency on Hadoop 2.2 to get the
necessary protobuf upgrade. From reading Cloudera's documentation and
multiple google searches, it is not clear to me how we can construct the
stack to continue to work.
Has anyone else requested info on getting this combination to work? From
others we've spoken to, they've basically said that we'll be forced to use
Yarn for Hadoop support in the very near future anyway so we should switch.
Since we colocate Spark with out HDFS nodes, it's hard to see how we would
run both Yarn and Mesos on the same servers.
Re: Mesos + Hadoop 2.2 for protobuf 2.5 dependency
Posted by Gary Malouf <ma...@gmail.com>.
I put this on the wrong mailing list - my apologies.
On Tue, Mar 25, 2014 at 7:57 PM, Gary Malouf <ma...@gmail.com> wrote:
> For various reasons, our team needs to keep all of our projects on the
> same protobuf version. We've now hit a point where we need to upgrade
> protobuf from 2.4.1 to 2.5.0 across the board in our projects and dependent
> platforms.
>
> *Current stack*: Mesos 0.15, Chronos, CDH 4.2.1-MRV1, Spark
> 0.9-pre-scala-2.10 build off master
>
> *Ideal stack after upgrade*: Mesos 0.17, Chronos, CDH5 beta2, Spark 0.9.1
> (hadoop 2.2 build)
>
> From what we understand, we need a dependency on Hadoop 2.2 to get the
> necessary protobuf upgrade. From reading Cloudera's documentation and
> multiple google searches, it is not clear to me how we can construct the
> stack to continue to work.
>
> Has anyone else requested info on getting this combination to work? From
> others we've spoken to, they've basically said that we'll be forced to use
> Yarn for Hadoop support in the very near future anyway so we should switch.
> Since we colocate Spark with out HDFS nodes, it's hard to see how we would
> run both Yarn and Mesos on the same servers.
>
Re: Mesos + Hadoop 2.2 for protobuf 2.5 dependency
Posted by Gary Malouf <ma...@gmail.com>.
Thanks Bernardo, I will update it on
https://github.com/apache/spark/blob/71d4ed271bcbddb154643bd44297ed77190e75cf/project/SparkBuild.scala
in our fork.
On Tue, Mar 25, 2014 at 9:49 PM, Bernardo Gomez Palacio <
bernardo.gomezpalacio@gmail.com> wrote:
> I built Spark for hadoop 2.2 with Mesos, make sure that the YARN env var is
> disabled. I updated the Mesos dep to point to a Mesos version that uses
> protobufs 2.5.0. That should work with CDH5.
>
>
>
> On Tuesday, March 25, 2014, Gary Malouf <ma...@gmail.com> wrote:
>
> > Did you build the spark hadoop 2.2 yarn version or just the regular one?
> > There is no CDH5 build available. We used spark 0.8.1 built with
> > cdh4.2.1-mrv1 before.
> >
> > We will give it a shot tomorrow and see what happens :)
> > On Mar 25, 2014 9:01 PM, "Bernardo Gomez Palacio" <
> > bernardo.gomezpalacio@gmail.com <javascript:;>> wrote:
> >
> > > In that setup I am running Spark on Mesos not YARN.
> > >
> > > On Tue, Mar 25, 2014 at 5:55 PM, Gary Malouf <malouf.gary@gmail.com
> <javascript:;>
> > >
> > > wrote:
> > > > Thank you for the feedback. Are you running Yarn as a part of this
> > > setup?
> > > > Seems like that would collide with Mesos. We have a job or 2 written
> > in
> > > > MRV1 that from what I understand can still be run with 2.2 somehow...
> > > > On Mar 25, 2014 8:49 PM, "Bernardo Gomez Palacio" <
> > > > bernardo.gomezpalacio@gmail.com <javascript:;>> wrote:
> > > >
> > > >> Hi Gary.
> > > >> I am currently using
> > > >> Mesos 0.16.0 build for Protobufs 2.5.0
> > > >> (https://github.com/Guavus/mesos/tree/releases/0.16.0-HDP2x)
> > > >> Hadoop version 2.2
> > > >> Spark 0.9.0 (
> > > >>
> > https://github.com/guavus/incubator-spark/tree/guavus/branch-0.9-hdp2.2
> > > )
> > > >>
> > > >> Things to know.
> > > >> Hadoop 1.0.4 uses protobufs 2.4.1
> > > >> Hadoop 2.2 and 2.3 uses protobufs 2.5.0
> > > >> Mesos < 0.17.0 uses protobufs 2.4.1
> > > >> Mesos >= 0.17.0 uses protobufs 2.5.0
> > > >>
> > > >> If you need Mesos 0.15.0 on protobufs 2.5.0 you could try to
> backport
> > > >> https://github.com/Guavus/mesos/tree/releases/0.16.0-HDP2x but if
> you
> > > >> keep CDH4 with protobufs 2.4.1 you will run into trouble.
> > > >>
> > > >> Let me know if this help.
> > > >> Bernardo.
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> On Tue, Mar 25, 2014 at 4:57 PM, Gary Malouf <malouf.gary@gmail.com
> <javascript:;>
> > >
> > > >> wrote:
> > > >> > For various reasons, our team needs to keep all of our projects on
> > the
> > > >> same
> > > >> > protobuf version. We've now hit a point where we need to upgrade
> > > >> protobuf
> > > >> > from 2.4.1 to 2.5.0 across the board in our projects and dependent
> > > >> > platforms.
> > > >> >
> > > >> > *Current stack*: Mesos 0.15, Chronos, CDH 4.2.1-MRV1, Spark
> > > >> > 0.9-pre-scala-2.10 build off master
> > > >> >
> > > >> > *Ideal stack after upgrade*: Mesos 0.17, Chronos, CDH5 beta2,
> Spark
> > > 0.9.1
> > > >> > (hadoop 2.2 build)
> > > >> >
> > > >> > From what we understand, we need a dependency on Hadoop 2.2 to get
> > the
> > > >> > necessary protobuf upgrade. From reading Cloudera's documentation
> > and
> > > >> > multiple google searches, it is not clear to me how we can
> construct
> > > the
> > > >> > stack to continue to work.
> > > >> >
> > > >> > Has anyone else requested info on getting this combination to
> work?
> > > From
> > > >> > others we've spoken to, they've basically said that we'll be
> forced
> > to
> > > >> use
> > > >> > Yarn for Hadoop support in the very near future anyway so we
> should
> > > >> switch.
> > > >> > Since we colocate Spark with out HDFS nodes, it's hard to see how
> > we
> > > >> would
> > > >> > run both Yarn and Mesos on the same servers.
> > > >>
> > >
> >
>
Re: Mesos + Hadoop 2.2 for protobuf 2.5 dependency
Posted by Bernardo Gomez Palacio <be...@gmail.com>.
I built Spark for hadoop 2.2 with Mesos, make sure that the YARN env var is
disabled. I updated the Mesos dep to point to a Mesos version that uses
protobufs 2.5.0. That should work with CDH5.
On Tuesday, March 25, 2014, Gary Malouf <ma...@gmail.com> wrote:
> Did you build the spark hadoop 2.2 yarn version or just the regular one?
> There is no CDH5 build available. We used spark 0.8.1 built with
> cdh4.2.1-mrv1 before.
>
> We will give it a shot tomorrow and see what happens :)
> On Mar 25, 2014 9:01 PM, "Bernardo Gomez Palacio" <
> bernardo.gomezpalacio@gmail.com <javascript:;>> wrote:
>
> > In that setup I am running Spark on Mesos not YARN.
> >
> > On Tue, Mar 25, 2014 at 5:55 PM, Gary Malouf <malouf.gary@gmail.com<javascript:;>
> >
> > wrote:
> > > Thank you for the feedback. Are you running Yarn as a part of this
> > setup?
> > > Seems like that would collide with Mesos. We have a job or 2 written
> in
> > > MRV1 that from what I understand can still be run with 2.2 somehow...
> > > On Mar 25, 2014 8:49 PM, "Bernardo Gomez Palacio" <
> > > bernardo.gomezpalacio@gmail.com <javascript:;>> wrote:
> > >
> > >> Hi Gary.
> > >> I am currently using
> > >> Mesos 0.16.0 build for Protobufs 2.5.0
> > >> (https://github.com/Guavus/mesos/tree/releases/0.16.0-HDP2x)
> > >> Hadoop version 2.2
> > >> Spark 0.9.0 (
> > >>
> https://github.com/guavus/incubator-spark/tree/guavus/branch-0.9-hdp2.2
> > )
> > >>
> > >> Things to know.
> > >> Hadoop 1.0.4 uses protobufs 2.4.1
> > >> Hadoop 2.2 and 2.3 uses protobufs 2.5.0
> > >> Mesos < 0.17.0 uses protobufs 2.4.1
> > >> Mesos >= 0.17.0 uses protobufs 2.5.0
> > >>
> > >> If you need Mesos 0.15.0 on protobufs 2.5.0 you could try to backport
> > >> https://github.com/Guavus/mesos/tree/releases/0.16.0-HDP2x but if you
> > >> keep CDH4 with protobufs 2.4.1 you will run into trouble.
> > >>
> > >> Let me know if this help.
> > >> Bernardo.
> > >>
> > >>
> > >>
> > >>
> > >> On Tue, Mar 25, 2014 at 4:57 PM, Gary Malouf <malouf.gary@gmail.com<javascript:;>
> >
> > >> wrote:
> > >> > For various reasons, our team needs to keep all of our projects on
> the
> > >> same
> > >> > protobuf version. We've now hit a point where we need to upgrade
> > >> protobuf
> > >> > from 2.4.1 to 2.5.0 across the board in our projects and dependent
> > >> > platforms.
> > >> >
> > >> > *Current stack*: Mesos 0.15, Chronos, CDH 4.2.1-MRV1, Spark
> > >> > 0.9-pre-scala-2.10 build off master
> > >> >
> > >> > *Ideal stack after upgrade*: Mesos 0.17, Chronos, CDH5 beta2, Spark
> > 0.9.1
> > >> > (hadoop 2.2 build)
> > >> >
> > >> > From what we understand, we need a dependency on Hadoop 2.2 to get
> the
> > >> > necessary protobuf upgrade. From reading Cloudera's documentation
> and
> > >> > multiple google searches, it is not clear to me how we can construct
> > the
> > >> > stack to continue to work.
> > >> >
> > >> > Has anyone else requested info on getting this combination to work?
> > From
> > >> > others we've spoken to, they've basically said that we'll be forced
> to
> > >> use
> > >> > Yarn for Hadoop support in the very near future anyway so we should
> > >> switch.
> > >> > Since we colocate Spark with out HDFS nodes, it's hard to see how
> we
> > >> would
> > >> > run both Yarn and Mesos on the same servers.
> > >>
> >
>
Re: Mesos + Hadoop 2.2 for protobuf 2.5 dependency
Posted by Gary Malouf <ma...@gmail.com>.
Did you build the spark hadoop 2.2 yarn version or just the regular one?
There is no CDH5 build available. We used spark 0.8.1 built with
cdh4.2.1-mrv1 before.
We will give it a shot tomorrow and see what happens :)
On Mar 25, 2014 9:01 PM, "Bernardo Gomez Palacio" <
bernardo.gomezpalacio@gmail.com> wrote:
> In that setup I am running Spark on Mesos not YARN.
>
> On Tue, Mar 25, 2014 at 5:55 PM, Gary Malouf <ma...@gmail.com>
> wrote:
> > Thank you for the feedback. Are you running Yarn as a part of this
> setup?
> > Seems like that would collide with Mesos. We have a job or 2 written in
> > MRV1 that from what I understand can still be run with 2.2 somehow...
> > On Mar 25, 2014 8:49 PM, "Bernardo Gomez Palacio" <
> > bernardo.gomezpalacio@gmail.com> wrote:
> >
> >> Hi Gary.
> >> I am currently using
> >> Mesos 0.16.0 build for Protobufs 2.5.0
> >> (https://github.com/Guavus/mesos/tree/releases/0.16.0-HDP2x)
> >> Hadoop version 2.2
> >> Spark 0.9.0 (
> >> https://github.com/guavus/incubator-spark/tree/guavus/branch-0.9-hdp2.2
> )
> >>
> >> Things to know.
> >> Hadoop 1.0.4 uses protobufs 2.4.1
> >> Hadoop 2.2 and 2.3 uses protobufs 2.5.0
> >> Mesos < 0.17.0 uses protobufs 2.4.1
> >> Mesos >= 0.17.0 uses protobufs 2.5.0
> >>
> >> If you need Mesos 0.15.0 on protobufs 2.5.0 you could try to backport
> >> https://github.com/Guavus/mesos/tree/releases/0.16.0-HDP2x but if you
> >> keep CDH4 with protobufs 2.4.1 you will run into trouble.
> >>
> >> Let me know if this help.
> >> Bernardo.
> >>
> >>
> >>
> >>
> >> On Tue, Mar 25, 2014 at 4:57 PM, Gary Malouf <ma...@gmail.com>
> >> wrote:
> >> > For various reasons, our team needs to keep all of our projects on the
> >> same
> >> > protobuf version. We've now hit a point where we need to upgrade
> >> protobuf
> >> > from 2.4.1 to 2.5.0 across the board in our projects and dependent
> >> > platforms.
> >> >
> >> > *Current stack*: Mesos 0.15, Chronos, CDH 4.2.1-MRV1, Spark
> >> > 0.9-pre-scala-2.10 build off master
> >> >
> >> > *Ideal stack after upgrade*: Mesos 0.17, Chronos, CDH5 beta2, Spark
> 0.9.1
> >> > (hadoop 2.2 build)
> >> >
> >> > From what we understand, we need a dependency on Hadoop 2.2 to get the
> >> > necessary protobuf upgrade. From reading Cloudera's documentation and
> >> > multiple google searches, it is not clear to me how we can construct
> the
> >> > stack to continue to work.
> >> >
> >> > Has anyone else requested info on getting this combination to work?
> From
> >> > others we've spoken to, they've basically said that we'll be forced to
> >> use
> >> > Yarn for Hadoop support in the very near future anyway so we should
> >> switch.
> >> > Since we colocate Spark with out HDFS nodes, it's hard to see how we
> >> would
> >> > run both Yarn and Mesos on the same servers.
> >>
>
Re: Mesos + Hadoop 2.2 for protobuf 2.5 dependency
Posted by Bernardo Gomez Palacio <be...@gmail.com>.
In that setup I am running Spark on Mesos not YARN.
On Tue, Mar 25, 2014 at 5:55 PM, Gary Malouf <ma...@gmail.com> wrote:
> Thank you for the feedback. Are you running Yarn as a part of this setup?
> Seems like that would collide with Mesos. We have a job or 2 written in
> MRV1 that from what I understand can still be run with 2.2 somehow...
> On Mar 25, 2014 8:49 PM, "Bernardo Gomez Palacio" <
> bernardo.gomezpalacio@gmail.com> wrote:
>
>> Hi Gary.
>> I am currently using
>> Mesos 0.16.0 build for Protobufs 2.5.0
>> (https://github.com/Guavus/mesos/tree/releases/0.16.0-HDP2x)
>> Hadoop version 2.2
>> Spark 0.9.0 (
>> https://github.com/guavus/incubator-spark/tree/guavus/branch-0.9-hdp2.2)
>>
>> Things to know.
>> Hadoop 1.0.4 uses protobufs 2.4.1
>> Hadoop 2.2 and 2.3 uses protobufs 2.5.0
>> Mesos < 0.17.0 uses protobufs 2.4.1
>> Mesos >= 0.17.0 uses protobufs 2.5.0
>>
>> If you need Mesos 0.15.0 on protobufs 2.5.0 you could try to backport
>> https://github.com/Guavus/mesos/tree/releases/0.16.0-HDP2x but if you
>> keep CDH4 with protobufs 2.4.1 you will run into trouble.
>>
>> Let me know if this help.
>> Bernardo.
>>
>>
>>
>>
>> On Tue, Mar 25, 2014 at 4:57 PM, Gary Malouf <ma...@gmail.com>
>> wrote:
>> > For various reasons, our team needs to keep all of our projects on the
>> same
>> > protobuf version. We've now hit a point where we need to upgrade
>> protobuf
>> > from 2.4.1 to 2.5.0 across the board in our projects and dependent
>> > platforms.
>> >
>> > *Current stack*: Mesos 0.15, Chronos, CDH 4.2.1-MRV1, Spark
>> > 0.9-pre-scala-2.10 build off master
>> >
>> > *Ideal stack after upgrade*: Mesos 0.17, Chronos, CDH5 beta2, Spark 0.9.1
>> > (hadoop 2.2 build)
>> >
>> > From what we understand, we need a dependency on Hadoop 2.2 to get the
>> > necessary protobuf upgrade. From reading Cloudera's documentation and
>> > multiple google searches, it is not clear to me how we can construct the
>> > stack to continue to work.
>> >
>> > Has anyone else requested info on getting this combination to work? From
>> > others we've spoken to, they've basically said that we'll be forced to
>> use
>> > Yarn for Hadoop support in the very near future anyway so we should
>> switch.
>> > Since we colocate Spark with out HDFS nodes, it's hard to see how we
>> would
>> > run both Yarn and Mesos on the same servers.
>>
Re: Mesos + Hadoop 2.2 for protobuf 2.5 dependency
Posted by Gary Malouf <ma...@gmail.com>.
Thank you for the feedback. Are you running Yarn as a part of this setup?
Seems like that would collide with Mesos. We have a job or 2 written in
MRV1 that from what I understand can still be run with 2.2 somehow...
On Mar 25, 2014 8:49 PM, "Bernardo Gomez Palacio" <
bernardo.gomezpalacio@gmail.com> wrote:
> Hi Gary.
> I am currently using
> Mesos 0.16.0 build for Protobufs 2.5.0
> (https://github.com/Guavus/mesos/tree/releases/0.16.0-HDP2x)
> Hadoop version 2.2
> Spark 0.9.0 (
> https://github.com/guavus/incubator-spark/tree/guavus/branch-0.9-hdp2.2)
>
> Things to know.
> Hadoop 1.0.4 uses protobufs 2.4.1
> Hadoop 2.2 and 2.3 uses protobufs 2.5.0
> Mesos < 0.17.0 uses protobufs 2.4.1
> Mesos >= 0.17.0 uses protobufs 2.5.0
>
> If you need Mesos 0.15.0 on protobufs 2.5.0 you could try to backport
> https://github.com/Guavus/mesos/tree/releases/0.16.0-HDP2x but if you
> keep CDH4 with protobufs 2.4.1 you will run into trouble.
>
> Let me know if this help.
> Bernardo.
>
>
>
>
> On Tue, Mar 25, 2014 at 4:57 PM, Gary Malouf <ma...@gmail.com>
> wrote:
> > For various reasons, our team needs to keep all of our projects on the
> same
> > protobuf version. We've now hit a point where we need to upgrade
> protobuf
> > from 2.4.1 to 2.5.0 across the board in our projects and dependent
> > platforms.
> >
> > *Current stack*: Mesos 0.15, Chronos, CDH 4.2.1-MRV1, Spark
> > 0.9-pre-scala-2.10 build off master
> >
> > *Ideal stack after upgrade*: Mesos 0.17, Chronos, CDH5 beta2, Spark 0.9.1
> > (hadoop 2.2 build)
> >
> > From what we understand, we need a dependency on Hadoop 2.2 to get the
> > necessary protobuf upgrade. From reading Cloudera's documentation and
> > multiple google searches, it is not clear to me how we can construct the
> > stack to continue to work.
> >
> > Has anyone else requested info on getting this combination to work? From
> > others we've spoken to, they've basically said that we'll be forced to
> use
> > Yarn for Hadoop support in the very near future anyway so we should
> switch.
> > Since we colocate Spark with out HDFS nodes, it's hard to see how we
> would
> > run both Yarn and Mesos on the same servers.
>
Re: Mesos + Hadoop 2.2 for protobuf 2.5 dependency
Posted by Bernardo Gomez Palacio <be...@gmail.com>.
Hi Gary.
I am currently using
Mesos 0.16.0 build for Protobufs 2.5.0
(https://github.com/Guavus/mesos/tree/releases/0.16.0-HDP2x)
Hadoop version 2.2
Spark 0.9.0 (https://github.com/guavus/incubator-spark/tree/guavus/branch-0.9-hdp2.2)
Things to know.
Hadoop 1.0.4 uses protobufs 2.4.1
Hadoop 2.2 and 2.3 uses protobufs 2.5.0
Mesos < 0.17.0 uses protobufs 2.4.1
Mesos >= 0.17.0 uses protobufs 2.5.0
If you need Mesos 0.15.0 on protobufs 2.5.0 you could try to backport
https://github.com/Guavus/mesos/tree/releases/0.16.0-HDP2x but if you
keep CDH4 with protobufs 2.4.1 you will run into trouble.
Let me know if this help.
Bernardo.
On Tue, Mar 25, 2014 at 4:57 PM, Gary Malouf <ma...@gmail.com> wrote:
> For various reasons, our team needs to keep all of our projects on the same
> protobuf version. We've now hit a point where we need to upgrade protobuf
> from 2.4.1 to 2.5.0 across the board in our projects and dependent
> platforms.
>
> *Current stack*: Mesos 0.15, Chronos, CDH 4.2.1-MRV1, Spark
> 0.9-pre-scala-2.10 build off master
>
> *Ideal stack after upgrade*: Mesos 0.17, Chronos, CDH5 beta2, Spark 0.9.1
> (hadoop 2.2 build)
>
> From what we understand, we need a dependency on Hadoop 2.2 to get the
> necessary protobuf upgrade. From reading Cloudera's documentation and
> multiple google searches, it is not clear to me how we can construct the
> stack to continue to work.
>
> Has anyone else requested info on getting this combination to work? From
> others we've spoken to, they've basically said that we'll be forced to use
> Yarn for Hadoop support in the very near future anyway so we should switch.
> Since we colocate Spark with out HDFS nodes, it's hard to see how we would
> run both Yarn and Mesos on the same servers.