You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Ankur Chauhan <an...@malloc64.com> on 2015/05/15 08:30:31 UTC
Spark on Mesos vs Yarn
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi,
This is both a survey type as well as a roadmap query question. It
seems like of the cluster options to run spark (i.e. via YARN and
Mesos), YARN seems to be getting a lot more attention and patches when
compared to Mesos.
Would it be correct to assume that spark on mesos is more or less a
dead or something like a maintenance-only feature and YARN is the
recommended way to go?
What is the roadmap for spark on mesos? and what is the roadmap for
spark on yarn. I like mesos so as much as I would like to see it
thrive I don't think spark community is active (or maybe it just
appears that way).
Another more community oriented question: what do most people use to
run spark in production or more-than-POC products? Why did you make
that decision?
There was a similar post form early 2014 where Metei answered that
mesos and yarn were equally important, but has this changed as spark
has now reached almost 1.4.0 stage?
- -- Ankur Chauhan
-----BEGIN PGP SIGNATURE-----
iQEcBAEBAgAGBQJVVZKGAAoJEOSJAMhvLp3L0vEIAI4edLB2rMGk+OTI4WujxX6k
Ud5NyFUpaQ8WDjOhwcWB9RK5EoM7X3wGzRcGza1HLVnvdSUBG8Ltabt47GsP2lo0
7H9y2GluUZg/RJXbN0Ehp6moWjAU1W/55POD3t87qeUdydUJVbgDYA/KovNa6i8s
Z/e8mfvOrFSJyuJi8KW2KcfOmB1i8VZH7b/zZqtfJKNGo/0dac/gez19vVPaXPa4
WNUN8dHcp0yiZnZ0PUTYNLhI58BXBCSmkEl2Ex7X3NBUGUgJ5HGHn6dpqqNhGvf3
yPw0B0q93NcExK/E4/I75nn4vh5wKLPLWT8U5btphmc7S6h8gWFMEJRHQCdtaUk=
=uYXZ
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark on Mesos vs Yarn
Posted by Timothy Chen <tn...@gmail.com>.
Hi Nik,
Bharath is mostly referring to Spark commiters in this thread.
Tim
On Tue, Jun 9, 2015 at 9:51 PM, Niklas Nielsen <ni...@mesosphere.io> wrote:
> Hi Bharath (and rest of Spark dev list!),
>
> Just a small shout out: I am a Apache Mesos Committer and would love to help
> out with anything you need to get this going.
>
> Cheers,
> Nik
>
> On 9 June 2015 at 21:10, Bharath Ravi Kumar <re...@gmail.com> wrote:
>>
>> All,
>>
>> Despite the common origin of spark & mesos, the stability and adoption of
>> mesos, and the age of the spark-mesos binding, I find the mesos support less
>> mature, with fundamental shortcomings (like framework auth) remaining
>> unresolved. If there's shortage of developer time, I'd be glad to
>> contribute, but it's unclear if the committer group has sufficient time (and
>> priority) to take the mesos support forward. While it has been stated often
>> that support for mesos & yarn are equally important, that doesn't seem to
>> translate to visible progress. I'd be glad if my observation is incorrect as
>> I seek better focus and long term commitment on the mesos support.
>> As for the specific issue (6284), I'm happy to build, testing & eventually
>> deploy the patch in our production cluster, but I'd rather see it becoming
>> mainstream.
>> Thanks for your consideration.
>>
>> -Bharath
>>
>>
>> On Thu, May 28, 2015 at 9:18 AM, Bharath Ravi Kumar <re...@gmail.com>
>> wrote:
>>>
>>> A follow up : considering that spark on mesos is indeed important to
>>> databricks, its partners and the community, fundamental issues like
>>> spark-6284 shouldn't be languishing for this long. A mesos cluster hosting
>>> diverse (i.e.multi-tenant) workloads is a common scenario in production for
>>> serious users. The ability to auth a framework & assign roles would be a
>>> fairly basic ask, one would imagine. Is the lack of time / effort a
>>> constraint? If so, I'd be glad to help (as mentioned in the jira).
>>>
>>> On Fri, May 15, 2015 at 5:29 PM, Iulian Dragoș
>>> <iu...@typesafe.com> wrote:
>>>>
>>>> Hi Ankur,
>>>>
>>>> Just to add a thought to Tim's excellent answer, Spark on Mesos is very
>>>> important to us and is the recommended deployment for our customers as
>>>> Typesafe.
>>>>
>>>> Thanks for pointing to your PR, I see Tim already went through a round
>>>> of reviews. It seems very useful, I'll give it a try as well.
>>>>
>>>> thanks,
>>>> iulian
>>>>
>>>>
>>>>
>>>> On Fri, May 15, 2015 at 9:53 AM, Ankur Chauhan <an...@malloc64.com>
>>>> wrote:
>>>>>
>>>>> -----BEGIN PGP SIGNED MESSAGE-----
>>>>> Hash: SHA1
>>>>>
>>>>> Hi Tim,
>>>>>
>>>>> Thanks for such a detailed email. I am excited to hear about the new
>>>>> features, I had a pull request going for adding "attribute based
>>>>> filtering in the mesos scheduler" but it hasn't received much love -
>>>>> https://github.com/apache/spark/pull/5563 . I am a fan of
>>>>> mesos/marathon/mesosphere and spark ecosystems and trying to push
>>>>> adoption at my workplace.
>>>>>
>>>>> It would love to see documentation, tutorials (anything actually) that
>>>>> would make mesos + spark a better and more fleshed out solution. Would
>>>>> it be possible for you to share some links to the JIRA and pull
>>>>> requests so that I can keep track on the progress/features.
>>>>>
>>>>> Again, thanks for replying.
>>>>>
>>>>> - -- Ankur Chauhan
>>>>>
>>>>> On 15/05/2015 00:39, Tim Chen wrote:
>>>>> > Hi Ankur,
>>>>> >
>>>>> > This is a great question as I've heard similar concerns about Spark
>>>>> > on Mesos.
>>>>> >
>>>>> > At the time when I started to contribute to Spark on Mesos approx
>>>>> > half year ago, the Mesos scheduler and related code hasn't really
>>>>> > got much attention from anyone and it was pretty much in
>>>>> > maintenance mode.
>>>>> >
>>>>> > As a Mesos PMC that is really interested in Spark I started to
>>>>> > refactor and check out different JIRAs and PRs around the Mesos
>>>>> > scheduler, and after that started to fix various bugs in Spark,
>>>>> > added documentation and also in fix related Mesos issues as well.
>>>>> >
>>>>> > Just recently for 1.4 we've merged in Cluster mode and Docker
>>>>> > support, and there are also pending PRs around framework
>>>>> > authentication, multi-role support, dynamic allocation, more finer
>>>>> > tuned coarse grain mode scheduling configurations, etc.
>>>>> >
>>>>> > And finally just want to mention that Mesosphere and Typesafe is
>>>>> > collaborating to bring a certified distribution
>>>>> >
>>>>> > (https://databricks.com/spark/certification/certified-spark-distributi
>>>>> on)
>>>>> > of Spark on Mesos and DCOS, and we will be pouring resources into
>>>>> > not just maintain Spark on Mesos but drive more features into the
>>>>> > Mesos scheduler and also in Mesos so stateful services can leverage
>>>>> > new APIs and features to make better scheduling decisions and
>>>>> > optimizations.
>>>>> >
>>>>> > I don't have a solidified roadmap to share yet, but we will be
>>>>> > discussing this and hopefully can share with the community soon.
>>>>> >
>>>>> > In summary Spark on Mesos is not dead or in maintenance mode, and
>>>>> > look forward to see a lot more changes from us and the community.
>>>>> >
>>>>> > Tim
>>>>> >
>>>>> > On Thu, May 14, 2015 at 11:30 PM, Ankur Chauhan
>>>>> > <ankur@malloc64.com <ma...@malloc64.com>> wrote:
>>>>> >
>>>>> > Hi,
>>>>> >
>>>>> > This is both a survey type as well as a roadmap query question. It
>>>>> > seems like of the cluster options to run spark (i.e. via YARN and
>>>>> > Mesos), YARN seems to be getting a lot more attention and patches
>>>>> > when compared to Mesos.
>>>>> >
>>>>> > Would it be correct to assume that spark on mesos is more or less
>>>>> > a dead or something like a maintenance-only feature and YARN is
>>>>> > the recommended way to go?
>>>>> >
>>>>> > What is the roadmap for spark on mesos? and what is the roadmap
>>>>> > for spark on yarn. I like mesos so as much as I would like to see
>>>>> > it thrive I don't think spark community is active (or maybe it
>>>>> > just appears that way).
>>>>> >
>>>>> > Another more community oriented question: what do most people use
>>>>> > to run spark in production or more-than-POC products? Why did you
>>>>> > make that decision?
>>>>> >
>>>>> > There was a similar post form early 2014 where Metei answered that
>>>>> > mesos and yarn were equally important, but has this changed as
>>>>> > spark has now reached almost 1.4.0 stage?
>>>>> >
>>>>> > -- Ankur Chauhan
>>>>> >
>>>>> > ---------------------------------------------------------------------
>>>>> >
>>>>> >
>>>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>>> > <ma...@spark.apache.org> For additional commands,
>>>>> > e-mail: user-help@spark.apache.org
>>>>> > <ma...@spark.apache.org>
>>>>> >
>>>>> >
>>>>> -----BEGIN PGP SIGNATURE-----
>>>>>
>>>>> iQEcBAEBAgAGBQJVVaXvAAoJEOSJAMhvLp3LzkIH/RLPVUoOcFo0Gij1NpZDszN4
>>>>> xWvesbOZszuqD8H1Dhyndz4RQKnrodyCE+NycFB+utd9epmuyGemmHpTnq18Gek6
>>>>> PR5jqmgza94dOy0rfuIVvba14ALZb4tO9SgkjyGujrpMlFYvxTjBYdYCAjfEOTx7
>>>>> A/vqaCzPSBRBmO8gWx07GWa4zI70qBSZ9KnV7dgtqfUUgPKdF4NnMZWRJjTO9Bp8
>>>>> tTmWMldqYPqI95wdeeqTGMH0XT6JAKAiCskf62DGadRBsOshrhmh5mAQzUFwoTpA
>>>>> w4uZ+qMrTsblBvOf9z++v0eY8VBiQpOyXfOBiYiCNRtSsGa0KvqwgF1S/yLeRs0=
>>>>> =4Aax
>>>>> -----END PGP SIGNATURE-----
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>>> For additional commands, e-mail: user-help@spark.apache.org
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> --
>>>> Iulian Dragos
>>>>
>>>> ------
>>>> Reactive Apps on the JVM
>>>> www.typesafe.com
>>>>
>>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: Spark on Mesos vs Yarn
Posted by Niklas Nielsen <ni...@mesosphere.io>.
Hi Bharath (and rest of Spark dev list!),
Just a small shout out: I am a Apache Mesos Committer and would love to
help out with anything you need to get this going.
Cheers,
Nik
On 9 June 2015 at 21:10, Bharath Ravi Kumar <re...@gmail.com> wrote:
> All,
>
> Despite the common origin of spark & mesos, the stability and adoption of
> mesos, and the age of the spark-mesos binding, I find the mesos support
> less mature, with fundamental shortcomings (like framework auth
> <https://issues.apache.org/jira/browse/SPARK-6284>) remaining unresolved.
> If there's shortage of developer time, I'd be glad to contribute, but it's
> unclear if the committer group has sufficient time (and priority) to take
> the mesos support forward. While it has been stated often that support for
> mesos & yarn are equally important, that doesn't seem to translate to
> visible progress. I'd be glad if my observation is incorrect as I seek
> better focus and long term commitment on the mesos support.
> As for the specific issue (6284), I'm happy to build, testing & eventually
> deploy the patch in our production cluster, but I'd rather see it becoming
> mainstream.
> Thanks for your consideration.
>
> -Bharath
>
>
> On Thu, May 28, 2015 at 9:18 AM, Bharath Ravi Kumar <re...@gmail.com>
> wrote:
>
>> A follow up : considering that spark on mesos is indeed important to
>> databricks, its partners and the community, fundamental issues like
>> spark-6284 shouldn't be languishing for this long. A mesos cluster hosting
>> diverse (i.e.multi-tenant) workloads is a common scenario in production
>> for serious users. The ability to auth a framework & assign roles would be
>> a fairly basic ask, one would imagine. Is the lack of time / effort a
>> constraint? If so, I'd be glad to help (as mentioned in the jira).
>>
>> On Fri, May 15, 2015 at 5:29 PM, Iulian Dragoș <
>> iulian.dragos@typesafe.com> wrote:
>>
>>> Hi Ankur,
>>>
>>> Just to add a thought to Tim's excellent answer, Spark on Mesos is very
>>> important to us and is the recommended deployment for our customers as
>>> Typesafe.
>>>
>>> Thanks for pointing to your PR, I see Tim already went through a round
>>> of reviews. It seems very useful, I'll give it a try as well.
>>>
>>> thanks,
>>> iulian
>>>
>>>
>>>
>>> On Fri, May 15, 2015 at 9:53 AM, Ankur Chauhan <an...@malloc64.com>
>>> wrote:
>>>
>>>> -----BEGIN PGP SIGNED MESSAGE-----
>>>> Hash: SHA1
>>>>
>>>> Hi Tim,
>>>>
>>>> Thanks for such a detailed email. I am excited to hear about the new
>>>> features, I had a pull request going for adding "attribute based
>>>> filtering in the mesos scheduler" but it hasn't received much love -
>>>> https://github.com/apache/spark/pull/5563 . I am a fan of
>>>> mesos/marathon/mesosphere and spark ecosystems and trying to push
>>>> adoption at my workplace.
>>>>
>>>> It would love to see documentation, tutorials (anything actually) that
>>>> would make mesos + spark a better and more fleshed out solution. Would
>>>> it be possible for you to share some links to the JIRA and pull
>>>> requests so that I can keep track on the progress/features.
>>>>
>>>> Again, thanks for replying.
>>>>
>>>> - -- Ankur Chauhan
>>>>
>>>> On 15/05/2015 00:39, Tim Chen wrote:
>>>> > Hi Ankur,
>>>> >
>>>> > This is a great question as I've heard similar concerns about Spark
>>>> > on Mesos.
>>>> >
>>>> > At the time when I started to contribute to Spark on Mesos approx
>>>> > half year ago, the Mesos scheduler and related code hasn't really
>>>> > got much attention from anyone and it was pretty much in
>>>> > maintenance mode.
>>>> >
>>>> > As a Mesos PMC that is really interested in Spark I started to
>>>> > refactor and check out different JIRAs and PRs around the Mesos
>>>> > scheduler, and after that started to fix various bugs in Spark,
>>>> > added documentation and also in fix related Mesos issues as well.
>>>> >
>>>> > Just recently for 1.4 we've merged in Cluster mode and Docker
>>>> > support, and there are also pending PRs around framework
>>>> > authentication, multi-role support, dynamic allocation, more finer
>>>> > tuned coarse grain mode scheduling configurations, etc.
>>>> >
>>>> > And finally just want to mention that Mesosphere and Typesafe is
>>>> > collaborating to bring a certified distribution
>>>> > (
>>>> https://databricks.com/spark/certification/certified-spark-distributi
>>>> on)
>>>> > of Spark on Mesos and DCOS, and we will be pouring resources into
>>>> > not just maintain Spark on Mesos but drive more features into the
>>>> > Mesos scheduler and also in Mesos so stateful services can leverage
>>>> > new APIs and features to make better scheduling decisions and
>>>> > optimizations.
>>>> >
>>>> > I don't have a solidified roadmap to share yet, but we will be
>>>> > discussing this and hopefully can share with the community soon.
>>>> >
>>>> > In summary Spark on Mesos is not dead or in maintenance mode, and
>>>> > look forward to see a lot more changes from us and the community.
>>>> >
>>>> > Tim
>>>> >
>>>> > On Thu, May 14, 2015 at 11:30 PM, Ankur Chauhan
>>>> > <ankur@malloc64.com <ma...@malloc64.com>> wrote:
>>>> >
>>>> > Hi,
>>>> >
>>>> > This is both a survey type as well as a roadmap query question. It
>>>> > seems like of the cluster options to run spark (i.e. via YARN and
>>>> > Mesos), YARN seems to be getting a lot more attention and patches
>>>> > when compared to Mesos.
>>>> >
>>>> > Would it be correct to assume that spark on mesos is more or less
>>>> > a dead or something like a maintenance-only feature and YARN is
>>>> > the recommended way to go?
>>>> >
>>>> > What is the roadmap for spark on mesos? and what is the roadmap
>>>> > for spark on yarn. I like mesos so as much as I would like to see
>>>> > it thrive I don't think spark community is active (or maybe it
>>>> > just appears that way).
>>>> >
>>>> > Another more community oriented question: what do most people use
>>>> > to run spark in production or more-than-POC products? Why did you
>>>> > make that decision?
>>>> >
>>>> > There was a similar post form early 2014 where Metei answered that
>>>> > mesos and yarn were equally important, but has this changed as
>>>> > spark has now reached almost 1.4.0 stage?
>>>> >
>>>> > -- Ankur Chauhan
>>>> >
>>>> > ---------------------------------------------------------------------
>>>> >
>>>> >
>>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>> > <ma...@spark.apache.org> For additional commands,
>>>> > e-mail: user-help@spark.apache.org
>>>> > <ma...@spark.apache.org>
>>>> >
>>>> >
>>>> -----BEGIN PGP SIGNATURE-----
>>>>
>>>> iQEcBAEBAgAGBQJVVaXvAAoJEOSJAMhvLp3LzkIH/RLPVUoOcFo0Gij1NpZDszN4
>>>> xWvesbOZszuqD8H1Dhyndz4RQKnrodyCE+NycFB+utd9epmuyGemmHpTnq18Gek6
>>>> PR5jqmgza94dOy0rfuIVvba14ALZb4tO9SgkjyGujrpMlFYvxTjBYdYCAjfEOTx7
>>>> A/vqaCzPSBRBmO8gWx07GWa4zI70qBSZ9KnV7dgtqfUUgPKdF4NnMZWRJjTO9Bp8
>>>> tTmWMldqYPqI95wdeeqTGMH0XT6JAKAiCskf62DGadRBsOshrhmh5mAQzUFwoTpA
>>>> w4uZ+qMrTsblBvOf9z++v0eY8VBiQpOyXfOBiYiCNRtSsGa0KvqwgF1S/yLeRs0=
>>>> =4Aax
>>>> -----END PGP SIGNATURE-----
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>> For additional commands, e-mail: user-help@spark.apache.org
>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> --
>>> Iulian Dragos
>>>
>>> ------
>>> Reactive Apps on the JVM
>>> www.typesafe.com
>>>
>>>
>>
>
Re: Spark on Mesos vs Yarn
Posted by Bharath Ravi Kumar <re...@gmail.com>.
All,
Despite the common origin of spark & mesos, the stability and adoption of
mesos, and the age of the spark-mesos binding, I find the mesos support
less mature, with fundamental shortcomings (like framework auth
<https://issues.apache.org/jira/browse/SPARK-6284>) remaining unresolved.
If there's shortage of developer time, I'd be glad to contribute, but it's
unclear if the committer group has sufficient time (and priority) to take
the mesos support forward. While it has been stated often that support for
mesos & yarn are equally important, that doesn't seem to translate to
visible progress. I'd be glad if my observation is incorrect as I seek
better focus and long term commitment on the mesos support.
As for the specific issue (6284), I'm happy to build, testing & eventually
deploy the patch in our production cluster, but I'd rather see it becoming
mainstream.
Thanks for your consideration.
-Bharath
On Thu, May 28, 2015 at 9:18 AM, Bharath Ravi Kumar <re...@gmail.com>
wrote:
> A follow up : considering that spark on mesos is indeed important to
> databricks, its partners and the community, fundamental issues like
> spark-6284 shouldn't be languishing for this long. A mesos cluster hosting
> diverse (i.e.multi-tenant) workloads is a common scenario in production
> for serious users. The ability to auth a framework & assign roles would be
> a fairly basic ask, one would imagine. Is the lack of time / effort a
> constraint? If so, I'd be glad to help (as mentioned in the jira).
>
> On Fri, May 15, 2015 at 5:29 PM, Iulian Dragoș <iulian.dragos@typesafe.com
> > wrote:
>
>> Hi Ankur,
>>
>> Just to add a thought to Tim's excellent answer, Spark on Mesos is very
>> important to us and is the recommended deployment for our customers as
>> Typesafe.
>>
>> Thanks for pointing to your PR, I see Tim already went through a round of
>> reviews. It seems very useful, I'll give it a try as well.
>>
>> thanks,
>> iulian
>>
>>
>>
>> On Fri, May 15, 2015 at 9:53 AM, Ankur Chauhan <an...@malloc64.com>
>> wrote:
>>
>>> -----BEGIN PGP SIGNED MESSAGE-----
>>> Hash: SHA1
>>>
>>> Hi Tim,
>>>
>>> Thanks for such a detailed email. I am excited to hear about the new
>>> features, I had a pull request going for adding "attribute based
>>> filtering in the mesos scheduler" but it hasn't received much love -
>>> https://github.com/apache/spark/pull/5563 . I am a fan of
>>> mesos/marathon/mesosphere and spark ecosystems and trying to push
>>> adoption at my workplace.
>>>
>>> It would love to see documentation, tutorials (anything actually) that
>>> would make mesos + spark a better and more fleshed out solution. Would
>>> it be possible for you to share some links to the JIRA and pull
>>> requests so that I can keep track on the progress/features.
>>>
>>> Again, thanks for replying.
>>>
>>> - -- Ankur Chauhan
>>>
>>> On 15/05/2015 00:39, Tim Chen wrote:
>>> > Hi Ankur,
>>> >
>>> > This is a great question as I've heard similar concerns about Spark
>>> > on Mesos.
>>> >
>>> > At the time when I started to contribute to Spark on Mesos approx
>>> > half year ago, the Mesos scheduler and related code hasn't really
>>> > got much attention from anyone and it was pretty much in
>>> > maintenance mode.
>>> >
>>> > As a Mesos PMC that is really interested in Spark I started to
>>> > refactor and check out different JIRAs and PRs around the Mesos
>>> > scheduler, and after that started to fix various bugs in Spark,
>>> > added documentation and also in fix related Mesos issues as well.
>>> >
>>> > Just recently for 1.4 we've merged in Cluster mode and Docker
>>> > support, and there are also pending PRs around framework
>>> > authentication, multi-role support, dynamic allocation, more finer
>>> > tuned coarse grain mode scheduling configurations, etc.
>>> >
>>> > And finally just want to mention that Mesosphere and Typesafe is
>>> > collaborating to bring a certified distribution
>>> > (https://databricks.com/spark/certification/certified-spark-distributi
>>> on)
>>> > of Spark on Mesos and DCOS, and we will be pouring resources into
>>> > not just maintain Spark on Mesos but drive more features into the
>>> > Mesos scheduler and also in Mesos so stateful services can leverage
>>> > new APIs and features to make better scheduling decisions and
>>> > optimizations.
>>> >
>>> > I don't have a solidified roadmap to share yet, but we will be
>>> > discussing this and hopefully can share with the community soon.
>>> >
>>> > In summary Spark on Mesos is not dead or in maintenance mode, and
>>> > look forward to see a lot more changes from us and the community.
>>> >
>>> > Tim
>>> >
>>> > On Thu, May 14, 2015 at 11:30 PM, Ankur Chauhan
>>> > <ankur@malloc64.com <ma...@malloc64.com>> wrote:
>>> >
>>> > Hi,
>>> >
>>> > This is both a survey type as well as a roadmap query question. It
>>> > seems like of the cluster options to run spark (i.e. via YARN and
>>> > Mesos), YARN seems to be getting a lot more attention and patches
>>> > when compared to Mesos.
>>> >
>>> > Would it be correct to assume that spark on mesos is more or less
>>> > a dead or something like a maintenance-only feature and YARN is
>>> > the recommended way to go?
>>> >
>>> > What is the roadmap for spark on mesos? and what is the roadmap
>>> > for spark on yarn. I like mesos so as much as I would like to see
>>> > it thrive I don't think spark community is active (or maybe it
>>> > just appears that way).
>>> >
>>> > Another more community oriented question: what do most people use
>>> > to run spark in production or more-than-POC products? Why did you
>>> > make that decision?
>>> >
>>> > There was a similar post form early 2014 where Metei answered that
>>> > mesos and yarn were equally important, but has this changed as
>>> > spark has now reached almost 1.4.0 stage?
>>> >
>>> > -- Ankur Chauhan
>>> >
>>> > ---------------------------------------------------------------------
>>> >
>>> >
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> > <ma...@spark.apache.org> For additional commands,
>>> > e-mail: user-help@spark.apache.org
>>> > <ma...@spark.apache.org>
>>> >
>>> >
>>> -----BEGIN PGP SIGNATURE-----
>>>
>>> iQEcBAEBAgAGBQJVVaXvAAoJEOSJAMhvLp3LzkIH/RLPVUoOcFo0Gij1NpZDszN4
>>> xWvesbOZszuqD8H1Dhyndz4RQKnrodyCE+NycFB+utd9epmuyGemmHpTnq18Gek6
>>> PR5jqmgza94dOy0rfuIVvba14ALZb4tO9SgkjyGujrpMlFYvxTjBYdYCAjfEOTx7
>>> A/vqaCzPSBRBmO8gWx07GWa4zI70qBSZ9KnV7dgtqfUUgPKdF4NnMZWRJjTO9Bp8
>>> tTmWMldqYPqI95wdeeqTGMH0XT6JAKAiCskf62DGadRBsOshrhmh5mAQzUFwoTpA
>>> w4uZ+qMrTsblBvOf9z++v0eY8VBiQpOyXfOBiYiCNRtSsGa0KvqwgF1S/yLeRs0=
>>> =4Aax
>>> -----END PGP SIGNATURE-----
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: user-help@spark.apache.org
>>>
>>>
>>
>>
>> --
>>
>> --
>> Iulian Dragos
>>
>> ------
>> Reactive Apps on the JVM
>> www.typesafe.com
>>
>>
>
Re: Spark on Mesos vs Yarn
Posted by Bharath Ravi Kumar <re...@gmail.com>.
A follow up : considering that spark on mesos is indeed important to
databricks, its partners and the community, fundamental issues like
spark-6284 shouldn't be languishing for this long. A mesos cluster hosting
diverse (i.e.multi-tenant) workloads is a common scenario in production
for serious users. The ability to auth a framework & assign roles would be
a fairly basic ask, one would imagine. Is the lack of time / effort a
constraint? If so, I'd be glad to help (as mentioned in the jira).
On Fri, May 15, 2015 at 5:29 PM, Iulian Dragoș <iu...@typesafe.com>
wrote:
> Hi Ankur,
>
> Just to add a thought to Tim's excellent answer, Spark on Mesos is very
> important to us and is the recommended deployment for our customers as
> Typesafe.
>
> Thanks for pointing to your PR, I see Tim already went through a round of
> reviews. It seems very useful, I'll give it a try as well.
>
> thanks,
> iulian
>
>
>
> On Fri, May 15, 2015 at 9:53 AM, Ankur Chauhan <an...@malloc64.com> wrote:
>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Hi Tim,
>>
>> Thanks for such a detailed email. I am excited to hear about the new
>> features, I had a pull request going for adding "attribute based
>> filtering in the mesos scheduler" but it hasn't received much love -
>> https://github.com/apache/spark/pull/5563 . I am a fan of
>> mesos/marathon/mesosphere and spark ecosystems and trying to push
>> adoption at my workplace.
>>
>> It would love to see documentation, tutorials (anything actually) that
>> would make mesos + spark a better and more fleshed out solution. Would
>> it be possible for you to share some links to the JIRA and pull
>> requests so that I can keep track on the progress/features.
>>
>> Again, thanks for replying.
>>
>> - -- Ankur Chauhan
>>
>> On 15/05/2015 00:39, Tim Chen wrote:
>> > Hi Ankur,
>> >
>> > This is a great question as I've heard similar concerns about Spark
>> > on Mesos.
>> >
>> > At the time when I started to contribute to Spark on Mesos approx
>> > half year ago, the Mesos scheduler and related code hasn't really
>> > got much attention from anyone and it was pretty much in
>> > maintenance mode.
>> >
>> > As a Mesos PMC that is really interested in Spark I started to
>> > refactor and check out different JIRAs and PRs around the Mesos
>> > scheduler, and after that started to fix various bugs in Spark,
>> > added documentation and also in fix related Mesos issues as well.
>> >
>> > Just recently for 1.4 we've merged in Cluster mode and Docker
>> > support, and there are also pending PRs around framework
>> > authentication, multi-role support, dynamic allocation, more finer
>> > tuned coarse grain mode scheduling configurations, etc.
>> >
>> > And finally just want to mention that Mesosphere and Typesafe is
>> > collaborating to bring a certified distribution
>> > (https://databricks.com/spark/certification/certified-spark-distributi
>> on)
>> > of Spark on Mesos and DCOS, and we will be pouring resources into
>> > not just maintain Spark on Mesos but drive more features into the
>> > Mesos scheduler and also in Mesos so stateful services can leverage
>> > new APIs and features to make better scheduling decisions and
>> > optimizations.
>> >
>> > I don't have a solidified roadmap to share yet, but we will be
>> > discussing this and hopefully can share with the community soon.
>> >
>> > In summary Spark on Mesos is not dead or in maintenance mode, and
>> > look forward to see a lot more changes from us and the community.
>> >
>> > Tim
>> >
>> > On Thu, May 14, 2015 at 11:30 PM, Ankur Chauhan
>> > <ankur@malloc64.com <ma...@malloc64.com>> wrote:
>> >
>> > Hi,
>> >
>> > This is both a survey type as well as a roadmap query question. It
>> > seems like of the cluster options to run spark (i.e. via YARN and
>> > Mesos), YARN seems to be getting a lot more attention and patches
>> > when compared to Mesos.
>> >
>> > Would it be correct to assume that spark on mesos is more or less
>> > a dead or something like a maintenance-only feature and YARN is
>> > the recommended way to go?
>> >
>> > What is the roadmap for spark on mesos? and what is the roadmap
>> > for spark on yarn. I like mesos so as much as I would like to see
>> > it thrive I don't think spark community is active (or maybe it
>> > just appears that way).
>> >
>> > Another more community oriented question: what do most people use
>> > to run spark in production or more-than-POC products? Why did you
>> > make that decision?
>> >
>> > There was a similar post form early 2014 where Metei answered that
>> > mesos and yarn were equally important, but has this changed as
>> > spark has now reached almost 1.4.0 stage?
>> >
>> > -- Ankur Chauhan
>> >
>> > ---------------------------------------------------------------------
>> >
>> >
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> > <ma...@spark.apache.org> For additional commands,
>> > e-mail: user-help@spark.apache.org
>> > <ma...@spark.apache.org>
>> >
>> >
>> -----BEGIN PGP SIGNATURE-----
>>
>> iQEcBAEBAgAGBQJVVaXvAAoJEOSJAMhvLp3LzkIH/RLPVUoOcFo0Gij1NpZDszN4
>> xWvesbOZszuqD8H1Dhyndz4RQKnrodyCE+NycFB+utd9epmuyGemmHpTnq18Gek6
>> PR5jqmgza94dOy0rfuIVvba14ALZb4tO9SgkjyGujrpMlFYvxTjBYdYCAjfEOTx7
>> A/vqaCzPSBRBmO8gWx07GWa4zI70qBSZ9KnV7dgtqfUUgPKdF4NnMZWRJjTO9Bp8
>> tTmWMldqYPqI95wdeeqTGMH0XT6JAKAiCskf62DGadRBsOshrhmh5mAQzUFwoTpA
>> w4uZ+qMrTsblBvOf9z++v0eY8VBiQpOyXfOBiYiCNRtSsGa0KvqwgF1S/yLeRs0=
>> =4Aax
>> -----END PGP SIGNATURE-----
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>
>
> --
>
> --
> Iulian Dragos
>
> ------
> Reactive Apps on the JVM
> www.typesafe.com
>
>
Re: Spark on Mesos vs Yarn
Posted by Iulian Dragoș <iu...@typesafe.com>.
Hi Ankur,
Just to add a thought to Tim's excellent answer, Spark on Mesos is very
important to us and is the recommended deployment for our customers as
Typesafe.
Thanks for pointing to your PR, I see Tim already went through a round of
reviews. It seems very useful, I'll give it a try as well.
thanks,
iulian
On Fri, May 15, 2015 at 9:53 AM, Ankur Chauhan <an...@malloc64.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hi Tim,
>
> Thanks for such a detailed email. I am excited to hear about the new
> features, I had a pull request going for adding "attribute based
> filtering in the mesos scheduler" but it hasn't received much love -
> https://github.com/apache/spark/pull/5563 . I am a fan of
> mesos/marathon/mesosphere and spark ecosystems and trying to push
> adoption at my workplace.
>
> It would love to see documentation, tutorials (anything actually) that
> would make mesos + spark a better and more fleshed out solution. Would
> it be possible for you to share some links to the JIRA and pull
> requests so that I can keep track on the progress/features.
>
> Again, thanks for replying.
>
> - -- Ankur Chauhan
>
> On 15/05/2015 00:39, Tim Chen wrote:
> > Hi Ankur,
> >
> > This is a great question as I've heard similar concerns about Spark
> > on Mesos.
> >
> > At the time when I started to contribute to Spark on Mesos approx
> > half year ago, the Mesos scheduler and related code hasn't really
> > got much attention from anyone and it was pretty much in
> > maintenance mode.
> >
> > As a Mesos PMC that is really interested in Spark I started to
> > refactor and check out different JIRAs and PRs around the Mesos
> > scheduler, and after that started to fix various bugs in Spark,
> > added documentation and also in fix related Mesos issues as well.
> >
> > Just recently for 1.4 we've merged in Cluster mode and Docker
> > support, and there are also pending PRs around framework
> > authentication, multi-role support, dynamic allocation, more finer
> > tuned coarse grain mode scheduling configurations, etc.
> >
> > And finally just want to mention that Mesosphere and Typesafe is
> > collaborating to bring a certified distribution
> > (https://databricks.com/spark/certification/certified-spark-distributi
> on)
> > of Spark on Mesos and DCOS, and we will be pouring resources into
> > not just maintain Spark on Mesos but drive more features into the
> > Mesos scheduler and also in Mesos so stateful services can leverage
> > new APIs and features to make better scheduling decisions and
> > optimizations.
> >
> > I don't have a solidified roadmap to share yet, but we will be
> > discussing this and hopefully can share with the community soon.
> >
> > In summary Spark on Mesos is not dead or in maintenance mode, and
> > look forward to see a lot more changes from us and the community.
> >
> > Tim
> >
> > On Thu, May 14, 2015 at 11:30 PM, Ankur Chauhan
> > <ankur@malloc64.com <ma...@malloc64.com>> wrote:
> >
> > Hi,
> >
> > This is both a survey type as well as a roadmap query question. It
> > seems like of the cluster options to run spark (i.e. via YARN and
> > Mesos), YARN seems to be getting a lot more attention and patches
> > when compared to Mesos.
> >
> > Would it be correct to assume that spark on mesos is more or less
> > a dead or something like a maintenance-only feature and YARN is
> > the recommended way to go?
> >
> > What is the roadmap for spark on mesos? and what is the roadmap
> > for spark on yarn. I like mesos so as much as I would like to see
> > it thrive I don't think spark community is active (or maybe it
> > just appears that way).
> >
> > Another more community oriented question: what do most people use
> > to run spark in production or more-than-POC products? Why did you
> > make that decision?
> >
> > There was a similar post form early 2014 where Metei answered that
> > mesos and yarn were equally important, but has this changed as
> > spark has now reached almost 1.4.0 stage?
> >
> > -- Ankur Chauhan
> >
> > ---------------------------------------------------------------------
> >
> >
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> > <ma...@spark.apache.org> For additional commands,
> > e-mail: user-help@spark.apache.org
> > <ma...@spark.apache.org>
> >
> >
> -----BEGIN PGP SIGNATURE-----
>
> iQEcBAEBAgAGBQJVVaXvAAoJEOSJAMhvLp3LzkIH/RLPVUoOcFo0Gij1NpZDszN4
> xWvesbOZszuqD8H1Dhyndz4RQKnrodyCE+NycFB+utd9epmuyGemmHpTnq18Gek6
> PR5jqmgza94dOy0rfuIVvba14ALZb4tO9SgkjyGujrpMlFYvxTjBYdYCAjfEOTx7
> A/vqaCzPSBRBmO8gWx07GWa4zI70qBSZ9KnV7dgtqfUUgPKdF4NnMZWRJjTO9Bp8
> tTmWMldqYPqI95wdeeqTGMH0XT6JAKAiCskf62DGadRBsOshrhmh5mAQzUFwoTpA
> w4uZ+qMrTsblBvOf9z++v0eY8VBiQpOyXfOBiYiCNRtSsGa0KvqwgF1S/yLeRs0=
> =4Aax
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
--
--
Iulian Dragos
------
Reactive Apps on the JVM
www.typesafe.com
Re: Spark on Mesos vs Yarn
Posted by Ankur Chauhan <an...@malloc64.com>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi Tim,
Thanks for such a detailed email. I am excited to hear about the new
features, I had a pull request going for adding "attribute based
filtering in the mesos scheduler" but it hasn't received much love -
https://github.com/apache/spark/pull/5563 . I am a fan of
mesos/marathon/mesosphere and spark ecosystems and trying to push
adoption at my workplace.
It would love to see documentation, tutorials (anything actually) that
would make mesos + spark a better and more fleshed out solution. Would
it be possible for you to share some links to the JIRA and pull
requests so that I can keep track on the progress/features.
Again, thanks for replying.
- -- Ankur Chauhan
On 15/05/2015 00:39, Tim Chen wrote:
> Hi Ankur,
>
> This is a great question as I've heard similar concerns about Spark
> on Mesos.
>
> At the time when I started to contribute to Spark on Mesos approx
> half year ago, the Mesos scheduler and related code hasn't really
> got much attention from anyone and it was pretty much in
> maintenance mode.
>
> As a Mesos PMC that is really interested in Spark I started to
> refactor and check out different JIRAs and PRs around the Mesos
> scheduler, and after that started to fix various bugs in Spark,
> added documentation and also in fix related Mesos issues as well.
>
> Just recently for 1.4 we've merged in Cluster mode and Docker
> support, and there are also pending PRs around framework
> authentication, multi-role support, dynamic allocation, more finer
> tuned coarse grain mode scheduling configurations, etc.
>
> And finally just want to mention that Mesosphere and Typesafe is
> collaborating to bring a certified distribution
> (https://databricks.com/spark/certification/certified-spark-distributi
on)
> of Spark on Mesos and DCOS, and we will be pouring resources into
> not just maintain Spark on Mesos but drive more features into the
> Mesos scheduler and also in Mesos so stateful services can leverage
> new APIs and features to make better scheduling decisions and
> optimizations.
>
> I don't have a solidified roadmap to share yet, but we will be
> discussing this and hopefully can share with the community soon.
>
> In summary Spark on Mesos is not dead or in maintenance mode, and
> look forward to see a lot more changes from us and the community.
>
> Tim
>
> On Thu, May 14, 2015 at 11:30 PM, Ankur Chauhan
> <ankur@malloc64.com <ma...@malloc64.com>> wrote:
>
> Hi,
>
> This is both a survey type as well as a roadmap query question. It
> seems like of the cluster options to run spark (i.e. via YARN and
> Mesos), YARN seems to be getting a lot more attention and patches
> when compared to Mesos.
>
> Would it be correct to assume that spark on mesos is more or less
> a dead or something like a maintenance-only feature and YARN is
> the recommended way to go?
>
> What is the roadmap for spark on mesos? and what is the roadmap
> for spark on yarn. I like mesos so as much as I would like to see
> it thrive I don't think spark community is active (or maybe it
> just appears that way).
>
> Another more community oriented question: what do most people use
> to run spark in production or more-than-POC products? Why did you
> make that decision?
>
> There was a similar post form early 2014 where Metei answered that
> mesos and yarn were equally important, but has this changed as
> spark has now reached almost 1.4.0 stage?
>
> -- Ankur Chauhan
>
> ---------------------------------------------------------------------
>
>
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> <ma...@spark.apache.org> For additional commands,
> e-mail: user-help@spark.apache.org
> <ma...@spark.apache.org>
>
>
-----BEGIN PGP SIGNATURE-----
iQEcBAEBAgAGBQJVVaXvAAoJEOSJAMhvLp3LzkIH/RLPVUoOcFo0Gij1NpZDszN4
xWvesbOZszuqD8H1Dhyndz4RQKnrodyCE+NycFB+utd9epmuyGemmHpTnq18Gek6
PR5jqmgza94dOy0rfuIVvba14ALZb4tO9SgkjyGujrpMlFYvxTjBYdYCAjfEOTx7
A/vqaCzPSBRBmO8gWx07GWa4zI70qBSZ9KnV7dgtqfUUgPKdF4NnMZWRJjTO9Bp8
tTmWMldqYPqI95wdeeqTGMH0XT6JAKAiCskf62DGadRBsOshrhmh5mAQzUFwoTpA
w4uZ+qMrTsblBvOf9z++v0eY8VBiQpOyXfOBiYiCNRtSsGa0KvqwgF1S/yLeRs0=
=4Aax
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark on Mesos vs Yarn
Posted by Tim Chen <ti...@mesosphere.io>.
Hi Ankur,
This is a great question as I've heard similar concerns about Spark on
Mesos.
At the time when I started to contribute to Spark on Mesos approx half year
ago, the Mesos scheduler and related code hasn't really got much attention
from anyone and it was pretty much in maintenance mode.
As a Mesos PMC that is really interested in Spark I started to refactor and
check out different JIRAs and PRs around the Mesos scheduler, and after
that started to fix various bugs in Spark, added documentation and also in
fix related Mesos issues as well.
Just recently for 1.4 we've merged in Cluster mode and Docker support, and
there are also pending PRs around framework authentication, multi-role
support, dynamic allocation, more finer tuned coarse grain mode scheduling
configurations, etc.
And finally just want to mention that Mesosphere and Typesafe is
collaborating to bring a certified distribution (
https://databricks.com/spark/certification/certified-spark-distribution) of
Spark on Mesos and DCOS, and we will be pouring resources into not just
maintain Spark on Mesos but drive more features into the Mesos scheduler
and also in Mesos so stateful services can leverage new APIs and features
to make better scheduling decisions and optimizations.
I don't have a solidified roadmap to share yet, but we will be discussing
this and hopefully can share with the community soon.
In summary Spark on Mesos is not dead or in maintenance mode, and look
forward to see a lot more changes from us and the community.
Tim
On Thu, May 14, 2015 at 11:30 PM, Ankur Chauhan <an...@malloc64.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hi,
>
> This is both a survey type as well as a roadmap query question. It
> seems like of the cluster options to run spark (i.e. via YARN and
> Mesos), YARN seems to be getting a lot more attention and patches when
> compared to Mesos.
>
> Would it be correct to assume that spark on mesos is more or less a
> dead or something like a maintenance-only feature and YARN is the
> recommended way to go?
>
> What is the roadmap for spark on mesos? and what is the roadmap for
> spark on yarn. I like mesos so as much as I would like to see it
> thrive I don't think spark community is active (or maybe it just
> appears that way).
>
> Another more community oriented question: what do most people use to
> run spark in production or more-than-POC products? Why did you make
> that decision?
>
> There was a similar post form early 2014 where Metei answered that
> mesos and yarn were equally important, but has this changed as spark
> has now reached almost 1.4.0 stage?
>
> - -- Ankur Chauhan
> -----BEGIN PGP SIGNATURE-----
>
> iQEcBAEBAgAGBQJVVZKGAAoJEOSJAMhvLp3L0vEIAI4edLB2rMGk+OTI4WujxX6k
> Ud5NyFUpaQ8WDjOhwcWB9RK5EoM7X3wGzRcGza1HLVnvdSUBG8Ltabt47GsP2lo0
> 7H9y2GluUZg/RJXbN0Ehp6moWjAU1W/55POD3t87qeUdydUJVbgDYA/KovNa6i8s
> Z/e8mfvOrFSJyuJi8KW2KcfOmB1i8VZH7b/zZqtfJKNGo/0dac/gez19vVPaXPa4
> WNUN8dHcp0yiZnZ0PUTYNLhI58BXBCSmkEl2Ex7X3NBUGUgJ5HGHn6dpqqNhGvf3
> yPw0B0q93NcExK/E4/I75nn4vh5wKLPLWT8U5btphmc7S6h8gWFMEJRHQCdtaUk=
> =uYXZ
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>