You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Reynold Xin <rx...@databricks.com> on 2015/11/04 00:54:06 UTC
Please reply if you use Mesos fine grained mode
If you are using Spark with Mesos fine grained mode, can you please respond
to this email explaining why you use it over the coarse grained mode?
Thanks.
Re: Please reply if you use Mesos fine grained mode
Posted by Jerry Lam <ch...@gmail.com>.
We "used" Spark on Mesos to build interactive data analysis platform
because the interactive session could be long and might not use Spark for
the entire session. It is very wasteful of resources if we used the
coarse-grained mode because it keeps resource for the entire session.
Therefore, fine-grained mode was used.
Knowing that Spark now supports dynamic resource allocation with coarse
grained mode, we were thinking about using it. However, we decided to
switch to Yarn because in addition to dynamic allocation, it has better
supports on security.
On Tue, Nov 3, 2015 at 7:22 PM, Soren Macbeth <so...@yieldbot.com> wrote:
> we use fine-grained mode. coarse-grained mode keeps JVMs around which
> often leads to OOMs, which in turn kill the entire executor, causing entire
> stages to be retried. In fine-grained mode, only the task fails and
> subsequently gets retried without taking out an entire stage or worse.
>
> On Tue, Nov 3, 2015 at 3:54 PM, Reynold Xin <rx...@databricks.com> wrote:
>
>> If you are using Spark with Mesos fine grained mode, can you please
>> respond to this email explaining why you use it over the coarse grained
>> mode?
>>
>> Thanks.
>>
>>
>
Re: Please reply if you use Mesos fine grained mode
Posted by Timothy Chen <tn...@gmail.com>.
Fine grain mode does reuse the same JVM but perhaps different placement or different allocated cores comparing to the same total memory allocation.
Tim
Sent from my iPhone
> On Nov 3, 2015, at 6:00 PM, Reynold Xin <rx...@databricks.com> wrote:
>
> Soren,
>
> If I understand how Mesos works correctly, even the fine grained mode keeps the JVMs around?
>
>
>> On Tue, Nov 3, 2015 at 4:22 PM, Soren Macbeth <so...@yieldbot.com> wrote:
>> we use fine-grained mode. coarse-grained mode keeps JVMs around which often leads to OOMs, which in turn kill the entire executor, causing entire stages to be retried. In fine-grained mode, only the task fails and subsequently gets retried without taking out an entire stage or worse.
>>
>>> On Tue, Nov 3, 2015 at 3:54 PM, Reynold Xin <rx...@databricks.com> wrote:
>>> If you are using Spark with Mesos fine grained mode, can you please respond to this email explaining why you use it over the coarse grained mode?
>>>
>>> Thanks.
>
Re: Please reply if you use Mesos fine grained mode
Posted by Timothy Chen <tn...@gmail.com>.
Fine grain mode does reuse the same JVM but perhaps different placement or different allocated cores comparing to the same total memory allocation.
Tim
Sent from my iPhone
> On Nov 3, 2015, at 6:00 PM, Reynold Xin <rx...@databricks.com> wrote:
>
> Soren,
>
> If I understand how Mesos works correctly, even the fine grained mode keeps the JVMs around?
>
>
>> On Tue, Nov 3, 2015 at 4:22 PM, Soren Macbeth <so...@yieldbot.com> wrote:
>> we use fine-grained mode. coarse-grained mode keeps JVMs around which often leads to OOMs, which in turn kill the entire executor, causing entire stages to be retried. In fine-grained mode, only the task fails and subsequently gets retried without taking out an entire stage or worse.
>>
>>> On Tue, Nov 3, 2015 at 3:54 PM, Reynold Xin <rx...@databricks.com> wrote:
>>> If you are using Spark with Mesos fine grained mode, can you please respond to this email explaining why you use it over the coarse grained mode?
>>>
>>> Thanks.
>
Re: Please reply if you use Mesos fine grained mode
Posted by Reynold Xin <rx...@databricks.com>.
Soren,
If I understand how Mesos works correctly, even the fine grained mode keeps
the JVMs around?
On Tue, Nov 3, 2015 at 4:22 PM, Soren Macbeth <so...@yieldbot.com> wrote:
> we use fine-grained mode. coarse-grained mode keeps JVMs around which
> often leads to OOMs, which in turn kill the entire executor, causing entire
> stages to be retried. In fine-grained mode, only the task fails and
> subsequently gets retried without taking out an entire stage or worse.
>
> On Tue, Nov 3, 2015 at 3:54 PM, Reynold Xin <rx...@databricks.com> wrote:
>
>> If you are using Spark with Mesos fine grained mode, can you please
>> respond to this email explaining why you use it over the coarse grained
>> mode?
>>
>> Thanks.
>>
>>
>
Re: Please reply if you use Mesos fine grained mode
Posted by Jerry Lam <ch...@gmail.com>.
We "used" Spark on Mesos to build interactive data analysis platform
because the interactive session could be long and might not use Spark for
the entire session. It is very wasteful of resources if we used the
coarse-grained mode because it keeps resource for the entire session.
Therefore, fine-grained mode was used.
Knowing that Spark now supports dynamic resource allocation with coarse
grained mode, we were thinking about using it. However, we decided to
switch to Yarn because in addition to dynamic allocation, it has better
supports on security.
On Tue, Nov 3, 2015 at 7:22 PM, Soren Macbeth <so...@yieldbot.com> wrote:
> we use fine-grained mode. coarse-grained mode keeps JVMs around which
> often leads to OOMs, which in turn kill the entire executor, causing entire
> stages to be retried. In fine-grained mode, only the task fails and
> subsequently gets retried without taking out an entire stage or worse.
>
> On Tue, Nov 3, 2015 at 3:54 PM, Reynold Xin <rx...@databricks.com> wrote:
>
>> If you are using Spark with Mesos fine grained mode, can you please
>> respond to this email explaining why you use it over the coarse grained
>> mode?
>>
>> Thanks.
>>
>>
>
Re: Please reply if you use Mesos fine grained mode
Posted by Reynold Xin <rx...@databricks.com>.
Soren,
If I understand how Mesos works correctly, even the fine grained mode keeps
the JVMs around?
On Tue, Nov 3, 2015 at 4:22 PM, Soren Macbeth <so...@yieldbot.com> wrote:
> we use fine-grained mode. coarse-grained mode keeps JVMs around which
> often leads to OOMs, which in turn kill the entire executor, causing entire
> stages to be retried. In fine-grained mode, only the task fails and
> subsequently gets retried without taking out an entire stage or worse.
>
> On Tue, Nov 3, 2015 at 3:54 PM, Reynold Xin <rx...@databricks.com> wrote:
>
>> If you are using Spark with Mesos fine grained mode, can you please
>> respond to this email explaining why you use it over the coarse grained
>> mode?
>>
>> Thanks.
>>
>>
>
Re: Please reply if you use Mesos fine grained mode
Posted by Soren Macbeth <so...@yieldbot.com>.
we use fine-grained mode. coarse-grained mode keeps JVMs around which often
leads to OOMs, which in turn kill the entire executor, causing entire
stages to be retried. In fine-grained mode, only the task fails and
subsequently gets retried without taking out an entire stage or worse.
On Tue, Nov 3, 2015 at 3:54 PM, Reynold Xin <rx...@databricks.com> wrote:
> If you are using Spark with Mesos fine grained mode, can you please
> respond to this email explaining why you use it over the coarse grained
> mode?
>
> Thanks.
>
>
Re: Please reply if you use Mesos fine grained mode
Posted by Soren Macbeth <so...@yieldbot.com>.
we use fine-grained mode. coarse-grained mode keeps JVMs around which often
leads to OOMs, which in turn kill the entire executor, causing entire
stages to be retried. In fine-grained mode, only the task fails and
subsequently gets retried without taking out an entire stage or worse.
On Tue, Nov 3, 2015 at 3:54 PM, Reynold Xin <rx...@databricks.com> wrote:
> If you are using Spark with Mesos fine grained mode, can you please
> respond to this email explaining why you use it over the coarse grained
> mode?
>
> Thanks.
>
>
Re: Please reply if you use Mesos fine grained mode
Posted by Iulian Dragoș <iu...@typesafe.com>.
Probably because only coarse-grained mode respects `spark.cores.max` right
now. See (and maybe review ;-)) #9027
<https://github.com/apache/spark/pull/9027> (sorry for the shameless plug).
iulian
On Wed, Nov 4, 2015 at 5:05 PM, Timothy Chen <tn...@gmail.com> wrote:
> Hi Chris,
>
> How does coarse grain mode gives you less starvation in your overloaded
> cluster? Is it just because it allocates all resources at once (which I
> think in a overloaded cluster allows less things to run at once).
>
> Tim
>
>
> On Nov 4, 2015, at 4:21 AM, Heller, Chris <ch...@akamai.com> wrote:
>
> We’ve been making use of both. Fine-grain mode makes sense for more ad-hoc
> work loads, and coarse-grained for more job like loads on a common data
> set. My preference is the fine-grain mode in all cases, but the overhead
> associated with its startup and the possibility that an overloaded cluster
> would be starved for resources makes coarse grain mode a reality at the
> moment.
>
> On Wednesday, 4 November 2015 5:24 AM, Reynold Xin <rx...@databricks.com>
> wrote:
>
>
> If you are using Spark with Mesos fine grained mode, can you please
> respond to this email explaining why you use it over the coarse grained
> mode?
>
> Thanks.
>
>
>
>
--
--
Iulian Dragos
------
Reactive Apps on the JVM
www.typesafe.com
Re: Please reply if you use Mesos fine grained mode
Posted by "Heller, Chris" <ch...@akamai.com>.
Correct. Its just that with coarse mode we grab the resources up front, so its either available or not. But using resources on demand, as with a fine grained mode, just means the potential to starve out an individual job. There is also the sharing of RDDs that coarse gives you which would need something like Tachyon to achieve in fine grain mode.
From: Timothy Chen <tn...@gmail.com>>
Date: Wednesday, November 4, 2015 at 11:05 AM
To: "Heller, Chris" <ch...@akamai.com>>
Cc: Reynold Xin <rx...@databricks.com>>, "dev@spark.apache.org<ma...@spark.apache.org>" <de...@spark.apache.org>>
Subject: Re: Please reply if you use Mesos fine grained mode
Hi Chris,
How does coarse grain mode gives you less starvation in your overloaded cluster? Is it just because it allocates all resources at once (which I think in a overloaded cluster allows less things to run at once).
Tim
On Nov 4, 2015, at 4:21 AM, Heller, Chris <ch...@akamai.com>> wrote:
We’ve been making use of both. Fine-grain mode makes sense for more ad-hoc work loads, and coarse-grained for more job like loads on a common data set. My preference is the fine-grain mode in all cases, but the overhead associated with its startup and the possibility that an overloaded cluster would be starved for resources makes coarse grain mode a reality at the moment.
On Wednesday, 4 November 2015 5:24 AM, Reynold Xin <rx...@databricks.com>> wrote:
If you are using Spark with Mesos fine grained mode, can you please respond to this email explaining why you use it over the coarse grained mode?
Thanks.
Re: Please reply if you use Mesos fine grained mode
Posted by Timothy Chen <tn...@gmail.com>.
Hi Chris,
How does coarse grain mode gives you less starvation in your overloaded cluster? Is it just because it allocates all resources at once (which I think in a overloaded cluster allows less things to run at once).
Tim
> On Nov 4, 2015, at 4:21 AM, Heller, Chris <ch...@akamai.com> wrote:
>
> We’ve been making use of both. Fine-grain mode makes sense for more ad-hoc work loads, and coarse-grained for more job like loads on a common data set. My preference is the fine-grain mode in all cases, but the overhead associated with its startup and the possibility that an overloaded cluster would be starved for resources makes coarse grain mode a reality at the moment.
>
> On Wednesday, 4 November 2015 5:24 AM, Reynold Xin <rx...@databricks.com> wrote:
>
>
> If you are using Spark with Mesos fine grained mode, can you please respond to this email explaining why you use it over the coarse grained mode?
>
> Thanks.
>
>
>
Re: Please reply if you use Mesos fine grained mode
Posted by "Heller, Chris" <ch...@akamai.com>.
We’ve been making use of both. Fine-grain mode makes sense for more ad-hoc work loads, and coarse-grained for more job like loads on a common data set. My preference is the fine-grain mode in all cases, but the overhead associated with its startup and the possibility that an overloaded cluster would be starved for resources makes coarse grain mode a reality at the moment.
On Wednesday, 4 November 2015 5:24 AM, Reynold Xin <rx...@databricks.com>> wrote:
If you are using Spark with Mesos fine grained mode, can you please respond to this email explaining why you use it over the coarse grained mode?
Thanks.
Re: Please reply if you use Mesos fine grained mode
Posted by MEETHU MATHEW <me...@yahoo.co.in>.
Hi,
We are using Mesos fine grained mode because we can have multiple instances of spark to share machines and each application get resources dynamically allocated. Thanks & Regards, Meethu M
On Wednesday, 4 November 2015 5:24 AM, Reynold Xin <rx...@databricks.com> wrote:
If you are using Spark with Mesos fine grained mode, can you please respond to this email explaining why you use it over the coarse grained mode?
Thanks.
Re: Please reply if you use Mesos fine grained mode
Posted by MEETHU MATHEW <me...@yahoo.co.in>.
Hi,
We are using Mesos fine grained mode because we can have multiple instances of spark to share machines and each application get resources dynamically allocated. Thanks & Regards, Meethu M
On Wednesday, 4 November 2015 5:24 AM, Reynold Xin <rx...@databricks.com> wrote:
If you are using Spark with Mesos fine grained mode, can you please respond to this email explaining why you use it over the coarse grained mode?
Thanks.