Re: Spark Streaming on mesos

[Iulian Dragoș <iu...@typesafe.com>]

On Mon, Nov 30, 2015 at 4:09 PM, Renjie Liu <li...@gmail.com> wrote:

> Hi, Lulian:
>

Please, it's Iulian, not Lulian.


> Are you sure that it'll be a long running process in fine-grained mode? I
> think you have a misunderstanding about it. An executor will be launched
> for some tasks, but not a long running process. When a group of tasks
> finished, it will get shutdown.
>

Sorry I missed your answer. Yes, I'm pretty sure, and if you have SSH
access to one of the slaves it's pretty easy to check. What makes you think
otherwise?


>
> On Mon, Nov 30, 2015 at 6:25 PM Iulian Dragoș <iu...@typesafe.com>
> wrote:
>
>> Hi,
>>
>> Latency isn't such a big issue as it sounds. Did you try it out and
>> failed some performance metrics?
>>
>> In short, the *Mesos* executor on a given slave is going to be
>> long-running (consuming memory, but no CPUs). Each Spark task will be
>> scheduled using Mesos CPU resources, but they don't suffer much latency.
>>
>> iulian
>>
>>
>> On Mon, Nov 30, 2015 at 4:17 AM, Renjie Liu <li...@gmail.com>
>> wrote:
>>
>>> Hi, Tim:
>>> Fine grain mode is not suitable for streaming applications since it need
>>> to start up an executor each time. When will the revamp get release? In the
>>> coming 1.6.0?
>>>
>>> On Sun, Nov 29, 2015 at 6:16 PM Timothy Chen <ti...@mesosphere.io> wrote:
>>>
>>>> Hi Renjie,
>>>>
>>>> You can set number of cores per executor with spark executor cores in
>>>> fine grain mode.
>>>>
>>>> If you want coarse grain mode to support that it will
>>>> Be supported in the near term as he coarse grain scheduler is getting
>>>> revamped now.
>>>>
>>>> Tim
>>>>
>>>> On Nov 28, 2015, at 7:31 PM, Renjie Liu <li...@gmail.com>
>>>> wrote:
>>>>
>>>> Hi, Nagaraj:
>>>>  Thanks for the response, but this does not solve my problem.
>>>> I think executor memory should be proportional to number of cores, or
>>>> number of core
>>>> in each executor should be the same.
>>>> On Sat, Nov 28, 2015 at 1:48 AM Nagaraj Chandrashekar <
>>>> nchandrashekar@innominds.com> wrote:
>>>>
>>>>> Hi Renjie,
>>>>>
>>>>> I have not setup Spark Streaming on Mesos but there is something
>>>>> called reservations in Mesos.  It supports both Static and Dynamic
>>>>> reservations.  Both types of reservations must have role defined. You may
>>>>> want to explore these options.   Excerpts from the Apache Mesos
>>>>> documentation.
>>>>>
>>>>> Cheers
>>>>> Nagaraj C
>>>>> Reservation
>>>>>
>>>>> Mesos provides mechanisms to reserve resources in specific slaves.
>>>>> The concept was first introduced with static reservation in 0.14.0
>>>>> which enabled operators to specify the reserved resources on slave startup.
>>>>> This was extended with dynamic reservation in 0.23.0 which enabled
>>>>> operators and authorized frameworks to dynamically reserve resources
>>>>> in the cluster.
>>>>>
>>>>> No breaking changes were introduced with dynamic reservation, which
>>>>> means the existing static reservation mechanism continues to be fully
>>>>> supported.
>>>>>
>>>>> In both types of reservations, resources are reserved for a role.
>>>>> Static Reservation (since 0.14.0)
>>>>>
>>>>> An operator can configure a slave with resources reserved for a role.
>>>>> The reserved resources are specified via the --resources flag. For
>>>>> example, suppose we have 12 CPUs and 6144 MB of RAM available on a slave
>>>>> and that we want to reserve 8 CPUs and 4096 MB of RAM for the ads role.
>>>>> We start the slave like so:
>>>>>
>>>>>     $ mesos-slave \
>>>>>       --master=<ip>:<port> \
>>>>>       --resources="cpus:4;mem:2048;cpus(ads):8;mem(ads):4096"
>>>>>
>>>>> We now have 8 CPUs and 4096 MB of RAM reserved for ads on this slave.
>>>>>
>>>>>
>>>>> From: Renjie Liu <li...@gmail.com>
>>>>> Date: Friday, November 27, 2015 at 9:57 PM
>>>>> To: "user@spark.apache.org" <us...@spark.apache.org>
>>>>> Subject: Spark Streaming on mesos
>>>>>
>>>>> Hi, all:
>>>>> I'm trying to run spark streaming on mesos and it seems that none of
>>>>> the scheduler is suitable for that. Fine grain scheduler will start an
>>>>> executor for each task so it will significantly increase the latency. While
>>>>> coarse grained mode can only set the max core numbers and executor memory
>>>>> but there's no way to set the number of cores for each executor. Has anyone
>>>>> deployed spark streaming on mesos? And what's your settings?
>>>>> --
>>>>> Liu, Renjie
>>>>> Software Engineer, MVAD
>>>>>
>>>> --
>>>> Liu, Renjie
>>>> Software Engineer, MVAD
>>>>
>>>> --
>>> Liu, Renjie
>>> Software Engineer, MVAD
>>>
>>
>>
>>
>> --
>>
>> --
>> Iulian Dragos
>>
>> ------
>> Reactive Apps on the JVM
>> www.typesafe.com
>>
>> --
> Liu, Renjie
> Software Engineer, MVAD
>



-- 

--
Iulian Dragos

------
Reactive Apps on the JVM
www.typesafe.com