Replacing Spark's native scheduler with Sparrow

[Nicholas Chammas <ni...@gmail.com>]

I just watched Kay's talk from 2013 on Sparrow
<https://www.youtube.com/watch?v=ayjH_bG-RC0>. Is replacing Spark's native
scheduler with Sparrow still on the books?

The Sparrow repo <https://github.com/radlab/sparrow> hasn't been updated
recently, and I don't see any JIRA issues about it.

It would be good to at least have a JIRA issue to track progress on this if
it's a long-term goal.

Nick

Re: Replacing Spark's native scheduler with Sparrow

[Nicholas Chammas <ni...@gmail.com>]

Yeah, perhaps I misunderstood what Michael was saying. But thanks for
pointing out the relevant UI functionality.

On Sat, Nov 8, 2014 at 1:22 AM, Kay Ousterhout <ke...@eecs.berkeley.edu>
wrote:

> I don't have much more info than what Shivaram said.  My sense is that,
> over time, task launch overhead with Spark has slowly grown as Spark
> supports more and more functionality.  However, I haven't seen it be as
> high as the 100ms Michael quoted (maybe this was for jobs with tasks that
> have much larger objects that take a long time to deserialize?).
> Fortunately, the UI now quantifies this: if you click "Show Additional
> Metrics", the scheduler delay (which basically represents the overhead of
> shipping the task to the worker and getting the result back), the task
> deserialization time, and the result serialization time all represent parts
> of the task launch overhead.  So, you can use the UI to get a sense of what
> this overhead is for the workload you're considering and whether it's worth
> optimizing.
>
> -Kay
>
> On Fri, Nov 7, 2014 at 9:43 PM, Shivaram Venkataraman <
> shivaram@eecs.berkeley.edu> wrote:
>
>> I think Kay might be able to give a better answer. The most recent
>> benchmark I remember had the number at at somewhere between 8.6ms and
>> 14.6ms depending on the Spark version (
>> https://github.com/apache/spark/pull/2030#issuecomment-52715181).
>> Another point to note is that this is the total time to run a null job, so
>> this includes scheduling + task launch + time to send back results etc.
>>
>> Shivaram
>>
>> On Fri, Nov 7, 2014 at 9:23 PM, Nicholas Chammas <
>> nicholas.chammas@gmail.com> wrote:
>>
>>> Hmm, relevant quote from section 3.3:
>>>
>>> newer frameworks like Spark [35] reduce the overhead to 5ms. To support
>>>> tasks that complete in hundreds of mil- liseconds, we argue for reducing
>>>> task launch overhead even further to 1ms so that launch overhead
>>>> constitutes at most 1% of task runtime. By maintaining an active thread
>>>> pool for task execution on each worker node and caching binaries, task
>>>> launch overhead can be reduced to the time to make a remote procedure call
>>>> to the slave machine to launch the task. Today’s datacenter networks easily
>>>> allow a RPC to complete within 1ms. In fact, re- cent work showed that 10μs
>>>> RPCs are possible in the short term [26]; thus, with careful engineering,
>>>> we be- lieve task launch overheads of 50μ s are attainable. 50μ s task
>>>> launch overheads would enable even smaller tasks that could read data from
>>>> in-memory or from flash stor- age in order to complete in milliseconds.
>>>
>>>
>>> So it looks like I misunderstood the current cost of task
>>> initialization. It's already as low as 5ms (and not 100ms)?
>>>
>>> Nick
>>>
>>> On Fri, Nov 7, 2014 at 11:15 PM, Shivaram Venkataraman <
>>> shivaram@eecs.berkeley.edu> wrote:
>>>
>>>>
>>>>
>>>> On Fri, Nov 7, 2014 at 8:04 PM, Nicholas Chammas <
>>>> nicholas.chammas@gmail.com> wrote:
>>>>
>>>>> Sounds good. I'm looking forward to tracking improvements in this area.
>>>>>
>>>>> Also, just to connect some more dots here, I just remembered that
>>>>> there is
>>>>> currently an initiative to add an IndexedRDD
>>>>> <https://issues.apache.org/jira/browse/SPARK-2365> interface. Some
>>>>> interesting use cases mentioned there include (emphasis added):
>>>>>
>>>>> To address these problems, we propose IndexedRDD, an efficient
>>>>> key-value
>>>>> > store built on RDDs. IndexedRDD would extend RDD[(Long, V)] by
>>>>> enforcing
>>>>> > key uniqueness and pre-indexing the entries for efficient joins and
>>>>> *point
>>>>> > lookups, updates, and deletions*.
>>>>>
>>>>>
>>>>> GraphX would be the first user of IndexedRDD, since it currently
>>>>> implements
>>>>> > a limited form of this functionality in VertexRDD. We envision a
>>>>> variety of
>>>>> > other uses for IndexedRDD, including *streaming updates* to RDDs,
>>>>> *direct
>>>>> > serving* from RDDs, and as an execution strategy for Spark SQL.
>>>>>
>>>>>
>>>>> Maybe some day we'll have Spark clusters directly serving up point
>>>>> lookups
>>>>> or updates. I imagine the tasks running on clusters like that would be
>>>>> tiny
>>>>> and would benefit from very low task startup times and scheduling
>>>>> latency.
>>>>> Am I painting that picture correctly?
>>>>>
>>>>> Yeah - we painted a similar picture in a short paper last year titled
>>>> "The Case for Tiny Tasks in Compute Clusters"
>>>> http://shivaram.org/publications/tinytasks-hotos13.pdf
>>>>
>>>>> Anyway, thanks for explaining the current status of Sparrow.
>>>>>
>>>>> Nick
>>>>>
>>>>
>>>>
>>>
>>
>

Re: Replacing Spark's native scheduler with Sparrow

[Tathagata Das <ta...@gmail.com>]

Too bad Nick, I dont have anything immediately ready that tests Spark
Streaming with those extreme settings. :)

On Mon, Nov 10, 2014 at 9:56 AM, Nicholas Chammas
<ni...@gmail.com> wrote:
> On Sun, Nov 9, 2014 at 1:51 AM, Tathagata Das <ta...@gmail.com>
> wrote:
>>
>> This causes a scalability vs. latency tradeoff - if your limit is 1000
>> tasks per second (simplifying from 1500), you could either configure
>> it to use 100 receivers at 100 ms batches (10 blocks/sec), or 1000
>> receivers at 1 second batches.
>
>
> This raises an interesting question, TD.
>
> Do we have a benchmark for Spark Streaming that tests it at the extreme for
> some key metric, perhaps processed messages per second per node? Something
> that would press Spark's ability to process tasks quickly enough.
>
> Given such a benchmark, it would probably be interesting to see how -- if at
> all -- Sparrow has an impact on Spark Streaming's performance.
>
> Nick

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org

Re: Replacing Spark's native scheduler with Sparrow

[Nicholas Chammas <ni...@gmail.com>]

On Sun, Nov 9, 2014 at 1:51 AM, Tathagata Das <ta...@gmail.com>
wrote:

> This causes a scalability vs. latency tradeoff - if your limit is 1000
> tasks per second (simplifying from 1500), you could either configure
> it to use 100 receivers at 100 ms batches (10 blocks/sec), or 1000
> receivers at 1 second batches.
>

This raises an interesting question, TD.

Do we have a benchmark for Spark Streaming that tests it at the extreme for
some key metric, perhaps processed messages per second per node? Something
that would press Spark's ability to process tasks quickly enough.

Given such a benchmark, it would probably be interesting to see how -- if
at all -- Sparrow has an impact on Spark Streaming's performance.

Nick

Re: Replacing Spark's native scheduler with Sparrow

[Tathagata Das <ta...@gmail.com>]

Let me chime in on the discussion as well. Spark Streaming is another
usecase where the scheduler's task-launching throughput and
task-latency can limit the batch interval and the overall latencies
achievable by Spark Streaming. Lets say we want to do batches of 20 ms
(for achieve end-to-end latencies < 50ms) with 100 receivers. If each
receiver chunks received data into 10 ms blocks (required for making
batches every 10 ms), then launches tasks to process those blocks, it
means 100 blocks / second X 100 receivers = 10000 tasks per seconds.
This causes a scalability vs. latency tradeoff - if your limit is 1000
tasks per second (simplifying from 1500), you could either configure
it to use 100 receivers at 100 ms batches (10 blocks/sec), or 1000
receivers at 1 second batches.

Note that I did the calculation without considering processing power
of the worker nodes, task closure size, workload characteristics, etc
and other bottlenecks in the system. This calculation therefore does
not reflect the current performance limits of Spark Streaming, which
are most likely much lower than above due to other overheads. However,
as we optimize things further, I know that the scheduler is going to
be one of the biggest hurdles, and only out-of-the-box approaches like
Sparrow can solve it.

TD

On Sat, Nov 8, 2014 at 10:26 AM, Michael Armbrust
<mi...@databricks.com> wrote:
>>
>> However, I haven't seen it be as
>> high as the 100ms Michael quoted (maybe this was for jobs with tasks that
>> have much larger objects that take a long time to deserialize?).
>>
>
> I was thinking more about the average end-to-end latency for launching a
> query that has 100s of partitions. Its also quite possible that SQLs task
> launch overhead is higher since we have never profiled how much is getting
> pulled into the closures.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org

Re: Replacing Spark's native scheduler with Sparrow

[Michael Armbrust <mi...@databricks.com>]

>
> However, I haven't seen it be as
> high as the 100ms Michael quoted (maybe this was for jobs with tasks that
> have much larger objects that take a long time to deserialize?).
>

I was thinking more about the average end-to-end latency for launching a
query that has 100s of partitions. Its also quite possible that SQLs task
launch overhead is higher since we have never profiled how much is getting
pulled into the closures.

Re: Replacing Spark's native scheduler with Sparrow

[Kay Ousterhout <ke...@eecs.berkeley.edu>]

I don't have much more info than what Shivaram said.  My sense is that,
over time, task launch overhead with Spark has slowly grown as Spark
supports more and more functionality.  However, I haven't seen it be as
high as the 100ms Michael quoted (maybe this was for jobs with tasks that
have much larger objects that take a long time to deserialize?).
Fortunately, the UI now quantifies this: if you click "Show Additional
Metrics", the scheduler delay (which basically represents the overhead of
shipping the task to the worker and getting the result back), the task
deserialization time, and the result serialization time all represent parts
of the task launch overhead.  So, you can use the UI to get a sense of what
this overhead is for the workload you're considering and whether it's worth
optimizing.

-Kay

On Fri, Nov 7, 2014 at 9:43 PM, Shivaram Venkataraman <
shivaram@eecs.berkeley.edu> wrote:

> I think Kay might be able to give a better answer. The most recent
> benchmark I remember had the number at at somewhere between 8.6ms and
> 14.6ms depending on the Spark version (
> https://github.com/apache/spark/pull/2030#issuecomment-52715181). Another
> point to note is that this is the total time to run a null job, so this
> includes scheduling + task launch + time to send back results etc.
>
> Shivaram
>
> On Fri, Nov 7, 2014 at 9:23 PM, Nicholas Chammas <
> nicholas.chammas@gmail.com> wrote:
>
>> Hmm, relevant quote from section 3.3:
>>
>> newer frameworks like Spark [35] reduce the overhead to 5ms. To support
>>> tasks that complete in hundreds of mil- liseconds, we argue for reducing
>>> task launch overhead even further to 1ms so that launch overhead
>>> constitutes at most 1% of task runtime. By maintaining an active thread
>>> pool for task execution on each worker node and caching binaries, task
>>> launch overhead can be reduced to the time to make a remote procedure call
>>> to the slave machine to launch the task. Today’s datacenter networks easily
>>> allow a RPC to complete within 1ms. In fact, re- cent work showed that 10μs
>>> RPCs are possible in the short term [26]; thus, with careful engineering,
>>> we be- lieve task launch overheads of 50μ s are attainable. 50μ s task
>>> launch overheads would enable even smaller tasks that could read data from
>>> in-memory or from flash stor- age in order to complete in milliseconds.
>>
>>
>> So it looks like I misunderstood the current cost of task initialization.
>> It's already as low as 5ms (and not 100ms)?
>>
>> Nick
>>
>> On Fri, Nov 7, 2014 at 11:15 PM, Shivaram Venkataraman <
>> shivaram@eecs.berkeley.edu> wrote:
>>
>>>
>>>
>>> On Fri, Nov 7, 2014 at 8:04 PM, Nicholas Chammas <
>>> nicholas.chammas@gmail.com> wrote:
>>>
>>>> Sounds good. I'm looking forward to tracking improvements in this area.
>>>>
>>>> Also, just to connect some more dots here, I just remembered that there
>>>> is
>>>> currently an initiative to add an IndexedRDD
>>>> <https://issues.apache.org/jira/browse/SPARK-2365> interface. Some
>>>> interesting use cases mentioned there include (emphasis added):
>>>>
>>>> To address these problems, we propose IndexedRDD, an efficient key-value
>>>> > store built on RDDs. IndexedRDD would extend RDD[(Long, V)] by
>>>> enforcing
>>>> > key uniqueness and pre-indexing the entries for efficient joins and
>>>> *point
>>>> > lookups, updates, and deletions*.
>>>>
>>>>
>>>> GraphX would be the first user of IndexedRDD, since it currently
>>>> implements
>>>> > a limited form of this functionality in VertexRDD. We envision a
>>>> variety of
>>>> > other uses for IndexedRDD, including *streaming updates* to RDDs,
>>>> *direct
>>>> > serving* from RDDs, and as an execution strategy for Spark SQL.
>>>>
>>>>
>>>> Maybe some day we'll have Spark clusters directly serving up point
>>>> lookups
>>>> or updates. I imagine the tasks running on clusters like that would be
>>>> tiny
>>>> and would benefit from very low task startup times and scheduling
>>>> latency.
>>>> Am I painting that picture correctly?
>>>>
>>>> Yeah - we painted a similar picture in a short paper last year titled
>>> "The Case for Tiny Tasks in Compute Clusters"
>>> http://shivaram.org/publications/tinytasks-hotos13.pdf
>>>
>>>> Anyway, thanks for explaining the current status of Sparrow.
>>>>
>>>> Nick
>>>>
>>>
>>>
>>
>

Re: Replacing Spark's native scheduler with Sparrow

[Shivaram Venkataraman <sh...@eecs.berkeley.edu>]

I think Kay might be able to give a better answer. The most recent
benchmark I remember had the number at at somewhere between 8.6ms and
14.6ms depending on the Spark version (
https://github.com/apache/spark/pull/2030#issuecomment-52715181). Another
point to note is that this is the total time to run a null job, so this
includes scheduling + task launch + time to send back results etc.

Shivaram

On Fri, Nov 7, 2014 at 9:23 PM, Nicholas Chammas <nicholas.chammas@gmail.com
> wrote:

> Hmm, relevant quote from section 3.3:
>
> newer frameworks like Spark [35] reduce the overhead to 5ms. To support
>> tasks that complete in hundreds of mil- liseconds, we argue for reducing
>> task launch overhead even further to 1ms so that launch overhead
>> constitutes at most 1% of task runtime. By maintaining an active thread
>> pool for task execution on each worker node and caching binaries, task
>> launch overhead can be reduced to the time to make a remote procedure call
>> to the slave machine to launch the task. Today’s datacenter networks easily
>> allow a RPC to complete within 1ms. In fact, re- cent work showed that 10μs
>> RPCs are possible in the short term [26]; thus, with careful engineering,
>> we be- lieve task launch overheads of 50μ s are attainable. 50μ s task
>> launch overheads would enable even smaller tasks that could read data from
>> in-memory or from flash stor- age in order to complete in milliseconds.
>
>
> So it looks like I misunderstood the current cost of task initialization.
> It's already as low as 5ms (and not 100ms)?
>
> Nick
>
> On Fri, Nov 7, 2014 at 11:15 PM, Shivaram Venkataraman <
> shivaram@eecs.berkeley.edu> wrote:
>
>>
>>
>> On Fri, Nov 7, 2014 at 8:04 PM, Nicholas Chammas <
>> nicholas.chammas@gmail.com> wrote:
>>
>>> Sounds good. I'm looking forward to tracking improvements in this area.
>>>
>>> Also, just to connect some more dots here, I just remembered that there
>>> is
>>> currently an initiative to add an IndexedRDD
>>> <https://issues.apache.org/jira/browse/SPARK-2365> interface. Some
>>> interesting use cases mentioned there include (emphasis added):
>>>
>>> To address these problems, we propose IndexedRDD, an efficient key-value
>>> > store built on RDDs. IndexedRDD would extend RDD[(Long, V)] by
>>> enforcing
>>> > key uniqueness and pre-indexing the entries for efficient joins and
>>> *point
>>> > lookups, updates, and deletions*.
>>>
>>>
>>> GraphX would be the first user of IndexedRDD, since it currently
>>> implements
>>> > a limited form of this functionality in VertexRDD. We envision a
>>> variety of
>>> > other uses for IndexedRDD, including *streaming updates* to RDDs,
>>> *direct
>>> > serving* from RDDs, and as an execution strategy for Spark SQL.
>>>
>>>
>>> Maybe some day we'll have Spark clusters directly serving up point
>>> lookups
>>> or updates. I imagine the tasks running on clusters like that would be
>>> tiny
>>> and would benefit from very low task startup times and scheduling
>>> latency.
>>> Am I painting that picture correctly?
>>>
>>> Yeah - we painted a similar picture in a short paper last year titled
>> "The Case for Tiny Tasks in Compute Clusters"
>> http://shivaram.org/publications/tinytasks-hotos13.pdf
>>
>>> Anyway, thanks for explaining the current status of Sparrow.
>>>
>>> Nick
>>>
>>
>>
>

Re: Replacing Spark's native scheduler with Sparrow

[Nicholas Chammas <ni...@gmail.com>]

Hmm, relevant quote from section 3.3:

newer frameworks like Spark [35] reduce the overhead to 5ms. To support
> tasks that complete in hundreds of mil- liseconds, we argue for reducing
> task launch overhead even further to 1ms so that launch overhead
> constitutes at most 1% of task runtime. By maintaining an active thread
> pool for task execution on each worker node and caching binaries, task
> launch overhead can be reduced to the time to make a remote procedure call
> to the slave machine to launch the task. Today’s datacenter networks easily
> allow a RPC to complete within 1ms. In fact, re- cent work showed that 10μs
> RPCs are possible in the short term [26]; thus, with careful engineering,
> we be- lieve task launch overheads of 50μ s are attainable. 50μ s task
> launch overheads would enable even smaller tasks that could read data from
> in-memory or from flash stor- age in order to complete in milliseconds.


So it looks like I misunderstood the current cost of task initialization.
It's already as low as 5ms (and not 100ms)?

Nick

On Fri, Nov 7, 2014 at 11:15 PM, Shivaram Venkataraman <
shivaram@eecs.berkeley.edu> wrote:

>
>
> On Fri, Nov 7, 2014 at 8:04 PM, Nicholas Chammas <
> nicholas.chammas@gmail.com> wrote:
>
>> Sounds good. I'm looking forward to tracking improvements in this area.
>>
>> Also, just to connect some more dots here, I just remembered that there is
>> currently an initiative to add an IndexedRDD
>> <https://issues.apache.org/jira/browse/SPARK-2365> interface. Some
>> interesting use cases mentioned there include (emphasis added):
>>
>> To address these problems, we propose IndexedRDD, an efficient key-value
>> > store built on RDDs. IndexedRDD would extend RDD[(Long, V)] by enforcing
>> > key uniqueness and pre-indexing the entries for efficient joins and
>> *point
>> > lookups, updates, and deletions*.
>>
>>
>> GraphX would be the first user of IndexedRDD, since it currently
>> implements
>> > a limited form of this functionality in VertexRDD. We envision a
>> variety of
>> > other uses for IndexedRDD, including *streaming updates* to RDDs,
>> *direct
>> > serving* from RDDs, and as an execution strategy for Spark SQL.
>>
>>
>> Maybe some day we'll have Spark clusters directly serving up point lookups
>> or updates. I imagine the tasks running on clusters like that would be
>> tiny
>> and would benefit from very low task startup times and scheduling latency.
>> Am I painting that picture correctly?
>>
>> Yeah - we painted a similar picture in a short paper last year titled
> "The Case for Tiny Tasks in Compute Clusters"
> http://shivaram.org/publications/tinytasks-hotos13.pdf
>
>> Anyway, thanks for explaining the current status of Sparrow.
>>
>> Nick
>>
>
>

Re: Replacing Spark's native scheduler with Sparrow

[Shivaram Venkataraman <sh...@eecs.berkeley.edu>]

On Fri, Nov 7, 2014 at 8:04 PM, Nicholas Chammas <nicholas.chammas@gmail.com
> wrote:

> Sounds good. I'm looking forward to tracking improvements in this area.
>
> Also, just to connect some more dots here, I just remembered that there is
> currently an initiative to add an IndexedRDD
> <https://issues.apache.org/jira/browse/SPARK-2365> interface. Some
> interesting use cases mentioned there include (emphasis added):
>
> To address these problems, we propose IndexedRDD, an efficient key-value
> > store built on RDDs. IndexedRDD would extend RDD[(Long, V)] by enforcing
> > key uniqueness and pre-indexing the entries for efficient joins and
> *point
> > lookups, updates, and deletions*.
>
>
> GraphX would be the first user of IndexedRDD, since it currently implements
> > a limited form of this functionality in VertexRDD. We envision a variety
> of
> > other uses for IndexedRDD, including *streaming updates* to RDDs, *direct
> > serving* from RDDs, and as an execution strategy for Spark SQL.
>
>
> Maybe some day we'll have Spark clusters directly serving up point lookups
> or updates. I imagine the tasks running on clusters like that would be tiny
> and would benefit from very low task startup times and scheduling latency.
> Am I painting that picture correctly?
>
> Yeah - we painted a similar picture in a short paper last year titled "The
Case for Tiny Tasks in Compute Clusters"
http://shivaram.org/publications/tinytasks-hotos13.pdf

> Anyway, thanks for explaining the current status of Sparrow.
>
> Nick
>

Re: Replacing Spark's native scheduler with Sparrow

[Nicholas Chammas <ni...@gmail.com>]

Sounds good. I'm looking forward to tracking improvements in this area.

Also, just to connect some more dots here, I just remembered that there is
currently an initiative to add an IndexedRDD
<https://issues.apache.org/jira/browse/SPARK-2365> interface. Some
interesting use cases mentioned there include (emphasis added):

To address these problems, we propose IndexedRDD, an efficient key-value
> store built on RDDs. IndexedRDD would extend RDD[(Long, V)] by enforcing
> key uniqueness and pre-indexing the entries for efficient joins and *point
> lookups, updates, and deletions*.


GraphX would be the first user of IndexedRDD, since it currently implements
> a limited form of this functionality in VertexRDD. We envision a variety of
> other uses for IndexedRDD, including *streaming updates* to RDDs, *direct
> serving* from RDDs, and as an execution strategy for Spark SQL.


Maybe some day we'll have Spark clusters directly serving up point lookups
or updates. I imagine the tasks running on clusters like that would be tiny
and would benefit from very low task startup times and scheduling latency.
Am I painting that picture correctly?

Anyway, thanks for explaining the current status of Sparrow.

Nick

Re: Replacing Spark's native scheduler with Sparrow

[Kay Ousterhout <ke...@eecs.berkeley.edu>]

On Fri, Nov 7, 2014 at 6:20 PM, Nicholas Chammas <nicholas.chammas@gmail.com
> wrote:

> If, for example, you have a cluster of 100 machines, this means the
>> scheduler can launch 150 tasks per machine per second.
>
>
> Did you mean 15 tasks per machine per second here? Or alternatively, 10
> machines?
>
Yes -- sorry for the terrible math there!

>
> I don't know of any existing Spark clusters that have a large enough
>> number of machines or short enough tasks to justify the added complexity of
>> distributing the scheduler.
>
>
> Actually, this was the reason I took interest in Sparrow--specifically,
> the idea of a Spark cluster handling many very short (<< 50 ms) tasks.
>
> At the recent Spark Committer Night
> <http://www.meetup.com/Spark-NYC/events/209271842/> in NYC, I asked
> Michael if he thought that Spark SQL could eventually completely fill the
> need for very low latency queries currently served by MPP databases like
> Redshift or Vertica. If I recall correctly, he said that the main obstacle
> to that was simply task startup time, which is on the order of 100 ms.
>
> Is there interest in (or perhaps an existing initiative related to)
> improving task startup times to the point where one could legitimately look
> at Spark SQL as a low latency database that can serve many users or
> applications at once? That would probably make a good use case for Sparrow,
> no?
>

Shorter tasks would indeed be a good use case for Sparrow, and was the
motivation behind the Sparrow work.  When evaluating Sparrow, we focused on
running SQL workloads where tasks were in the 50-100ms range (detailed in
the paper <http://people.csail.mit.edu/matei/papers/2013/sosp_sparrow.pdf>).

I know Evan, who I added here, has been looking at task startup times in
the context of ML workloads; this motivated some recent work (e.g.,
https://issues.apache.org/jira/browse/SPARK-3984) to improve metrics shown
in the UI to describe task launch overhead.  For jobs we've looked at, task
startup time was at most tens of milliseconds (I also remember this being
the case when we ran short tasks on Sparrow).  Decreasing this seems like
it would be widely beneficial, especially if there are cases where it's
more like 100ms, as Michael alluded.  Hopefully some of the improved UI
reporting will help to understand the degree to which this is (or is not)
an issue.  I'm not sure how much Evan is attempting to quantify the
overhead versus fix it -- so I'll let him chime in here.


> Nick
>
>

Re: Replacing Spark's native scheduler with Sparrow

[Nicholas Chammas <ni...@gmail.com>]

>
> If, for example, you have a cluster of 100 machines, this means the
> scheduler can launch 150 tasks per machine per second.


Did you mean 15 tasks per machine per second here? Or alternatively, 10
machines?

I don't know of any existing Spark clusters that have a large enough number
> of machines or short enough tasks to justify the added complexity of
> distributing the scheduler.


Actually, this was the reason I took interest in Sparrow--specifically, the
idea of a Spark cluster handling many very short (<< 50 ms) tasks.

At the recent Spark Committer Night
<http://www.meetup.com/Spark-NYC/events/209271842/> in NYC, I asked Michael
if he thought that Spark SQL could eventually completely fill the need for
very low latency queries currently served by MPP databases like Redshift or
Vertica. If I recall correctly, he said that the main obstacle to that was
simply task startup time, which is on the order of 100 ms.

Is there interest in (or perhaps an existing initiative related to)
improving task startup times to the point where one could legitimately look
at Spark SQL as a low latency database that can serve many users or
applications at once? That would probably make a good use case for Sparrow,
no?

Nick

Re: Replacing Spark's native scheduler with Sparrow

[Kay Ousterhout <ke...@eecs.berkeley.edu>]

Hi Nick,

This hasn't yet been directly supported by Spark because of a lack of
demand.  The last time I ran a throughput test on the default Spark
scheduler (~1 year ago, so this may have changed), it could launch
approximately 1500 tasks / second.  If, for example, you have a cluster of
100 machines, this means the scheduler can launch 150 tasks per machine per
second.  I don't know of any existing Spark clusters that have a large
enough number of machines or short enough tasks to justify the added
complexity of distributing the scheduler.  Eventually I hope to see Spark
used on much larger clusters, such that Sparrow will be necessary!

-Kay

On Fri, Nov 7, 2014 at 3:05 PM, Nicholas Chammas <nicholas.chammas@gmail.com
> wrote:

> I just watched Kay's talk from 2013 on Sparrow
> <https://www.youtube.com/watch?v=ayjH_bG-RC0>. Is replacing Spark's native
> scheduler with Sparrow still on the books?
>
> The Sparrow repo <https://github.com/radlab/sparrow> hasn't been updated
> recently, and I don't see any JIRA issues about it.
>
> It would be good to at least have a JIRA issue to track progress on this if
> it's a long-term goal.
>
> Nick
>