You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Andrew Or (JIRA)" <ji...@apache.org> on 2015/12/03 20:23:11 UTC

[jira] [Created] (SPARK-12133) Support dynamic allocation in Spark Streaming

Andrew Or created SPARK-12133:
---------------------------------

             Summary: Support dynamic allocation in Spark Streaming
                 Key: SPARK-12133
                 URL: https://issues.apache.org/jira/browse/SPARK-12133
             Project: Spark
          Issue Type: Bug
          Components: Spark Core, Streaming
            Reporter: Andrew Or


Dynamic allocation is a feature that allows your cluster resources to scale up and down based on the workload. Currently it doesn't work well with Spark streaming because of several reasons:

(1) Your executors may never be idle since they run something every N seconds
(2) You should have at least one receiver running always
(3) The existing heuristics don't take into account length of batch queue
...

The goal of this JIRA is to provide better support for using dynamic allocation in streaming. The PR will revert the changes added in SPARK-10955, which warns against using dynamic allocation with streaming.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org