[jira] [Created] (SPARK-7398) Add back-pressure to Spark Streaming

["François Garillot (JIRA)" <ji...@apache.org>]

François Garillot created SPARK-7398:
----------------------------------------

             Summary: Add back-pressure to Spark Streaming
                 Key: SPARK-7398
                 URL: https://issues.apache.org/jira/browse/SPARK-7398
             Project: Spark
          Issue Type: Bug
          Components: Streaming
    Affects Versions: 1.3.1
            Reporter: François Garillot


Spark Streaming has trouble dealing with situations where 
 batch processing time > batch interval
Meaning a high throughput of input data w.r.t. Spark's ability to remove data from the queue.

If this throughput is sustained for long enough, it leads to an unstable situation where the memory of the Receiver's Executor is overflowed. This aims at transmitting a back-pressure signal back to data ingestion to help with dealing with that high throughput, in a backwards-compatible way.

The design doc can be found here:
https://docs.google.com/document/d/15m8N717n4hwFya_pJHUbiv5s50Mq1SZZzL9ftOhms4E/edit?usp=sharing



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org