[jira] [Created] (SPARK-17539) Streaming Backpressure Starves DirectStream When Used In Combination With Receivers

["Jeff Nadler (JIRA)" <ji...@apache.org>]

Jeff Nadler created SPARK-17539:
-----------------------------------

             Summary: Streaming Backpressure Starves DirectStream When Used In Combination With Receivers
                 Key: SPARK-17539
                 URL: https://issues.apache.org/jira/browse/SPARK-17539
             Project: Spark
          Issue Type: Bug
          Components: Streaming
    Affects Versions: 1.5.0
         Environment: Spark 1.5.0 under CDH 5.5.1 w/ Kafka 0.8.2.1
            Reporter: Jeff Nadler


Backpressure seems to work fine for Direct Streams when used alone.

We use Receivers (x4) to consume Topic A, and a Direct Stream to consume Topic B in a single Streaming app.    Both go through updateStateByKey and are later joined.

In this configuration, backpressure works as expected for the Receivers but not for the Direct Stream.   After running for some time there is no scheduling delay - there is some at startup, but it burns off in a minute or two.    

The Direct Stream will get 'stuck' throttled at 1 msg / partition / sec.   As soon as I turn off backpressure this problem goes away.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org