[jira] [Commented] (SPARK-18371) Spark Streaming backpressure bug - generates a batch with large number of records

["Sebastian Arzt (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-18371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15984776#comment-15984776 ] 

Sebastian Arzt commented on SPARK-18371:
----------------------------------------

I deep dived into it and found a simple solution. The problem is that [maxRateLimitPerPartition|https://github.com/apache/spark/blob/branch-2.0/external/kafka-0-8/src/main/scala/org/apache/spark/streaming/kafka/DirectKafkaInputDStream.scala#L94] returns {{None}} for an unintended case. {{None}} should only be returned if there is no lag as indicated by this [condition|https://github.com/apache/spark/blob/branch-2.0/external/kafka-0-8/src/main/scala/org/apache/spark/streaming/kafka/DirectKafkaInputDStream.scala#L114]. However, this condition is also true if all backpressureRates are [rounded|https://github.com/apache/spark/blob/branch-2.0/external/kafka-0-8/src/main/scala/org/apache/spark/streaming/kafka/DirectKafkaInputDStream.scala#L107] to zero. I propose a solution, where rounding is omitted at all. I has the nice side-effect that the backpressure in more fine-grained and not only an integral multiple of the [batchDuration|https://github.com/apache/spark/blob/branch-2.0/external/kafka-0-8/src/main/scala/org/apache/spark/streaming/kafka/DirectKafkaInputDStream.scala#L117] in seconds. I will open a pull request for it soon.

> Spark Streaming backpressure bug - generates a batch with large number of records
> ---------------------------------------------------------------------------------
>
>                 Key: SPARK-18371
>                 URL: https://issues.apache.org/jira/browse/SPARK-18371
>             Project: Spark
>          Issue Type: Bug
>          Components: DStreams
>    Affects Versions: 2.0.0
>            Reporter: mapreduced
>         Attachments: GiantBatch2.png, GiantBatch3.png, Giant_batch_at_23_00.png, Look_at_batch_at_22_14.png
>
>
> When the streaming job is configured with backpressureEnabled=true, it generates a GIANT batch of records if the processing time + scheduled delay is (much) larger than batchDuration. This creates a backlog of records like no other and results in batches queueing for hours until it chews through this giant batch.
> Expectation is that it should reduce the number of records per batch in some time to whatever it can really process.
> Attaching some screen shots where it seems that this issue is quite easily reproducible.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org