You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dongjoon Hyun (JIRA)" <ji...@apache.org> on 2019/08/05 06:03:00 UTC
[jira] [Resolved] (SPARK-28603) Spark Streaming application
receives inconsistent input events per batch interval
[ https://issues.apache.org/jira/browse/SPARK-28603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dongjoon Hyun resolved SPARK-28603.
-----------------------------------
Resolution: Cannot Reproduce
> Spark Streaming application receives inconsistent input events per batch interval
> ---------------------------------------------------------------------------------
>
> Key: SPARK-28603
> URL: https://issues.apache.org/jira/browse/SPARK-28603
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.6.3
> Reporter: Raja
> Priority: Major
> Attachments: Batch_Inconsistencies.JPG
>
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> We have a 2 sec batch interval for a Spark Streaming application. The Spark is configured to receive from RabbitMQ queue and batch interval was chosen based on the resources available in the Cluster and the processing time taken without causing scheduling delays. For each run we have defined the MaxReceiverRate, BlockInterval and BackPressure enabled to deliver consistent performance for each batch.
> For example, the MaxReceiverRate was given "75", BlockInterval = 50ms and backPressure enabled, we expect for 2 sec batch - 150 msgs should be delivered for a batch to process. Most of the time we are able to achieve this performance, but except for few cases, where few batches will receive "0" events and a following batch receives say 3000 msgs (> greater than the maxReceiverRate). we are not sure of this unexpected behavior of the batch sizing, because of which our application is causing great scheduling delays because of which the application processing is unable to catch up to the incoming msg rates.
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org