[jira] [Updated] (SPARK-11152) Streaming UI: Input sizes are 0 for makeup batches started from a checkpoint

["Hyukjin Kwon (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-11152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hyukjin Kwon updated SPARK-11152:
---------------------------------
    Labels: bulk-closed  (was: )

> Streaming UI: Input sizes are 0 for makeup batches started from a checkpoint 
> -----------------------------------------------------------------------------
>
>                 Key: SPARK-11152
>                 URL: https://issues.apache.org/jira/browse/SPARK-11152
>             Project: Spark
>          Issue Type: Bug
>          Components: DStreams, Web UI
>            Reporter: Yongjia Wang
>            Priority: Major
>              Labels: bulk-closed
>
> When a streaming job is resumed from a checkpoint at batch time x, and say the current time when we resume this streaming job is x+10. In this scenario, since Spark will schedule the missing batches from x+1 to x+10 without any metadata, the behavior is to pack up all the backlogged inputs into batch x+1, then assign any new inputs into x+2 to x+10 immediately without waiting. This results in tiny batches that capture inputs only during the back to back scheduling intervals. This behavior is very reasonable. However, the streaming UI does not show correctly the input sizes for all these makeup batches - they are all 0 from batch x to x+10. Fixing this would be very helpful. This happens when I use Kafka direct streaming, I assume this would happen for all other streaming sources as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org