[jira] [Commented] (SPARK-14230) Config the start time (jitter) for streaming jobs

["Apache Spark (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-14230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15215403#comment-15215403 ] 

Apache Spark commented on SPARK-14230:
--------------------------------------

User 'liyintang' has created a pull request for this issue:
https://github.com/apache/spark/pull/12026

> Config the start time (jitter) for streaming jobs
> -------------------------------------------------
>
>                 Key: SPARK-14230
>                 URL: https://issues.apache.org/jira/browse/SPARK-14230
>             Project: Spark
>          Issue Type: Improvement
>          Components: Streaming
>            Reporter: Liyin Tang
>
> Currently, RecurringTimer will normalize the start time. For instance, if batch duration is 1 min, all the job will start exactly at 1 min boundary. 
> This actually adds some burden to the streaming source. Assuming the source is Kafka, and there is a list of streaming jobs with 1 min batch duration, then at first few seconds of each min, high network traffic will be observed in Kafka. This makes Kafka capacity planning tricky. 
> It will be great to have an option in the streaming context to set the job start time. In this way, user can add a jitter for the start time for each, and make Kafka fetch_request much smooth across the duration window.
> {code}
> class RecurringTimer {
>   def getStartTime(): Long = {
>     (math.floor(clock.currentTime.toDouble / period) + 1).toLong * period + jitter
>   }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org