You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Sachin Aggarwal <di...@gmail.com> on 2016/02/18 12:14:02 UTC

explaination for parent.slideDuration in ReducedWindowedDStream

While reading code i came across parent.slideDuration in
ReducedWindowedDStream class

val currentWindow = new Interval(currentTime - windowDuration +
parent.slideDuration,
  currentTime)

//  _____________________________
// |  previous window   _________|___________________
// |___________________|       current window        |  --------------> Time
//                     |_____________________________|
//
// |________ _________|          |________ _________|
//          |                             |
//          V                             V
//       old RDDs                     new RDDs
//

Here currentWindow can be expresed as currentTime - windowDuration
*for what we need parent.slideDuration.*

same is repeated in further expressions also,

val oldRDDs =
      reducedStream.slice(previousWindow.beginTime,
currentWindow.beginTime - parent.slideDuration)
    logDebug("# old RDDs = " + oldRDDs.size)

    // Get the RDDs of the reduced values in "new time steps"
    val newRDDs =
      reducedStream.slice(previousWindow.endTime +
parent.slideDuration, currentWindow.endTime)
    logDebug("# new RDDs = " + newRDDs.size)


-- 

Thanks & Regards

Sachin Aggarwal
7760502772