You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Sachin Aggarwal <di...@gmail.com> on 2016/02/18 12:14:02 UTC
explaination for parent.slideDuration in ReducedWindowedDStream
While reading code i came across parent.slideDuration in
ReducedWindowedDStream class
val currentWindow = new Interval(currentTime - windowDuration +
parent.slideDuration,
currentTime)
// _____________________________
// | previous window _________|___________________
// |___________________| current window | --------------> Time
// |_____________________________|
//
// |________ _________| |________ _________|
// | |
// V V
// old RDDs new RDDs
//
Here currentWindow can be expresed as currentTime - windowDuration
*for what we need parent.slideDuration.*
same is repeated in further expressions also,
val oldRDDs =
reducedStream.slice(previousWindow.beginTime,
currentWindow.beginTime - parent.slideDuration)
logDebug("# old RDDs = " + oldRDDs.size)
// Get the RDDs of the reduced values in "new time steps"
val newRDDs =
reducedStream.slice(previousWindow.endTime +
parent.slideDuration, currentWindow.endTime)
logDebug("# new RDDs = " + newRDDs.size)
--
Thanks & Regards
Sachin Aggarwal
7760502772