You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@spark.apache.org by "wyphao.2007" <wy...@163.com> on 2015/03/31 04:46:27 UTC

How to get removed RDD from windows？

I want to get removed RDD from windows as follow, The old RDDs will removed from current window, 
//  _____________________________
// |  previous window   _________|___________________
// |___________________|       current window        |  --------------> Time
//                     |_____________________________|
//
// |________ _________|          |________ _________|
//          |                             |
//          V                             V
//       old RDDs                     new RDDs
//
I find  the slice function in DStream class can return the DStream between fromTime to  toTime. But when I use the function as follow:


    val now = System.currentTimeMillis()
    result.slice(new Time(now - 30 * 1000), new Time(now - 30 * 1000 + result.slideDuration.milliseconds)).foreach(item => println("xxx" + item))
    ssc.start()


30 is the window's duration,Then I got zeroTime has not been initialized exception. 


Is anyone can help me? thx!