You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Xi Liu (JIRA)" <ji...@apache.org> on 2014/09/12 08:15:33 UTC

[jira] [Created] (SPARK-3505) Augmenting SparkStreaming updateStateByKey API with timestamp

Xi Liu created SPARK-3505:
-----------------------------

             Summary: Augmenting SparkStreaming updateStateByKey API with timestamp
                 Key: SPARK-3505
                 URL: https://issues.apache.org/jira/browse/SPARK-3505
             Project: Spark
          Issue Type: Improvement
          Components: Streaming
    Affects Versions: 1.0.0
            Reporter: Xi Liu
            Priority: Minor
             Fix For: 1.1.0


The current updateStateByKey API in Spark Streaming does not expose timestamp to the application. 

In our use case, the application need to know the batch timestamp to decide whether to keep the state or not. And we do not want to use real system time because we want to decouple the two (because the same code base is used for streaming and offline processing).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org