You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by wobuxiangtong <gi...@git.apache.org> on 2017/12/14 11:02:45 UTC

[GitHub] spark issue #16374: [SPARK-18925][STREAMING] Reduce memory usage of mapWithS...

Github user wobuxiangtong commented on the issue:

    https://github.com/apache/spark/pull/16374
  
    Excuse me：
        I am using mapwithstate to storage data in sparkstreaming。What confused me is that rememberDuration must more than checkpointDuration。I read the code of MapWithStateRDD and changed is as the follow question and have done some tests  which indicated that it does not matter even though hold only one RDD in storage and no data will lose。it will cut down a lot of memory if doing so like this.
        Maybe i am wrong. Could you help me? the question is here:
    https://stackoverflow.com/questions/47784439/what-will-happen-if-i-change-the-rememberduration-in-dstream-less-than-checkpoin


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org