You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Burak Yavuz (JIRA)" <ji...@apache.org> on 2018/06/15 21:00:00 UTC

[jira] [Resolved] (SPARK-24525) Provide an option to limit MemorySink memory usage

     [ https://issues.apache.org/jira/browse/SPARK-24525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Burak Yavuz resolved SPARK-24525.
---------------------------------
       Resolution: Fixed
    Fix Version/s: 2.4.0

Resolved by [https://github.com/apache/spark/pull/21559]

> Provide an option to limit MemorySink memory usage
> --------------------------------------------------
>
>                 Key: SPARK-24525
>                 URL: https://issues.apache.org/jira/browse/SPARK-24525
>             Project: Spark
>          Issue Type: Improvement
>          Components: Structured Streaming
>    Affects Versions: 2.3.1
>            Reporter: Mukul Murthy
>            Assignee: Mukul Murthy
>            Priority: Major
>             Fix For: 2.4.0
>
>
> MemorySink stores stream results in memory and is mostly used for testing and displaying streams, but for large streams, this can OOM the driver. We should add an option to limit the number of rows and the total size of a memory sink and not add any new data once either limit is hit. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org