[jira] [Created] (SPARK-3466) Limit size of results that a driver collects for each action

["Matei Zaharia (JIRA)" <ji...@apache.org>]

Matei Zaharia created SPARK-3466:
------------------------------------

             Summary: Limit size of results that a driver collects for each action
                 Key: SPARK-3466
                 URL: https://issues.apache.org/jira/browse/SPARK-3466
             Project: Spark
          Issue Type: Improvement
          Components: Spark Core
            Reporter: Matei Zaharia


Right now, operations like collect() and take() can crash the driver if they bring back too many data. We should add a spark.driver.maxResultSize setting (or something like that) that will make the driver abort a job if its result is too big. We can set it to some fraction of the driver's memory by default, or to something like 100 MB.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org