You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Yuri Makhno (JIRA)" <ji...@apache.org> on 2015/06/07 13:29:00 UTC

[jira] [Created] (SPARK-8147) Add ability to decorate RDD iterators

Yuri Makhno created SPARK-8147:
----------------------------------

             Summary: Add ability to decorate RDD iterators
                 Key: SPARK-8147
                 URL: https://issues.apache.org/jira/browse/SPARK-8147
             Project: Spark
          Issue Type: Improvement
          Components: Spark Core
    Affects Versions: 1.3.1
            Reporter: Yuri Makhno


In Spark all computations are done through iterators which are created by RDD.iterator method. It would be good if we can specify some RDDIteratorDecoratorFactory in SparkConf and be able to decorate all RDD iterators created in executor JVM. 

For us it would be extremely useful because we want to control executor's memory and prevent OutOfMemory on executor but instead fail job with NotEnoughMemory reason in case when we see that we don't have more memory to do this. Also we want to collect some computation statistics on executor.

I can provide PR in case this improvement is approved.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org