You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@systemml.apache.org by "Matthias Boehm (JIRA)" <ji...@apache.org> on 2017/03/10 09:01:04 UTC

[jira] [Created] (SYSTEMML-1390) Avoid unnecessary caching of parfor spark datapartition-execute input

Matthias Boehm created SYSTEMML-1390:
----------------------------------------

             Summary: Avoid unnecessary caching of parfor spark datapartition-execute input
                 Key: SYSTEMML-1390
                 URL: https://issues.apache.org/jira/browse/SYSTEMML-1390
             Project: SystemML
          Issue Type: Sub-task
            Reporter: Matthias Boehm


This task aims to avoid unnecessary input caching for parfor spark datapartition-execute jobs (with grouping) in order to reduce the memory pressure and thus garbage collection overhead during shuffle and subsequent execution. We only apply this for the general case with grouping and if the input is a persisted rdd which has not yet been cached.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)