You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@systemml.apache.org by "Matthias Boehm (JIRA)" <ji...@apache.org> on 2018/02/28 04:16:00 UTC

[jira] [Created] (SYSTEMML-2163) Performance large partitioned broadcasts

Matthias Boehm created SYSTEMML-2163:
----------------------------------------

             Summary: Performance large partitioned broadcasts
                 Key: SYSTEMML-2163
                 URL: https://issues.apache.org/jira/browse/SYSTEMML-2163
             Project: SystemML
          Issue Type: Bug
            Reporter: Matthias Boehm


Due to Spark's limitations with broadcasts larger than 2G, in SystemML we use partitioned broadcasts that split a large side input into potentially many broadcast variables. For historic reasons the meta data is still maintained in the individual partitioned blocks. However, for many operations this meta data is accessed on the first partitioned block which leads to potentially unnecessary broadcast fetches.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)