You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@systemml.apache.org by "LI Guobao (JIRA)" <ji...@apache.org> on 2018/06/24 20:37:00 UTC

[jira] [Updated] (SYSTEMML-2418) Distributing data to workers

     [ https://issues.apache.org/jira/browse/SYSTEMML-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

LI Guobao updated SYSTEMML-2418:
--------------------------------
    Description: In the context of ps, the training data will be partitioned according to the different schemes. This conversion is executed in driver node and the partitioned data should be distributed to workers via broadcast. Due to the 2G limitation of spark broadcast, we could leverage the _PartitionedBroadcast_ class to do this conversion. Afterwards, the partitioned broadcast object can be passed to workers for launching its job.

> Distributing data to workers
> ----------------------------
>
>                 Key: SYSTEMML-2418
>                 URL: https://issues.apache.org/jira/browse/SYSTEMML-2418
>             Project: SystemML
>          Issue Type: Sub-task
>            Reporter: LI Guobao
>            Assignee: LI Guobao
>            Priority: Major
>
> In the context of ps, the training data will be partitioned according to the different schemes. This conversion is executed in driver node and the partitioned data should be distributed to workers via broadcast. Due to the 2G limitation of spark broadcast, we could leverage the _PartitionedBroadcast_ class to do this conversion. Afterwards, the partitioned broadcast object can be passed to workers for launching its job.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)