You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@singa.apache.org by "wangwei (JIRA)" <ji...@apache.org> on 2015/09/13 13:06:45 UTC

[jira] [Created] (SINGA-71) Fix a buf from repeated averaging gradients from workers

wangwei created SINGA-71:
----------------------------

             Summary: Fix a buf from repeated averaging gradients from workers
                 Key: SINGA-71
                 URL: https://issues.apache.org/jira/browse/SINGA-71
             Project: Singa
          Issue Type: Bug
            Reporter: wangwei
            Assignee: wangwei


For workers from the same group, if they share parameters, gradients of parameters should be averaged. SINGA averages the gradients by
* aggregates gradients from all workers
* calls Updater::Update() passing the scale_grad = 1.0 / worker_group_size. 
* The Update function multiplies the gradient with scale_grad.

The bug was from the Trainer::HandleUpdate() which averaged the gradients from local workers. The bug has been fixed by removing the averaging operation. Gradients from local workers are (only) summed before sending to servers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)