You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@singa.apache.org by "wangwei (JIRA)" <ji...@apache.org> on 2015/08/08 15:35:45 UTC

[jira] [Created] (SINGA-46) Fix a bug in updater.cc to scale the gradients

wangwei created SINGA-46:
----------------------------

             Summary: Fix a bug in updater.cc to scale the gradients
                 Key: SINGA-46
                 URL: https://issues.apache.org/jira/browse/SINGA-46
             Project: Singa
          Issue Type: Bug
            Reporter: wangwei


The Updater's Update function has an argument grad_scale (default value is 1.0), which is used to scale the gradients of parameters. For instance, when n workers compute over one mini-batch (each worker is assigned 1/n records), then their gradients should be averaged. We can do the average by passing grad_scale=1/n.

Some updaters in updater.cc, e.g., AdaGradUpdater, forget to scale the gradients by grad_scale. The bug can be fixed by
{code}
if (grad_scale != 1)
    grad *= grad_scale;
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)