[jira] [Commented] (SINGA-46) Fix a bug in updater.cc to scale the gradients

["ASF subversion and git services (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SINGA-46?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14693174#comment-14693174 ] 

ASF subversion and git services commented on SINGA-46:
------------------------------------------------------

Commit 6b34ff4e539ade046d916fa2af52af425a304f2d in incubator-singa's branch refs/heads/master from Wei Wang
[ https://git-wip-us.apache.org/repos/asf?p=incubator-singa.git;h=6b34ff4 ]

SINGA-46 Fix a bug in updater.cc to scale the gradients

Scale gradients in Updater::Update() before updating parameters.
Format code in updater.h and updater.cc.


> Fix a bug in updater.cc to scale the gradients
> ----------------------------------------------
>
>                 Key: SINGA-46
>                 URL: https://issues.apache.org/jira/browse/SINGA-46
>             Project: Singa
>          Issue Type: Bug
>            Reporter: wangwei
>
> The Updater's Update function has an argument grad_scale (default value is 1.0), which is used to scale the gradients of parameters. For instance, when n workers compute over one mini-batch (each worker is assigned 1/n records), then their gradients should be averaged. We can do the average by passing grad_scale=1/n.
> Some updaters in updater.cc, e.g., AdaGradUpdater, forget to scale the gradients by grad_scale. The bug can be fixed by
> {code}
> if (grad_scale != 1)
>     grad *= grad_scale;
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)