[jira] [Commented] (SINGA-315) Reduce memory footprint by Python generator for parameter gradient

["ASF subversion and git services (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SINGA-315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16022788#comment-16022788 ] 

ASF subversion and git services commented on SINGA-315:
-------------------------------------------------------

Commit ea078dca9cfafdda9d5a15ae2f2823897d217292 in incubator-singa's branch refs/heads/master from wangwei
[ https://git-wip-us.apache.org/repos/asf?p=incubator-singa.git;h=ea078dc ]

SINGA-315 Reduce memory footprint by Python generator for parameter gradient

Update the API of net::backward() function.
1. add arguments dy and output. dy for the input gradient tensor(s), e.g. from the loss functions.
   output is a list of layer names, whose output gradient tensor(s) would be returned in addition to the param gradient tensor(s).
2. returnes a generator iterator that generates (param_names, param_values, param_grads, out_grads) after processing each layer.
The callee function can update the parameters and release the gradient tensors in layerwise.


> Reduce memory footprint by Python generator for parameter gradient
> ------------------------------------------------------------------
>
>                 Key: SINGA-315
>                 URL: https://issues.apache.org/jira/browse/SINGA-315
>             Project: Singa
>          Issue Type: New Feature
>            Reporter: wangwei
>
> The parameter gradient tensors are stored in memory until BP is finished.
> This is not necessary as we can update the parameter once its gradient is ready. Then we free the gradient tensor. 
> In this way, we reduce the memory footprint by avoiding store all gradient tensors.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)