You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2019/06/05 22:32:56 UTC
[GitHub] [incubator-mxnet] kshitij12345 edited a comment on issue #15120:
[bug] fix higher grad log
kshitij12345 edited a comment on issue #15120: [bug] fix higher grad log
URL: https://github.com/apache/incubator-mxnet/pull/15120#issuecomment-499271979
> @kshitij12345 I have some question about the equation `expected_head_grad = (grad_op(x) * head_grad_grads).asnumpy()` in your test.
>
> My understanding from the chain rule:
>
> ```
> Given y =f(x)
> dL/dx = dL/dy * dy/dx --> this is the first backward pass. Let dL/dy be y_grad, we get dL/dx (noted as x_grad)
>
> Now we rewrite the above the equation:
>
> input0: y_grad
> input1: x
> output: x_grad = y_grad * f'(x)
>
> Another backward pass for this would be:
> dL/d y_grad = dL/d x_grad * f'(x)
> dL/dx = dL/d x_grad * y_grad * f''(x)
> ```
>
> What is the meaning of dL/d y_grad? Are we treating y_grad as another input variable here?
>
> Many thanks for your clarification.
I was following on the basis of this graph that I had in my mind.
Even I am not sure about the mathematical meaning of it. Also in usual scenario it would essentially be an intermediate node. Like in the picture.
However since we are returning it might as well test for it. And in case of the current test, `head_grads` as depicted by `dL/dy` is an input leaf node in the second graph. So I expect that we will be able to accumulate gradients in it (similar to Pytorch).
I believe the graph for second order should be like this.
I hope it makes sense. Thank You.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services