You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2021/08/09 00:26:39 UTC
[GitHub] [incubator-mxnet] KexinFeng opened a new pull request #20500: Add feature of retain_grad
KexinFeng opened a new pull request #20500:
URL: https://github.com/apache/incubator-mxnet/pull/20500
## Description ##
The PR adds the support for fetching the gradients of intermediate variables in a computation graph.
The motivation of this feature comes from this [issue](https://github.com/apache/incubator-mxnet/issues/11865).
## Checklist ##
### Essentials ###
- [ ] PR's title starts with a category (e.g. [BUGFIX], [MODEL], [TUTORIAL], [FEATURE], [DOC], etc)
- [ ] Changes are complete (i.e. I finished coding on this PR)
- [ ] All changes have test coverage
- [ ] Code is well-documented
### Changes ###
- [ ] Feature1, tests, (and when applicable, API doc)
- [ ] Feature2, tests, (and when applicable, API doc)
## Comments ##
- Interesting edge cases to note here: 1. the gradient-queried node is the output node, `z.retain_grad()`. 2. when it is used combined with detaching computation.
- Note that `u.retain_grad()` is called outside the scope of ag.record(). Otherwise, `u.retain_grad()` would be recorded as operation and trigger building of computational node. This is conceptually not desired.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #20500: [FEATURE] Add feature of retain_grad
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #20500:
URL: https://github.com/apache/incubator-mxnet/pull/20500#issuecomment-907538204
Jenkins CI successfully triggered : [unix-cpu]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] KexinFeng commented on pull request #20500: [FEATURE] Add feature of retain_grad
Posted by GitBox <gi...@apache.org>.
KexinFeng commented on pull request #20500:
URL: https://github.com/apache/incubator-mxnet/pull/20500#issuecomment-901444416
@mxnet-bot run ci [centos-gpu, unix-gpu]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #20500: [FEATURE] Add feature of retain_grad
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #20500:
URL: https://github.com/apache/incubator-mxnet/pull/20500#issuecomment-900792918
Jenkins CI successfully triggered : [clang, windows-cpu, miscellaneous, centos-cpu, unix-cpu, edge, website, windows-gpu, sanity, centos-gpu, unix-gpu]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] barry-jin commented on pull request #20500: [FEATURE] Add feature of retain_grad
Posted by GitBox <gi...@apache.org>.
barry-jin commented on pull request #20500:
URL: https://github.com/apache/incubator-mxnet/pull/20500#issuecomment-921964213
@mxnet-bot run ci [unix-gpu, centos-gpu]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #20500: [FEATURE] Add feature of retain_grad
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #20500:
URL: https://github.com/apache/incubator-mxnet/pull/20500#issuecomment-900792918
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] KexinFeng commented on pull request #20500: [FEATURE] Add feature of retain_grad
Posted by GitBox <gi...@apache.org>.
KexinFeng commented on pull request #20500:
URL: https://github.com/apache/incubator-mxnet/pull/20500#issuecomment-907538194
@mxnet-bot run ci [unix-cpu]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] KexinFeng commented on pull request #20500: [FEATURE] Add feature of retain_grad
Posted by GitBox <gi...@apache.org>.
KexinFeng commented on pull request #20500:
URL: https://github.com/apache/incubator-mxnet/pull/20500#issuecomment-900792859
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] barry-jin commented on a change in pull request #20500: [FEATURE] Add feature of retain_grad
Posted by GitBox <gi...@apache.org>.
barry-jin commented on a change in pull request #20500:
URL: https://github.com/apache/incubator-mxnet/pull/20500#discussion_r689870529
##########
File path: tests/python/unittest/test_autograd.py
##########
@@ -519,3 +519,33 @@ def test_gradient():
dx.backward()
assert abs(x.grad.asscalar() - 2.71828175) < 1e-7
+def test_retain_grad_drop_grad():
+ x = nd.array([1,2,3,4])
+ x.attach_grad()
+ y = nd.array([5,6,7,8])
+ y.attach_grad()
+
+ with mx.autograd.record():
+ u = x * y
+ z = u * x
+
+ u.attach_grad()
+ z.attach_grad()
+ out_grad = nd.array([10, 10, 10, 10])
+ z.backward(out_grad, retain_graph=True)
+
+ assert (u.grad == out_grad * x).asnumpy().all()
+ assert (z.grad == out_grad).asnumpy().all()
+ assert (x.grad == out_grad * 2 * x * y).asnumpy().all()
+ assert (y.grad == out_grad * x*x).asnumpy().all()
+
+ u.drop_grad()
+ z.drop_grad()
+ y.drop_grad()
+ out_grad = nd.array([0.1, 0.1, 0.1, 0.1])
+ z.backward(out_grad)
+
+ assert u.grad is None
+ assert z.grad is None
+ assert (x.grad == out_grad * 2 * x * y).asnumpy().all()
+ assert y.grad is None
Review comment:
Could you add some tests to test retain grad on gluon blocks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] barry-jin commented on pull request #20500: [FEATURE] Add feature of retain_grad
Posted by GitBox <gi...@apache.org>.
barry-jin commented on pull request #20500:
URL: https://github.com/apache/incubator-mxnet/pull/20500#issuecomment-923197853
@mxnet-bot run ci [centos-gpu]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #20500: [FEATURE] Add feature of retain_grad
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #20500:
URL: https://github.com/apache/incubator-mxnet/pull/20500#issuecomment-901444464
Jenkins CI successfully triggered : [centos-gpu, unix-gpu]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] KexinFeng commented on pull request #20500: [FEATURE] Add feature of retain_grad
Posted by GitBox <gi...@apache.org>.
KexinFeng commented on pull request #20500:
URL: https://github.com/apache/incubator-mxnet/pull/20500#issuecomment-898072721
@mxnet-bot run ci [all]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #20500: [FEATURE] Add feature of retain_grad
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #20500:
URL: https://github.com/apache/incubator-mxnet/pull/20500#issuecomment-898072772
Jenkins CI successfully triggered : [unix-gpu, miscellaneous, centos-gpu, unix-cpu, windows-cpu, centos-cpu, clang, website, windows-gpu, sanity, edge]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] KexinFeng commented on pull request #20500: [FEATURE] Add feature of retain_grad
Posted by GitBox <gi...@apache.org>.
KexinFeng commented on pull request #20500:
URL: https://github.com/apache/incubator-mxnet/pull/20500#issuecomment-907522613
@mxnet-bot run ci [website]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #20500: [FEATURE] Add feature of retain_grad
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #20500:
URL: https://github.com/apache/incubator-mxnet/pull/20500#issuecomment-923197903
Jenkins CI successfully triggered : [centos-gpu]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] KexinFeng commented on pull request #20500: [FEATURE] Add feature of retain_grad
Posted by GitBox <gi...@apache.org>.
KexinFeng commented on pull request #20500:
URL: https://github.com/apache/incubator-mxnet/pull/20500#issuecomment-900792859
@mxnet-bot run ci [all]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] KexinFeng commented on pull request #20500: [FEATURE] Add feature of retain_grad
Posted by GitBox <gi...@apache.org>.
KexinFeng commented on pull request #20500:
URL: https://github.com/apache/incubator-mxnet/pull/20500#issuecomment-907566517
@mxnet-bot run ci [unix-cpu]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] KexinFeng commented on a change in pull request #20500: [FEATURE] Add feature of retain_grad
Posted by GitBox <gi...@apache.org>.
KexinFeng commented on a change in pull request #20500:
URL: https://github.com/apache/incubator-mxnet/pull/20500#discussion_r693526823
##########
File path: tests/python/unittest/test_autograd.py
##########
@@ -519,3 +519,33 @@ def test_gradient():
dx.backward()
assert abs(x.grad.asscalar() - 2.71828175) < 1e-7
+def test_retain_grad_drop_grad():
+ x = nd.array([1,2,3,4])
+ x.attach_grad()
+ y = nd.array([5,6,7,8])
+ y.attach_grad()
+
+ with mx.autograd.record():
+ u = x * y
+ z = u * x
+
+ u.attach_grad()
+ z.attach_grad()
+ out_grad = nd.array([10, 10, 10, 10])
+ z.backward(out_grad, retain_graph=True)
+
+ assert (u.grad == out_grad * x).asnumpy().all()
+ assert (z.grad == out_grad).asnumpy().all()
+ assert (x.grad == out_grad * 2 * x * y).asnumpy().all()
+ assert (y.grad == out_grad * x*x).asnumpy().all()
+
+ u.drop_grad()
+ z.drop_grad()
+ y.drop_grad()
+ out_grad = nd.array([0.1, 0.1, 0.1, 0.1])
+ z.backward(out_grad)
+
+ assert u.grad is None
+ assert z.grad is None
+ assert (x.grad == out_grad * 2 * x * y).asnumpy().all()
+ assert y.grad is None
Review comment:
Yes, it's added now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] KexinFeng commented on pull request #20500: [FEATURE] Add feature of retain_grad
Posted by GitBox <gi...@apache.org>.
KexinFeng commented on pull request #20500:
URL: https://github.com/apache/incubator-mxnet/pull/20500#issuecomment-901133112
@mxnet-bot run ci [unix-cpu]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #20500: [FEATURE] Add feature of retain_grad
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #20500:
URL: https://github.com/apache/incubator-mxnet/pull/20500#issuecomment-901133172
Jenkins CI successfully triggered : [unix-cpu]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] KexinFeng commented on pull request #20500: [FEATURE] Add feature of retain_grad
Posted by GitBox <gi...@apache.org>.
KexinFeng commented on pull request #20500:
URL: https://github.com/apache/incubator-mxnet/pull/20500#issuecomment-901133985
@mxnet-bot run ci [macosx-x86_64, linkcheck]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #20500: [FEATURE] Add feature of retain_grad
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #20500:
URL: https://github.com/apache/incubator-mxnet/pull/20500#issuecomment-907566522
Jenkins CI successfully triggered : [unix-cpu]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #20500: Add feature of retain_grad
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #20500:
URL: https://github.com/apache/incubator-mxnet/pull/20500#issuecomment-894881871
Hey @KexinFeng , Thanks for submitting the PR
All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands:
- To trigger all jobs: @mxnet-bot run ci [all]
- To trigger specific jobs: @mxnet-bot run ci [job1, job2]
***
**CI supported jobs**: [centos-gpu, centos-cpu, windows-cpu, unix-gpu, website, sanity, edge, miscellaneous, unix-cpu, windows-gpu, clang]
***
_Note_:
Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin.
All CI tests must pass before the PR can be merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #20500: [FEATURE] Add feature of retain_grad
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #20500:
URL: https://github.com/apache/incubator-mxnet/pull/20500#issuecomment-901133997
None of the jobs entered are supported.
Jobs entered by user: [macosx-x86_64, linkcheck]
CI supported Jobs: [centos-cpu, edge, windows-cpu, sanity, clang, centos-gpu, unix-gpu, website, miscellaneous, unix-cpu, windows-gpu]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] barry-jin commented on a change in pull request #20500: [FEATURE] Add feature of retain_grad
Posted by GitBox <gi...@apache.org>.
barry-jin commented on a change in pull request #20500:
URL: https://github.com/apache/incubator-mxnet/pull/20500#discussion_r689863276
##########
File path: python/mxnet/ndarray/ndarray.py
##########
@@ -2885,6 +2885,11 @@ def attach_grad(self, grad_req='write', stype=None):
ctypes.pointer(mx_uint(grad_req)),
ctypes.pointer(grad.handle)))
+ def drop_grad(self):
Review comment:
Could you also add drop_grad to MXNet NumPy array package.
##########
File path: tests/python/unittest/test_autograd.py
##########
@@ -519,3 +519,33 @@ def test_gradient():
dx.backward()
assert abs(x.grad.asscalar() - 2.71828175) < 1e-7
+def test_retain_grad_drop_grad():
+ x = nd.array([1,2,3,4])
+ x.attach_grad()
+ y = nd.array([5,6,7,8])
+ y.attach_grad()
+
+ with mx.autograd.record():
+ u = x * y
+ z = u * x
+
+ u.attach_grad()
+ z.attach_grad()
+ out_grad = nd.array([10, 10, 10, 10])
+ z.backward(out_grad, retain_graph=True)
+
+ assert (u.grad == out_grad * x).asnumpy().all()
+ assert (z.grad == out_grad).asnumpy().all()
+ assert (x.grad == out_grad * 2 * x * y).asnumpy().all()
+ assert (y.grad == out_grad * x*x).asnumpy().all()
+
+ u.drop_grad()
+ z.drop_grad()
+ y.drop_grad()
+ out_grad = nd.array([0.1, 0.1, 0.1, 0.1])
+ z.backward(out_grad)
+
+ assert u.grad is None
+ assert z.grad is None
+ assert (x.grad == out_grad * 2 * x * y).asnumpy().all()
+ assert y.grad is None
Review comment:
Could you also add some tests to test retain grad on gluon blocks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #20500: [FEATURE] Add feature of retain_grad
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #20500:
URL: https://github.com/apache/incubator-mxnet/pull/20500#issuecomment-907522625
Jenkins CI successfully triggered : [website]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #20500: [FEATURE] Add feature of retain_grad
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #20500:
URL: https://github.com/apache/incubator-mxnet/pull/20500#issuecomment-921964281
Jenkins CI successfully triggered : [centos-gpu, unix-gpu]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] barry-jin merged pull request #20500: [FEATURE] Add feature of retain_grad
Posted by GitBox <gi...@apache.org>.
barry-jin merged pull request #20500:
URL: https://github.com/apache/incubator-mxnet/pull/20500
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] KexinFeng commented on a change in pull request #20500: [FEATURE] Add feature of retain_grad
Posted by GitBox <gi...@apache.org>.
KexinFeng commented on a change in pull request #20500:
URL: https://github.com/apache/incubator-mxnet/pull/20500#discussion_r693526722
##########
File path: python/mxnet/ndarray/ndarray.py
##########
@@ -2885,6 +2885,11 @@ def attach_grad(self, grad_req='write', stype=None):
ctypes.pointer(mx_uint(grad_req)),
ctypes.pointer(grad.handle)))
+ def drop_grad(self):
Review comment:
Yes it's added now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org