You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2020/05/22 04:42:35 UTC
[GitHub] [incubator-mxnet] MoisesHer opened a new pull request #18385: Safe accumulation for computing gradient in Embedding & Take
MoisesHer opened a new pull request #18385:
URL: https://github.com/apache/incubator-mxnet/pull/18385
## Description ##
This PR adds the option to compute gradient of Embedding and Take with safe accumulation,
i.e. when dtype is FP16, accumulation will be done in FP32.
It fixes the underflow problem that was causing issue [https://github.com/apache/incubator-mxnet/issues/17703](https://github.com/apache/incubator-mxnet/issues/17703)
This was happening when number of indices is very large, and vocabulary/data small
For activating it user can set: MXNET_SAFE_ACCUMULATION=1
## Checklist ##
### Essentials ###
Please feel free to remove inapplicable items for your PR.
- [ ] Changes are complete (i.e. I finished coding on this PR)
- [x] All changes have test coverage:
- Unit tests are added for both Take and Embedding using a
- [x] Code is well-documented:
- [x] To the best of my knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change
### Changes ###
SAFE_ACCUMULATION was added in the following operators
- [x] _backward_Embedding<cpu> (required modification within mshadow)
- [x] _backward_Embedding<gpu>
- [x] _backward_take <cpu> for any axis (required modification within mshadow for axis=0)
- [x] _backward_take <gpu> for any axis (required modification within mshadow for axis=0)
## Comments ##
TODO:
- [x] _backward_take unit test (cpu / gpu) failing. Non deterministic error, which happens after re-running _backward_take. Looks like deallocation/allocation mem issue.
- [x] Performance review
- [x] Include this option into Automatic Mixed Precision to avoid issues
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] szha commented on pull request #18385: Safe accumulation for computing gradient in Embedding & Take
Posted by GitBox <gi...@apache.org>.
szha commented on pull request #18385:
URL: https://github.com/apache/incubator-mxnet/pull/18385#issuecomment-663777166
@mxnet-bot run ci [all]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #18385: Safe accumulation for computing gradient in Embedding & Take
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #18385:
URL: https://github.com/apache/incubator-mxnet/pull/18385#issuecomment-632476192
Hey @MoisesHer , Thanks for submitting the PR
All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands:
- To trigger all jobs: @mxnet-bot run ci [all]
- To trigger specific jobs: @mxnet-bot run ci [job1, job2]
***
**CI supported jobs**: [centos-cpu, miscellaneous, website, sanity, windows-gpu, windows-cpu, unix-gpu, centos-gpu, clang, edge, unix-cpu]
***
_Note_:
Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin.
All CI tests must pass before the PR can be merged.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #18385: Safe accumulation for computing gradient in Embedding & Take
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #18385:
URL: https://github.com/apache/incubator-mxnet/pull/18385#issuecomment-636527077
Jenkins CI successfully triggered : [windows-gpu]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] MoisesHer commented on pull request #18385: Safe accumulation for computing gradient in Embedding & Take
Posted by GitBox <gi...@apache.org>.
MoisesHer commented on pull request #18385:
URL: https://github.com/apache/incubator-mxnet/pull/18385#issuecomment-636222460
@mxnet-bot run ci [miscellaneous]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] szha commented on pull request #18385: Safe accumulation for computing gradient in Embedding & Take
Posted by GitBox <gi...@apache.org>.
szha commented on pull request #18385:
URL: https://github.com/apache/incubator-mxnet/pull/18385#issuecomment-650911225
Hi. What’s the status for this PR?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] MoisesHer commented on pull request #18385: Safe accumulation for computing gradient in Embedding & Take
Posted by GitBox <gi...@apache.org>.
MoisesHer commented on pull request #18385:
URL: https://github.com/apache/incubator-mxnet/pull/18385#issuecomment-651243080
> Hi. What’s the status for this PR?
The PR is completed, but failing because Windows is ignoring the environment variable MXNET_SAFE_ACCUMULATION
I think [PR #18424 ](https://github.com/apache/incubator-mxnet/pull/18424) can solve this isse
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] MoisesHer commented on pull request #18385: Safe accumulation for computing gradient in Embedding & Take
Posted by GitBox <gi...@apache.org>.
MoisesHer commented on pull request #18385:
URL: https://github.com/apache/incubator-mxnet/pull/18385#issuecomment-636527053
@mxnet-bot run ci [windows-gpu]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] szha merged pull request #18385: Safe accumulation for computing gradient in Embedding & Take
Posted by GitBox <gi...@apache.org>.
szha merged pull request #18385:
URL: https://github.com/apache/incubator-mxnet/pull/18385
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #18385: Safe accumulation for computing gradient in Embedding & Take
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #18385:
URL: https://github.com/apache/incubator-mxnet/pull/18385#issuecomment-636387282
Jenkins CI successfully triggered : [windows-gpu, miscellaneous]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] MoisesHer commented on pull request #18385: Safe accumulation for computing gradient in Embedding & Take
Posted by GitBox <gi...@apache.org>.
MoisesHer commented on pull request #18385:
URL: https://github.com/apache/incubator-mxnet/pull/18385#issuecomment-636387271
@mxnet-bot run ci [miscellaneous, windows-gpu]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #18385: Safe accumulation for computing gradient in Embedding & Take
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #18385:
URL: https://github.com/apache/incubator-mxnet/pull/18385#issuecomment-636222498
Jenkins CI successfully triggered : [miscellaneous]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] szha commented on a change in pull request #18385: Safe accumulation for computing gradient in Embedding & Take
Posted by GitBox <gi...@apache.org>.
szha commented on a change in pull request #18385:
URL: https://github.com/apache/incubator-mxnet/pull/18385#discussion_r466788279
##########
File path: tests/python/gpu/test_operator_gpu.py
##########
@@ -1547,47 +1551,91 @@ def test_embedding_helper(data_types, weight_types, low_pad, high_pad):
@with_seed()
def test_take_with_type():
sym = mx.sym.take(name='take')
- for data_ndim in range(2, 5):
- for idx_ndim in range(1, 4):
- data_shape = ()
- for _ in range(data_ndim):
- data_shape += (np.random.randint(low=3, high=6), )
- idx_shape = ()
- for _ in range(idx_ndim):
- idx_shape += (np.random.randint(low=3, high=5), )
- ctx_list = [{'ctx': mx.gpu(0), 'take_indices': idx_shape,
- 'take_a': data_shape,
- 'type_dict': {'take_indices': np.float64,
- 'take_a': np.float64}},
- {'ctx': mx.gpu(0), 'take_indices': idx_shape,
- 'take_a': data_shape,
- 'type_dict': {'take_indices': np.float32,
- 'take_a': np.float32}},
- {'ctx': mx.gpu(0), 'take_indices': idx_shape,
- 'take_a': data_shape,
- 'type_dict': {'take_indices': np.float16,
- 'take_a': np.float16}},
- {'ctx': mx.cpu(0), 'take_indices': idx_shape,
- 'take_a': data_shape,
- 'type_dict': {'take_indices': np.float64,
- 'take_a': np.float64}},
- {'ctx': mx.cpu(0), 'take_indices': idx_shape,
- 'take_a': data_shape,
- 'type_dict': {'take_indices': np.float32,
- 'take_a': np.float32}},
- {'ctx': mx.cpu(0), 'take_indices': idx_shape,
- 'take_a': data_shape,
- 'type_dict': {'take_indices': np.float16,
- 'take_a': np.float16}}]
- arg_params = {'take_indices': np.random.randint(low=0,
- high=data_shape[0],
- size=idx_shape),
- 'take_a': np.random.normal(size=data_shape)}
- check_consistency(sym, ctx_list,
- grad_req={'take_indices': 'null',
- 'take_a': 'write'},
- arg_params=arg_params)
-
+ for safe_accumulation in ['0', '1', None]:
+ for data_ndim in range(2, 5):
+ for idx_ndim in range(1, 4):
Review comment:
use `pytest.mark.parametrize(...)`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #18385: Safe accumulation for computing gradient in Embedding & Take
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #18385:
URL: https://github.com/apache/incubator-mxnet/pull/18385#issuecomment-663777193
Jenkins CI successfully triggered : [centos-gpu, miscellaneous, windows-gpu, edge, sanity, unix-cpu, website, unix-gpu, centos-cpu, clang, windows-cpu]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org