You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2020/06/06 05:07:36 UTC
[GitHub] [incubator-mxnet] wkcn opened a new pull request #18500: [Bug Fixed] Fix batch norm when grad_req is `add`
wkcn opened a new pull request #18500:
URL: https://github.com/apache/incubator-mxnet/pull/18500
## Description ##
Fix #18499
## Checklist ##
### Essentials ###
Please feel free to remove inapplicable items for your PR.
- [ ] The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant [JIRA issue](https://issues.apache.org/jira/projects/MXNET/issues) created (except PRs with tiny changes)
- [ ] Changes are complete (i.e. I finished coding on this PR)
- [ ] All changes have test coverage:
- Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
- Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
- Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
- [ ] Code is well-documented:
- For user-facing API changes, API doc string has been updated.
- For new C++ functions in header files, their functionalities and arguments are documented.
- For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
- Check the API doc at https://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
- [ ] To the best of my knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change
### Changes ###
Fix the bug in the following implementation
[x] The naive batch norm
[x] CUDNN Batch Norm
[x] MKLDNN Batch Norm
[x] Add unittest for `grad_req=add`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] szha commented on pull request #18500: [Bug Fixed] Fix batch norm when grad_req is `add`
Posted by GitBox <gi...@apache.org>.
szha commented on pull request #18500:
URL: https://github.com/apache/incubator-mxnet/pull/18500#issuecomment-640773890
thanks for the fix of a long standing problem
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] wkcn commented on pull request #18500: [Bug Fixed] Fix batch norm when grad_req is `add`
Posted by GitBox <gi...@apache.org>.
wkcn commented on pull request #18500:
URL: https://github.com/apache/incubator-mxnet/pull/18500#issuecomment-640132560
@mxnet-bot run ci [centos-gpu]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #18500: [Bug Fixed] Fix batch norm when grad_req is `add`
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #18500:
URL: https://github.com/apache/incubator-mxnet/pull/18500#issuecomment-640132571
Jenkins CI successfully triggered : [centos-gpu]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] wkcn commented on a change in pull request #18500: [Bug Fixed] Fix batch norm when grad_req is `add`
Posted by GitBox <gi...@apache.org>.
wkcn commented on a change in pull request #18500:
URL: https://github.com/apache/incubator-mxnet/pull/18500#discussion_r436437897
##########
File path: tests/python/unittest/test_operator.py
##########
@@ -1944,20 +1960,27 @@ def _test_batchnorm_impl(op, shape, axis, cudnn_off, output_mean_var):
), running_var.asnumpy(), atol=atol, rtol=rtol)
assert_almost_equal(data.grad.asnumpy(),
- dX.asnumpy(), atol=atol, rtol=rtol)
- assert_almost_equal(
- bn_gamma.grad.asnumpy(), dW.asnumpy(), atol=atol, rtol=rtol)
+ adX.asnumpy(), atol=atol, rtol=rtol)
+ if not fix_gamma:
+ assert_almost_equal(
+ bn_gamma.grad.asnumpy(), adW.asnumpy(),
+ atol=atol, rtol=rtol)
+ else:
+ assert((bn_gamma.asnumpy() == 1).all())
assert_almost_equal(
- bn_beta.grad.asnumpy(), db.asnumpy(), atol=atol, rtol=rtol)
+ bn_beta.grad.asnumpy(), adb.asnumpy(), atol=atol, rtol=rtol)
for op in [mx.nd.BatchNorm, mx.nd.contrib.SyncBatchNorm]:
- for shape in [(24, 2), (24, 3, 4), (24, 4, 4, 4), (24, 8, 4, 4), (24, 5, 6, 4, 4)]:
+ for shape in [(24, 2), (24, 3, 4), (24, 4, 4, 4),
+ (24, 8, 4, 4), (24, 5, 6, 4, 4)]:
for axis in range(len(shape)):
- for cudnn_off in [False, True]:
- for output_mean_var in [False, True]:
- _test_batchnorm_impl(op, shape, axis,
- cudnn_off, output_mean_var)
-
+ for fix_gamma in [False, True]:
+ for grad_req in ['write', 'add']:
+ for cudnn_off in [False, True]:
+ for output_mean_var in [False, True]:
Review comment:
Solved. I will submit the PR later.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] wkcn commented on a change in pull request #18500: [Bug Fixed] Fix batch norm when grad_req is `add`
Posted by GitBox <gi...@apache.org>.
wkcn commented on a change in pull request #18500:
URL: https://github.com/apache/incubator-mxnet/pull/18500#discussion_r436437897
##########
File path: tests/python/unittest/test_operator.py
##########
@@ -1944,20 +1960,27 @@ def _test_batchnorm_impl(op, shape, axis, cudnn_off, output_mean_var):
), running_var.asnumpy(), atol=atol, rtol=rtol)
assert_almost_equal(data.grad.asnumpy(),
- dX.asnumpy(), atol=atol, rtol=rtol)
- assert_almost_equal(
- bn_gamma.grad.asnumpy(), dW.asnumpy(), atol=atol, rtol=rtol)
+ adX.asnumpy(), atol=atol, rtol=rtol)
+ if not fix_gamma:
+ assert_almost_equal(
+ bn_gamma.grad.asnumpy(), adW.asnumpy(),
+ atol=atol, rtol=rtol)
+ else:
+ assert((bn_gamma.asnumpy() == 1).all())
assert_almost_equal(
- bn_beta.grad.asnumpy(), db.asnumpy(), atol=atol, rtol=rtol)
+ bn_beta.grad.asnumpy(), adb.asnumpy(), atol=atol, rtol=rtol)
for op in [mx.nd.BatchNorm, mx.nd.contrib.SyncBatchNorm]:
- for shape in [(24, 2), (24, 3, 4), (24, 4, 4, 4), (24, 8, 4, 4), (24, 5, 6, 4, 4)]:
+ for shape in [(24, 2), (24, 3, 4), (24, 4, 4, 4),
+ (24, 8, 4, 4), (24, 5, 6, 4, 4)]:
for axis in range(len(shape)):
- for cudnn_off in [False, True]:
- for output_mean_var in [False, True]:
- _test_batchnorm_impl(op, shape, axis,
- cudnn_off, output_mean_var)
-
+ for fix_gamma in [False, True]:
+ for grad_req in ['write', 'add']:
+ for cudnn_off in [False, True]:
+ for output_mean_var in [False, True]:
Review comment:
Solved. I will submit the PR later.
##########
File path: tests/python/unittest/test_operator.py
##########
@@ -1944,20 +1960,27 @@ def _test_batchnorm_impl(op, shape, axis, cudnn_off, output_mean_var):
), running_var.asnumpy(), atol=atol, rtol=rtol)
assert_almost_equal(data.grad.asnumpy(),
- dX.asnumpy(), atol=atol, rtol=rtol)
- assert_almost_equal(
- bn_gamma.grad.asnumpy(), dW.asnumpy(), atol=atol, rtol=rtol)
+ adX.asnumpy(), atol=atol, rtol=rtol)
+ if not fix_gamma:
+ assert_almost_equal(
+ bn_gamma.grad.asnumpy(), adW.asnumpy(),
+ atol=atol, rtol=rtol)
+ else:
+ assert((bn_gamma.asnumpy() == 1).all())
assert_almost_equal(
- bn_beta.grad.asnumpy(), db.asnumpy(), atol=atol, rtol=rtol)
+ bn_beta.grad.asnumpy(), adb.asnumpy(), atol=atol, rtol=rtol)
for op in [mx.nd.BatchNorm, mx.nd.contrib.SyncBatchNorm]:
- for shape in [(24, 2), (24, 3, 4), (24, 4, 4, 4), (24, 8, 4, 4), (24, 5, 6, 4, 4)]:
+ for shape in [(24, 2), (24, 3, 4), (24, 4, 4, 4),
+ (24, 8, 4, 4), (24, 5, 6, 4, 4)]:
for axis in range(len(shape)):
- for cudnn_off in [False, True]:
- for output_mean_var in [False, True]:
- _test_batchnorm_impl(op, shape, axis,
- cudnn_off, output_mean_var)
-
+ for fix_gamma in [False, True]:
+ for grad_req in ['write', 'add']:
Review comment:
Solved. I will submit the PR later.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] szha merged pull request #18500: [Bug Fixed] Fix batch norm when grad_req is `add`
Posted by GitBox <gi...@apache.org>.
szha merged pull request #18500:
URL: https://github.com/apache/incubator-mxnet/pull/18500
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] wkcn commented on pull request #18500: [Bug Fixed] Fix batch norm when grad_req is `add`
Posted by GitBox <gi...@apache.org>.
wkcn commented on pull request #18500:
URL: https://github.com/apache/incubator-mxnet/pull/18500#issuecomment-640178213
Hi @sxjscience , @zhreshold and @TaoLv , could you please help take a review?
The bug of batch norm has been fixed. I change the following implementations of batch norm.
name | filename
-|-
naive cpu | src/operator/nn/batch_norm.cc
naive gpu | src/operator/nn/batch_norm.cu
cudnn|src/operator/nn/cudnn/cudnn_batch_norm-inl.h
mkldnn | src/operator/nn/mkldnn/mkldnn_batch_norm-inl.h
Thank you!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] wkcn edited a comment on pull request #18500: [Bug Fixed] Fix batch norm when grad_req is `add`
Posted by GitBox <gi...@apache.org>.
wkcn edited a comment on pull request #18500:
URL: https://github.com/apache/incubator-mxnet/pull/18500#issuecomment-640336128
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] wkcn commented on pull request #18500: [Bug Fixed] Fix batch norm when grad_req is `add`
Posted by GitBox <gi...@apache.org>.
wkcn commented on pull request #18500:
URL: https://github.com/apache/incubator-mxnet/pull/18500#issuecomment-640336128
Hi @szha , mx.npx.batch_norm is the alias of mx.nd.BatchNorm. I do not know how to add the test into the npx side.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] wkcn commented on a change in pull request #18500: [Bug Fixed] Fix batch norm when grad_req is `add`
Posted by GitBox <gi...@apache.org>.
wkcn commented on a change in pull request #18500:
URL: https://github.com/apache/incubator-mxnet/pull/18500#discussion_r436437919
##########
File path: tests/python/unittest/test_operator.py
##########
@@ -1944,20 +1960,27 @@ def _test_batchnorm_impl(op, shape, axis, cudnn_off, output_mean_var):
), running_var.asnumpy(), atol=atol, rtol=rtol)
assert_almost_equal(data.grad.asnumpy(),
- dX.asnumpy(), atol=atol, rtol=rtol)
- assert_almost_equal(
- bn_gamma.grad.asnumpy(), dW.asnumpy(), atol=atol, rtol=rtol)
+ adX.asnumpy(), atol=atol, rtol=rtol)
+ if not fix_gamma:
+ assert_almost_equal(
+ bn_gamma.grad.asnumpy(), adW.asnumpy(),
+ atol=atol, rtol=rtol)
+ else:
+ assert((bn_gamma.asnumpy() == 1).all())
assert_almost_equal(
- bn_beta.grad.asnumpy(), db.asnumpy(), atol=atol, rtol=rtol)
+ bn_beta.grad.asnumpy(), adb.asnumpy(), atol=atol, rtol=rtol)
for op in [mx.nd.BatchNorm, mx.nd.contrib.SyncBatchNorm]:
- for shape in [(24, 2), (24, 3, 4), (24, 4, 4, 4), (24, 8, 4, 4), (24, 5, 6, 4, 4)]:
+ for shape in [(24, 2), (24, 3, 4), (24, 4, 4, 4),
+ (24, 8, 4, 4), (24, 5, 6, 4, 4)]:
for axis in range(len(shape)):
- for cudnn_off in [False, True]:
- for output_mean_var in [False, True]:
- _test_batchnorm_impl(op, shape, axis,
- cudnn_off, output_mean_var)
-
+ for fix_gamma in [False, True]:
+ for grad_req in ['write', 'add']:
Review comment:
Solved. I will submit the PR later.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] szha commented on a change in pull request #18500: [Bug Fixed] Fix batch norm when grad_req is `add`
Posted by GitBox <gi...@apache.org>.
szha commented on a change in pull request #18500:
URL: https://github.com/apache/incubator-mxnet/pull/18500#discussion_r436426078
##########
File path: tests/python/unittest/test_operator.py
##########
@@ -1944,20 +1960,27 @@ def _test_batchnorm_impl(op, shape, axis, cudnn_off, output_mean_var):
), running_var.asnumpy(), atol=atol, rtol=rtol)
assert_almost_equal(data.grad.asnumpy(),
- dX.asnumpy(), atol=atol, rtol=rtol)
- assert_almost_equal(
- bn_gamma.grad.asnumpy(), dW.asnumpy(), atol=atol, rtol=rtol)
+ adX.asnumpy(), atol=atol, rtol=rtol)
+ if not fix_gamma:
+ assert_almost_equal(
+ bn_gamma.grad.asnumpy(), adW.asnumpy(),
+ atol=atol, rtol=rtol)
+ else:
+ assert((bn_gamma.asnumpy() == 1).all())
assert_almost_equal(
- bn_beta.grad.asnumpy(), db.asnumpy(), atol=atol, rtol=rtol)
+ bn_beta.grad.asnumpy(), adb.asnumpy(), atol=atol, rtol=rtol)
for op in [mx.nd.BatchNorm, mx.nd.contrib.SyncBatchNorm]:
- for shape in [(24, 2), (24, 3, 4), (24, 4, 4, 4), (24, 8, 4, 4), (24, 5, 6, 4, 4)]:
+ for shape in [(24, 2), (24, 3, 4), (24, 4, 4, 4),
+ (24, 8, 4, 4), (24, 5, 6, 4, 4)]:
for axis in range(len(shape)):
- for cudnn_off in [False, True]:
- for output_mean_var in [False, True]:
- _test_batchnorm_impl(op, shape, axis,
- cudnn_off, output_mean_var)
-
+ for fix_gamma in [False, True]:
+ for grad_req in ['write', 'add']:
Review comment:
in gluon, the grad req can actually differ in gamma and beta. could you test the different combinations?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] szha commented on a change in pull request #18500: [Bug Fixed] Fix batch norm when grad_req is `add`
Posted by GitBox <gi...@apache.org>.
szha commented on a change in pull request #18500:
URL: https://github.com/apache/incubator-mxnet/pull/18500#discussion_r436425275
##########
File path: tests/python/unittest/test_operator.py
##########
@@ -1944,20 +1960,27 @@ def _test_batchnorm_impl(op, shape, axis, cudnn_off, output_mean_var):
), running_var.asnumpy(), atol=atol, rtol=rtol)
assert_almost_equal(data.grad.asnumpy(),
- dX.asnumpy(), atol=atol, rtol=rtol)
- assert_almost_equal(
- bn_gamma.grad.asnumpy(), dW.asnumpy(), atol=atol, rtol=rtol)
+ adX.asnumpy(), atol=atol, rtol=rtol)
+ if not fix_gamma:
+ assert_almost_equal(
+ bn_gamma.grad.asnumpy(), adW.asnumpy(),
+ atol=atol, rtol=rtol)
+ else:
+ assert((bn_gamma.asnumpy() == 1).all())
assert_almost_equal(
- bn_beta.grad.asnumpy(), db.asnumpy(), atol=atol, rtol=rtol)
+ bn_beta.grad.asnumpy(), adb.asnumpy(), atol=atol, rtol=rtol)
for op in [mx.nd.BatchNorm, mx.nd.contrib.SyncBatchNorm]:
- for shape in [(24, 2), (24, 3, 4), (24, 4, 4, 4), (24, 8, 4, 4), (24, 5, 6, 4, 4)]:
+ for shape in [(24, 2), (24, 3, 4), (24, 4, 4, 4),
+ (24, 8, 4, 4), (24, 5, 6, 4, 4)]:
for axis in range(len(shape)):
- for cudnn_off in [False, True]:
- for output_mean_var in [False, True]:
- _test_batchnorm_impl(op, shape, axis,
- cudnn_off, output_mean_var)
-
+ for fix_gamma in [False, True]:
+ for grad_req in ['write', 'add']:
+ for cudnn_off in [False, True]:
+ for output_mean_var in [False, True]:
Review comment:
use `@pytest.mark.parametrize` instead.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #18500: [Bug Fixed] Fix batch norm when grad_req is `add`
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #18500:
URL: https://github.com/apache/incubator-mxnet/pull/18500#issuecomment-639985096
Hey @wkcn , Thanks for submitting the PR
All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands:
- To trigger all jobs: @mxnet-bot run ci [all]
- To trigger specific jobs: @mxnet-bot run ci [job1, job2]
***
**CI supported jobs**: [centos-gpu, centos-cpu, unix-gpu, sanity, windows-cpu, miscellaneous, windows-gpu, clang, website, edge, unix-cpu]
***
_Note_:
Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin.
All CI tests must pass before the PR can be merged.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org