You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2020/03/02 02:13:29 UTC
[GitHub] [incubator-mxnet] ChaiBapchya opened a new pull request #17735: Fix
OpPerf in Master
ChaiBapchya opened a new pull request #17735: Fix OpPerf in Master
URL: https://github.com/apache/incubator-mxnet/pull/17735
## Description ##
Change 1
After merging PR #17449 and #17400
refactor of optimizer was incomplete due to both PRs not knowing changes made by each other.
While #17449 added set of variables for large tensor, #17400 refactored 2 variables from gamma1 gamma2 to rho and momentum
Fixing that conflict here
Change 2
Upon running entire opperf suite for CUDA=ON, CUDNN=ON, it was found BatchNorm fails here
```
<function BatchNorm at 0x7f46869e3bf8>
Traceback (most recent call last):
File "incubator-mxnet/benchmark/opperf/opperf.py", line 213, in <module>
sys.exit(main())
File "incubator-mxnet/benchmark/opperf/opperf.py", line 193, in main
benchmark_results = run_all_mxnet_operator_benchmarks(ctx=ctx, dtype=dtype, profiler=profiler, int64_tensor=int64_tensor, warmup=warmup, runs=runs)
File "incubator-mxnet/benchmark/opperf/opperf.py", line 99, in run_all_mxnet_operator_benchmarks
mxnet_operator_benchmark_results.append(run_nn_basic_operators_benchmarks(ctx=ctx, dtype=dtype, profiler=profiler, int64_tensor=int64_tensor, warmup=warmup, runs=runs))
File "/home/ubuntu/incubator-mxnet/benchmark/opperf/nd_operations/nn_basic_operators.py", line 143, in run_nn_basic_operators_benchmarks
mx_nn_basic_op_results = run_op_benchmarks(mx_nn_basic_ops, dtype, ctx, profiler, int64_tensor, warmup, runs)
File "/home/ubuntu/incubator-mxnet/benchmark/opperf/utils/benchmark_utils.py", line 211, in run_op_benchmarks
warmup=warmup, runs=runs)
File "/home/ubuntu/incubator-mxnet/benchmark/opperf/utils/benchmark_utils.py", line 178, in run_performance_test
benchmark_result = _run_nd_operator_performance_test(op, inputs, run_backward, warmup, runs, kwargs_list, profiler)
File "/home/ubuntu/incubator-mxnet/benchmark/opperf/utils/benchmark_utils.py", line 115, in _run_nd_operator_performance_test
_, _ = benchmark_helper_func(op, warmup, **kwargs_list[0])
File "/home/ubuntu/incubator-mxnet/benchmark/opperf/utils/profiler_utils.py", line 200, in cpp_profile_it
res = func(*args, **kwargs)
File "/home/ubuntu/incubator-mxnet/benchmark/opperf/utils/ndarray_utils.py", line 60, in nd_forward_backward_and_profile
nd.waitall()
File "/home/ubuntu/incubator-mxnet/python/mxnet/ndarray/ndarray.py", line 206, in waitall
check_call(_LIB.MXNDArrayWaitAll())
File "/home/ubuntu/incubator-mxnet/python/mxnet/base.py", line 246, in check_call
raise get_last_ffi_error()
mxnet.base.MXNetError: Traceback (most recent call last):
File "/home/ubuntu/incubator-mxnet/src/operator/nn/./cudnn/cudnn_batch_norm-inl.h", line 62
MXNetError: Check failed: param.eps >= 1e-5 (1e-08 vs. 1e-05) : CuDNN requires eps to be no less than 1e-05
```
## Checklist ##
### Essentials ###
Please feel free to remove inapplicable items for your PR.
- [ ] Changes are complete (i.e. I finished coding on this PR)
- [ ] All changes have test coverage:
- [ ] Code is well-documented:
- [ ] To the best of my knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] apeforest commented on a change in pull request
#17735: Fix OpPerf in Master
Posted by GitBox <gi...@apache.org>.
apeforest commented on a change in pull request #17735: Fix OpPerf in Master
URL: https://github.com/apache/incubator-mxnet/pull/17735#discussion_r387498647
##########
File path: benchmark/opperf/rules/default_params.py
##########
@@ -431,7 +433,7 @@
"delta" : DEFAULT_DELTA,
"lr" : DEFAULT_LR,
"lrs" : DEFAULT_LRS,
- "wds" : DEFAULT_LRS,
+ "wd" : DEFAULT_LR,
Review comment:
Yes, I think it's better to create a new var even if its value is the same as other var. It improves readability.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] apeforest commented on a change in pull request
#17735: Fix OpPerf in Master
Posted by GitBox <gi...@apache.org>.
apeforest commented on a change in pull request #17735: Fix OpPerf in Master
URL: https://github.com/apache/incubator-mxnet/pull/17735#discussion_r386583235
##########
File path: benchmark/opperf/rules/default_params.py
##########
@@ -431,7 +433,7 @@
"delta" : DEFAULT_DELTA,
"lr" : DEFAULT_LR,
"lrs" : DEFAULT_LRS,
- "wds" : DEFAULT_LRS,
+ "wd" : DEFAULT_LR,
Review comment:
Isn't wd represent for weight_decay? Why does it take the DEFAULT_LR here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] leezu merged pull request #17735: Fix OpPerf in
Master
Posted by GitBox <gi...@apache.org>.
leezu merged pull request #17735: Fix OpPerf in Master
URL: https://github.com/apache/incubator-mxnet/pull/17735
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] ChaiBapchya commented on issue #17735: Fix OpPerf
in Master
Posted by GitBox <gi...@apache.org>.
ChaiBapchya commented on issue #17735: Fix OpPerf in Master
URL: https://github.com/apache/incubator-mxnet/pull/17735#issuecomment-596769165
@mxnet-label-bot update [pr-awaiting-merge]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] ChaiBapchya commented on a change in pull request
#17735: Fix OpPerf in Master
Posted by GitBox <gi...@apache.org>.
ChaiBapchya commented on a change in pull request #17735: Fix OpPerf in Master
URL: https://github.com/apache/incubator-mxnet/pull/17735#discussion_r387849531
##########
File path: benchmark/opperf/rules/default_params.py
##########
@@ -431,7 +433,7 @@
"delta" : DEFAULT_DELTA,
"lr" : DEFAULT_LR,
"lrs" : DEFAULT_LRS,
- "wds" : DEFAULT_LRS,
+ "wd" : DEFAULT_LR,
Review comment:
Alright addressed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] leezu commented on issue #17735: Fix OpPerf in
Master
Posted by GitBox <gi...@apache.org>.
leezu commented on issue #17735: Fix OpPerf in Master
URL: https://github.com/apache/incubator-mxnet/pull/17735#issuecomment-593524608
Restarted windows CI
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] ChaiBapchya commented on issue #17735: Fix OpPerf
in Master
Posted by GitBox <gi...@apache.org>.
ChaiBapchya commented on issue #17735: Fix OpPerf in Master
URL: https://github.com/apache/incubator-mxnet/pull/17735#issuecomment-593259061
Ran the entire opperf suite after making the changes
### Command
```
python incubator-mxnet/benchmark/opperf/opperf.py --ctx gpu --output-format md --output-file mxnet_opperf_gpu_fix_opperf_lamb.md --warmup 1 --runs 1
```
```
python incubator-mxnet/benchmark/opperf/opperf.py --ctx cpu --output-format md --output-file mxnet_opperf_cpu_fix_opperf_lamb.md --warmup 1 --runs 1
```
### CPU & GPU
https://gist.github.com/ChaiBapchya/517445961bcd57da6b7b5d23fa5d3dc0
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] ChaiBapchya commented on a change in pull request
#17735: Fix OpPerf in Master
Posted by GitBox <gi...@apache.org>.
ChaiBapchya commented on a change in pull request #17735: Fix OpPerf in Master
URL: https://github.com/apache/incubator-mxnet/pull/17735#discussion_r386652795
##########
File path: benchmark/opperf/rules/default_params.py
##########
@@ -431,7 +433,7 @@
"delta" : DEFAULT_DELTA,
"lr" : DEFAULT_LR,
"lrs" : DEFAULT_LRS,
- "wds" : DEFAULT_LRS,
+ "wd" : DEFAULT_LR,
Review comment:
using the same value as lr [0.1,0.5,0.9]
so instead of creating additional variables I tried to reuse existing one as much as possible. can create a new var if needed?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services