You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mxnet.apache.org by GitBox <gi...@apache.org> on 2021/03/25 21:21:58 UTC
[GitHub] [incubator-mxnet] barry-jin opened a new issue #20092: test_norm, test_batchnorm and test_layer_norm dramatically slow down
barry-jin opened a new issue #20092:
URL: https://github.com/apache/incubator-mxnet/issues/20092
It looks like test_norm, test_layer_norm became the slowest tests after openmp submodule being removed in #19953 .
unix-cpu Python3: CPU pytest slowest 50 for commit #19970
```
[2021-03-01T06:10:05.320Z] ============================= slowest 50 durations =============================
[2021-03-01T06:10:05.320Z] 246.53s call tests/python/unittest/test_operator.py::test_broadcast_binary_op
[2021-03-01T06:10:05.320Z] 185.37s call tests/python/unittest/test_operator.py::test_order
[2021-03-01T06:10:05.320Z] 110.09s call tests/python/unittest/test_operator.py::test_psroipooling
[2021-03-01T06:10:05.320Z] 97.39s call tests/python/unittest/test_operator.py::test_layer_norm[float32-0.001-0.001-in_shape_l1-finite_grad_check_l1-0]
[2021-03-01T06:10:05.320Z] 95.54s call tests/python/unittest/test_operator.py::test_layer_norm[float64-0.0001-0.0001-in_shape_l2-finite_grad_check_l2-0]
[2021-03-01T06:10:05.321Z] 90.15s call tests/python/unittest/test_operator.py::test_layer_norm[float64-0.0001-0.0001-in_shape_l2-finite_grad_check_l2-1]
[2021-03-01T06:10:05.321Z] 88.62s call tests/python/unittest/test_operator.py::test_layer_norm[float32-0.001-0.001-in_shape_l1-finite_grad_check_l1-1]
[2021-03-01T06:10:05.321Z] 71.50s call tests/python/unittest/test_operator.py::test_convolution_dilated_impulse_response
[2021-03-01T06:10:05.321Z] 71.45s call tests/python/unittest/test_operator.py::test_bilinear_resize_op
[2021-03-01T06:10:05.321Z] 43.48s call tests/python/unittest/test_operator.py::test_convolution_independent_gradients
[2021-03-01T06:10:05.321Z] 42.75s call tests/python/unittest/test_operator.py::test_stack
[2021-03-01T06:10:05.321Z] 24.36s call tests/python/unittest/test_operator.py::test_multi_proposal_op
[2021-03-01T06:10:05.321Z] 24.18s call tests/python/unittest/test_operator.py::test_laop_2
[2021-03-01T06:10:05.321Z] 21.11s call tests/python/unittest/test_operator.py::test_layer_norm[float16-0.01-0.01-in_shape_l0-finite_grad_check_l0-1]
[2021-03-01T06:10:05.321Z] 20.08s call tests/python/unittest/test_operator.py::test_layer_norm[float16-0.01-0.01-in_shape_l0-finite_grad_check_l0-0]
[2021-03-01T06:10:05.321Z] 18.79s call tests/python/unittest/test_operator.py::test_reduce
[2021-03-01T06:10:05.321Z] 10.60s call tests/python/unittest/test_operator.py::test_batchnorm[True-False-False-shape2-BatchNorm]
[2021-03-01T06:10:05.321Z] 10.39s call tests/python/unittest/test_operator.py::test_batchnorm_training
[2021-03-01T06:10:05.321Z] 10.11s call tests/python/unittest/test_operator.py::test_batchnorm[True-True-False-shape2-BatchNorm]
[2021-03-01T06:10:05.321Z] 9.05s call tests/python/unittest/test_operator.py::test_l2_normalization
```
unix-cpu Python3: CPU pytest slowest 50 for commit #19984
```
[2021-03-06T03:23:55.682Z] ============================= slowest 50 durations =============================
[2021-03-06T03:23:55.682Z] 955.18s call tests/python/unittest/test_operator.py::test_norm
[2021-03-06T03:23:55.682Z] 840.74s call tests/python/unittest/test_operator.py::test_batchnorm[False-False-False-shape2-BatchNorm]
[2021-03-06T03:23:55.682Z] 671.00s call tests/python/unittest/test_operator.py::test_reduce
[2021-03-06T03:23:55.682Z] 610.03s call tests/python/unittest/test_operator.py::test_layer_norm[float32-0.001-0.001-in_shape_l1-finite_grad_check_l1-1]
[2021-03-06T03:23:55.682Z] 555.06s call tests/python/unittest/test_operator.py::test_layer_norm[float64-0.0001-0.0001-in_shape_l2-finite_grad_check_l2-1]
[2021-03-06T03:23:55.682Z] 483.95s call tests/python/unittest/test_operator.py::test_batchnorm[False-True-False-shape2-BatchNorm]
[2021-03-06T03:23:55.682Z] 320.25s call tests/python/unittest/test_operator.py::test_layer_norm[float32-0.001-0.001-in_shape_l1-finite_grad_check_l1-0]
[2021-03-06T03:23:55.682Z] 315.74s call tests/python/unittest/test_operator.py::test_layer_norm[float64-0.0001-0.0001-in_shape_l2-finite_grad_check_l2-0]
[2021-03-06T03:23:55.682Z] 293.05s call tests/python/unittest/test_operator.py::test_batchnorm[True-False-False-shape2-SyncBatchNorm]
[2021-03-06T03:23:55.682Z] 289.04s call tests/python/unittest/test_operator.py::test_layer_norm[float16-0.01-0.01-in_shape_l0-finite_grad_check_l0-0]
[2021-03-06T03:23:55.682Z] 238.83s call tests/python/unittest/test_operator.py::test_laop_2
[2021-03-06T03:23:55.682Z] 230.21s call tests/python/unittest/test_operator.py::test_batchnorm[False-False-False-shape2-SyncBatchNorm]
[2021-03-06T03:23:55.682Z] 222.51s call tests/python/unittest/test_operator.py::test_batchnorm[True-False-False-shape2-BatchNorm]
[2021-03-06T03:23:55.682Z] 209.34s call tests/python/unittest/test_operator.py::test_broadcast_binary_op
[2021-03-06T03:23:55.682Z] 201.85s call tests/python/unittest/test_operator.py::test_batchnorm[True-True-False-shape2-BatchNorm]
[2021-03-06T03:23:55.682Z] 194.73s call tests/python/unittest/test_operator.py::test_layer_norm[float16-0.01-0.01-in_shape_l0-finite_grad_check_l0-1]
[2021-03-06T03:23:55.682Z] 156.82s call tests/python/unittest/test_operator.py::test_batchnorm[False-True-True-shape2-BatchNorm]
[2021-03-06T03:23:55.682Z] 132.63s call tests/python/unittest/test_operator.py::test_order
[2021-03-06T03:23:55.682Z] 129.84s call tests/python/unittest/test_operator.py::test_batchnorm[False-False-True-shape2-BatchNorm]
[2021-03-06T03:23:55.682Z] 107.90s call tests/python/unittest/test_operator.py::test_psroipooling
[2021-03-06T03:23:55.682Z] 101.68s call tests/python/unittest/test_operator.py::test_batchnorm[False-False-True-shape2-SyncBatchNorm]
```
_Originally posted by @barry-jin in https://github.com/apache/incubator-mxnet/issues/20091#issuecomment-807490069_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] akarbown commented on issue #20092: test_norm, test_batchnorm and test_layer_norm dramatically slow down
Posted by GitBox <gi...@apache.org>.
akarbown commented on issue #20092:
URL: https://github.com/apache/incubator-mxnet/issues/20092#issuecomment-944487638
@barry-jin - can we close this issue - it seems to be resolved by the #20367, isn't it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] leezu commented on issue #20092: test_norm, test_batchnorm and test_layer_norm dramatically slow down
Posted by GitBox <gi...@apache.org>.
leezu commented on issue #20092:
URL: https://github.com/apache/incubator-mxnet/issues/20092#issuecomment-807545019
Please provide more info how you established a connection to #19953. It looks like it's a non-deterministic issue happening in the **centos**-cpu Python3: CPU
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] barry-jin closed issue #20092: test_norm, test_batchnorm and test_layer_norm dramatically slow down
Posted by GitBox <gi...@apache.org>.
barry-jin closed issue #20092:
URL: https://github.com/apache/incubator-mxnet/issues/20092
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] barry-jin edited a comment on issue #20092: test_norm, test_batchnorm and test_layer_norm dramatically slow down
Posted by GitBox <gi...@apache.org>.
barry-jin edited a comment on issue #20092:
URL: https://github.com/apache/incubator-mxnet/issues/20092#issuecomment-944488907
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] leezu commented on issue #20092: test_norm, test_batchnorm and test_layer_norm dramatically slow down
Posted by GitBox <gi...@apache.org>.
leezu commented on issue #20092:
URL: https://github.com/apache/incubator-mxnet/issues/20092#issuecomment-807566815
May be fixed by https://github.com/apache/incubator-mxnet/pull/20093
Let's check the CI timing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] leezu removed a comment on issue #20092: test_norm, test_batchnorm and test_layer_norm dramatically slow down
Posted by GitBox <gi...@apache.org>.
leezu removed a comment on issue #20092:
URL: https://github.com/apache/incubator-mxnet/issues/20092#issuecomment-807545019
Please provide more info how you established a connection to #19953. It looks like it's a non-deterministic issue happening in the **centos**-cpu Python3: CPU
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] barry-jin commented on issue #20092: test_norm, test_batchnorm and test_layer_norm dramatically slow down
Posted by GitBox <gi...@apache.org>.
barry-jin commented on issue #20092:
URL: https://github.com/apache/incubator-mxnet/issues/20092#issuecomment-944488907
Closes via #20367, Thanks @akarbown for the fix!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org