[GitHub] [incubator-mxnet] zixuanweeei commented on issue #17183: [MKL-DNN] MKL-DNN RNN backward path enhancement

[GitBox <gi...@apache.org>]

zixuanweeei commented on issue #17183: [MKL-DNN] MKL-DNN RNN backward path enhancement
URL: https://github.com/apache/incubator-mxnet/pull/17183#issuecomment-570533279
 
 
   > @zixuanweeei Do you have any clue about the perf degradeation on forward pass? As the PR also done some refactor to the common code of forward and backward.
   
   Another perf test containing more warm-up loops gave the result below, which compared the performance between 89fe1f6 and 622d84398. The refactor that have some impact on the forward pass is related to *reorder*. But *reorder* is only executed in the initialization process at the first iteration. RNN primitives may take more time to become stabilized. I think we can get this PR into master at first for settling the problem of gradients explode and release 1.6.x. 
   
   | directions | cell_type | layer_num | hidden_size | Fwk Gap | Bwd Gap |
   |------------|-----------|-----------|-------------|--------:|--------:|
   | 1          | lstm      | 1         | 512         | -0.09%  | 13.40%  |
   | 1          | lstm      | 1         | 1024        | 0.93%   | 8.47%   |
   | 2          | lstm      | 1         | 512         | 0.24%   | 8.03%   |
   | 2          | lstm      | 1         | 1024        | 0.44%   | 21.48%  |
   | 1          | gru       | 1         | 512         | 0.40%   | 11.57%  |
   | 1          | gru       | 1         | 1024        | -0.81%  | 12.09%  |
   | 2          | gru       | 1         | 512         | 0.21%   | 7.71%   |
   | 2          | gru       | 1         | 1024        | 1.91%   | 5.63%   |
   | 1          | rnn_relu  | 1         | 512         | 2.03%   | 18.97%  |
   | 1          | rnn_relu  | 1         | 1024        | 10.51%  | 10.48%  |
   | 2          | rnn_relu  | 1         | 512         | 1.51%   | 9.22%   |
   | 2          | rnn_relu  | 1         | 1024        | -0.42%  | 9.12%   |
   | 1          | rnn_tanh  | 1         | 512         | 2.87%   | 5.28%   |
   | 1          | rnn_tanh  | 1         | 1024        | 0.47%   | 8.09%   |
   | 2          | rnn_tanh  | 1         | 512         | 0.96%   | 6.88%   |
   | 2          | rnn_tanh  | 1         | 1024        | 0.27%   | 9.15%   |

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services