You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mxnet.apache.org by GitBox <gi...@apache.org> on 2022/03/10 09:00:18 UTC
[GitHub] [incubator-mxnet] mazeltovlee opened a new issue #20947: Inconsistent behavior of mx.sym.LeakyReLU between CPU and GPU
mazeltovlee opened a new issue #20947:
URL: https://github.com/apache/incubator-mxnet/issues/20947
## Description
When sending `mx.sym.LeakyReLU` with `slop=0.0` a `-Inf` array, it will output `0` in CPU but `NaN` in GPU. In contrast, LeakyRelu behaves the same on the `Inf` array (output `Inf` in both CPU and GPU)
## To Reproduce
```
import os
import mxnet as mx
net = mx.symbol.Variable('data', shape=(0, 10,10,3))
net = mx.sym.LeakyReLU(data=net, slope=0.0, act_type="leaky")
mx.viz.plot_network(symbol=net)
import numpy as np
input_shape = (10, 10,10,3)
x = np.random.rand(*input_shape)
x *= -np.Inf
e = net.bind(mx.gpu(), {'data': mx.nd.array(x,mx.gpu())})
output_gpu = e.forward()
print("The GPU output is: ", output_gpu)
e_cpu = net.bind(mx.cpu(), {'data': mx.nd.array(x,mx.cpu())})
output_cpu = e_cpu.forward()
print("The CPU output is: ", output_cpu)
```
### Steps to reproduce
Run above code with GPU or access the CoLab link: https://colab.research.google.com/drive/1jIyOFR0n8j15Yy10b1EwUKWj6v7EmJAF?usp=sharing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] mazeltovlee commented on issue #20947: Inconsistent behavior of mx.sym.LeakyReLU between CPU and GPU
Posted by GitBox <gi...@apache.org>.
mazeltovlee commented on issue #20947:
URL: https://github.com/apache/incubator-mxnet/issues/20947#issuecomment-1064680256
Yes, I have tried `-Inf` to `mx.sym.relu` on both GPU and CPU, it will output 0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] DickJC123 commented on issue #20947: Inconsistent behavior of mx.sym.LeakyReLU between CPU and GPU
Posted by GitBox <gi...@apache.org>.
DickJC123 commented on issue #20947:
URL: https://github.com/apache/incubator-mxnet/issues/20947#issuecomment-1064618024
For negative input `x`, isn't the leaky relu output `slope*x`? So the proper output = `-inf * 0` = `Nan`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] DickJC123 edited a comment on issue #20947: Inconsistent behavior of mx.sym.LeakyReLU between CPU and GPU
Posted by GitBox <gi...@apache.org>.
DickJC123 edited a comment on issue #20947:
URL: https://github.com/apache/incubator-mxnet/issues/20947#issuecomment-1064618024
For negative input `x`, isn't the leaky relu output `slope * x`? So the proper output = `0 * -inf` = `Nan`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] DickJC123 edited a comment on issue #20947: Inconsistent behavior of mx.sym.LeakyReLU between CPU and GPU
Posted by GitBox <gi...@apache.org>.
DickJC123 edited a comment on issue #20947:
URL: https://github.com/apache/incubator-mxnet/issues/20947#issuecomment-1064618024
For negative input `x`, isn't the leaky relu output `slope * x`? So the proper output = `0 * -inf` = `Nan`.
On the other hand, one could take the view:
```
if (slope == 0.0)
<behavior is relu>
else
<behavior is leaky relu>
```
... which would suggest the output should be 0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] mazeltovlee commented on issue #20947: Inconsistent behavior of mx.sym.LeakyReLU between CPU and GPU
Posted by GitBox <gi...@apache.org>.
mazeltovlee commented on issue #20947:
URL: https://github.com/apache/incubator-mxnet/issues/20947#issuecomment-1063832341
I also find giving the LeakyReLU `NaN` will also trigger inconsistency: given NaN, LeakyReLU in GPU mode will output NaN while it outputs 0 in CPU mode. I seems that, given a NaN input, the expected behavior should be NaN instead of 0. But for the -inf input, the expected behavior should be 0 instead of NaN?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] DickJC123 edited a comment on issue #20947: Inconsistent behavior of mx.sym.LeakyReLU between CPU and GPU
Posted by GitBox <gi...@apache.org>.
DickJC123 edited a comment on issue #20947:
URL: https://github.com/apache/incubator-mxnet/issues/20947#issuecomment-1064618024
For negative input `x`, isn't the leaky relu output `slope*x`? So the proper output = `0 * -inf` = `Nan`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org