You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@singa.apache.org by GitBox <gi...@apache.org> on 2020/02/12 11:45:22 UTC
[GitHub] [singa] XJDKC opened a new pull request #598: fix softmax nan
problem
XJDKC opened a new pull request #598: fix softmax nan problem
URL: https://github.com/apache/singa/pull/598
Here I change the algorithm of the cudnnSoftmaxForward function.
The original algorithm was CUDNN_SOFTMAX_FAST which may lead to overflow when the input numbers are too large. Change it to CUDNN_SOFTMAX_ACCURATE will solve this problem.
For example: If cudnn softmax is used in mlp.py, it will lead to this problem.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] XJDKC commented on issue #598: fix softmax nan problem
Posted by GitBox <gi...@apache.org>.
XJDKC commented on issue #598: fix softmax nan problem
URL: https://github.com/apache/singa/pull/598#issuecomment-585176483
Thank you for your reminding.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #598: fix softmax nan
problem
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #598: fix softmax nan problem
URL: https://github.com/apache/singa/pull/598#issuecomment-585170944
I strongly support this change for two reasons:
1. The current setting fails some network structure/dataset, which make the functions not universal for all applications.
2. SoftMax forward spent only very short time within the CNN training, e.g. resnet50 has many conv layers but only one SoftMax layer, so the time spent in SoftMax is not the bottleneck.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #598: fix softmax nan
problem
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #598: fix softmax nan problem
URL: https://github.com/apache/singa/pull/598#issuecomment-585171699
sorry, could you also change CUDNN_SOFTMAX_FAST to CUDNN_SOFTMAX_ACCURATE in SoftMaxbackward in tensor_math_cuda.cc ? I am not sure if both needed to be matched
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #598: fix softmax nan
problem
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #598: fix softmax nan problem
URL: https://github.com/apache/singa/pull/598#issuecomment-585175871
thanks. I think it is ready for merge
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] nudles merged pull request #598: fix softmax nan problem
Posted by GitBox <gi...@apache.org>.
nudles merged pull request #598: fix softmax nan problem
URL: https://github.com/apache/singa/pull/598
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services