[GitHub] 0xfiras commented on issue #13798: C++ generate wrong prediction results on GPU

[GitBox <gi...@apache.org>]

0xfiras commented on issue #13798: C++ generate wrong prediction results on GPU
URL: https://github.com/apache/incubator-mxnet/issues/13798#issuecomment-453794900
 
 
   I am facing the same issue right now, but my case is a bit different.
   
   I am training using the c++ API on the GPU. When I inspected the output values of the first batches, I can clearly see that the output is incorrect: negative prediction values after a relu.
   
   After a lot of printing of input and output, just to make sure I am not printing or editing the labels, I concluded that the problem is most probably from the C++ API, which brought me here.
   
   I switched to CPU, to check if I can reproduce the bug. And I did. Output from CPU is correct, i.e. positive.
   
   I stumbled upon this issue while investigating what I thought to be a differrent issue: when I print the *training* labels before and after a forward pass I get an occasional mismatch. I will get a single batch item have one of its labels flip from 0 to 1.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services