[GitHub] asitstands edited a comment on issue #11268: A binary RBM example

[GitBox <gi...@apache.org>]

asitstands edited a comment on issue #11268: A binary RBM example
URL: https://github.com/apache/incubator-mxnet/pull/11268#issuecomment-400526464
 
 
   @yifeim Thanks for the suggestions!
   
   - The gradient in this example is the standard one from the straightforward calculation of the gradient of the log likelihood, to be used in the stochastic gradient descent (SGD). The expressions in the tutorial you linked replace the average over the Gibbs samples of hidden units with the probability of a hidden unit to be one. They are interchangeable in practice, while strictly speaking the latter is not exact for SGD.
   - There are lots of literature on RBM and each of them has its own presentation of the math. No one may directly match with the code in this PR. Github does not support math rendering with the markdown so explaining the math of RBM in the README is difficult and I think that is also out of scope of this PR. If someone wants to understand the example code (not merely using), then they should understand the math and I think that the differences in the literature may not be hard to be adjusted.
   - Yes, the Bernoulli sampling method you provide is the standard idiom! I forgot to restore it after some tests with `multinomial` ( #10970). Thanks.
   - `interaction_weight` sounds good. I'll use the name. But I prefer explicit initialization.
   - Monitoring the progress of the RBM learning is not trivial because the calculation of the loss (objective function) is practically impossible. There is no golden standard for the monitoring as far as I know. So I left it empty. But anyway adding one to this example would not be bad. My personal preference is using the annealed importance sampling and some people recommend it (e.g. [a guide by Hinton](https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=0ahUKEwiwwPHs5PLbAhWKi7wKHWIiCOwQFggrMAA&url=https%3A%2F%2Fwww.cs.toronto.edu%2F~hinton%2Fabsps%2FguideTR.pdf&usg=AOvVaw2--ZzyoAVNqnBr993kZeF8)). I'll add it.
   
   I'll update the code soon.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services