[jira] [Created] (SINGA-115) Print layer debug information in the neural net graph file

["wangwei (JIRA)" <ji...@apache.org>]

wangwei created SINGA-115:
-----------------------------

             Summary: Print layer debug information in the neural net graph file
                 Key: SINGA-115
                 URL: https://issues.apache.org/jira/browse/SINGA-115
             Project: Singa
          Issue Type: New Feature
            Reporter: wangwei


It is non-trivial to debug the code for deep learning, e.g., the BP algorithm, the hybrid partitioning and layer implementation. 

In SINGA, we print the neural net in INFO log as json string, which can be converted into an image with the net graph (nodes are layers). This graph can be used to check the neural net configuration, e.g., layer connection and neural net partitioning. However, it does not collect the run time data, e.g., gradient norm or value norm of each layer, which is important to debug  accuracy etc. bugs.

In this ticket, we will collect the gradient and value norm of each layer and each Param object. These information will be printed as attributes (or sub-nodes) of the layer node in the neural net graph. Users/developers can located the bugs by inspecting the graph after converting the json string into an image.

Particularly, uses can set the disp_freq to 1 and running steps to a small number, e.g., 5. Then 5 neural net graphs will be printed, one per step. The debug option should be turned on in the job.conf file for printing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)