[GitHub] [singa] nudles edited a comment on issue #696: Refactor autograd module

[GitBox <gi...@apache.org>]

nudles edited a comment on issue #696:
URL: https://github.com/apache/singa/issues/696#issuecomment-628365342


   > 
   > 
   > > 1. Before we call Module.forward(), we can randomly fill the placeholder tensors.
   > > 2. We can make Layer.init() optional. To implement a new layer, the parameter initialization can be done within the `__call__` method or in a `init()` method. It is up to the contributor.
   > > 
   > > Any comments on the drawbacks?
   > > @dcslin @XJDKC
   > 
   > For some models, it cannot use the random inputs, such as BERT within ONNX, some nodes may compute the indices of a tensor, and the next node may split the tensor by using these indices. If we randomly generate the inputs, this case always fails.
   
   Good point. Then we can config the data type when creating the placeholder and initialize the placeholder according to this data type. But how to initialize? randomly or set to 0? there could still be some issues. So the better way is to use real data instead of placeholders..
   > 
   > By the way, I prefer the idea of:
   > 
   > ```python
   > # another option is to define a compile method
   >     def compile(self, inputs, is_train, use_graph, graph_alg):
   >         self.forward(*inputs)
   > ```
   > 
   > However, I'd like to add a method to compute the shape based on the inputs of each node instead of calling the forward function:
   > 
   > ```python
   > def compute_output_shape(self, input_shape):
   >     # print(input_shape) # [(None, 10), (None, 12)]
   >     return (None, input_shape[0][1] + input_shape[1][1] + 2)
   > ```
   > 
   
   Do you need this one for onnx loading?
   
   > Let me think about it, I'll comment the detailed API later.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org