[GitHub] feevos commented on issue #10101: gluon feature request: proper registration/initialization of layers inside a list for custom (Hybrid)Blocks

[GitBox <gi...@apache.org>]

feevos commented on issue #10101: gluon feature request: proper registration/initialization of layers inside a list for custom (Hybrid)Blocks
URL: https://github.com/apache/incubator-mxnet/issues/10101#issuecomment-372936577
 
 
   Hi @szha, thank you for your reply. I've done so in simpler architectures as you describe but now I want to try something more advanced. 
   
   The basic idea is that one can have a set of layers that live in a list, ```layers_list```. Then one can form a sparse connectivity matrix, ```Sij``` where each row corresponds to the connections of ```layer_i``` to ```layer_j```. The connectivity matrix will be an individual inside an evolutionary algorithm. The architecture of the network is defined by ```Sij```. For example, a simple Sequential module, where one stacks 4 layers 
   
   ```Python
   net = Sequential()
   for i in range(3):
       net.add(Dense(5))
   ```
   can be represented with the following connectivity matrix: 
   ```Python
      | 1   2   3   4
   ------------------- 
   1  | 0  1   0   0
   2  | 0  0   1   0
   3  | 0  0   0   1 
   4  | 0  0   0   0 
    ```
   Starting from row, ```layer_1``` connects to ```layer_2```, ```layer_2``` to ```layer_3``` and so on. Layer 4 has no connectivy( last layer). But if we want more advanced topology of the network (like ```layer_1``` connecting with ```layer_2``` and ```layer_3```)
   ```Python
      | 1   2   3   4
   ------------------- 
   1  | 0  1   1   0
   2  | 0  0   1   0
   3  | 0  0   0   1 
   4  | 0  0   0   0 
    ```
   the ```Sequential``` breaks down. It is possible again to formulate it with Sequential but I think it lacks the flexibility of indexing.  
   
   Now assuming one has the layers in a container (a list in this example, I can think of dictionary usage as well), ```layers_list```, and ```Sij``` is a sparse matrix, one can formulate a ```forward``` function (design prototype, not the true solution, [here](https://stackoverflow.com/questions/4319014/iterating-through-a-scipy-sparse-vector-or-matrix) is an example of iterating over sparse matrix): 
   
   ```Python 
   def hybrid_forward(self, F, input):
       out = self.first_layer(input)
       cx = Sij.tocoo()    
       # This for loop iterates over non zero elements. 
       for i,j,_ in itertools.izip(cx.row, cx.col, cx.data):
           out = self.layer_list[j](self.layer_list[i](out))
           out = F.relu(out)
       return out
   ``` 
   
   The basic idea is to create a DAG on the fly, using lists and a connectivity matrix is the first way that comes into my mind of implemeting this (I may be wrong, am pretty sure there are perhaps better ways of doing so, but I don't know any). I think this functionality, in combination with the flexibility of gluon imperative style, can help a lot of people play with variable architectures. 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services