You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2020/05/27 05:45:48 UTC
[GitHub] [incubator-mxnet] acphile opened a new issue #18412: Simplifying mxnet.gluon.block APIs
acphile opened a new issue #18412:
URL: https://github.com/apache/incubator-mxnet/issues/18412
## Motivations
Currently the implementation of mxnet.gluon.block is not so pythonic and there are many redundancies
### 1. overlaps between Block._params and Block._reg_params
when we want to self-define a model, we currently need to use the code as follows:
```
class Net(nn.HybridBlock):
def __init__(self, **kwargs):
super(HybridNet, self).__init__(**kwargs)
with self.name_scope():
self.hidden1 = nn.Dense(256, activation='relu')
self.a=self.params.get('a', shape=(1, ))
```
There are several shortcomings when using this form of registration:
a. adding parameter ‘a’ will lead to double recordings in both self._params and self._reg_params, which is a redundancy. And there is also a discrepancy in Block:
i. In the method “collect_params”, we use “_params” to get all parameters
ii. while in the method “_collect_params_with_prefix” (and methods “load_parameters” accordingly), we use “_reg_params” to get all parameters.
b. Currently if we do not use “with self.name_scope():” for children blocks, it will lead to wrong name scopes. For the following example, we actually can not get the parameters of self.hidden1 from the result of collect_params
```
class HybridNet(nn.HybridBlock):
def __init__(self, **kwargs):
super(HybridNet, self).__init__(**kwargs)
self.hidden1 = nn.Dense(256, activation='relu')
with self.name_scope():
self.hidden2 = nn.Dense(10, activation='relu')
def hybrid_forward(self, F, x):
x = self.hidden2(self.hidden1(x))
return x
>>> net = HybridNet()
>>> net.initialize()
>>> print(net.collect_params())
hybridnet0_ (
Parameter dense0_weight (shape=(256, -1), dtype=float32)
Parameter dense0_bias (shape=(256,), dtype=float32)
Parameter hybridnet0_dense0_weight (shape=(10, -1), dtype=float32)
Parameter hybridnet0_dense0_bias (shape=(10,), dtype=float32)
)
```
From the above example we can also find that the parameter names are not related to the attributes’ names, which is not straightforward.
In all, we find that using name_scope and ParameterDict is not user-friendly. Thus we plan to remove such redundancies and simplify the definitions of children blocks and parameters, like:
```
class Net(nn.HybridBlock):
def __init__(self, **kwargs):
super(HybridNet, self).__init__(**kwargs)
self.hidden1 = nn.Dense(256, activation='relu')
self.a=gluon.parameter.Parameter(name="a", shape=(1, ))
```
And we also can get correct results in the following operations like “collect_params”: For the above example, when calling “collect_params” we should get the form like:
{
“a”: xxx
“hidden1_weight”: xxx
“hidden1_bias”: xxx
}
which is like the origianl form used in _collect_params_with_prefix()
For using name_scope in hybridization, we use new method “set_prefix()” to recursively add prefix for all parameters inside, using the name of attributes (similar form as the keys in _collect_params_with_prefix())
For example:
```
net=Net()
>>> print(net.collect_params())
{
“a”: Parameter a (...)
“hidden1_weight”: Parameter weight (...)
“hidden1_bias”: Parameter bias (...)
}
>>> net.set_prefix()
>>> print(net.collect_params())
{
“a”: Parameter a (...)
“hidden1_weight”: Parameter hidden1_weight (...)
“hidden1_bias”: Parameter hidden1_bias (...)
}
```
Shared parameter would be added with the prefix where it first occurs.
### 2. parameter sharing
Currently, we use parameter “params” in the definition of Block for parameter sharing. It means before the __init__ of Block, shared parameters already recorded in self._params.shared. And currently Block forbids overriding parameters.
We think that this is not convenient. A most common way to share parameter is like what Pytorch does, like
```
self.hidden1.weight=self.hidden2.weight
```
But note that in the case where we have a HybridBlock and the block has been hybridized, then we shouldn't allow overriding the parameter but ask the user to unhybridize the Block first.
To further allow sharing parameters recursively, we plan to add an API:
```
def share_parameters(self, params : Dict):
```
We plan to use the structured based form (like what is used in “_collect_params_with_prefix()”) to represent each parameter recursively. For example, we denote “self.hidden1.weight” as “hidden_weight”
In all, we plan to make the following improvements:
1. remove parameters “prefix” and “params” in the “\_\_init\_\_" function.
2. remove the use of self._params(ParameterDict) in Block
3. allow parameter attribute overriding in non-hydridization case.
4. add the method “share_parameters" to recursively share parameters in children blocks.
## Detailed improvements
### For Class Block
1. remove attribute _empty_prefix, _prefix, _params, _profiler_scope_name, _name and their corresponding properties.
```
@property
def params(self):
"""Returns this :py:class:`Block`'s parameter dictionary (does not include its
children's parameters)."""
return self._reg_params
```
2. use the structured format for self._scope
a. self._scope is only intended to be used in hybridize() internally
3. implement collect_params with _collect_params_with_prefix
a. def _collect_params_with_prefix(self, prefix='', select=None):
4. change the implementation of save_params to save_parameters
5. move the implementation of ParameterDict.initialize to Block.initialize
6. move ParameterDict.zero_grad() to Block.zero_grad()
a. call model.zero_grad() instead of model.collect_params().zero_grad()
7. move ParameterDict.reset_ctx(ctx) to Block.reset_ctx(ctx)
8. move ParameterDict.setattr to Block.setattr
9. add method share_parameters(self, params : OrderedDict):
a. return self
10. add method set_prefix():
a. return self
b. when calling Block.initialize(), set_prefix() would be called first internally.
### For Class HybridBlock:
1. before hybridize() is called, set_prefix would be called first.
```
net = Net()
net.initialize()
net.hybridize()
```
or
```
net = Net().set_prefix()
net.hybridize()
```
### For Parameter:
1. add attribute self._prefix (which is set automatically by Block.set_prefix())
2. use self._prefix + self._name for attribute self.name
### For other children class:
1. remove params and prefix
2. use Parameter instead of self.params.get like
```
#self.i2h_weight = self.params.get('i2h_weight', shape=(hidden_size, input_size),
# init=i2h_weight_initializer,
# allow_deferred_init=True)
self.i2h_weight = Parameter(shape=(hidden_size, input_size), init=i2h_weight_initializer,
allow_deferred_init=True)
```
### For _RNNLayer (gluon.rnn.rnn_layer._RNNlayer)
1. remove the implementation of _unfuse()
2. remove the implementation of _collect_params_with_prefix() (Seemingly It is only for backward compatibility)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] acphile commented on issue #18412: Simplifying mxnet.gluon.block APIs
Posted by GitBox <gi...@apache.org>.
acphile commented on issue #18412:
URL: https://github.com/apache/incubator-mxnet/issues/18412#issuecomment-642383991
Thanks @leezu . Based on the suggestions, I remove the method `set_prefix` and separate the param.name and structure based naming scheme. Here are the details
## Parameter name
Once a parameter is created, `param.name` would not be changed in the following operations. It is in the form of `param_{uuid4}_{name}`, where `name` is from `__init __` parameter. Here `name` is optional, default `weight`. It is mainly used to denote which default initialization should be used.
We use `param.name` as the name of a parameter's symbol representation.
## collect_params()
It returns a `dict`, where the keys are structural names of parameters, like
`{'hidden1.weight': Parameter (shape=(3, -1), dtype=float32), 'hidden1.bias': Parameter (shape=(3,), dtype=float32)}`
Note that we use `.` as the linking character again because the structured based naming scheme is no longer used in the symbol representation.
## Save and Load
For `HybridBlock`, there are two ways to save and load parameters:
### save_parameters() and load_parameters()
In `save_parameters()`, we use `structural name` to save parameters, and they should be loaded by `load_parameters()`, which loads parameters based on a model's structure.
### HybridBlock.export and SymbolBlock.imports
In `export`, we only save parameters using `param.name` without `structural name`. The param file should be loaded in SymbolBlock.imports.
## SymbolBlock
When using `SymbolBlock.imports`, keys in `self.param` would be the loaded parameters' names `param.name`.
While in `SymbolBlock(outputs, inputs, params=None)`, if you provide like `params=net.collect_params()`, keys in `self.param` would be structural names of `net`'s parameters (keys in net.collect_params() ). It is often used in this situation that a `SymbolBlock` is a children block of another `HybridBlock`. Otherwise, keys in `self.param` would be the loaded parameters' names `param.name`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] acphile commented on issue #18412: Simplifying mxnet.gluon.block APIs
Posted by GitBox <gi...@apache.org>.
acphile commented on issue #18412:
URL: https://github.com/apache/incubator-mxnet/issues/18412#issuecomment-639213050
## A example
```
from mxnet import np, npx
from mxnet.gluon import nn, parameter
npx.set_np()
class Net(nn.HybridBlock):
def __init__(self, **kwargs):
super(Net, self).__init__(**kwargs)
self.hidden1 = nn.Dense(3)
def hybrid_forward(self, F, x):
x = self.hidden1(x)
return x
class Net2(nn.HybridBlock):
def __init__(self, **kwargs):
super(Net2, self).__init__(**kwargs)
self.hidden2 = nn.Dense(3, activation='relu')
def hybrid_forward(self, F, x):
x = self.hidden2(x)
return x
>>> x = np.random.normal(size=(5, 10))
>>> net = Net()
>>> net.initialize()
>>> print(net.collect_params())
{'hidden1_weight': Parameter hidden1_weight (shape=(3, -1), dtype=float32), 'hidden1_bias': Parameter hidden1_bias (shape=(3,), dtype=float32)}
>>> print(net(x))
[[ 0.09921001 0.04954842 0.12571132]
[-0.06151271 -0.17121975 -0.18948194]
[ 0.0051947 -0.08211827 0.02048509]
[ 0.10466634 -0.08711289 -0.18864125]
[ 0.26467288 -0.08746998 0.16121587]]
>>> net2=Net2()
>>> net2.initialize()
>>> print(net2.collect_params())
{'hidden2_weight': Parameter hidden2_weight (shape=(3, -1), dtype=float32), 'hidden2_bias': Parameter hidden2_bias (shape=(3,), dtype=float32)}
>>> net2.hidden2.share_parameters(net.hidden1.collect_params())
"""
equals to
net2.hidden2.weight=net.hidden1.weight
net2.hidden2.bias=net.hidden1.bias
"""
>>> print(net2.hidden2.weight is net.hidden1.weight)
True
>>> print(net2.hidden2.bias is net.hidden1.bias)
True
>>> net2.initialize()
/home/ubuntu/incubator-mxnet/python/mxnet/gluon/block.py:410: UserWarning: Parameter hidden1_weight has generated its symbol representation, which could be used in some cached graph. Skip the operation that sets its name as hidden2_weight.
self._set_prefix(recorded, prefix)
/home/ubuntu/incubator-mxnet/python/mxnet/gluon/block.py:410: UserWarning: Parameter hidden1_bias has generated its symbol representation, which could be used in some cached graph. Skip the operation that sets its name as hidden2_bias.
self._set_prefix(recorded, prefix)
/home/ubuntu/incubator-mxnet/python/mxnet/gluon/block.py:694: UserWarning: Parameter 'hidden1_weight' is already initialized, ignoring. Set force_reinit=True to re-initialize.
v.initialize(None, ctx, init, force_reinit=force_reinit)
/home/ubuntu/incubator-mxnet/python/mxnet/gluon/block.py:694: UserWarning: Parameter 'hidden1_bias' is already initialized, ignoring. Set force_reinit=True to re-initialize.
v.initialize(None, ctx, init, force_reinit=force_reinit)
>>> net2.hybridize()
>>> print(net2.collect_params())
{'hidden2_weight': Parameter hidden1_weight (shape=(3, 10), dtype=float32), 'hidden2_bias': Parameter hidden1_bias (shape=(3,), dtype=float32)}
>>> print(net2(x)) # should equal to relu(net1(x))
[[ 0.09921001 0.04954842 0.12571132]
[-0. -0. -0. ]
[ 0.0051947 -0. 0.02048509]
[ 0.10466634 -0. -0. ]
[ 0.26467288 -0. 0.16121587]]
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] sxjscience edited a comment on issue #18412: Simplifying mxnet.gluon.block APIs
Posted by GitBox <gi...@apache.org>.
sxjscience edited a comment on issue #18412:
URL: https://github.com/apache/incubator-mxnet/issues/18412#issuecomment-638495686
Is there a small example to showcase how we can share parameters in the new API?
Let me clarify a little bit about the hello-world-like examples.
If we check the documentation of TorchScript: https://pytorch.org/docs/stable/jit.html, there are lots of tiny examples, which explains the usage of the module. May be we need to add some examples to help users understand how to use the new Gluon.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu closed issue #18412: Simplifying mxnet.gluon.block APIs
Posted by GitBox <gi...@apache.org>.
leezu closed issue #18412:
URL: https://github.com/apache/incubator-mxnet/issues/18412
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] sxjscience commented on issue #18412: Simplifying mxnet.gluon.block APIs
Posted by GitBox <gi...@apache.org>.
sxjscience commented on issue #18412:
URL: https://github.com/apache/incubator-mxnet/issues/18412#issuecomment-638495686
Is there a small example to showcase how we can share parameters in the new API?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu commented on issue #18412: Simplifying mxnet.gluon.block APIs
Posted by GitBox <gi...@apache.org>.
leezu commented on issue #18412:
URL: https://github.com/apache/incubator-mxnet/issues/18412#issuecomment-639983273
Thanks @acphile! I think we can consider if the `param.name` attribute is really needed. Instead we can solely rely on the structure based naming scheme. This is because I'm afraid that the `set_prefix` approach may not be completely robust (users may access the `param.name` before the prefix is set). I put an example at the code review: https://github.com/apache/incubator-mxnet/pull/18413#pullrequestreview-425619757 What do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org