You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@systemml.apache.org by "Mike Dusenberry (JIRA)" <ji...@apache.org> on 2016/03/01 00:50:18 UTC
[jira] [Comment Edited] (SYSTEMML-547) Implement built-in functions
for max and average pooling
[ https://issues.apache.org/jira/browse/SYSTEMML-547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15172891#comment-15172891 ]
Mike Dusenberry edited comment on SYSTEMML-547 at 2/29/16 11:49 PM:
--------------------------------------------------------------------
Great, this will be a good place to start for the {{pool}} operator. Of course, this brings up a good area of discussion regarding the system & API design:
1. How many operations do we want to push down into the core Java codebase as built-in functions vs. lifting them into DML itself? For example, would we want to also add a {{pool2d_grad}} built-in operator as well to handle the gradient calculation for {{pool2d}} as used in a "backward" pass? Do we return both the output and the gradients with the original {{pool2d}} function? Alternatively, do we lift these into a library of DML UDFs and optimize with rewrite rules? I think we'll find that there are tradeoffs in any direction.
Overall, whether in a DML API or as a drop-in for TensorFlow/Caffe/Torch/etc., we'll want to encapsulate layer types that contain the "forward" & "backward" logic so that they can easily be used in training and prediction. I am in favor of simplifying our eventual API as much as possible so that implementing new layers & ideas from experimentations/papers is easy. For our DML API, it would be nice to have the layers defined in DML, and have the associated {{forward}} and {{backward}} functions of the layers use optimized built-in functions (or optimizer rules) as necessary. Then, it would be quite simple for a new layer to be written in DML, and later optimized for performance as needed using either optimizer rules or built-in functions. Our TensorFlow/Caffe/etc. drop-in API could also make use of these layers and/or built-in functions.
As for the padding, I agree that it is not commonly (if at all) used with pooling, so an API simplification similar to TensorFlow's functions would be nice. Of note though, their padding calculations appear to be based on more-complicated legacy scheme [1], so I suggest that we follow the more common formulations for the case when it is used.
Also, +1 for CS231n; great class, and Andrej Karpathy is a fantastic reference.
[1]: [https://github.com/tensorflow/tensorflow/issues/196#issuecomment-156849844 | https://github.com/tensorflow/tensorflow/issues/196#issuecomment-156849844]
was (Author: mwdusenb@us.ibm.com):
Great, this will be a good place to start for the {{pool}} operator. Of course, this brings up a good area of discussion regarding the system & API design:
1. How many operations do we want to push down into the core Java codebase as built-in functions vs. lifting them into DML itself? For example, would we want to also add a {{pool2d_grad}} built-in operator as well to handle the gradient calculation for {{pool2d}} as used in a "backward" pass? Do we return both the output and the gradients with the original {{pool2d}} function? Alternatively, do we lift these into a library of DML UDFs and optimize with rewrite rules? I think we'll find that there are tradeoffs in any direction.
Overall, whether in a DML API or as a drop-in for TensorFlow/Caffe/Torch/etc., we'll want to encapsulate layer types that contain the "forward" & "backward" logic so that they can easily be used in training and prediction. I am in favor of simplifying our eventual API as much as possible so that implementing new layers & ideas from experimentations/papers is easy. For our DML API, it would be nice to have the layers defined in DML, and have the associated {{forward}} and {{backward}} functions of the layers use optimized built-in functions (or optimizer rules) as necessary. Then, it would be quite simple for a new layer to be written in DML, and later optimized for performance as needed using either optimizer rules or built-in functions. Our TensorFlow/Caffe/etc. drop-in API could also make use of these layers and/or built-in functions.
As for the padding, I agree that it is not commonly (if at all) used with pooling, so an API simplification similar to TensorFlow's functions would be nice. Of note though, their padding calculations appear to be based on more-complicated legacy scheme [1], so I suggest that we follow the more common formulations for the case when it is used.
Also, +1 for CS231n; great class, and Andrej Karpathy is a fantastic reference.
> Implement built-in functions for max and average pooling
> --------------------------------------------------------
>
> Key: SYSTEMML-547
> URL: https://issues.apache.org/jira/browse/SYSTEMML-547
> Project: SystemML
> Issue Type: New Feature
> Components: Parser, Runtime
> Reporter: Niketan Pansare
> Assignee: Nakul Jindal
> Priority: Minor
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> pool2d(input, pool_size, stride_length, border_mode="valid", pool_mode="max")
> Performs downscaling of the input matrix.
> The arguments to this function are:
> 1. input is a 2-dimensional matrix.
> 2. pool_size is a required integer parameter.
> 3. stride_length is an optional Int parameter. The default value is 1.
> 4. border_mode is an optional String parameter. The valid values are "same" and "valid".
> 5. pool_mode is an optional String parameter. The valid values are "max" and "avg". We can later add additional operators here (such as sum).
> For detailed documentation, see Theano's pool_2d function: https://github.com/Theano/Theano/blob/master/theano/tensor/signal/pool.py#L40
> An an example, our pool2d(input=X, pool_size=2, stride_length=1, border_mode="valid", pool_mode="avg") invocation is similar to Theano's
> pool_2d(X, ds=(2,2), st=(1,1), ignore_border=True, padding=(0, 0), mode="average_exc_pad")
> Since padding=(0,0) is the most common padding (probably the only one most people will use), I thought of simplifying the interface by borrowing concepts from TensorFlow's functions max_pool and avg_pool. See https://www.tensorflow.org/versions/r0.7/api_docs/python/nn.html#avg_pool
> The above example will translate into following TensorFlow code:
> tf.nn.avg_pool(X, pool_size=(1,2,2,1), strides=(1,1,1,1), padding="VALID")
> Another good reference to understanding pooling operation is http://cs231n.github.io/convolutional-networks/#pool
> [~mwdusenb@us.ibm.com], [~nakul02], [~prithvi_r_s], [~reinwald@us.ibm.com]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)